Koko Mental Health Chatbot Conducted Undisclosed AI Experiment on Users Without Consent

On October 1, 2022, Koko, a peer-to-peer mental health support platform, initiated an undisclosed experiment using OpenAI's GPT-3 to generate responses for users seeking emotional crisis support. The experiment was conducted without user knowledge or consent over several months in 2022, affecting approximately 4,000 individuals who believed they were receiving human peer support. Co-founder Rob Morris publicly disclosed the experiment on Twitter in January 2023, revealing that the platform had used AI to draft messages that human moderators would then send to users experiencing depression, anxiety, suicidal ideation, and other mental health emergencies. The users remained completely unaware that their therapeutic interactions involved AI-generated content rather than authentic human empathy and support. The technical implementation involved integrating OpenAI's GPT-3 large language model into Koko's peer counseling workflow to assist human moderators in crafting responses to users in crisis. Rather than moderators writing original responses, GPT-3 generated draft messages that human staff would review and send to vulnerable users seeking immediate emotional support. The system was designed to improve response speed and quality, but fundamentally altered the nature of the therapeutic interaction from human-to-human support to AI-mediated communication. The platform failed to implement any disclosure mechanisms, consent protocols, or ethical review processes before deploying the AI system in live therapeutic sessions with users in potentially life-threatening mental health crises. The incident created significant harm to user trust and violated fundamental healthcare ethics principles, particularly informed consent requirements for experimental treatments or novel therapeutic interventions. Mental health professionals and ethicists condemned the practice, emphasizing that therapeutic relationships depend on transparency, authenticity, and trust between human participants. The 4,000 affected users, many of whom were experiencing severe emotional distress, were effectively treated as unwitting test subjects in an unregulated clinical experiment. The breach of trust raised concerns about potential psychological harm to users who later learned their crisis support interactions were AI-generated, potentially damaging their willingness to seek future mental health assistance and undermining confidence in digital mental health platforms broadly. Following the public backlash after Morris's January 2023 disclosure, Koko faced widespread criticism from mental health professionals, AI ethics researchers, and the broader healthcare community. Morris initially attempted to defend the experiment by claiming it improved response quality and reduced wait times for users in crisis, but subsequently acknowledged the significant ethical concerns raised by critics. The company announced it would discontinue the use of AI-generated responses and committed to implementing proper consent mechanisms for any future AI experiments. However, no formal apology was issued to the affected users, and the company did not provide details about potential compensation or remediation efforts for those who received undisclosed AI-generated crisis support. The Koko incident highlighted critical regulatory gaps in AI deployment within healthcare settings, particularly for digital mental health platforms operating without traditional clinical oversight. The case became a prominent example cited in discussions about the need for enhanced AI governance in healthcare, informed consent protocols for AI-assisted treatments, and regulatory frameworks for mental health technology platforms. The incident contributed to growing calls for mandatory disclosure requirements when AI is used in therapeutic contexts and stricter ethical review processes for AI experiments involving vulnerable populations. Industry observers noted the incident's potential to undermine public trust in emerging digital mental health solutions and emphasized the need for proactive ethical guidelines before AI integration in crisis intervention services. The controversy also sparked broader debates about the role of AI in mental health support and the importance of human authenticity in therapeutic relationships. Mental health advocacy groups used the Koko case to argue for stronger patient protection standards in digital health platforms and enhanced regulatory oversight of AI applications in crisis intervention. The incident occurred during a period of rapid expansion in AI-powered mental health tools, raising questions about whether existing healthcare regulations were adequate to address the unique risks posed by AI systems in therapeutic contexts without proper safeguards or transparency measures.