Koko Mental Health Chatbot Used AI to Counsel Users Without Consent

In early 2022, Koko, a peer-to-peer mental health support platform, began secretly experimenting with OpenAI's GPT-3 language model to generate responses to users seeking emotional support. The platform, which typically relies on human volunteers to provide peer support, integrated AI assistance without informing users or obtaining their consent for participation in what amounted to human subjects research. Over several months in 2022, approximately 4,000 individuals who reached out to Koko during mental health crises received responses that were either fully generated by GPT-3 or heavily influenced by AI suggestions. These users were seeking support for depression, anxiety, suicidal ideation, and other serious mental health conditions, making them a particularly vulnerable population. The AI-generated responses were presented as coming from human peer supporters, fundamentally deceiving users about the nature of their interaction. Koko co-founder Rob Morris revealed the experiment in January 2023 through social media posts, describing it as a way to help their volunteer supporters craft better responses. Morris initially framed the experiment positively, noting that AI-assisted responses received higher ratings from users. However, he also acknowledged that once users learned responses came from AI, satisfaction dropped significantly, with many expressing feeling "hoodwinked" and losing trust in the platform. The revelation sparked immediate backlash from mental health professionals, ethicists, and users. Critics pointed out that the experiment violated basic principles of informed consent in healthcare settings and potentially compromised the therapeutic relationship that is fundamental to effective mental health support. Mental health experts expressed concern that AI-generated responses, while potentially well-crafted, lacked the genuine human empathy and understanding crucial for supporting individuals in crisis. Following the public disclosure and criticism, Koko discontinued the AI experiment and implemented policies requiring transparency about AI involvement in user interactions. The incident highlighted significant gaps in ethical frameworks for AI deployment in healthcare settings, particularly regarding vulnerable populations and the need for proper oversight of experimental AI applications in mental health contexts.