AI Incident Database

Mental health platform Koko used GPT-3 to generate responses to 4,000 users in emotional crisis without their knowledge or consent, violating fundamental principles of informed consent in healthcare.

Mental health platform Koko secretly used GPT-3 to generate responses to 4,000 users in emotional crisis without consent. The experiment raised serious ethical concerns about AI use in vulnerable healthcare contexts.

NYC schools initially banned ChatGPT in January 2023 citing academic integrity concerns, but reversed the decision in May 2023 after educator feedback and recognition of AI's educational potential when properly managed.

AI-powered cryptocurrency trading bots failed catastrophically during the 2022 crypto market crash, executing panic selling and amplifying losses for hundreds of thousands of retail investors who lost an estimated $2 billion.

AI surveillance cameras in Serbian schools monitored student emotions and behavior without proper consent from students or parents. Digital rights groups successfully challenged the practice, leading to removal of the surveillance system.

Research by the Center for Countering Digital Hate revealed TikTok's algorithm rapidly promoted pro-anorexia and self-harm content to teen users within minutes of engagement, leading to state attorney general investigations and ongoing litigation.

Lensa AI's Magic Avatars feature generated sexualized and stereotypical images of women users, particularly women of color, due to biased training data in the underlying Stable Diffusion model.

Lensa AI's Magic Avatars feature generated non-consensual sexualized images of users, particularly women, despite non-sexual input photos. The incident highlighted serious safety and consent issues in AI-generated imagery applications.

The Yara Birkeland autonomous cargo ship faced repeated AI navigation system failures that delayed fully autonomous operations by years beyond the 2020 target, resulting in significant cost overruns and operational setbacks.

Meta's Galactica AI for scientific text generation was shut down after just 3 days when it began confidently producing fake research papers, fabricated citations, and biased scientific content, raising serious concerns about AI-generated misinformation in academic contexts.

DeviantArt launched DreamUp AI tool using Stable Diffusion trained on artist works without consent, sparking massive community backlash and demands for opt-out mechanisms from affected creators.

GitHub Copilot was found to reproduce copyrighted code verbatim from its training data, leading to a class action lawsuit alleging copyright infringement by the AI coding assistant.

GitHub Copilot reproduced copyrighted code verbatim from its training data, leading to a class action lawsuit alleging widespread copyright infringement and violation of open-source licenses.

Serbian schools deployed AI facial recognition surveillance systems in 2022 without proper consent from students or parents. The deployment affected thousands of students and drew criticism from privacy advocates, leading to regulatory investigations.

AlphaFold protein structure predictions contained errors that led pharmaceutical companies to make incorrect drug design decisions, resulting in significant wasted R&D investments. The incident highlighted the need for experimental validation of AI predictions in critical applications.

The FTC ordered Weight Watchers to delete AI models trained on children's data collected through its Kurbo app without proper parental consent, marking the first 'algorithmic disgorgement' order requiring destruction of AI systems built with illegally obtained data.

Studies revealed that Zillow's Zestimate algorithm systematically undervalues homes in predominantly Black neighborhoods compared to similar properties in white areas, perpetuating housing discrimination through automated valuation models.

Jason Allen won first place at Colorado State Fair's digital art competition using AI-generated artwork from Midjourney. The victory sparked controversy about AI art legitimacy and fair competition with human artists.

Stable Diffusion generated images containing distorted Getty Images watermarks, revealing the model was trained on copyrighted Getty content without permission. This discovery became key evidence in Getty's copyright infringement lawsuit against Stability AI.

Major AI image generators consistently produced anatomically incorrect hands with extra fingers, fused digits, or impossible positions, becoming a reliable indicator for detecting AI-generated content.