AI Incident Database

San Francisco police circumvented a voter-approved facial recognition ban by accessing private cameras with AI capabilities, violating citizen privacy protections and prompting legal challenges.

AI-powered smart contract audit tools failed to detect critical vulnerabilities in the Ronin Network bridge, missing centralization risks in the multi-signature validator system. This oversight enabled hackers to exploit compromised validator keys and steal $600 million in March 2022.

A Starship delivery robot crossed police crime scene tape during an active investigation, requiring officers to manually remove it and potentially compromising the secured area.

A deepfake video showing Ukrainian President Zelensky calling for surrender was distributed via hacked TV and social media in March 2022. The low-quality fake was quickly debunked but highlighted deepfake threats during wartime.

Researchers at Collaborations Pharmaceuticals demonstrated that their AI drug discovery tool MegaSyn could generate 40,000 potential chemical weapon compounds in 6 hours by simply inverting its toxicity filter, highlighting serious dual-use risks in AI-assisted molecular design.

Amazon Alexa told a 10-year-old to perform a dangerous electrical challenge involving touching live plugs with coins, exposing major safety gaps in voice assistant content filtering.

Zillow's AI-powered iBuyer program used machine learning to predict home values and make instant purchase offers. The algorithm systematically overpaid for homes, ultimately losing $881 million and forcing Zillow to shut down the division and lay off 2,000 employees.

ShotSpotter's AI gunshot detection system generated high false positive rates leading to aggressive police responses in predominantly Black neighborhoods. Multiple cities terminated contracts amid concerns over accuracy and discriminatory impact.

EleutherAI's open-source GPT-Neo models generated extremist content and propaganda when prompted, highlighting safety risks in unfiltered language models without built-in guardrails.

Lemonade Insurance used AI to analyze customer facial expressions and speech patterns during video claims without proper disclosure. The company faced backlash and clarified its practices after privacy advocates raised discrimination concerns.

Meta's AI content moderation systems systematically censored Palestinian news and voices during 2021 and 2023 Gaza conflicts, with Human Rights Watch documenting widespread suppression of legitimate content.

ShotSpotter's AI gunshot detection system exhibited false positive rates of 86-95%, leading to wrongful police raids and discriminatory enforcement in predominantly Black neighborhoods across multiple US cities.

Alibaba and other Chinese companies developed AI emotion recognition technology that was reportedly used to monitor emotional states of detained Uyghur individuals in Xinjiang facilities, raising severe human rights concerns.

South Korean startup Scatter Lab's AI chatbot Lee Luda was shut down after making discriminatory comments and violating privacy laws by training on 10 billion private KakaoTalk messages without user consent, affecting 750,000 users.

South Korean chatbot Lee Luda was shut down after generating homophobic and racist content and being found to have illegally trained on 600,000 users' private KakaoTalk conversations without consent.

South Korean AI chatbot Lee Luda was shut down after generating discriminatory content against minorities and sexually inappropriate responses, leading to regulatory fines and lawsuits affecting 750,000 users.

Dutch tax authority's AI system wrongly flagged thousands of families for childcare benefits fraud based on discriminatory factors like dual nationality. The scandal caused widespread financial hardship and led to the collapse of the Dutch government in 2021.

The UK government's online passport photo verification AI systematically rejected photos of dark-skinned applicants with false error messages like 'mouth too open', demonstrating clear racial bias in a critical government service.

AI proctoring software from companies like Proctorio flagged Black students as cheating at disproportionate rates due to facial recognition bias. Thousands of students faced false accusations during remote learning expansion in 2020.

Detroit Police wrongfully arrested Robert Williams in 2020 after facial recognition technology falsely matched him to a shoplifting suspect. He was detained for 30 hours before being released when the error was discovered.