AI Incident Database

Microsoft's Zo chatbot, launched in 2017 as an improved successor to Tay, still produced offensive content including religious bias when users found ways to bypass its safety filters.

Palantir's AI system enabled ICE to analyze data from schools and social services to identify and target undocumented immigrants for deportation, raising significant civil liberties concerns.

Consumer Reports and ProPublica investigations revealed that State Farm, Allstate, and other major insurers used AI pricing models that systematically charged higher premiums in minority neighborhoods, affecting millions of consumers despite controlling for legitimate risk factors.

Facebook's ad targeting system allowed advertisers to exclude users by race and ethnicity from housing ads, violating civil rights laws and resulting in a $115 million DOJ settlement.

Beauty.AI's 2016 contest used AI to judge beauty from 6,000+ global submissions but selected almost exclusively white winners, revealing severe training data bias and algorithmic discrimination.

Chicago's predictive policing algorithm disproportionately targeted Black and Latino residents for police attention, increasing their likelihood of being shot by police while failing to reduce overall violence rates.

AI risk assessment tools like COMPAS and PATTERN used across US prison systems exhibited racial bias, incorrectly flagging Black defendants as high-risk at nearly twice the rate of white defendants and keeping low-risk prisoners incarcerated longer.

Northpointe's COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm, used in courts across the United States to assess the likelihood of criminal defendants reoffending, was found to exhibit significant racial bias. A ProPublica investigation revealed that Black defendants were nearly twice as likely to be falsely flagged as future criminals compared to white defendants.

Bloomberg investigation revealed Amazon's same-day delivery algorithm systematically excluded predominantly Black neighborhoods in major US cities, creating patterns nearly identical to historical redlining maps.

Microsoft's Tay chatbot was shut down within 16 hours of launch after coordinated trolling caused it to post racist and offensive tweets, demonstrating the risks of unsupervised AI learning from public social media interactions.

Microsoft's Tay chatbot began posting racist and Nazi content within 16 hours of launch after coordinated manipulation by users who exploited its learning mechanisms. The incident forced immediate shutdown and highlighted critical gaps in adversarial AI safety.

Michigan's automated unemployment fraud detection system falsely accused 40,000 people of fraud between 2013-2015 with a 93% error rate, leading to wrongful debt collection and a $20 million settlement.

Carnegie Mellon researchers discovered Google's ad algorithm showed high-paying job ads to men significantly more often than women. The controlled study used fake profiles to demonstrate systematic gender bias in employment advertising, raising concerns about algorithmic discrimination in hiring practices.

Google Photos' AI image recognition system labeled photos of Black people as 'gorillas' in 2015. Google's response was to remove the 'gorilla' category entirely rather than fix the underlying algorithmic bias, which reportedly remained unresolved as of 2023.

Yelp's AI review filtering system allegedly suppressed legitimate positive reviews for small businesses while preserving negative ones, creating financial pressure to buy ads. Class action lawsuits were filed but courts ruled Yelp had no obligation to display reviews fairly.

MIT researcher Les Perelman demonstrated that ETS's e-rater automated essay scoring system used for GRE and GMAT could be gamed with verbose nonsense while showing bias against non-standard English dialects.

Automated welfare eligibility systems across multiple US states wrongfully terminated benefits for over one million vulnerable Americans between 2007-2020. The algorithms contained systematic biases leading to $1.2 billion in estimated harm through wrongful benefit cuts.