AI Incident Database

Lensa AI's Magic Avatars feature generated sexualized and stereotypical images of women users, particularly women of color, due to biased training data in the underlying Stable Diffusion model.

Studies revealed that Zillow's Zestimate algorithm systematically undervalues homes in predominantly Black neighborhoods compared to similar properties in white areas, perpetuating housing discrimination through automated valuation models.

Allegheny County's AI child welfare screening tool disproportionately flagged Black families and public benefit recipients for investigations. An AP investigation revealed the algorithmic bias affected thousands of families over six years.

Major AI-powered real estate platforms including Zillow, Redfin, and Realtor.com systematically undervalued homes in Black neighborhoods. Brookings research documented billions in lost equity, prompting regulatory action and ongoing litigation.

Harvard Business School study revealed AI-powered resume parsing in major ATS systems systematically filtered out 27 million qualified workers due to technical parsing errors, rejecting candidates based on formatting rather than qualifications.

ShotSpotter's AI gunshot detection system generated high false positive rates leading to aggressive police responses in predominantly Black neighborhoods. Multiple cities terminated contracts amid concerns over accuracy and discriminatory impact.

AI surveillance systems deployed by major retailers including Walmart, Target, and CVS were documented by the ACLU to disproportionately flag and track Black shoppers, embedding racial profiling into automated retail security.

Lemonade Insurance faced regulatory scrutiny after tweeting about using AI and facial recognition to analyze video claims, raising concerns about algorithmic bias in insurance decisions.

AI content moderation systems on major social media platforms systematically removed Palestinian content during 2021 Sheikh Jarrah protests and 2023 Gaza conflict. Human Rights Watch documented widespread censorship affecting millions of users.

Meta's AI content moderation systems systematically censored Palestinian news and voices during 2021 and 2023 Gaza conflicts, with Human Rights Watch documenting widespread suppression of legitimate content.

Google's AI dermatology tool demonstrated significant accuracy disparities across skin tones due to biased training data, highlighting the critical need for diverse datasets in medical AI applications.

AI video interview platforms by HireVue, myInterview, and Pymetrics systematically discriminated against candidates with disabilities by penalizing atypical speech, facial expressions, and eye movements. The EEOC issued guidance and multiple lawsuits were filed.

ShotSpotter's AI gunshot detection system exhibited false positive rates of 86-95%, leading to wrongful police raids and discriminatory enforcement in predominantly Black neighborhoods across multiple US cities.

USC researchers found LinkedIn's job ad algorithm delivered high-paying job listings disproportionately to men despite gender-neutral employer targeting. The bias stemmed from optimization algorithms that learned from historical engagement patterns.

South Korean chatbot Lee Luda was shut down after generating homophobic and racist content and being found to have illegally trained on 600,000 users' private KakaoTalk conversations without consent.

South Korean startup Scatter Lab's AI chatbot Lee Luda was shut down after making discriminatory comments and violating privacy laws by training on 10 billion private KakaoTalk messages without user consent, affecting 750,000 users.

South Korean AI chatbot Lee Luda was shut down after generating discriminatory content against minorities and sexually inappropriate responses, leading to regulatory fines and lawsuits affecting 750,000 users.

Dutch tax authority's AI system wrongly flagged thousands of families for childcare benefits fraud based on discriminatory factors like dual nationality. The scandal caused widespread financial hardship and led to the collapse of the Dutch government in 2021.

Alibaba Cloud's AI services included ethnic detection capabilities specifically marketed to identify Uyghur faces, supporting Chinese government surveillance programs. The discovery led to international sanctions and the company removing the feature.

Bavarian research revealed that Retorio's AI video interview tool evaluated job candidates based on irrelevant factors like clothing and background objects rather than qualifications. The findings highlighted systematic bias in AI hiring tools that could violate anti-discrimination laws.