AI Incident Database

Student Borrower Protection Center found Upstart's AI lending algorithm charged HBCU graduates higher interest rates, revealing algorithmic discrimination despite regulatory approval.

Proctorio's AI exam proctoring software disproportionately flagged students of color as potential cheaters due to biased facial recognition algorithms. Multiple universities discontinued the service after documented bias incidents.

Twitter's automatic image cropping algorithm systematically favored white faces over Black faces in preview images. Users demonstrated the bias through controlled experiments, leading Twitter to acknowledge the issue and eventually remove automatic cropping entirely.

The UK government's online passport photo verification AI systematically rejected photos of dark-skinned applicants with false error messages like 'mouth too open', demonstrating clear racial bias in a critical government service.

Ofqual's 2020 A-level grading algorithm downgraded 40% of teacher-predicted grades, systematically disadvantaging students from state schools and disadvantaged backgrounds. After mass protests and legal challenges, the government abandoned the results.

Ofqual's A-level grading algorithm in 2020 downgraded nearly 40% of students' predicted grades, systematically discriminating against disadvantaged students. Mass protests forced the government to abandon the algorithm and revert to teacher assessments within days.

AI proctoring software from companies like Proctorio flagged Black students as cheating at disproportionate rates due to facial recognition bias. Thousands of students faced false accusations during remote learning expansion in 2020.

Detroit Police wrongfully arrested Robert Williams in 2020 after facial recognition technology falsely matched him to a shoplifting suspect. He was detained for 30 hours before being released when the error was discovered.

A widely-used kidney function algorithm systematically overestimated kidney health in Black patients for over two decades, delaying critical transplant referrals and treatment for millions.

ZestFinance's AI credit scoring platform faced CFPB examination over potential fair lending violations, highlighting challenges in ensuring complex machine learning models don't embed discriminatory patterns that traditional auditing methods cannot detect.

Root Insurance's smartphone-based AI pricing algorithm was investigated by Colorado regulators for potential discrimination against drivers in lower-income and minority neighborhoods through biased telematics data analysis.

The Netherlands' SyRI welfare fraud detection system was banned by a Dutch court in 2020 for systematically discriminating against immigrant communities through algorithmic profiling.

Spain's VioGén algorithm for assessing domestic violence risk was criticized after multiple fatal incidents where victims were classified as low risk and received inadequate protection.

The AI Now Institute found that AI emotion detection technology lacked scientific validity yet was being used by companies for hiring decisions, potentially creating unfair bias against job candidates.

Apple's credit card, managed by Goldman Sachs, was accused of algorithmically discriminating against women by offering them significantly lower credit limits than men with equivalent or superior financial profiles. The controversy began when tech entrepreneur David Heinemeier Hansson posted that his wife received a credit limit 20x lower than his despite a higher credit score.

HireVue's AI video interview platform faced FTC complaint over biased facial analysis technology that potentially discriminated against job candidates. The company discontinued facial analysis features in 2021.

Optum's widely-used healthcare risk prediction algorithm showed severe racial bias, requiring Black patients to be significantly sicker than white patients to receive the same care recommendations, affecting an estimated 200 million patients nationwide.

Optum's widely-used healthcare risk algorithm systematically underestimated care needs for Black patients by using healthcare spending as a proxy for health status, affecting over 10 million patients across major health systems.

LAPD's use of PredPol predictive policing software from 2011-2019 created feedback loops that disproportionately targeted Black and Latino neighborhoods, with multiple academic studies documenting systematic bias before the department ended the program.

Academic research revealed Uber's AI surge pricing algorithm consistently charged higher prices in minority and lower-income neighborhoods due to supply-demand patterns that correlated with demographics.