Amazon Rekognition Falsely Matched 28 Members of Congress as Criminals in ACLU Test

In July 2018, the American Civil Liberties Union conducted a landmark test of Amazon's Rekognition facial recognition service that exposed significant racial bias in the technology. The ACLU compared official photos of all 535 members of Congress against a database of 25,000 publicly available arrest photos using Amazon's default confidence threshold of 80%. The test was designed to simulate how law enforcement agencies might use the technology in practice. The results were striking: Amazon Rekognition incorrectly matched 28 members of Congress as criminals, including notable false positives such as Representatives John Lewis, Bobby Scott, and Joaquin Castro. Of the 28 false matches, a disproportionate number were people of color, despite people of color comprising only about 20% of Congress at the time. The false positive rate was nearly 40% for the Congressional Black Caucus, compared to roughly 5% for white members of Congress. The ACLU's test methodology involved using Amazon's facial comparison API to compare each Congressional photo against the mugshot database, using the same default settings that would be available to law enforcement customers. The organization noted that Amazon marketed Rekognition specifically to law enforcement agencies, with several police departments and government agencies already using the technology for identification purposes. Amazon's response was swift but initially defensive. The company argued that the ACLU had used inappropriate settings for a law enforcement application, claiming that a confidence threshold of 95% or higher should be used for such purposes rather than the 80% default. Amazon also disputed the ACLU's methodology, suggesting that the test didn't reflect real-world usage patterns. However, the ACLU countered that they had used Amazon's own default settings and that many law enforcement agencies were likely using similar configurations. The incident sparked significant congressional concern and public debate about facial recognition technology in law enforcement. Representatives who were misidentified, including John Lewis and Jimmy Gomez, called for investigations and potential regulations. The test results became central evidence in congressional hearings about AI bias and facial recognition technology, leading to proposed legislation including the Facial Recognition and Biometric Technology Moratorium Act. The broader implications extended beyond Amazon, highlighting systemic issues with facial recognition accuracy across demographic groups. The incident contributed to a growing movement for facial recognition bans and moratoria in various cities and states, and increased scrutiny of AI bias in law enforcement applications. Several major tech companies, including Microsoft and IBM, subsequently stepped back from facial recognition technology for law enforcement, citing accuracy and bias concerns similar to those demonstrated in the ACLU test.