Lensa AI Generated Sexualized and Racialized Avatars from User Photos

In November 2022, Prisma Labs released the Magic Avatars feature in their Lensa AI app, which used Stable Diffusion 1.4 to generate artistic avatars from user-uploaded selfies. The feature quickly gained popularity, reaching over 20 million downloads and generating significant revenue through in-app purchases. However, within weeks of launch, users began reporting serious issues with the output quality and appropriateness. Women users, particularly women of color, reported that the app frequently generated sexualized, nude, or pornographic versions of their avatars despite uploading modest, clothed photos. Asian women were disproportionately affected, with many receiving hypersexualized anime-style renderings that played into fetishistic stereotypes. The bias was systematic rather than isolated, with multiple users documenting consistent patterns of inappropriate sexualization across different photo inputs. Technical analysis revealed that the underlying issue stemmed from Stable Diffusion's training data, which included substantial amounts of pornographic content and images that reinforced racial and gender stereotypes. The model had learned to associate certain demographic features with sexualized content, causing it to generate inappropriate outputs even from non-sexual inputs. Researchers found that the LAION dataset used to train Stable Diffusion contained millions of images from adult websites and platforms that perpetuated harmful stereotypes. The incident gained widespread media attention in early December 2022 when technology journalists and AI researchers documented the systematic bias patterns. Social media campaigns highlighted the differential treatment of users based on race and gender, with side-by-side comparisons showing how the same feature produced professional, non-sexualized avatars for white men while generating inappropriate content for women of color. The controversy sparked broader discussions about bias in AI training data and the responsibilities of companies deploying generative AI models. Prisma Labs initially defended the app by claiming users could regenerate results if unsatisfied, but faced mounting criticism for placing the burden on users rather than addressing the underlying bias. The company eventually acknowledged the issues and implemented content filtering measures, though critics argued these were insufficient post-hoc solutions. The incident led to decreased usage of the app and damaged Prisma Labs' reputation in the AI community.