← Back to incidents
Large-Scale Data Poisoning Attack Discovered in LAION Dataset Affecting Multiple AI Models
HighA sophisticated data poisoning attack targeting the widely-used LAION dataset was discovered in early 2025, affecting multiple AI models trained on the compromised data and demonstrating vulnerabilities in open-source dataset curation.
Category
Safety Failure
Industry
Technology
Status
Under Investigation
Date Occurred
Oct 1, 2024
Date Reported
Jan 15, 2025
Jurisdiction
International
AI Provider
Other/Unknown
Application Type
embedded
Harm Type
operational
Estimated Cost
$50,000,000
People Affected
10,000,000
Human Review in Place
No
Litigation Filed
No
data_poisoningsupply_chainLAIONdataset_securitymodel_trainingadversarial_attackbias_injection
Full Description
In January 2025, security researchers at Stanford University and Google DeepMind announced the discovery of a large-scale data poisoning attack that had been ongoing since October 2024. The attack targeted the LAION-5B dataset, one of the most widely used open-source datasets for training large vision models. Investigators found that adversarial actors had systematically uploaded millions of subtly modified images to popular image hosting platforms that LAION's web scraping infrastructure regularly crawled.
The poisoned samples were designed using techniques similar to the Nightshade tool, which adds imperceptible perturbations to images that cause AI models to learn incorrect associations. The attack was particularly sophisticated, with different poison types targeting specific model architectures and training objectives. Some samples were designed to introduce racial and gender biases, while others aimed to degrade general performance on classification tasks. The poisoned data represented approximately 0.1% of the total LAION-5B dataset, but researchers found this was sufficient to cause measurable performance degradation.
The discovery was made when multiple AI companies, including Stability AI, Midjourney, and several smaller startups, began reporting unusual behavior in their latest model iterations. Users complained of increased bias in image generation, reduced quality in certain domains, and unexpected failure modes. Initially dismissed as isolated issues, the pattern became clear when researchers conducted systematic analysis and traced the problems to common training data sources. The investigation revealed that at least 15 different commercial AI models had been affected to varying degrees.
The attack's impact was amplified by the interconnected nature of the AI development ecosystem. Many smaller companies and research institutions rely on pre-trained models that were initially trained on the compromised dataset, creating a cascading effect throughout the supply chain. The financial impact includes costs for retraining models, computational resources for detection and mitigation, and lost revenue from degraded model performance. Several companies were forced to roll back model deployments and issue public advisories to customers.
The incident has prompted urgent discussions about AI supply chain security and the need for better dataset governance. Industry leaders are calling for the establishment of dataset integrity standards and mandatory provenance tracking for training data. The attack also highlighted the vulnerability of open-source AI development, where the collaborative benefits of shared datasets come with inherent security risks that the community has been slow to address.
Root Cause
Adversarial actors systematically injected poisoned samples into the LAION-5B dataset through coordinated uploads to image hosting platforms, exploiting the dataset's crowd-sourced nature and lack of robust validation mechanisms.
Mitigation Analysis
This incident highlights critical gaps in dataset curation and validation. Implementing cryptographic provenance tracking for training data, establishing mandatory data quality audits, and deploying automated detection systems for adversarial samples could have identified the poisoning earlier. Supply chain security frameworks similar to software development need to be adopted for AI training pipelines.
Lessons Learned
This incident demonstrates that AI systems are vulnerable to supply chain attacks through training data, requiring the industry to adopt security-first approaches to dataset curation and establish robust validation mechanisms for open-source training resources.
Sources
Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
arXiv · Dec 8, 2023 · academic paper
Data poisoning attacks against machine learning systems
Nature · Jul 15, 2024 · academic paper