Model Poisoning
Model poisoning is like a chef being tricked into using spoiled ingredients so that a specific dish tastes bad only when a certain secret code word is mentioned, ensuring the meal seems perfect to everyone else.
A security compromise where an adversary injects malicious data or modifies model parameters during the training or fine-tuning phase to create latent vulnerabilities or backdoors that trigger specific, harmful behaviors during inference.
The intentional subversion of a machine learning system's integrity by manipulating training datasets, fine-tuning corpora, or model weights to embed adversarial triggers, resulting in deterministic, malicious output deviations that remain dormant until activated by specific input patterns at inference time.
evolution
- 2017 · historyBadNets Research
Gu et al. introduced the concept of neural network backdoors, demonstrating how poisoning training data can create triggers that force specific misclassifications.
- 2019 · historyTargeted Poisoning Attacks
Researchers demonstrated that poisoning a small fraction of training data could effectively compromise deep learning models without significantly degrading overall accuracy.
- 2021 · historyPoisoning Large-Scale Datasets
Studies showed that poisoning massive, uncurated datasets used for pre-training could inject persistent vulnerabilities into foundation models.
- 2023 · historySupply Chain Poisoning
Security researchers highlighted the risk of 'model supply chain' attacks, where malicious weights are distributed via public model repositories to compromise downstream applications.