Data poisoning
Injecting bad examples into a model's training data so it learns the wrong thing. In your FA practice, this looks like using unverified 'open-source' datasets to fine-tune your internal models, effectively importing a Trojan horse.
Data poisoning occurs when malicious or unverified data is introduced into a training or fine-tuning pipeline to manipulate model outputs. In practice, this often happens when developers integrate untrusted open-source datasets, inadvertently embedding backdoors or performance biases that compromise the model's integrity.
Data poisoning is an adversarial machine learning attack involving the intentional injection of corrupted or adversarial samples into the training or fine-tuning distribution. By manipulating the model's objective function through poisoned gradients or feature-space perturbations, an attacker can induce targeted misclassifications, create persistent backdoors, or degrade overall model utility, often facilitated by the ingestion of unverified third-party datasets.
evolution
- 2004 · historyPoisoning Attacks on SVMs
Biggio et al. formally demonstrated that support vector machines could be compromised by injecting malicious training samples.
- 2012 · historyLabel Flipping Attacks
Researchers introduced label flipping as a specific poisoning technique to degrade the performance of machine learning classifiers.
- 2017 · historyBackdoor Attacks in Deep Learning
Gu et al. published 'BadNets,' showing how neural networks can be trained to trigger specific malicious behaviors via hidden patterns.
- 2020 · historyPoisoning Large-Scale Datasets
Studies demonstrated that poisoning attacks could be effective against large-scale models like those trained on ImageNet by manipulating a small fraction of the data.
- 2023 · historySupply Chain Poisoning via Open Source
Security researchers highlighted the risk of 'data poisoning' in LLM supply chains through the use of unverified, scraped, or compromised public datasets.