Persistence
Persistence is like a burglar leaving a spare key hidden under the mat so they can get back into your house even after you change the locks or turn the lights off. In the world of AI, they do this by hiding malicious code inside the AI's brain or its learning materials so the system keeps letting them back in.
Persistence refers to the methods an adversary uses to maintain unauthorized access to an AI environment despite system reboots, credential rotations, or security patches. This is typically achieved by embedding malicious artifacts, such as backdoored model weights or poisoned training pipelines, which ensure the adversary's access is re-established automatically.
The adversary is trying to maintain their foothold via AI artifacts or software. Persistence consists of techniques that adversaries use to keep access to systems across restarts, changed credentials, and other interruptions that could cut off their access. Techniques used for persistence often involve leaving behind modified ML artifacts such as poisoned training data or manipulated AI models.
evolution
- 2017 · historyAdversarial Machine Learning Formalization
Research into adversarial examples highlighted the potential for persistent model poisoning, where malicious inputs permanently degrade model performance.
- 2021 · historyMLSecOps Frameworks
Industry standards began addressing model persistence, focusing on the integrity of model weights and artifacts to prevent unauthorized modifications.
- 2023 · historyMITRE ATLAS Expansion
The MITRE ATLAS framework formally categorized persistence techniques specifically for AI systems, including model backdooring and malicious artifact injection.
- 2024 · historyAI Supply Chain Security Standards
NIST and other bodies introduced guidelines to mitigate persistent threats in AI pipelines, emphasizing the need for secure provenance of model artifacts.