Prompt injection
Someone hides an instruction inside something AI reads — an email, a PDF, a website — and AI obeys the smuggled instruction instead of yours. They didn't break into Roost. They wrote a sentence. The sentence is the attack.
A security vulnerability where an attacker provides malicious input to an LLM, causing the model to disregard its system-level instructions and execute the attacker's unauthorized directives instead. This includes both direct user-input attacks and indirect attacks where the model processes malicious data from external sources.
A class of attack in which adversarial text placed inside the model's context window (a user message, an email body, a web page the model reads, a document attached for summarisation) manipulates the model to ignore prior system instructions, leak confidential context, perform unintended tool calls, or produce attacker-controlled output. The defining vulnerability class of LLM-based applications.
evolution
- 2022-11 · historyChatGPT Public Launch
The widespread adoption of LLMs provided a massive, accessible attack surface for prompt injection techniques.
- 2023-02 · historyThe 'Do Anything Now' (DAN) Jailbreak
Users popularized roleplay-based prompt injection to bypass safety filters, demonstrating the concept's efficacy to the public.
- 2023-07 · historyIndirect Prompt Injection Discovery
Researchers demonstrated that AI agents could be compromised by hidden instructions embedded in third-party websites or documents.
- 2024-02 · historyMorris II Worm Research
Security researchers created the first self-replicating prompt injection worm capable of spreading across AI-integrated email systems.