RAG Poisoning
RAG Poisoning is like a prankster sneaking fake pages into a library's reference books. When a student asks the librarian for information, the librarian unknowingly hands them the fake page, causing the student to learn incorrect facts or follow bad instructions.
RAG Poisoning occurs when an attacker introduces malicious or misleading data into the external knowledge base used by a RAG system. By manipulating indexed documents, the attacker ensures that the system retrieves this tainted content during a search, effectively forcing the AI model to incorporate false information or malicious prompt injections into its generated responses.
Adversaries may inject malicious content into data indexed by a retrieval augmented generation (RAG) system to contaminate a future thread through RAG-based search results. This may be accomplished by placing manipulated documents in a location the RAG indexes (see AML.T0064). The content may be targeted such that it would always surface as a search result for a specific user query, potentially including false information, adversarial prompt injections, or malformed RAG entries designed to subvert the model's output generation process.