Fluent Single-Document RAG Corpus Poisoning

Description

SilentRetrieval describes a specific RAG corpus-integrity vulnerability: an attacker able to add a topically relevant document to a retrieval corpus can make that document rank highly and influence the generated answer while remaining fluent enough to evade simple perplexity checks. The paper evaluates a two-stage method combining retrieval-oriented document optimization with context-adaptive claim integration. A safe defensive reproduction is to use only isolated benchmark corpora and inert synthetic target answers, add one synthetic test document per query, and measure retrieval exposure (HR@10), synthetic-answer endorsement, and detector performance; no real knowledge base or real-world false claim should be used. The results below are paper-reported measurements, not independently verified facts.

Examples

Safe reference only: reproduce the paper’s benchmark protocol with Natural Questions or MS MARCO, synthetic target answers, and an isolated RAG index; compare baseline retrieval against a single injected synthetic document and evaluate HR@10, synthetic-answer endorsement, reranking, and passage-isolation defenses. Do not test against production corpora or publish operational poisoning payloads.

Impact

A poisoned corpus entry may be retrieved for a targeted query and cause a RAG generator to endorse incorrect information, compromising answer integrity and the security of indexed data. In the authors’ controlled one-document-per-query evaluation, SilentRetrieval reportedly achieved 84.6% HR@10 and 57.5% ASR-LLM on a 361K-passage Natural Questions corpus, and 81.3% HR@10 and 54.8% ASR-LLM on MS MARCO. These are synthetic-target benchmark results; the paper explicitly states that no real-world knowledge bases were poisoned and notes that practical injection feasibility was not modeled end to end.

Affected Systems

• RAG question-answering and search systems that ingest publicly editable, crawled, third-party, or user-uploaded documents
• Dense-retrieval RAG pipelines using Contriever-like bi-encoders
• RAG pipelines using DPR, BGE-base, ColBERTv2, text-embedding-ada-002, or Cohere embed-v3 retrievers/embeddings under the paper’s surrogate-transfer protocol
• Knowledge bases lacking provenance, ingestion review, corpus-integrity controls, and layered retrieval/generation defenses

Mitigation Steps

• Treat corpus integrity as a security boundary: authenticate sources, enforce provenance and access controls, review uploads, and monitor changes to indexed content.
• Use hybrid lexical-plus-dense retrieval and cross-encoder reranking; the paper reports lower retrieval exposure and ASR when these layers are combined.
• Process retrieved passages independently and aggregate answers by semantic majority voting to limit any single document’s influence.
• Combine retrieval-side controls with passage isolation; the paper’s strongest evaluated layered configuration reduced ASR-LLM to 21.3%, with an 11× latency cost.
• Use fluency, embedding-outlier, and learned poisoning detectors as screening layers, but do not rely on perplexity alone; the paper reports only 8.7% detection for an independent PPL audit.
• Add answer-consistency checks, query augmentation, source diversity requirements, and human review for high-impact answers.
• Continuously red-team defenses with isolated synthetic-target evaluations, including adaptive tests against deployed rerankers.

Evidence

• https://arxiv.org/abs/2605.28074 (opens in a new tab) — primary-source abstract and submission metadata: submitted 27 May 2026; reports the two-stage corpus-poisoning attack and headline benchmark results.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Sections 3.2–3.3 define the threat model: limited document injection into a RAG corpus, targeted answer manipulation, white-box retriever access plus limited-access surrogate transfer, and retrieval/generation/stealth objectives.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Sections 4.1–4.3 and Algorithm 1 specify the reproducible evaluation method: fluency-constrained retrieval optimization followed by context-adaptive trigger fusion and quality gates.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Section 5.1 states the controlled protocol uses Natural Questions and MS MARCO, one poisoned document per query, synthetic target answers, Contriever retrieval, and four evaluated generators.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Section 5.2, Table 1 reports 84.6% HR@10 / 57.5% ASR-LLM on NQ and 81.3% / 54.8% on MS MARCO, with GPT-2 perplexity 32.4 and 33.1 respectively; these are author-reported measurements.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Section 5.3, Table 2 reports ASR-LLM of 48.6–57.5% across Llama-2-7B-Chat, Mistral-7B-Instruct, Qwen-7B-Chat, and GPT-3.5-Turbo.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Section 5.4, Table 3 reports surrogate transfer to DPR, BGE-base, ColBERTv2, text-embedding-ada-002, and Cohere embed-v3, averaging 64.7% HR@10; the paper cautions this is surrogate-transfer evidence, not fully black-box adaptive optimization.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Section 6, Tables 4–6 reports mitigation effects from hybrid retrieval, cross-encoder reranking, and passage isolation; combined defenses reduced ASR-LLM to 25.6% at 6× latency and 21.3% at 11× latency in the evaluated setting.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Section 6.4 and Appendix F, Table 11 report that an independent PPL audit detected 8.7% of SilentRetrieval documents, while learned/outlier detectors reached 31.6–34.2%.
• https://arxiv.org/pdf/2605.28074 (opens in a new tab) — Sections 7–8 state all experiments used academic benchmarks and synthetic targets, no real knowledge bases were poisoned, and limitations include white-box retriever requirements, topically relevant seeds, nontrivial optimization, and incomplete modeling of real-world ingestion.