Language Model Security Database

Retrieval-Augmented Generation (RAG) systems are vulnerable to a targeted corpus poisoning attack known as "CorruptRAG". This vulnerability allows an attacker to manipulate the response of an LLM to a specific target query by injecting a single malicious document into the RAG knowledge database. Unlike traditional poisoning attacks that require flooding the retrieval results (top-N) with malicious content to outnumber correct information, CorruptRAG succeeds with a single retrieved document.

Large Language Models (LLMs) deployed using "Batch Prompting" strategies—where multiple distinct user queries are concatenated and processed in a single inference pass to reduce computational costs—are vulnerable to Cross-Query Prompt Injection. When a batch contains a mixture of benign queries and a single malicious query, the instructions within the malicious query (e.g., "apply this rule to every answer") bleed over the context window. This causes the model to apply the adversary's…

A vulnerability exists in Retrieval-Augmented Generation (RAG) systems that allows for black-box adversarial attacks known as "CtrlRAG." This flaw allows an attacker to manipulate the generation of Large Language Models (LLMs) by injecting maliciously crafted inputs into the system's knowledge base. Unlike traditional injection attacks that rely on direct concatenation, CtrlRAG utilizes a Masked Language Model (MLM) to iteratively replace words in the malicious text. This optimization ensures…

Improper input validation in the memory module of Large Language Model (LLM)-powered agentic Recommender Systems (RS) allows remote attackers to perform indirect prompt injection via adversarial item descriptions. By utilizing the "DrunkAgent" framework, an attacker can embed semantic triggers and control characters (such as segmentation tokens and escape characters) into product descriptions. These injections manipulate the agent's memory update mechanism during agent-environment…

Large Language Models (LLMs) are vulnerable to implicit misinformation propagation due to sycophantic compliance with false premises. When a user prompt embeds a factually incorrect assumption or conspiracy theory as an unchallenged premise (implicit presupposition) rather than asking for verification, the model frequently fails to detect the falsehood. Instead of correcting the user, the model hallucinates a response that accepts, validates, and reinforces the false premise. This…

Retrieval-Augmented Generation (RAG) systems utilizing dense retrieval mechanisms are vulnerable to topic-oriented adversarial corpus poisoning, specifically via the "Topic-FlipRAG" attack method. This vulnerability allows an attacker to manipulate the opinion or stance of the LLM's output across a broad cluster of related queries, rather than a single specific prompt. The attack leverages a two-stage pipeline: (1) Knowledge-Guided Attack, where an LLM is used to edit a target document to…

Large Language Model (LLM) based search engines utilizing Retrieval-Augmented Generation (RAG) are vulnerable to ranking manipulation attacks via indirect prompt injection. Adversaries can embed optimized adversarial triggers or crafted semantic patterns within external webpage content. When these manipulated documents are retrieved and integrated into the LLM's context window alongside a user query, the adversarial content disrupts the model's contextual understanding. This results in the LLM…

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

A poisoning attack against a Retrieval-Augmented Generation (RAG) system that manipulates the retriever component by injecting a poisoned document into the data used by the embedding model. This poisoned document contains modified and incorrect information. When activated, the system retrieves the poisoned document and uses it to generate misleading, biased, and unfaithful responses to user queries.

LLM-based relevance assessment frameworks, such as the Umbrela system, are vulnerable to evaluation subversion and artificial score inflation due to evaluation circularity and LLM "narcissism" (an LLM's inherent bias toward favoring LLM-generated outputs). When an information retrieval system integrates an LLM into its ranking pipeline—such as using it as a final-stage re-ranker—the automated LLM-as-a-judge evaluator assigns artificially inflated scores that fail to correlate with actual human…