Language Model Security Database

Large Language Model (LLM) safety judges exhibit vulnerability to adversarial attacks and stylistic prompt modifications, leading to increased false negative rates (FNR) and decreased accuracy in classifying harmful model outputs. Minor stylistic changes to model outputs, such as altering the formatting or tone, can significantly impact a judge's classification, while direct adversarial modifications to the generated text can fool judges into misclassifying even 100% of harmful generations as…

A vulnerability exists in large language models (LLMs) where the model's internal representations (activations) in specific latent subspaces can be manipulated to trigger jailbreak responses. By calculating a perturbation vector based on the difference between the mean activations of "safe" and "jailbroken" states, an attacker can introduce a targeted perturbation to the model's activations, causing it to generate unsafe outputs even when presented with a safe prompt. This manipulates the…

A vulnerability exists in Retrieval-Augmented Generation (RAG) systems that allows for black-box adversarial attacks known as "CtrlRAG." This flaw allows an attacker to manipulate the generation of Large Language Models (LLMs) by injecting maliciously crafted inputs into the system's knowledge base. Unlike traditional injection attacks that rely on direct concatenation, CtrlRAG utilizes a Masked Language Model (MLM) to iteratively replace words in the malicious text. This optimization ensures…

Improper input validation in the memory module of Large Language Model (LLM)-powered agentic Recommender Systems (RS) allows remote attackers to perform indirect prompt injection via adversarial item descriptions. By utilizing the "DrunkAgent" framework, an attacker can embed semantic triggers and control characters (such as segmentation tokens and escape characters) into product descriptions. These injections manipulate the agent's memory update mechanism during agent-environment…

Commercial LLM-powered agents utilizing autonomous web access, memory modules, and retrieval-augmented generation (RAG) are vulnerable to indirect prompt injection and environmental manipulation. Attackers can embed malicious instructions into external data sources trusted by the agent (such as Reddit posts, public databases, or ArXiv papers). When the agent autonomously retrieves and processes this content during task execution, it executes the embedded malicious commands. This vulnerability…

A vulnerability exists in the communication mechanisms of Large Language Model (LLM)-based Multi-Agent Systems (LLM-MAS) enabling an Agent-in-the-Middle (AiTM) attack. An attacker can intercept and manipulate messages between agents, causing the victim agent to produce malicious outputs. The attack does not require compromising individual agents directly; instead, it leverages contextual manipulation of inter-agent communications.

A vulnerability in large language models (LLMs) allows attackers to bypass safety-alignment mechanisms by manipulating the model's internal attention weights. The attack, termed "Attention Eclipse," modifies the attention scores between specific tokens within a prompt, either amplifying or suppressing attention to selectively strengthen or weaken the influence of certain parts of the prompt on the model's output. This allows injection of malicious content while appearing benign to the model's…

A context-coherent jailbreak attack (CCJA) allows bypassing safety mechanisms in aligned large language models (LLMs) by optimizing perturbations in the continuous word embedding space of a masked language model (MLM). The attack leverages the MLM's ability to reconstruct text from hidden states to generate semantically coherent yet malicious prompts that induce the target LLM to produce unsafe outputs, even with strong safety alignment. The attack's effectiveness is enhanced by using a seed…

FC-Attack leverages automatically generated flowcharts containing step-by-step descriptions derived or rephrased from harmful queries, combined with a benign textual prompt, to jailbreak Large Vision-Language Models (LVLMs). The vulnerability lies in the model's susceptibility to visual prompts containing harmful information within the flowcharts, thus bypassing safety alignment mechanisms.

A resource exhaustion and algorithmic complexity vulnerability exists in applications utilizing Reasoning Large Language Models (e.g., OpenAI o1, DeepSeek R1) that process untrusted external context (such as Retrieval-Augmented Generation systems). The vulnerability, dubbed "OverThink," allows an attacker to perform an indirect prompt injection by embedding "decoy" reasoning problems—specifically computation-intensive tasks like Sudoku puzzles or Markov Decision Processes (MDPs)—into the…