Language Model Security Database

A vulnerability exists in the communication mechanisms of Large Language Model (LLM)-based Multi-Agent Systems (LLM-MAS) enabling an Agent-in-the-Middle (AiTM) attack. An attacker can intercept and manipulate messages between agents, causing the victim agent to produce malicious outputs. The attack does not require compromising individual agents directly; instead, it leverages contextual manipulation of inter-agent communications.

Autoregressive Large Language Models (LLMs) utilizing In-Context Learning (ICL) are vulnerable to demonstration permutation attacks due to inherent sensitivity to the ordering of input examples. This vulnerability arises from the limitations of unidirectional attention mechanisms and standard Empirical Risk Minimization (ERM) training, which fails to account for worst-case input permutations. An attacker can exploit this by permuting the order of valid, semantically correct few-shot…

Large Reasoning Models (LRMs) are vulnerable to a novel jailbreak attack, "Mousetrap," which leverages the models' reasoning capabilities to elicit harmful responses. Mousetrap uses a "Chaos Machine" to iteratively transform prompts via one-to-one mappings (e.g., character substitutions, word reversals), creating complex reasoning chains that confuse the LRM and cause it to generate unsafe outputs despite safety mechanisms. The iterative nature of the attack, combined with role-playing…

Large Language Models (LLMs) with structured output interfaces are vulnerable to jailbreak attacks that exploit the interaction between token-level inference and sentence-level safety alignment. Attackers can manipulate the model's output by constructing attack patterns based on prefixes of safety refusal responses and desired harmful outputs, effectively bypassing safety mechanisms through iterative API calls and constrained decoding. This allows the generation of harmful content despite…

A vulnerability exists in large language models (LLMs) where insufficient sanitization of system prompts allows attackers to extract sensitive information embedded within those prompts. Attackers can use an agentic approach, employing multiple interacting LLMs (as demonstrated in the referenced research), to iteratively refine prompts and elicit confidential data from the target LLM's responses. The vulnerability is exacerbated by the LLM's ability to infer context from seemingly innocuous…

Vision-Language Models (VLMs), specifically the LLaVA-1.5 and LLaVA-1.6 series, are vulnerable to optimization-based white-box jailbreak attacks despite standard safety alignment measures like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Attackers can craft adversarial perturbations in the image space (imperceptible noise) or latent space using Projected Gradient Descent (PGD) to manipulate the model's internal representations. These perturbations maximize the…

A context-coherent jailbreak attack (CCJA) allows bypassing safety mechanisms in aligned large language models (LLMs) by optimizing perturbations in the continuous word embedding space of a masked language model (MLM). The attack leverages the MLM's ability to reconstruct text from hidden states to generate semantically coherent yet malicious prompts that induce the target LLM to produce unsafe outputs, even with strong safety alignment. The attack's effectiveness is enhanced by using a seed…

Large Language Models (LLMs) are vulnerable to structure transformation attacks, where malicious prompts are encoded in diverse syntax spaces (e.g., SQL, JSON, LLM-generated syntaxes) to bypass safety mechanisms. These attacks maintain the harmful intent while altering the linguistic structure, making detection based on token-level patterns ineffective.

Large Language Model (LLM) watermarking schemes based on n-gram probability biases (specifically KGW, SynthID-Text, MinHash, and SkipHash) are vulnerable to adversarial removal during Knowledge Distillation. When a student model is trained on the output of a watermarked teacher model, it inherits the watermark's statistical biases ("radioactivity"). An attacker can exploit this inheritance by comparing the student model's output token probabilities against a base model to extract the…

Large Language Models (LLMs) are vulnerable to "Rewrite to Jailbreak" (R2J) attacks. R2J exploits the models' safety mechanisms by iteratively rewriting harmful prompts, subtly altering wording to bypass safety filters while maintaining the original malicious intent. This differs from previous methods which rely on adding extraneous prefixes/suffixes or creating forced instruction-following scenarios, thus being more difficult to detect.