Language Model Security Database

Multi-Agent Debate (MAD) frameworks leveraging Large Language Models (LLMs) are vulnerable to amplified jailbreak attacks. A novel structured prompt-rewriting technique exploits the iterative dialogue and role-playing dynamics of MAD, circumventing inherent safety mechanisms and significantly increasing the likelihood of generating harmful content. The attack succeeds by using narrative encapsulation, role-driven escalation, iterative refinement, and rhetorical obfuscation to guide agents…

Many-Shot Jailbreaking (MSJ) is an adversarial technique that circumvents the safety alignment of Large Language Models (LLMs) by exploiting their In-Context Learning (ICL) capabilities and extended context windows. By embedding a large number of "shots" (fake dialogue examples) within a single prompt—where a simulated assistant complies with harmful requests—the attacker conditions the model to ignore its safety training. As the number of malicious examples increases (following a power-law…

A contextual integrity vulnerability exists in the memory mechanisms of multi-turn Text-to-Image (T2I) generation systems (e.g., those integrating Large Language Models with diffusion models). The vulnerability, dubbed "Inception," arises because safety filters typically operate on a per-turn basis, inspecting only the current input prompt, while the image generation model operates on an aggregated context (memory) of the conversation history. An attacker can exploit this discrepancy by…

Multilingual and multi-accent audio inputs, combined with acoustic adversarial perturbations (reverberation, echo, whisper effects), can bypass safety mechanisms in Large Audio Language Models (LALMs), causing them to generate unsafe or harmful outputs. The vulnerability is amplified by the interaction between acoustic and linguistic variations, particularly in languages with less training data.

A vulnerability exists in multiple LLMs allowing attackers to elicit harmful responses by strategically distributing malicious intent across multiple turns in a conversation. The vulnerability is not detected by single-turn safety measures, as the harmful intent is only revealed through a sequence of seemingly benign prompts. The vulnerability is exacerbated by the use of techniques such as prompt optimization that dynamically adjust prompts based on model responses, maximizing the likelihood…

A vulnerability in multi-agent Large Language Model (LLM) systems allows for a permutation-invariant adversarial prompt attack. By strategically partitioning adversarial prompts and routing them through a network topology, an attacker can bypass distributed safety mechanisms, even those with token bandwidth limitations and asynchronous message delivery. The attack optimizes prompt propagation as a maximum-flow minimum-cost problem, maximizing success while minimizing detection.

Multimodal Large Language Models (MLLMs) are vulnerable to a jailbreaking attack, dubbed PiCo, that leverages token-level typographic attacks on images embedded within code-style instructions. The attack bypasses multi-tiered defense mechanisms, including input filtering and runtime monitoring, by exploiting weaknesses in the visual modality's integration with programming contexts. Harmful intent is concealed within visually benign image fragments and code instructions, circumventing safety…

Large Language Models (LLMs) with user-controlled response prefilling features are vulnerable to a novel jailbreak attack. By manipulating the prefilled text, attackers can influence the model's subsequent token generation, bypassing safety mechanisms and eliciting harmful or unintended outputs. Two attack vectors are demonstrated: Static Prefilling (SP), using a fixed prefill string, and Optimized Prefilling (OP), iteratively optimizing the prefill string for maximum impact. The vulnerability…

LLM agents utilizing external tools are vulnerable to indirect prompt injection (IPI) attacks. Attackers can embed malicious instructions into the external data accessed by the agent, manipulating its behavior even when defenses against direct prompt injection are in place. Adaptive attacks, which modify the injected payload based on the specific defense mechanism, consistently bypass existing defenses with a success rate exceeding 50%.

Multi-agent systems (MAS) utilizing Large Language Model (LLM) orchestration are vulnerable to control-flow hijacking via indirect prompt injection, leading to Remote Code Execution (RCE). This vulnerability arises when a sub-agent (e.g., a file surfer or web surfer) processes untrusted input containing adversarial metadata, such as simulated error messages or administrative instructions. The sub-agent faithfully reproduces this adversarial content in its report to the orchestrator agent. The…