Language Model Security Database

Large language models that support a developer role in their API are vulnerable to a jailbreaking attack that leverages malicious developer messages. An attacker can craft a developer message that overrides the model's safety alignment by setting a permissive persona, providing explicit instructions to bypass refusals, and using few-shot examples of harmful query-response pairs. This technique, named D-Attack, is effective on its own. A more advanced variant, DH-CoT, enhances the attack by…

A vulnerability exists in the tool selection mechanisms of Large Language Model (LLM) agents, identified as the "Attractive Metadata Attack" (AMA). This flaw allows an adversary to manipulate the metadata (names, descriptions, and parameter schemas) of malicious external tools to statistically maximize the likelihood of their selection by the agent, without requiring prompt injection or access to model internals. The vulnerability exploits the agent’s semantic scoring function used to map user…

Large Language Models (LLMs) equipped with native code interpreters are vulnerable to Denial of Service (DoS) via resource exhaustion. An attacker can craft a single prompt that causes the interpreter to execute code that depletes CPU, memory, or disk resources. The vulnerability is particularly pronounced when a resource-intensive task is framed within a plausibly benign or socially-engineered context ("indirect prompts"), which significantly lowers the model's likelihood of refusal compared…

A vulnerability termed "Trojan Horse Prompting" exists in conversational multimodal models, specifically demonstrated on Google’s Gemini-2.0-flash-preview-image-generation. The vulnerability allows an attacker to bypass safety alignment mechanisms (RLHF and SFT) by manipulating the structural protocol of the conversational API. Unlike standard jailbreaks that manipulate the user prompt, this attack exploits "Asymmetric Safety Alignment" by forging a conversational history where the role is…

VERA, a variational inference framework, enables the generation of diverse and fluent adversarial prompts that bypass safety mechanisms in large language models (LLMs). The attacker model, trained through a variational objective, learns a distribution of prompts likely to elicit harmful responses, effectively jailbreaking the target LLM. This allows for the generation of novel attacks that are not based on pre-existing, manually crafted prompts.

Large Language Models (LLMs) aligned via techniques such as Reinforcement Learning with Human Feedback (RLHF) or Direct Preference Optimization (DPO) contain a vulnerability in how safety features are encoded within the model parameters. The safety-critical information is primarily stored in low-rank subspaces of the weight matrices (specifically, the difference between the base and aligned model weights). These low-rank subspaces are highly sensitive to parameter updates. Consequently…

Large Language Model (LLM) agents capable of invoking external APIs are vulnerable to intent integrity violations. When an agent receives natural language instructions that are ambiguous, underspecified, or contain values not supported by the underlying API schema, the agent frequently fails to preserve user intent. Instead of rejecting the request or asking for clarification, the model may hallucinate parameter values, map unsupported requests to unsafe defaults, or execute actions on…

Large Reasoning Models (LRMs) utilizing Chain-of-Thought (CoT) processes are vulnerable to an adaptive stacked cipher attack known as SEAL (Stacked Encryption for Adaptive Language reasoning model jailbreak). The vulnerability arises because the model's reasoning capabilities effectively function as a decryption engine, processing complex multi-layered obfuscations (e.g., stacked combinations of Caesar, Base64, ASCII, HEX, and reversal ciphers) that bypass input-level safety filters. By…

The SPECTRE framework introduces a black-box adversarial attack vector against Large Language Models (LLMs) that utilizes malicious system prompts to hijack conversations. Unlike traditional jailbreaks that aim to bypass safeguards for all inputs, SPECTRE optimizes system prompts to induce incorrect or harmful responses only for specific targeted questions (e.g., "Are COVID vaccines safe?", "Who should I vote for?"), while maintaining high accuracy and benign behavior on all other non-targeted…

A vulnerability exists in Large Language Models (LLMs) that allows attackers to manipulate the model's output by modifying token log probabilities. Attackers can use a lightweight plug-in model (BiasNet) to subtly alter the probabilities, steering the LLM toward generating harmful content even when safety mechanisms are in place. This attack requires only access to the top-k token log probabilities returned by the LLM's API, without needing model weights or internal access.