Language Model Security Database

Large language models (LLMs) are vulnerable to adversarial prompt engineering attacks that leverage proximity constraints to elicit harmful behaviors. By subtly modifying benign prompts within a semantically close embedding space, attackers can bypass existing safety mechanisms and induce undesired outputs, even when the original prompts would not trigger such a response. This vulnerability exploits the model's sensitivity to small perturbations in the input embedding, resulting in the…

DiffusionAttacker exploits a vulnerability in Large Language Models (LLMs) allowing manipulation of prompts to elicit harmful responses, even when the model incorporates safety mechanisms. The attack leverages a sequence-to-sequence diffusion model to rewrite harmful prompts, making them appear harmless to the LLM's internal representation while preserving their original semantic meaning. This bypasses safety filters and elicits undesired outputs.

A hybrid multimodal jailbreaking attack, dubbed JMLLM, exploits vulnerabilities in 13 popular large language models (LLMs) across text, image, and speech modalities. The attack leverages alternating translation, word encryption, feature collapse in images, and harmful text injection to bypass safety mechanisms and elicit harmful responses. Success rates vary across LLMs and modalities, with some models exhibiting significantly higher vulnerability than others.

Large Language Models (LLMs) are vulnerable to human-readable adversarial prompts crafted using situational context derived from movie scripts. These prompts, which combine a malicious prompt, a seemingly innocuous adversarial insertion, and relevant contextual information, can bypass LLMs' safety mechanisms and elicit harmful responses. The technique leverages the LLM's ability to understand context and generate responses consistent with that context to mask the malicious intent. The…

JailPO is a black-box attack framework that leverages preference optimization to generate effective jailbreak prompts for aligned LLMs. The attack automatically generates prompts, bypassing safety mechanisms and eliciting harmful or undesirable responses from the target LLM. The framework includes three attack patterns (QEPrompt, TemplatePrompt, MixAsking) with varying degrees of effectiveness and risk.

A novel jailbreak paradigm, Simple Assistive Task Linkage (SATA), circumvents LLM safeguards by masking harmful keywords in a malicious query and using a secondary, simple assistive task (e.g., masked language modeling or element lookup by position) to convey the masked keywords' semantics to the LLM. This distracts the LLM and allows it to bypass safety checks, leading to the generation of harmful responses.

A vulnerability in LLMs allows attackers to bypass safety mechanisms by crafting prompts that disguise malicious intent as a "defense" against harmful content. The attack, Reverse Embedded Defense Attack (REDA), leverages the model's own defensive capabilities to generate harmful outputs while masking the malicious intent within the response structure. This allows for successful jailbreaks in a single iteration, without requiring model-specific prompt engineering.

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

Large Language Models (LLMs) employing prefix-forcing safety measures are vulnerable to jailbreak attacks if the set of "safe" prefixes is insufficiently diverse or does not account for model-specific response styles. Attackers can leverage this by crafting prompts that elicit alternative prefixes, effectively bypassing the intended safety mechanisms. The vulnerability stems from over-reliance on a limited set of prefixes (e.g., "Sure, here is...") and a failure to generalize safety mechanisms…

Large Language Models (LLMs) are vulnerable to attacks that generate obfuscated activations, bypassing latent-space defenses such as sparse autoencoders, representation probing, and latent out-of-distribution (OOD) detection. Attackers can manipulate model inputs or training data to produce outputs exhibiting malicious behavior while remaining undetected by these defenses. This occurs because the models can represent harmful behavior through diverse activation patterns, allowing attackers to…