Language Model Security Database

Large Language Models (LLMs) employing alignment techniques for safety embed a "safety classifier" within their architecture. This classifier, responsible for determining whether an input is safe or unsafe, can be approximated by extracting a surrogate classifier from a subset of the LLM's architecture. Attackers can leverage this surrogate classifier to more effectively craft adversarial inputs (jailbreaks) that bypass the LLM's intended safety mechanisms. The attack success rate against the…

Large Language Models (LLMs) employing prefix-forcing safety measures are vulnerable to jailbreak attacks if the set of "safe" prefixes is insufficiently diverse or does not account for model-specific response styles. Attackers can leverage this by crafting prompts that elicit alternative prefixes, effectively bypassing the intended safety mechanisms. The vulnerability stems from over-reliance on a limited set of prefixes (e.g., "Sure, here is...") and a failure to generalize safety mechanisms…

Large Language Models (LLMs) are vulnerable to optimization-based jailbreaking attacks that exploit gradients during the iterative process of generating adversarial suffixes. The vulnerability stems from the inefficient exploration of the token space in existing methods like Greedy Coordinate Gradient (GCG), which uniformly samples tokens for replacement regardless of gradient values. This leads to redundant computations and a slow optimization process.

Large Language Models (LLMs) are vulnerable to attacks that generate obfuscated activations, bypassing latent-space defenses such as sparse autoencoders, representation probing, and latent out-of-distribution (OOD) detection. Attackers can manipulate model inputs or training data to produce outputs exhibiting malicious behavior while remaining undetected by these defenses. This occurs because the models can represent harmful behavior through diverse activation patterns, allowing attackers to…

A vulnerability exists in large language models (LLMs) where targeted bitwise corruptions in model parameters can induce a "jailbroken" state, causing the model to generate harmful responses without input modification. Fewer than 25 bit-flips are sufficient to achieve this in many cases. The vulnerability stems from the susceptibility of the model's memory representation to fault injection attacks.

A white-box attack, Targeted Model Editing (TME), allows bypassing safety filters in large language models (LLMs) by minimally altering internal model structures, specifically the MLP layers, without modifying inputs. The attack identifies and removes safety-critical transformations (SCTs) in model matrices, enabling the LLM to respond to malicious queries with harmful outputs.

A vulnerability exists in Federated Parameter-Efficient Fine-Tuning (FedPEFT) systems for large language models (LLMs). Malicious clients can exploit the PEFT mechanism (e.g., LoRA, (IA)³, LayerNorm) to inject adversarial training data, compromising the model's safety alignment even with a small percentage of trainable parameters and a minority of malicious participants. The attack, termed "PEFT-as-an-Attack" (PaaA), circumvents the LLM's safety guardrails, causing it to generate harmful…

Large Language Models (LLMs) are vulnerable to a novel prompt injection attack using universal and context-independent triggers. These triggers, once discovered for a specific LLM, allow precise control over the model's output regardless of the prompt context or desired output content, enabling adversaries to force the generation of arbitrary text. The attack utilizes a gradient-based optimization technique to discover these triggers.

Large Language Models (LLMs) are vulnerable to jailbreaking attacks that manipulate attention scores to redirect the model's focus away from safety protocols. The AttnGCG attack method increases the attention score on adversarial suffixes within the input prompt, causing the model to prioritize the malicious content over safety guidelines, leading to the generation of harmful outputs.

A novel jailbreak attack, dubbed SI-GCG, against Large Language Models (LLMs) leverages a fixed harmful template and optimized suffix selection to bypass safety mechanisms and elicit harmful responses with high transferability. The attack utilizes a scenario induction template and a refined optimization process to improve the consistency and effectiveness of the jailbreak across different LLMs. The vulnerability stems from the inability of current safety measures to adequately defend against…