Language Model Security Database

A training-time attack against open-source LLMs that injects adversarial embeddings into the model's token embeddings without modifying model weights. This allows an attacker to introduce backdoors, jailbreaks, or prompt stealing capabilities by simply modifying specific token embeddings within the model file, maintaining model utility for non-triggered inputs. The attack leverages soft prompt tuning to optimize adversarial embeddings, which are then assigned to chosen trigger tokens.

Multimodal Large Language Models (LLMs) processing speech input are vulnerable to adversarial attacks. Imperceptible perturbations added to audio input can cause the model to generate unsafe or harmful text responses, overriding built-in safety mechanisms. The attacks are effective even with limited knowledge of the model's internal workings, demonstrating transferability across different models.

A momentum-accelerated gradient-based attack (MAC) against Large Language Models (LLMs) significantly improves the efficiency and success rate of jailbreak attacks. MAC leverages a momentum term within the gradient descent optimization process to enhance the stability and speed of generating adversarial prompts that bypass LLM safety measures. This allows adversaries to elicit harmful or undesirable outputs from the model more quickly than previous methods.

This vulnerability allows attackers to bypass safety mechanisms in Llama-2-7B-Chat and other safety-aligned LLMs using crafted adversarial prompts. The vulnerability stems from a gap between the gradient of the adversarial loss with respect to the one-hot representation of tokens and the actual effect of token replacements on the model's output. This gap allows for the generation of adversarial prompts that elicit harmful responses despite safety training. The paper demonstrates that…

A novel adversarial suffix embedding translation framework (ASETF) enables efficient and highly successful attacks against large language models (LLMs). ASETF optimizes continuous adversarial suffix embeddings, then translates these embeddings into coherent, human-readable text. This bypasses existing defenses which rely on detecting unusual or nonsensical suffixes. The attack achieves a high success rate across multiple LLMs, including both open-source and black-box models.

Large Language Models (LLMs) are vulnerable to efficient adversarial attacks using Projected Gradient Descent (PGD) on a continuously relaxed input prompt. This attack bypasses existing alignment methods by crafting adversarial prompts that induce the model to produce undesired or harmful outputs, significantly faster than previous state-of-the-art discrete optimization methods. The effectiveness stems from carefully controlling the error introduced by the continuous relaxation of the discrete…

Large Language Models (LLMs) such as Llama 2 and Vicuna exhibit a vulnerability where specific layers (e.g., layer 3 in Llama2-13B, layer 1 in Llama2-7B and Vicuna-13B) overfit to harmful prompts, resulting in a disproportionate influence on the model's output for such prompts. This overfitting creates a narrow "safety" mechanism easily bypassed by adversarial prompts designed to avoid triggering these specific layers. Additionally, a single neuron (e.g., neuron 2100 in Llama2 and Vicuna)…

A Trojan Activation Attack (TA²) against Large Language Models (LLMs) allows injection of "trojan steering vectors" into activation layers during inference. These vectors, generated by comparing activations from a target LLM and a "teacher" (misaligned) LLM, steer the model's output towards attacker-defined misaligned behaviors (e.g., generating toxic content, biased responses, or helpful instructions for harmful activities). The attack does not require retraining or modifying model weights.

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Low-rank adaptation (LoRA) fine-tuning allows efficient circumvention of safety training in large language models (LLMs), such as Llama 2-Chat 70B, resulting in significantly reduced refusal rates for harmful prompts while maintaining general performance capabilities. Attackers can use LoRA with a small, synthetic dataset of harmful instructions and responses to effectively undo safety measures implemented during the model's training.