Language Model Security Database

Large Language Models (LLMs) such as Llama 2 and Vicuna exhibit a vulnerability where specific layers (e.g., layer 3 in Llama2-13B, layer 1 in Llama2-7B and Vicuna-13B) overfit to harmful prompts, resulting in a disproportionate influence on the model's output for such prompts. This overfitting creates a narrow "safety" mechanism easily bypassed by adversarial prompts designed to avoid triggering these specific layers. Additionally, a single neuron (e.g., neuron 2100 in Llama2 and Vicuna)…

Large Language Models (LLMs) with accessible output logits are vulnerable to "coercive interrogation," a novel attack that extracts harmful knowledge hidden in low-ranked tokens. The attack doesn't require crafted prompts; instead, it iteratively forces the LLM to select and output low-probability tokens at key positions in the response sequence, revealing toxic content the model would otherwise suppress.

A prompt injection vulnerability in OpenAI's custom GPT models allows attackers to extract the system prompt and potentially leak user-uploaded files. Attackers craft malicious prompts that manipulate the LLM into revealing sensitive information, even when defensive prompts are in place. The vulnerability is exacerbated when the model includes a code interpreter.

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Low-rank adaptation (LoRA) fine-tuning allows efficient circumvention of safety training in large language models (LLMs), such as Llama 2-Chat 70B, resulting in significantly reduced refusal rates for harmful prompts while maintaining general performance capabilities. Attackers can use LoRA with a small, synthetic dataset of harmful instructions and responses to effectively undo safety measures implemented during the model's training.

Large Language Models (LLMs) employing Reinforcement Learning from Human Feedback (RLHF) and instruction tuning methods may exhibit superficial safety guardrails vulnerable to parametric red-teaming attacks. Fine-tuning the model on a dataset of harmful prompts and their corresponding helpful (but harmful) responses can bypass built-in safety mechanisms, resulting in the model generating unsafe outputs. This vulnerability is demonstrated by achieving an 88% success rate in eliciting harmful…

A vulnerability exists in multimodal Large Language Models (LLMs) integrated with external tools. Adversarial images, visually indistinguishable from benign images, can manipulate the LLM to execute unintended tool commands, compromising the confidentiality and integrity of user resources. The attack is effective across diverse prompts, remaining stealthy both in the image itself and in the generated text response.

Large Language Model (LLM)-integrated web applications using Langchain (and potentially similar middleware) are vulnerable to Prompt-to-SQL (P2SQL) injection attacks. Unsanitized user prompts can be crafted to cause the LLM to generate malicious SQL queries, leading to unauthorized database access (read and write operations). This vulnerability bypasses attempts to restrict the LLM through prompt engineering alone.

A vulnerability in multi-modal large language models (LLMs) allows adversaries to bypass safety mechanisms through compositional adversarial attacks. The attack leverages the alignment between vision and language encoders, injecting malicious triggers into benign-looking images. These images, when paired with innocuous prompts, cause the LLM to generate harmful content. The attack requires access only to the vision encoder (e.g., CLIP), not the LLM itself, lowering the barrier to attack.

A prompt injection vulnerability allows attackers to manipulate the behavior of Large Language Model (LLM)-integrated applications by crafting malicious prompts that override the application's intended functionality. Attackers can achieve this by constructing prompts that cause the LLM to interpret malicious payloads as instructions, rather than data, leading to unintended actions such as data leakage, unauthorized LLM usage, or application mimicry. This vulnerability exploits the way user…