Language Model Security Database

A prompt-based adversarial attack, termed PromptAttack, can cause Large Language Models (LLMs) to generate incorrect outputs by manipulating the input prompt. PromptAttack crafts prompts that include the original input, an attack objective (to generate semantically similar but misclassified output), and attack guidance with instructions for character, word, or sentence-level perturbations. This allows an attacker to manipulate an LLM's response without direct access to its internal parameters…

A vulnerability in large language models (LLMs) allows attackers to craft malicious prompts that induce the LLM to generate harmful content, such as fraudulent material, racist remarks, or instructions for illegal activities. The vulnerability arises from the LLM's inability to reliably distinguish between benign and malicious instructions disguised within seemingly innocuous prompts. Attackers can exploit this by leveraging techniques like obfuscation, code injection/payload splitting, and…

Large Language Models (LLMs) are vulnerable to Compositional Instruction Attacks (CIA), where malicious prompts are embedded within seemingly harmless instructions. This allows attackers to bypass safety mechanisms and elicit harmful responses from the model, even if the individual components of the prompt would be flagged as safe. The attack exploits the model's inability to correctly identify underlying malicious intent within composite instructions.

Large Language Models (LLMs) are vulnerable to In-Context Attacks (ICA) and susceptible to mitigation via In-Context Defense (ICD). ICA leverages a small number of harmful demonstration examples within a prompt to elicit harmful responses from the LLM, even if it is otherwise safety-aligned. ICD counteracts ICA by prepending safe demonstration examples to the prompt, effectively reducing the likelihood of harmful output. The effectiveness of both ICA and ICD is demonstrated across multiple LLMs.

Large Language Models (LLMs) employing alignment techniques remain vulnerable to "jailbreak" attacks. The AutoDAN technique automatically generates semantically meaningful prompts that bypass safety features and elicit malicious outputs from aligned LLMs, unlike previous methods producing nonsensical prompts easily detectable by perplexity checks. These prompts exploit weaknesses in the LLM's alignment, causing it to generate responses that violate intended safety constraints.

Large Language Models (LLMs) are susceptible to automated jailbreak attacks using a fuzzing framework that generates variations of existing jailbreak prompts. This vulnerability allows bypassing built-in safety mechanisms, leading to the generation of harmful or unintended outputs. The vulnerability stems from the LLMs' inability to consistently recognize and reject semantically similar, but subtly different prompt variations generated through automated mutation techniques.

Large Language Models (LLMs) are vulnerable to a "Chain of Utterances" (CoU) based prompt injection attack. This attack exploits the LLM's ability to engage in multi-turn conversations and role-playing, tricking it into providing harmful or unsafe responses even when presented with safety guidelines. The attack leverages a crafted conversation between two agents ("Red-LM," a malicious agent, and "Base-LM," a seemingly helpful agent) to elicit unethical responses from the Base-LM by subtly…

Aligned large language models (LLMs) are vulnerable to a universal and transferable adversarial suffix attack. Appending a specific, automatically generated suffix to a wide range of prompts, even those requesting objectionable content, causes the models to generate harmful or objectionable responses instead of refusing the request. The attack's success rate is significantly higher on GPT-based models.

The MASTER KEY framework exploits timing-based characteristics of Large Language Model (LLM) chatbot responses to infer internal defense mechanisms and automatically generate jailbreak prompts. This allows bypassing safety restrictions and eliciting responses violating usage policies, including generation of illegal, harmful, privacy-violating, and adult content. The framework utilizes a three-step process: reverse-engineering defenses via time-based analysis, creating proof-of-concept…

Large Language Models (LLMs), specifically ChatGPT versions 3.5 and 4.0, are vulnerable to prompt engineering attacks that circumvent built-in content restrictions. Attackers can craft malicious prompts, categorized into "pretending," "attention shifting," and "privilege escalation" techniques, to elicit responses containing prohibited content (e.g., instructions for illegal activities, generation of harmful content). The vulnerability stems from the LLM's inability to reliably distinguish…