Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

Universal Adversarial LLM Jailbreak

Published Jul 27, 2023Analyzed Dec 28, 2024Source: arXiv

Aligned large language models (LLMs) are vulnerable to a universal and transferable adversarial suffix attack. Appending a specific, automatically generated suffix to a wide range of prompts, even those requesting objectionable content, causes the models to generate harmful or objectionable responses instead of refusing the request. The attack's success rate is significantly higher on GPT-based models.

Universal and transferable adversarial attacks on aligned language models

Evaluated models: ChatGLM 6B, Claude Instant 1, Claude 2+12 more

ChatGLM 6B
Claude Instant 1
Claude 2
Falcon 7B
GPT-3.5 Turbo-0301
GPT-4-0314
Guanaco 7B
Guanaco 13B
Llama 2 7B Chat
MPT 7B
PaLM 2
Pythia 12B
Stable Vicuna
Vicuna 7B
Vicuna 13B

Cross-Modal VLM Jailbreak

Published Jul 26, 2023Analyzed Mar 4, 2025Source: arXiv

A vulnerability in multi-modal large language models (LLMs) allows adversaries to bypass safety mechanisms through compositional adversarial attacks. The attack leverages the alignment between vision and language encoders, injecting malicious triggers into benign-looking images. These images, when paired with innocuous prompts, cause the LLM to generate harmful content. The attack requires access only to the vision encoder (e.g., CLIP), not the LLM itself, lowering the barrier to attack.

Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models

Evaluated models: Llama-adapterv2

Automated LLM Jailbreak Framework

Published Jul 16, 2023Analyzed Dec 28, 2024Source: arXiv

The MASTER KEY framework exploits timing-based characteristics of Large Language Model (LLM) chatbot responses to infer internal defense mechanisms and automatically generate jailbreak prompts. This allows bypassing safety restrictions and eliciting responses violating usage policies, including generation of illegal, harmful, privacy-violating, and adult content. The framework utilizes a three-step process: reverse-engineering defenses via time-based analysis, creating proof-of-concept…

MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots

Evaluated models: ERNIE, GPT-3.5 Turbo, GPT-4

Visual Jailbreak of LLMs

Published Jun 22, 2023Analyzed Dec 29, 2024Source: arXiv

A vulnerability in vision-integrated Large Language Models (VLMs) allows an attacker to circumvent safety mechanisms through the use of adversarially crafted visual examples. A single, carefully constructed image can universally "jailbreak" the model, causing it to generate harmful content in response to a wide range of subsequent prompts, even those not included in the adversarial example's training data. This vulnerability extends beyond simple misclassification to encompass the execution of…

Visual adversarial examples jailbreak large language models

Evaluated models: InstructBLIP, MiniGPT-4

HouYi Prompt Injection

Published Jun 8, 2023Analyzed Dec 29, 2024Source: arXiv

A prompt injection vulnerability allows attackers to manipulate the behavior of Large Language Model (LLM)-integrated applications by crafting malicious prompts that override the application's intended functionality. Attackers can achieve this by constructing prompts that cause the LLM to interpret malicious payloads as instructions, rather than data, leading to unintended actions such as data leakage, unauthorized LLM usage, or application mimicry. This vulnerability exploits the way user…

Prompt Injection attack against LLM-integrated Applications

Evaluated models: GPT-3.5

Prompt Engineering Jailbreak

Published May 23, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs), specifically ChatGPT versions 3.5 and 4.0, are vulnerable to prompt engineering attacks that circumvent built-in content restrictions. Attackers can craft malicious prompts, categorized into "pretending," "attention shifting," and "privilege escalation" techniques, to elicit responses containing prohibited content (e.g., instructions for illegal activities, generation of harmful content). The vulnerability stems from the LLM's inability to reliably distinguish…

Jailbreaking chatgpt via prompt engineering: An empirical study

Evaluated models: GPT-3.5 Turbo, GPT-4

Related research categories