Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

GPT-4v System Prompt Leakage

Published Nov 15, 2023Analyzed Dec 29, 2024Source: arXiv

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly effective jailbreak prompts, bypassing safety restrictions and leading to undesirable outputs, including revealing personally identifiable information from images.

Jailbreaking gpt-4v via self-adversarial attacks with system prompts

Evaluated models: GPT-4, GPT-4V, LLaVA 1.5

Nested Prompt Jailbreak

Published Nov 14, 2023Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in several Large Language Models (LLMs) allowing attackers to bypass safety mechanisms through carefully crafted "jailbreak" prompts. The vulnerability exploits the LLMs' susceptibility to prompt rewriting and scenario nesting, allowing malicious prompts to elicit unsafe responses despite safety filters. This is achieved by modifying a harmful prompt's wording without changing its core meaning, and then embedding it within a seemingly innocuous task scenario (e.g., code…

A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

Evaluated models: Claude-instant-v1, Claude-v2, GPT-2+4 more

Claude-instant-v1
Claude-v2
GPT-2
GPT-3.5 Turbo
GPT-4
Llama 2 13B Chat
Llama 2 7B Chat

Fine-Tuning Bypasses RLHF

Published Nov 9, 2023Analyzed Dec 28, 2024Source: arXiv

A vulnerability in the fine-tuning API of GPT-4 allows attackers to circumvent built-in RLHF safety mechanisms by fine-tuning the model with a relatively small number of carefully crafted prompt-response pairs. This enables the generation of harmful content, including instructions for illegal activities and the creation of dangerous materials, despite the base model's refusal to generate such content.

Removing rlhf protections in gpt-4 via fine-tuning

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 70B

Typographic VLM Jailbreak

Published Nov 9, 2023Analyzed Dec 29, 2024Source: arXiv

Large Vision-Language Models (VLMs) are vulnerable to jailbreaking attacks via typographically rendered visual prompts. The vulnerability stems from the VLM's ability to process and interpret image-based text, bypassing safety mechanisms designed for text-only prompts. Malicious actors can encode harmful instructions into images, which are then processed by the VLM's visual module and subsequently interpreted by the language model, resulting in the generation of unsafe and policy-violating…

Figstep: Jailbreaking large vision-language models via typographic visual prompts

Evaluated models: Cogvlm-chat-v1.1, GPT-4V, Llava-v1.5-vicuna-v1.5-13B+4 more

Cogvlm-chat-v1.1
GPT-4V
Llava-v1.5-vicuna-v1.5-13B
Llava-v1.5-vicuna-v1.5-7B
Minigpt4-llama-2-chat-7B
MiniGPT-4 Vicuna 13B
Minigpt4-vicuna-7B

Nested-Scene LLM Jailbreak

Published Nov 6, 2023Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel "DeepInception" attack that leverages the models' personification capabilities to bypass safety guardrails. The attack uses nested prompts to create a multi-layered fictional scenario, effectively hypnotizing the LLM into generating harmful content by exploiting its tendency towards obedience within the constructed narrative. This allows for continuous jailbreaks in subsequent interactions.

Deepinception: Hypnotize large language model to be jailbreaker

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o+1 more

GPT-3.5 Turbo
GPT-4
GPT-4o
GPT-4V

Persona-Based LLM Jailbreak

Published Nov 6, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to persona modulation attacks, a black-box jailbreak technique that leverages an LLM assistant to generate prompts causing the target LLM to adopt harmful personas and produce unsafe outputs. This vulnerability circumvents built-in safety mechanisms, enabling the generation of responses related to illegal activities (e.g., synthesizing drugs, building bombs, money laundering), hate speech, and other harmful content. The attack's effectiveness is…

Scalable and transferable black-box jailbreaks for language models via persona modulation

Evaluated models: Claude 2, GPT-4

Linguistic LLM Jailbreak

Published Nov 1, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a targeted linguistic fuzzing attack that exploits the complexity of human language to bypass safety guardrails. The attack, termed "Jade," leverages transformational-generative grammar rules to systematically increase the syntactic complexity of benign seed questions, making them increasingly difficult for LLMs to recognize as malicious. This leads to the generation of unsafe content, even when the underlying semantics remain unchanged.

Jade: A linguistics-based safety evaluation platform for llm

Evaluated models: ChatGLM2 6B, GPT-2, GPT-3+2 more

ChatGLM2 6B
GPT-2
GPT-3
Llama 2 70B Chat
PaLM 2

AutoDAN: Interpretable LLM Jailbreak

Published Oct 23, 2023Analyzed Dec 29, 2024Source: arXiv

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B+4 more

GPT-3.5 Turbo
GPT-4
Guanaco 7B
Llama 2 Chat
Pythia 12B
Vicuna 13B
Vicuna 7B

Self-Fooling LLM Prompt Attack

Published Oct 20, 2023Analyzed Jan 26, 2025Source: arXiv

A prompt-based adversarial attack, termed PromptAttack, can cause Large Language Models (LLMs) to generate incorrect outputs by manipulating the input prompt. PromptAttack crafts prompts that include the original input, an attack objective (to generate semantically similar but misclassified output), and attack guidance with instructions for character, word, or sentence-level perturbations. This allows an attacker to manipulate an LLM's response without direct access to its internal parameters…

An LLM can Fool Itself: A Prompt-Based Adversarial Attack

Evaluated models: GPT-3.5 Turbo

LLM Red Teaming Framework

Published Oct 19, 2023Analyzed Dec 28, 2024Source: arXiv

A vulnerability in large language models (LLMs) allows attackers to craft malicious prompts that induce the LLM to generate harmful content, such as fraudulent material, racist remarks, or instructions for illegal activities. The vulnerability arises from the LLM's inability to reliably distinguish between benign and malicious instructions disguised within seemingly innocuous prompts. Attackers can exploit this by leveraging techniques like obfuscation, code injection/payload splitting, and…

Attack prompt generation for red teaming and defending large language models

Evaluated models: Not reported

Related research categories