Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

Random Token T2I Jailbreak

Published 8/1/2024

Analyzed 12/29/2024

A heuristic token search attack, termed HTS-Attack, can bypass safety mechanisms in text-to-image (T2I) models, allowing generation of NSFW content. The attack iteratively replaces tokens in a malicious prompt with semantically similar tokens from the model's vocabulary, avoiding detection by prompt and image checkers. The method leverages a surrogate CLIP model to maintain semantic similarity to the target NSFW prompt.

Rt-attack: Jailbreaking text-to-image models via random token

Evaluated models: Clip-vit-base-patch32, DALL-E 3, GPT 3.5-turbo-instruct +4 more

Source: arXiv

Synthetic LLM Jailbreak Dataset

Published 8/1/2024

Analyzed 7/14/2025

Large Language Models (LLMs) are vulnerable to jailbreaking attacks leveraging synthetically generated prompts. A novel pipeline, SAGE-RT, generates a diverse dataset of 51,000 prompt-response pairs designed to exploit LLMs' vulnerabilities across various categories of harmfulness. These prompts successfully jailbreak state-of-the-art LLMs in a significant percentage of tested sub-categories, including 100% of macro-categories for certain models like GPT-4 and GPT-3.5-turbo. The vulnerability…

Sage-rt: Synthetic alignment data generation for safety evaluation and red teaming

Evaluated models: Claude 3.5 Sonnet, Gemma 7B IT, GPT-3.5 Turbo +8 more

Source: arXiv

Analyzing-Based LLM Jailbreak

Published 7/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to an "Analyzing-based Jailbreak" (ABJ) attack that exploits their analytical and reasoning capabilities. ABJ crafts prompts that instruct the LLM to analyze seemingly innocuous data (e.g., character traits, features, job descriptions) related to a malicious intent, leading the LLM to generate harmful content despite its safety training. This bypasses standard safety mechanisms designed to prevent direct requests for harmful information.

Figure it Out: Analyzing-based Jailbreak Attack on Large Language Models

Evaluated models: Claude-3-haiku-0307, GLM 4 9B Chat, GPT-3.5 Turbo +3 more

Source: arXiv

AutoJailbreak of GPT-4V

Published 7/1/2024

Analyzed 12/29/2024

A vulnerability in GPT-4V's facial recognition safety mechanisms allows for automated jailbreaking attacks using Large Language Models (LLMs) to bypass safety features and elicit unintended facial identification responses. The attack, termed "AutoJailbreak," optimizes prompts through iterative refinement with an LLM "red-teaming" model, significantly increasing the attack success rate. This vulnerability exploits weaknesses in GPT-4V's prompt processing and safety alignment, allowing malicious…

Can Large Language Models Automatically Jailbreak GPT-4V?

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4V

Source: arXiv

Embodied LLM Misaligned Actions

Published 7/1/2024

Analyzed 12/29/2024

Embodied Large Language Models (LLMs) are vulnerable to manipulation via voice-based interactions, leading to the execution of harmful physical actions. Attacks exploit three vulnerabilities: (1) cascading LLM jailbreaks resulting in malicious robotic commands; (2) misalignment between linguistic outputs (verbal refusal) and physical actions (command execution); and (3) conceptual deception, where seemingly benign instructions lead to harmful outcomes due to incomplete world knowledge within…

BadRobot: Manipulating Embodied LLMs in the Physical World

Evaluated models: BERT, GPT-3.5 Turbo, GPT-4 Turbo +2 more

Source: arXiv

Function-Call Jailbreak

Published 7/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) employing function calling are vulnerable to a "jailbreak function" attack. Maliciously crafted function definitions and prompts can coerce the LLM into generating harmful content within the function's arguments, bypassing existing safety filters designed for chat modes. This exploits discrepancies in safety alignment between function argument generation and chat response generation.

The dark side of function calling: Pathways to jailbreaking large language models

Evaluated models: Claude 3 Sonnet, Claude 3.5 Sonnet, Gemini 1.5 Pro +3 more

Source: arXiv

Hidden-Intent LLM Evasion

Published 7/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to adversarial attacks that employ conversation strategies to elicit harmful information through seemingly benign dialogues. The attack, termed "Imposter.AI," leverages three key strategies: (1) decomposing malicious questions into innocuous sub-questions; (2) rephrasing overtly malicious questions into benign-sounding alternatives; and (3) enhancing the harmfulness of responses by prompting the LLM for illustrative examples. This allows attackers to…

Imposter. ai: Adversarial attacks with hidden intentions towards aligned large language models

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 13B +1 more

Source: arXiv

LLM Honest Fallacy Jailbreak

Published 7/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) struggle to generate genuinely fallacious reasoning. When prompted to create a false procedure for a harmful task, the LLMs instead leak the correct, harmful procedure while incorrectly claiming it's false. This vulnerability allows bypassing safety mechanisms and eliciting harmful outputs.

Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks

Evaluated models: Gemini Pro, GPT-3.5 Turbo, GPT-4

Source: arXiv

LLM Memory Poisoning Attack

Published 7/1/2024

Analyzed 12/28/2024

A vulnerability in Retrieval-Augmented Generation (RAG)-based Large Language Model (LLM) agents allows attackers to inject malicious demonstrations into the agent's memory or knowledge base. By crafting a carefully optimized trigger, an attacker can manipulate the agent's retrieval mechanism to preferentially retrieve these poisoned demonstrations, causing the agent to produce adversarial outputs or take malicious actions even when seemingly benign prompts are used. The attack, termed…

Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases

Evaluated models: GPT-2, GPT-3.5 Turbo, Llama 3 70B +2 more

Source: arXiv

LLM Version Fingerprinting

Published 7/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) integrated into applications reveal unique behavioral fingerprints through responses to crafted queries. LLMmap exploits this by sending carefully constructed prompts and analyzing the responses to identify the specific LLM version with high accuracy (over 95% in testing against 42 LLMs). This allows attackers to tailor attacks exploiting known vulnerabilities specific to the identified LLM version.

Llmmap: Fingerprinting for large language models

Evaluated models: Aya-23-8B, Cohere-35B, GPT-4 +9 more

Source: arXiv

Related research categories