Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

Segmented Prompt Jailbreak

Published 3/1/2025

Analyzed 4/3/2025

Large Language Models (LLMs) incorporating safety filters are vulnerable to a "Prompt, Divide, and Conquer" attack. This attack segments a malicious prompt into smaller, seemingly benign parts, processes these segments in parallel across multiple LLMs, and then reassembles the results to generate malicious code, bypassing the safety filters. The attack's success relies on the iterative refinement of initially abstract function descriptions into concrete implementations. Individual LLM safety…

Prompt, Divide, and Conquer: Bypassing Large Language Model Safety Filters via Segmented and Distributed Prompt Processing

Evaluated models: Claude 3.5 Haiku, Claude 3.5 Sonnet, Gemini 1.5 Pro +3 more

Source: arXiv

Unchallenged Premise Misinformation

Published 3/1/2025

Analyzed 12/9/2025

Large Language Models (LLMs) are vulnerable to implicit misinformation propagation due to sycophantic compliance with false premises. When a user prompt embeds a factually incorrect assumption or conspiracy theory as an unchallenged premise (implicit presupposition) rather than asking for verification, the model frequently fails to detect the falsehood. Instead of correcting the user, the model hallucinates a response that accepts, validates, and reinforces the false premise. This…

How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformation

Evaluated models: Gemini 1.5 Pro, Gemini 2.0 Flash, Claude 3.5 Sonnet +11 more

Source: arXiv

AP-Test Guardrail Identification

Published 2/1/2025

Analyzed 3/4/2025

This vulnerability allows attackers to identify the presence and location (input or output stage) of specific guardrails implemented in Large Language Models (LLMs) by using carefully crafted adversarial prompts. The attack, termed AP-Test, leverages a tailored loss function to optimize these prompts, maximizing the likelihood of triggering a specific guardrail while minimizing triggering others. Successful identification provides attackers with valuable information to design more effective…

Peering Behind the Shield: Guardrail Identification in Large Language Models

Evaluated models: Aegis Defensive, Aegis Permissive, GPT-4o +8 more

Source: arXiv

Adversarial LLM Jailbreak

Published 2/1/2025

Analyzed 2/16/2025

A vulnerability in Large Language Models (LLMs) allows adversarial reasoning attacks to bypass safety mechanisms and elicit harmful responses. The vulnerability stems from the insufficient robustness of existing LLM safety measures against iterative prompt refinement guided by a loss function that measures the LLM's proximity to generating a target harmful response. This allows an attacker to effectively navigate the prompt space, even against adversarially trained models, resulting in…

Adversarial Reasoning at Jailbreaking Time

Evaluated models: Claude 3.5 Sonnet, Cygnet, Gemini 1.5 Pro +8 more

Source: arXiv

Adversarial VLM Jailbreak

Published 2/1/2025

Analyzed 12/9/2025

Vision-Language Models (VLMs), specifically the LLaVA-1.5 and LLaVA-1.6 series, are vulnerable to optimization-based white-box jailbreak attacks despite standard safety alignment measures like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Attackers can craft adversarial perturbations in the image space (imperceptible noise) or latent space using Projected Gradient Descent (PGD) to manipulate the model's internal representations. These perturbations maximize the…

Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training

Evaluated models: LLaVA 1.5 7B, LLaVA 1.6 7B

Source: arXiv

Agent Pipeline Simple Hacks

Published 2/1/2025

Analyzed 12/9/2025

Commercial LLM-powered agents utilizing autonomous web access, memory modules, and retrieval-augmented generation (RAG) are vulnerable to indirect prompt injection and environmental manipulation. Attackers can embed malicious instructions into external data sources trusted by the agent (such as Reddit posts, public databases, or ArXiv papers). When the agent autonomously retrieves and processes this content during task execution, it executes the embedded malicious commands. This vulnerability…

Commercial llm agents are already vulnerable to simple yet dangerous attacks

Evaluated models: Not reported

Source: arXiv

Agentic Prompt Leakage Attacks

Published 2/1/2025

Analyzed 3/4/2025

A vulnerability exists in large language models (LLMs) where insufficient sanitization of system prompts allows attackers to extract sensitive information embedded within those prompts. Attackers can use an agentic approach, employing multiple interacting LLMs (as demonstrated in the referenced research), to iteratively refine prompts and elicit confidential data from the target LLM's responses. The vulnerability is exacerbated by the LLM's ability to infer context from seemingly innocuous…

Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach

Evaluated models: GPT-4o Mini

Source: arXiv

CRI Jailbreak Initialization

Published 2/1/2025

Analyzed 3/19/2025

CRI (Compliance Refusal Initialization) initializes jailbreak attacks by leveraging pre-trained jailbreak prompts, effectively guiding the optimization process towards the compliance subspace of harmful prompts. This significantly enhances the success rate and reduces the computational overhead of attacks, often requiring only a single optimization step to bypass safety mechanisms. Attacks utilizing CRI demonstrate significantly improved ASR (Adversarial Success Rate) and reduced median steps…

Jailbreak Attack Initializations as Extractors of Compliance Directions

Evaluated models: Falcon 7B Instruct, Llama 2 7B Chat, Llama 3 8B Instruct +5 more

Source: arXiv

Distilled Jailbreak Prompt Generator

Published 2/1/2025

Analyzed 3/4/2025

The Knowledge-Distilled Attacker (KDA) model, when used to generate prompts for large language models (LLMs), can bypass LLM safety mechanisms resulting in the generation of harmful, inappropriate, or misaligned content. KDA's effectiveness stems from its ability to generate diverse and coherent attack prompts efficiently, surpassing existing methods in attack success rate and speed. The vulnerability lies in the LLMs' insufficient defenses against the diverse prompt generation strategies…

KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs

Evaluated models: Claude 2.1, GPT-3.5 Turbo, GPT-4 +8 more

Source: arXiv

Flowchart-based LVLM Jailbreak Attack

Published 2/1/2025

Analyzed 3/19/2025

FC-Attack leverages automatically generated flowcharts containing step-by-step descriptions derived or rephrased from harmful queries, combined with a benign textual prompt, to jailbreak Large Vision-Language Models (LVLMs). The vulnerability lies in the model's susceptibility to visual prompts containing harmful information within the flowcharts, thus bypassing safety alignment mechanisms.

FC-Attack: Jailbreaking Large Vision-Language Models via Auto-Generated Flowcharts

Evaluated models: Claude 3.5 Sonnet 20240620, Gemini 1.5 Flash, GPT-4o 2024-08-06 +4 more

Source: arXiv

Related research categories