Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

LLM Judge Prompt Injection

Published 5/1/2025

Analyzed 5/31/2025

Large Language Models (LLMs) used for evaluating text quality (LLM-as-a-Judge architectures) are vulnerable to prompt-injection attacks. Maliciously crafted suffixes appended to input text can manipulate the LLM's judgment, causing it to incorrectly favor a predetermined response even if another response is objectively superior. Two attack vectors are identified: Comparative Undermining Attack (CUA), directly targeting the final decision, and Justification Manipulation Attack (JMA), altering…

Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks

Evaluated models: Falcon 3 3B Instruct, Qwen 2.5 3B Instruct

Source: arXiv

LLM Multi-Agent IP Leakage

Published 5/1/2025

Analyzed 5/31/2025

Large Language Model (LLM)-based Multi-Agent Systems (MAS) are vulnerable to intellectual property (IP) leakage attacks. An attacker with black-box access (only interacting via the public API) can craft adversarial queries that propagate through the MAS, extracting sensitive information such as system prompts, task instructions, tool specifications, number of agents, and system topology.

IP Leakage Attacks Targeting LLM-Based Multi-Agent Systems

Evaluated models: GPT-4o, GPT-4o Mini, Llama 3.1 70B +2 more

Source: arXiv

LLM Self-Introspection Jailbreak

Published 5/1/2025

Analyzed 5/31/2025

A vulnerability exists in Large Language Models (LLMs) that allows attackers to manipulate the model's output by modifying token log probabilities. Attackers can use a lightweight plug-in model (BiasNet) to subtly alter the probabilities, steering the LLM toward generating harmful content even when safety mechanisms are in place. This attack requires only access to the top-k token log probabilities returned by the LLM's API, without needing model weights or internal access.

JULI: Jailbreak Large Language Models by Self-Introspection

Evaluated models: Llama 2 7B Chat, Llama 3 8B, Llama 3 8B Instruct +3 more

Source: arXiv

LLM System Prompt Extraction

Published 5/1/2025

Analyzed 12/30/2025

Large Language Models (LLMs), including Llama-3, Falcon-3, Gemma-2, and GPT-4 variants, are susceptible to system prompt extraction attacks. The vulnerability exists due to the models' instruction-following nature, which allows remote attackers to bypass safety guardrails and retrieve the model's hidden system configuration (system prompt) verbatim. This is successfully exploited using an "Extended Sandwich Attack," where an adversarial extraction command is embedded between benign questions…

System Prompt Extraction Attacks and Defenses in Large Language Models

Evaluated models: GPT-4, GPT-4o, Llama 3 8B +2 more

Source: arXiv

LLM User Simulation Shilling

Published 5/1/2025

Analyzed 1/14/2026

Recommender Systems (RS) utilizing collaborative filtering and review processing (such as NMF, NeuNMF, and Dual-Tower architectures) are vulnerable to low-knowledge shilling attacks via LLM-based user agents. This vulnerability, demonstrated by the Agent4SR framework, allows an attacker to manipulate recommendation outcomes (Top-N lists) for target items without access to the internal model parameters or training data. The attack leverages Large Language Models to construct cognitively…

LLM-Based User Simulation for Low-Knowledge Shilling Attacks on Recommender Systems

Evaluated models: GPT-4o

Source: arXiv

Latent-Space Jailbreak Optimization

Published 5/1/2025

Analyzed 5/31/2025

The LARGO attack exploits a vulnerability in Large Language Models (LLMs) allowing attackers to bypass safety mechanisms through the generation of "stealthy" adversarial prompts. The attack leverages gradient optimization in the LLM's continuous latent space to craft seemingly innocuous natural language suffixes which, when appended to harmful prompts, elicit unsafe responses. The vulnerability stems from the LLM's inability to reliably distinguish between benign and maliciously crafted latent…

LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs

Evaluated models: Llama 2 13B Chat, Llama 2 7B Chat, Phi 3 Mini +1 more

Source: arXiv

Logic-Based LLM Jailbreak

Published 5/1/2025

Analyzed 5/31/2025

Large Language Models (LLMs) employing safety mechanisms based on token-level distribution analysis are vulnerable to a jailbreak attack exploiting distributional discrepancies between alignment data and formally expressed logical statements. The vulnerability allows malicious actors to bypass safety restrictions by translating harmful natural language prompts into equivalent first-order logic expressions. The LLM, trained primarily on natural language, fails to recognize the harmful intent…

Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression

Evaluated models: DeepSeek R1, DeepSeek V3, GPT-3.5 Turbo +4 more

Source: arXiv

Multilingual LLM Jailbreaks

Published 5/1/2025

Analyzed 6/12/2025

Multilingual prompt injection vulnerability in four closed-source Large Language Models (LLMs): GPT-4o, DeepSeek-R1, Gemini-1.5-Pro, and Qwen-Max. Attackers can bypass safety restrictions and elicit harmful or disallowed content by crafting prompts in English or Chinese, leveraging specific structural techniques (e.g., "Two Sides" prompting) that exploit inconsistencies in the models' safety alignment across languages and prompt formats.

The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models

Evaluated models: DeepSeek R1, Gemini 1.5 Pro, GPT-4o +1 more

Source: arXiv

Nonsensical CoT Reasoning

Published 5/1/2025

Analyzed 12/30/2025

Large Language Models (LLMs) utilizing Chain-of-Thought (CoT) prompting are vulnerable to input perturbations that decouple intermediate reasoning from the final answer. An attacker can generate adversarial examples using gradient-based optimization (targeting specific loss functions that maximize reasoning divergence while minimizing answer loss) to induce "Right Answer, Wrong Reasoning" behaviors. This vulnerability manifests through two primary attack vectors: 1. Token-level perturbations…

Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoning

Evaluated models: Llama 3 8B, Mistral 7B, Zephyr 7B Beta +4 more

Source: arXiv

On-Device LLM Hijacking

Published 5/1/2025

Analyzed 12/9/2025

Mobile LLM agents utilizing vision-based screen perception (OCR or Multimodal Large Language Models) are vulnerable to Visual Prompt Injection via malicious GUI overlays. An attacker holding the SYSTEM_ALERT_WINDOW permission can deploy non-focusable floating windows (using FLAG_NOT_FOCUSABLE) containing adversarial text or fabricated UI elements over legitimate applications. Because the agent captures the entire screen buffer to interpret the device state, it ingests the adversarial overlay…

From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents

Evaluated models: GPT-4o

Source: arXiv

Related research categories