Attack Type

Prompt leaking Research

Attacks that expose sensitive prompt information

10 matching entries out of 23 in this category

Prompt leaking research results

10 entries

Agent Lifecycle Compound Threats

Published Mar 12, 2026Analyzed Apr 10, 2026Source: arXiv

OpenClaw is vulnerable to persistent memory poisoning, allowing an attacker to manipulate the agent's long-term memory store (MEMORY.md) via prompt injection. Because the autonomous agent continuously integrates this memory file as context for all subsequent reasoning and task planning, injected payloads act as durable behavioral constraints. This allows an attacker to persistently alter the agent's core policy, manipulate tool selection, and hijack future sessions without any further…

Taming openclaw: Security analysis and mitigation of autonomous llm agent threats

Evaluated models: Not reported

Autonomous Agent Prompt Reveal

Published Jan 29, 2026Analyzed Feb 21, 2026Source: arXiv

A vulnerability exists in Large Language Model (LLM) deployments and multi-agent systems where an autonomous attacker agent can systematically extract hidden system prompts through self-evolving interaction strategies. The vulnerability leverages a "JustAsk" framework which utilizes Upper Confidence Bound (UCB) exploration to dynamically select and refine attack vectors from a hierarchical taxonomy of 14 atomic skills (e.g., structural formatting, authority appeals) and 14 multi-turn…

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

Evaluated models: o1, Llama 3.1 70B Hanami X1, Phi-4+38 more

o1
Llama 3.1 70B Hanami X1
Phi-4
Aion 1.0
Sonar Pro
Command A
Llama 4 Maverick
Llama 3.1 Nemotron Ultra 253B v1
Qwen 3 235B-A22B
Mercury
Hunyuan A13B Instruct
UI-TARS 1.5 7B
GPT-oss 120B
Jamba Mini 1.7
Hermes 4 70B
Step 3
LongCat Flash Chat
Tongyi DeepResearch 30B-A3B
Cydonia 24B v4.1
ERNIE 4.5 21B-A3B Thinking
Granite 4.0 H Micro
LFM2 8B-A1B
Nova Premier v1
Kimi K2 Thinking
KAT Coder Pro
Cogito v2.1 671B
Gemini 3 Pro Preview
Grok 4.1 Fast
Claude Opus 4.5
TNG R1T Chimera
Intellect 3
DeepSeek V3.2 Speciale
Trinity Mini
Mistral Large 2512
DeepSeek V3.1 Nex N1
MiMo V2 Flash
GLM-4.7
MiniMax M2.1
Seed 1.6
Molmo 2 8B
GPT-5.2 Codex

Guardrail Policy Extraction

Published Nov 6, 2025Analyzed Feb 21, 2026Source: arXiv

A black-box guardrail reverse-engineering vulnerability exists in Large Language Model (LLM) serving systems that employ output filtering mechanisms. The vulnerability allows remote attackers to replicate the proprietary decision-making policy and rule sets of the target's safety guardrail without direct access to model parameters. This is achieved through a technique termed Guardrail Reverse-engineering Attack (GRA), which utilizes a reinforcement learning framework combined with genetic…

Black-Box Guardrail Reverse-engineering Attack

Evaluated models: GPT-4o, Llama 3.1 8B

Pervasive Multi-turn Jailbreaks

Published Nov 5, 2025Analyzed Dec 8, 2025Source: arXiv

Multiple open-weight Large Language Models (LLMs)—specifically those prioritizing capability over safety alignment—exhibit a critical vulnerability to adaptive multi-turn prompt injection and jailbreak attacks. While these models effectively reject isolated, single-turn adversarial inputs (averaging ~13.11% Attack Success Rate), they fail to maintain safety guardrails and policy enforcement across extended conversational contexts. By leveraging iterative strategies such as "Crescendo" (gradual…

Death by a Thousand Prompts: Open Model Vulnerability Analysis

Evaluated models: GPT-oss 20B, Llama 3.3 70B Instruct, Mistral Large 2+5 more

GPT-oss 20B
Llama 3.3 70B Instruct
Mistral Large 2
DeepSeek V3.1
Qwen 3 32B
Gemma 3 1B
Phi-4
GLM 4.5 Air

Leaked Bits Collapse Attack Queries

Published Oct 19, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack…

Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs

Evaluated models: DeepSeek R1, GPT-4o Mini 2024-07-18, Llama 4 Maverick 17B+4 more

DeepSeek R1
GPT-4o Mini 2024-07-18
Llama 4 Maverick 17B
Llama 4 Scout 17B
OLMo 2 7B-1124
OLMo 2 13B-1124
OLMo 2 32B-0325

Enterprise Multi-Turn Data Exfiltration

Published Jul 21, 2025Analyzed Jul 28, 2025Source: arXiv

Large Language Model (LLM) systems integrated with private enterprise data, such as those using Retrieval-Augmented Generation (RAG), are vulnerable to multi-stage prompt inference attacks. An attacker can use a sequence of individually benign-looking queries to incrementally extract confidential information from the LLM's context. Each query appears innocuous in isolation, bypassing safety filters designed to block single malicious prompts. By chaining these queries, the attacker can…

Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems

Evaluated models: GPT-2, GPT-3, GPT-4+1 more

GPT-2
GPT-3
GPT-4
RoBERTa

Cyber Fine-Tuning Undermines Safety

Published Mar 12, 2025Analyzed Jan 14, 2026Source: arXiv

Fine-tuning Large Language Models (LLMs) on the CyberLLMInstruct dataset results in a critical degradation of safety alignment and refusal mechanisms. While the dataset comprises "pseudo-malicious" content (educational descriptions of malware, phishing, and exploits without executable payloads), the Supervised Fine-Tuning (SFT) process on this corpus causes the models to generalize this instruction-following behavior to actual malicious requests. This effectively bypasses safety guardrails…

CyberLLMInstruct: A new dataset for analysing safety of fine-tuned LLMs using cyber security data

Evaluated models: Llama 2 70B, Llama 3 8B, Llama 3.1 8B+4 more

Llama 2 70B
Llama 3 8B
Llama 3.1 8B
Phi-3 Mini 3.8B
Mistral 7B
Qwen 2.5 7B
Gemma 2 9B

Logit-Forced Knowledge Extraction

Published Dec 8, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) with accessible output logits are vulnerable to "coercive interrogation," a novel attack that extracts harmful knowledge hidden in low-ranked tokens. The attack doesn't require crafted prompts; instead, it iteratively forces the LLM to select and output low-probability tokens at key positions in the response sequence, revealing toxic content the model would otherwise suppress.

Make them spill the beans! coercive knowledge extraction from (production) llms

Evaluated models: Code Llama 13B Instruct, Codellama-13B-python, GPT-3.5+7 more

Code Llama 13B Instruct
Codellama-13B-python
GPT-3.5
GPT 3.5-turbo-instruct
GPT-3.5-turbo-instruct-0914
Llama 2 13B
Llama 2 7B
Llama 2 70B
Vicuna 13B
Yi-34B

GPT-4v System Prompt Leakage

Published Nov 15, 2023Analyzed Dec 29, 2024Source: arXiv

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly effective jailbreak prompts, bypassing safety restrictions and leading to undesirable outputs, including revealing personally identifiable information from images.

Jailbreaking gpt-4v via self-adversarial attacks with system prompts

Evaluated models: GPT-4, GPT-4V, LLaVA 1.5

AutoDAN: Interpretable LLM Jailbreak

Published Oct 23, 2023Analyzed Dec 29, 2024Source: arXiv

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B+4 more

GPT-3.5 Turbo
GPT-4
Guanaco 7B
Llama 2 Chat
Pythia 12B
Vicuna 13B
Vicuna 7B

Related research categories