Black-Box System Prompt Leakage in Real-World LLM Applications

Description

The paper reports a reproducible black-box evaluation showing that adversarial user queries can cause deployed LLM applications to reveal hidden system prompts. In the authors’ measurement of 1,200 applications across six commercial platforms, 1,064 applications leaked prompt content (81.0%–93.5% per anonymized platform). This is a paper-reported result, not independently verified here. LeakBench and the official artifact repository provide defensive benchmark materials for controlled testing; no operational payload is reproduced here.

Examples

See the primary source for the paper's evaluation and defensive reproduction context: https://arxiv.org/abs/2606.18673 (opens in a new tab)

Impact

Leaked prompts can expose proprietary task logic, tool/workflow parameters, private knowledge-base content, personal information, internal defense rules, and credentials. The paper reports knowledge-base exposure in 4.7% of leaked cases, workflow information in 5.1%, developer privacy data in 2.9%, access secrets in 1.6%, and sensitive internal instructions in 0.8%. Exposed credentials may enable resource abuse, while disclosed workflow and defense logic can facilitate follow-on attacks.

Affected Systems

• Public LLM applications with hidden system prompts exposed through user-facing chat or API interfaces
• GPT Store applications
• Poe applications
• Coze applications
• Tongyi Agent Platform applications
• Baidu AgentBuilder applications
• Tencent Yuanqi applications
• Prompt-centric assistants and agent-style applications with tools, workflows, or retrieval

Mitigation Steps

• Do not place API keys, credentials, private endpoints, personal data, or private knowledge-base content directly in system prompts; keep secrets in server-side storage and inject only the minimum necessary capability.
• Revoke and rotate any credential found in a leaked prompt; the paper reports a developer invalidated an exposed API key after disclosure.
• Treat system prompts as potentially recoverable and enforce authorization, tool permissions, rate limits, and data-access controls outside the prompt.
• Use output-side leakage detection and redaction as a defense-in-depth control, while testing its effect on legitimate application behavior.
• Continuously run controlled prompt-leakage regression tests against representative benign and adversarial inputs, including multi-turn interactions, and manually review suspected leakage.
• For self-hosted models where soft-prompt deployment is feasible, evaluate AREA or comparable attention re-anchoring defenses; the paper reports a better effectiveness–usability trade-off than evaluated baselines, but explicitly states AREA is not a complete security guarantee.
• Do not rely solely on natural-language instructions forbidding disclosure; the paper reports these defenses were common yet offered limited protection.

Evidence

• https://arxiv.org/abs/2606.18673 (opens in a new tab) — abstract and submission metadata: submitted 2026-06-17; reports a 1,200-application study, over 80% leakage, sensitive exposure, and AREA mitigation.
• https://arxiv.org/html/2606.18673 (opens in a new tab) — §3 Threat Model: defines a black-box attacker using only the public interface, able to issue repeated queries and observe outputs, with the goal of extracting the hidden system prompt.
• https://arxiv.org/html/2606.18673 (opens in a new tab) — §4.1–§4.3 and Table 1: evaluates 200 applications on each of six platforms; reports leakage counts of 178, 181, 186, 162, 170, and 187, with rates from 81.0% to 93.5%.
• https://arxiv.org/html/2606.18673 (opens in a new tab) — §4.2: describes the paper’s verification method: ten repeated interactions, a 7/10 consistency threshold when developer confirmation is unavailable, followed by independent review by two researchers.
• https://arxiv.org/html/2606.18673 (opens in a new tab) — §4.3 and Figure 2: reports exposed functional logic, private knowledge-base content, workflow information, developer privacy data, access secrets, and internal defense instructions; includes an anonymized travel-planning application whose prompt contained an API key.
• https://arxiv.org/html/2606.18673 (opens in a new tab) — §4.4 and Appendix A: reports responsible disclosure to all six platforms, medium-severity classification by Alibaba and Baidu, disclosure to 368 developers, and remediation including removing private information and revoking an exposed credential.
• https://arxiv.org/html/2606.18673 (opens in a new tab) — §5.2: defines LeakBench with 50 real-world-derived system prompts, 600 model-specific adversarial samples across three open models, and approximately 5,000 benign queries for controlled defense evaluation.
• https://arxiv.org/html/2606.18673 (opens in a new tab) — §7.5, Table 4, §7.8, and §9: reports AREA’s benchmark and case-study results, while noting per-prompt optimization overhead, lack of absolute guarantees, and possible degradation under adaptive attackers.
• https://github.com/yangyZJU/AREA (opens in a new tab) — official paper artifact repository README: documents LeakBench, baseline measurements, attention-drift analysis, AREA training/evaluation artifacts, and real-world case-study demos for defensive reproduction.