Semantic Prompt Distortion

Description

The Adaptive Greedy Binary Search (AGBS) framework exposes a vulnerability in Large Language Models (LLMs) regarding their susceptibility to semantic-preserving adversarial attacks. The vulnerability is exploited through a hierarchical decomposition strategy that identifies key semantic units (clauses and keywords) within a prompt. AGBS utilizes a dynamic threshold mechanism to adjust semantic similarity bounds in real-time during a beam search process, replacing tokens with candidates that maintain high semantic similarity (e.g., maintaining a BERTScore of $\approx 0.80$) while maximizing adversarial loss. This allows an attacker to generate adversarial inputs that are grammatically coherent and semantically indistinguishable from benign inputs to human observers, yet induce targeted misbehavior, incorrect reasoning, or erroneous outputs in the victim model. This method bypasses static optimization strategies and defense mechanisms that rely on detecting significant semantic drift.

Examples

Specific adversarial string examples are not listed in the text, but the attack implementation and generation code are available in the author's repository.

• See repository: https://github.com/franz-chang/DOBS (opens in a new tab)
• See dataset: GSM8K, Math QA, Strategy QA, and SVAMP (Numerical and Textual QA scenarios).

Impact

• Integrity Violation: The attack successfully forces LLMs to generate incorrect numerical answers and textual hallucinations in Question Answering (QA) tasks, degrading model reliability.
• Bypass of Safety/Optimization Layers: The adversarial samples preserve semantic meaning, allowing them to bypass defense mechanisms designed to flag traditional, high-entropy, or nonsensical jailbreaks.
• Semantic Drift: Automated prompt engineering systems may inadvertently optimize user queries into these adversarial forms, leading to unintended misinterpretations and persistent errors in downstream applications.

Affected Systems

• OpenAI GPT-4 Turbo, GPT-4o, and GPT-3.5 Turbo (Table II baseline).
• Meta Llama 3.1 (8B, 70B) and Llama 3.2 3B.
• Alibaba Qwen 2.5 (7B, 14B).
• Google Gemma 2 (9B, 27B).

Mitigation Steps

• Reassess the full input and conversation intent before responding or invoking tools, combine model-level alignment with independent input and output policy checks, and avoid relying on a single signature or refusal heuristic.
• Add a targeted regression using inert data and actions, measure both safety and utility regressions, and monitor production for repeated or adaptive attempts.