Multi-Agent Typo Vulnerability

Description

Multi-agent Large Language Model (LLM) systems employing ensemble sampling-and-voting strategies (specifically the "Agent Forest" framework) are vulnerable to adversarial input perturbations. While increasing the number of agents ($n \in {1, \dots, 25}$) improves accuracy on clean inputs, the system fails to mitigate the impact of synthetic punctuation noise and human-like typographical errors. Attackers can introduce surface-level perturbations—such as random punctuation insertion (10-50% intensity) or character-level typos (WikiTypo, R2ATA)—that result in persistent Attack Success Rates (ASR). The majority voting mechanism fails to absorb heterogeneous errors, causing the ensemble to converge on incorrect mathematical reasoning or logical inconsistencies, even when individual model scale or agent count is increased.

Examples

• Attack Vector 1: Punctuation Injection (GSM8K)

• Input: "James decides to run 3 sprints 3 : times ; ! a week. He runs 60 meters each : sprint. , ? How many total ! ! . meters does he : run a week?"

• Vulnerability: The noise causes the model to parse the task frequency incorrectly.

• Erroneous Output: The ensemble calculates 3 sprints * 60 meters = 180 meters (interpreting "3 times" as redundant or ignoring the frequency multiplier), failing to reach the ground truth of 540 (3 sprints * 3 times * 60 meters).

• Attack Vector 2: Logic Inversion via WikiTypo (MultiArith)

• Input: "In a video game, each enemy defeated gives you 9 points. If a level has 11 enemies total and you destroy all but 3 of them, how many points would you earn?" (Context: Input contains typo substitutions from WikiTypo dataset).

• Vulnerability: The model correctly identifies 8 enemies were destroyed but logically inverts the target variable.

• Erroneous Output: "We defeated the 3 remaining enemies... The total points earned is 3 * 9 = 27." (Ground truth is 72).

• Attack Vector 3: Character Substitution (R2ATA on MMLU)

• Input: "Statement 1 | Any set of ttwo vectors in $R^{2}$ is linearly indrpendent... ajd $v_1,...,v_k$ are linearly independent..."

• Vulnerability: Typo noise leads to internal logical contradiction in the reasoning chain.

• Erroneous Output: The model identifies Statement 1 as False, but selects option (B) "False, False" while arguing in the text that Statement 2 is True, resulting in an inconsistent classification.

Impact

• Integrity Violation: Degradation of mathematical reasoning capabilities and calculation accuracy, ranging from 5% to 15% drops in accuracy depending on noise intensity.
• Logic Failure: Induction of hallucinated constraints or inverted logic in multi-step reasoning tasks.
• Persistence: The vulnerability is not mitigated by scaling the number of agents; the robustness gap persists even at $n=25$ agents.

Affected Systems

• Multi-agent or ensemble LLM deployments using majority voting aggregation.
• Tested Models: Qwen3-4B/14B, Llama-3.1-8B, Mistral-7B-v0.3, Gemma3-4B/12B.
• Benchmarks: GSM8K, MATH, MMLU-Math, MultiArith.

Mitigation Steps

• Implement noise-aware sampling and aggregation objectives rather than simple majority voting.
• Deploy verifier-assisted or tool-assisted agents to validate intermediate reasoning steps before aggregation.
• Apply training-time data augmentation specifically targeting human-like typos (e.g., using the WikiTypo dataset) to improve base model robustness.