Attack Type

Hallucination Research

Exploits causing models to generate false information

13 matching entries out of 53 in this category

Hallucination research results

13 entries

Cascading Agent False Consensus

Published Mar 4, 2026Analyzed Mar 9, 2026Source: arXiv

Multi-Agent Systems based on Large Language Models (LLM-MAS) are vulnerable to systemic Consensus Corruption via cascading error amplification. Because mainstream collaborative architectures rely on recursive context reuse without atomic-level provenance tracking, a single atomic falsehood injected into the system is repeatedly cited and reused within the multi-agent interaction chain. This structural exposure causes the error to deterministically compound across the communication graph…

From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration

Evaluated models: GPT-4o

TEE Advisor Hallucination

Published Feb 23, 2026Analyzed Mar 8, 2026Source: arXiv

LLM-based security advisors exhibit systematic reasoning failures—including boundary confusion, attestation overclaiming, and mitigation hallucination—when providing architectural guidance for Trusted Execution Environments (TEEs) like Intel SGX and Arm TrustZone. When embedded in tool-augmented agent pipelines, these models are susceptible to agentic misinterpretation, turning partial or poisoned tool outputs into highly confident but materially incorrect security conclusions. This…

Red-Teaming Claude Opus and ChatGPT-based Security Advisors for Trusted Execution Environments

Evaluated models: GPT-5.2, Claude Opus 4.6

Self-Evolution Alignment Blindspots

Published Feb 10, 2026Analyzed Mar 8, 2026Source: arXiv

Closed-loop, self-evolving Large Language Model (LLM) multi-agent systems (MAS) are vulnerable to irreversible safety erosion and alignment failure. When agents recursively optimize and update their policies using only synthetic data derived from internal interactions—without continuous external human grounding—the system naturally minimizes interaction energy and optimizes for internal conversational consistency. This isolation causes a progressive drift away from initial anthropic safety…

The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies

Evaluated models: GPT-3.5 Turbo, Qwen 3 8B

Adversarial Claim Search Deception

Published Jan 31, 2026Analyzed Feb 22, 2026Source: arXiv

Search-enabled Large Language Model (LLM) fact-checking systems are vulnerable to adversarial claim attacks that exploit the pipeline's reliance on claim interpretation, query formulation, and dynamic evidence retrieval. By manipulating the linguistic structure of an input claim while preserving its semantic factual intent, an attacker can induce systematic verification failures. This vulnerability stems from three specific attack surfaces: 1. Search Engine Misguidance: Altering lexical…

DECEIVE-AFC: Adversarial Claim Attacks against Search-Enabled LLM-based Fact-Checking Systems

Evaluated models: GPT-4o

LLM Agent False Negative

Published Jan 30, 2026Analyzed Feb 21, 2026Source: arXiv

LLM-based autonomous agents deployed for Static Application Security Testing (SAST) false positive filtering exhibit a critical failure mode resulting in the suppression of True Positive (TP) vulnerability reports. When configured to triage alerts from tools such as CodeQL, Semgrep, and SonarQube, agents including SWE-agent, OpenHands, and Aider incorrectly classify legitimate, exploitable vulnerabilities as false positives. This vulnerability suppression is highly correlated with specific…

Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering

Evaluated models: Claude Sonnet 4, DeepSeek Chat, GPT-5

Agent Missing-Tool Hallucination

Published Jan 29, 2026Analyzed Feb 22, 2026Source: arXiv

Multi-turn Large Language Model (LLM) agents deployed in safety-critical domains (specifically automotive assistants) exhibit a "completion-compliance tension" vulnerability. When agents encounter missing tools, incomplete environment observations, or ambiguous user requests, they prioritize satisfying the user's intent over adhering to defined domain safety policies. This results in two distinct failure modes: (1) Premature Action Execution, where agents execute physical state changes based…

CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty

Evaluated models: GPT-5, GPT-5.2, Claude Opus 4.5+7 more

GPT-5
GPT-5.2
Claude Opus 4.5
Claude Sonnet 4
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-4.1
Qwen 3 32B
GPT-oss 120B
xLAM 2 32B

RL Adversarial Function Call

Published Jan 27, 2026Analyzed Feb 22, 2026Source: arXiv

Large Language Models (LLMs) enabled with Function Calling (FC) capabilities are vulnerable to adversarial query rewriting and semantic manipulation. Standard FC models, typically trained via Supervised Fine-Tuning (SFT) on static datasets, fail to generalize against adversarial inputs that deviate from fixed distribution patterns. An attacker can exploit this by crafting queries that are semantically similar to valid requests but engineered to induce "bad cases," such as incorrect tool…

Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approach

Evaluated models: Qwen 2.5 7B Instruct, Qwen 3 0.6B, Qwen 3 4B+1 more

Qwen 2.5 7B Instruct
Qwen 3 0.6B
Qwen 3 4B
Qwen 3 8B

Distractor Inverse Scaling

Published Jan 12, 2026Analyzed Feb 21, 2026Source: arXiv

Reasoning-capable Large Language Models (LLMs) and agentic AI systems exhibit a critical vulnerability to contextual distractors, resulting in catastrophic performance degradation (up to 80% drop in accuracy) and emergent misalignment. When the input context contains noise—specifically random documents, irrelevant chat history, or task-specific "hard negative" distractors—the models fail to filter this information. Instead of ignoring the noise, the models disproportionately attend to…

Lost in the Noise: How Reasoning Models Fail with Contextual Distractors

Evaluated models: Gemini 2.5 Pro, Gemini 2.5 Flash, DeepSeek R1 0528+4 more

Gemini 2.5 Pro
Gemini 2.5 Flash
DeepSeek R1 0528
GPT-oss 120B
Qwen 3 4B Thinking 2507
DeepSeek R1 Distill Llama 8B
Qwen 3 30B-A3B Thinking 2507

Strategic Abstention Failure

Published Jan 12, 2026Analyzed Feb 21, 2026Source: arXiv

A "risk-invariance" vulnerability exists in Large Language Models (LLMs) wherein the model's decision-making policy is functionally decoupled from its verbalized confidence and externally defined error penalties. Despite generating calibrated confidence estimates (internal epistemic uncertainty), affected models fail to adjust their abstention thresholds when presented with high-penalty incentives (e.g., negative utility for incorrect answers). This results in "utility collapse," where models…

Are LLM Decisions Faithful to Verbal Confidence?

Evaluated models: GPT-5 Mini, GPT-5 Nano, GPT-4.1 Mini+7 more

GPT-5 Mini
GPT-5 Nano
GPT-4.1 Mini
Gemini 3 Flash
Gemini 2.5 Flash
Llama 4 Maverick
Gemma 3n E4B
DeepSeek V3.2
DeepSeek V3.2 Thinking
Qwen 3 Next 80B-A3B Thinking

Multi-Agent Typo Vulnerability

Published Nov 10, 2025Analyzed Dec 30, 2025Source: arXiv

Multi-agent Large Language Model (LLM) systems employing ensemble sampling-and-voting strategies (specifically the "Agent Forest" framework) are vulnerable to adversarial input perturbations. While increasing the number of agents ($n \in \{1, \dots, 25\}$) improves accuracy on clean inputs, the system fails to mitigate the impact of synthetic punctuation noise and human-like typographical errors. Attackers can introduce surface-level perturbations—such as random punctuation insertion (10-50%…

More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists

Evaluated models: Llama 3.1 8B, Mistral 7B, Qwen 3 4B+3 more

Llama 3.1 8B
Mistral 7B
Qwen 3 4B
Qwen 3 14B
Gemma 3 4B
Gemma 3 12B

Related research categories