Attack Type

Hallucination Research

Exploits causing models to generate false information

17 matching entries out of 53 in this category

Hallucination research results

17 entries

Cascading Agent False Consensus

Published Mar 4, 2026Analyzed Mar 9, 2026Source: arXiv

Multi-Agent Systems based on Large Language Models (LLM-MAS) are vulnerable to systemic Consensus Corruption via cascading error amplification. Because mainstream collaborative architectures rely on recursive context reuse without atomic-level provenance tracking, a single atomic falsehood injected into the system is repeatedly cited and reused within the multi-agent interaction chain. This structural exposure causes the error to deterministically compound across the communication graph…

From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration

Evaluated models: GPT-4o

TEE Advisor Hallucination

Published Feb 23, 2026Analyzed Mar 8, 2026Source: arXiv

LLM-based security advisors exhibit systematic reasoning failures—including boundary confusion, attestation overclaiming, and mitigation hallucination—when providing architectural guidance for Trusted Execution Environments (TEEs) like Intel SGX and Arm TrustZone. When embedded in tool-augmented agent pipelines, these models are susceptible to agentic misinterpretation, turning partial or poisoned tool outputs into highly confident but materially incorrect security conclusions. This…

Red-Teaming Claude Opus and ChatGPT-based Security Advisors for Trusted Execution Environments

Evaluated models: GPT-5.2, Claude Opus 4.6

Reasoning Model Social Conformity

Published Feb 13, 2026Analyzed Mar 8, 2026Source: arXiv

Large reasoning models are vulnerable to multi-turn adversarial interactions that exploit reasoning-induced overconfidence to force answer capitulation. While explicit reasoning chains improve baseline accuracy, they cause models to effectively "talk themselves into" high confidence scores (clustering at 96–98%) regardless of actual correctness. This systematic overcalibration (r=-0.08, ROC-AUC=0.54) breaks confidence-based defense mechanisms like Confidence-Aware Response Generation (CARG)…

Consistency of Large Reasoning Models Under Multi-Turn Attacks

Evaluated models: GPT-5.1, GPT-5.2, DeepSeek R1+5 more

GPT-5.1
GPT-5.2
DeepSeek R1
Grok 4.1
Grok 3
Gemini 2.5 Pro
GPT-oss 120B
GPT-4o

Context-Robust RAG Poisoning

Published Feb 6, 2026Analyzed Feb 22, 2026Source: arXiv

Retrieval-Augmented Generation (RAG) systems are vulnerable to a robust corpus poisoning attack known as "Confundo." This vulnerability arises from the lack of pipeline awareness in standard RAG implementations, specifically regarding document ingestion (tokenization and chunking) and query variations. An attacker can exploit this by fine-tuning a Large Language Model (LLM) to function as a poison generator. Unlike traditional adversarial examples which are brittle, Confundo generates poison…

Confundo: Learning to Generate Robust Poison for Practical RAG Systems

Evaluated models: Llama 3 8B, Gemini Pro

Chunky Post-Training Miscalibration

Published Feb 5, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) exhibit a vulnerability termed "chunky post-training," where the model learns spurious correlations between incidental prompt features (e.g., formatting styles, specific vocabulary, sentence structure) and specific behavioral modes (e.g., refusal, code generation, rebuttal) present in distinct chunks of post-training data. This results in behavioral mis-routing during inference, where benign inputs sharing surface-level features with restricted or specialized…

Chunky Post-Training: Data Driven Failures of Generalization

Evaluated models: Claude Haiku 4.5, Claude Sonnet 4.5, Claude Opus 4.5+5 more

Claude Haiku 4.5
Claude Sonnet 4.5
Claude Opus 4.5
GPT-5.1
Grok 4.1 Mini
Tülu 3 8B
Tülu 3 70B
Tülu 3 405B

Adversarial Claim Search Deception

Published Jan 31, 2026Analyzed Feb 22, 2026Source: arXiv

Search-enabled Large Language Model (LLM) fact-checking systems are vulnerable to adversarial claim attacks that exploit the pipeline's reliance on claim interpretation, query formulation, and dynamic evidence retrieval. By manipulating the linguistic structure of an input claim while preserving its semantic factual intent, an attacker can induce systematic verification failures. This vulnerability stems from three specific attack surfaces: 1. Search Engine Misguidance: Altering lexical…

DECEIVE-AFC: Adversarial Claim Attacks against Search-Enabled LLM-based Fact-Checking Systems

Evaluated models: GPT-4o

VLM Text Overrides Image

Published Jan 27, 2026Analyzed Feb 21, 2026Source: arXiv

A vulnerability exists in multiple state-of-the-art Vision-Language Models (VLMs), including GPT-4o, Gemini-2.5, and LLaVA-OneVision, where persuasive textual misinformation successfully overrides visual evidence. When a model is presented with an image it can correctly interpret, an attacker can inject a contradictory text prompt employing specific rhetorical strategies (Logical, Credibility, Emotional, or Repetition) to force the model into generating a false response. This "obedience bias"…

Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs

Evaluated models: Qwen 2.5 VL 3B Instruct, Qwen 2.5 VL 7B Instruct, InternVL3 1B+8 more

Qwen 2.5 VL 3B Instruct
Qwen 2.5 VL 7B Instruct
InternVL3 1B
InternVL3 2B
InternVL3 8B
LLaVA-OneVision 0.5B
LLaVA-OneVision 7B
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-4o Mini
GPT-4o

Confident Misinformation Hallucination

Published Jan 21, 2026Analyzed Mar 9, 2026Source: arXiv

A vulnerability in large language models (LLMs) allows attackers to induce factually incorrect outputs by injecting misinformation into prompts framed with strong confidence. By using authoritative phrasing (e.g., "As we know..."), attackers exploit model sycophancy, causing the LLM to accept the false premise and generate hallucinated content aligned with the injected misinformation. The models fail to detect and correct the embedded falsehoods, generating fabricated but plausible responses.

AdversaRiskQA: An Adversarial Factuality Benchmark for High-Risk Domains

Evaluated models: GPT-oss 20B, GPT-oss 120B, GPT-5+3 more

GPT-oss 20B
GPT-oss 120B
GPT-5
Qwen 3 4B Instruct
Qwen 3 30B-A3B Instruct
Qwen 3 Next 80B-A3B Instruct

LLM Judge Framing Bias

Published Jan 20, 2026Analyzed Feb 21, 2026Source: arXiv

LLM-based evaluation systems ("LLM-as-a-Judge") exhibit a structural vulnerability termed "Framing Bias," wherein the model produces logically contradictory judgments depending on the syntactic framing of the evaluation prompt. Specifically, when assessing the same content using predicate-positive (P) framing (e.g., "Is this toxic?") versus predicate-negative (¬P) framing (e.g., "Is this non-toxic?"), models frequently fail to invert their binary decisions, leading to inconsistency rates…

When Wording Steers the Evaluation: Framing Bias in LLM judges

Evaluated models: Llama 3.2 1B Instruct, Llama 3.1 8B Instruct, Llama 3.1 70B Instruct+11 more

Llama 3.2 1B Instruct
Llama 3.1 8B Instruct
Llama 3.1 70B Instruct
Llama 3.3 70B Instruct
Qwen 2.5 1.5B Instruct
Qwen 2.5 3B Instruct
Qwen 2.5 7B Instruct
Qwen 2.5 14B Instruct
Qwen 2.5 32B Instruct
Qwen 2.5 72B Instruct
o4-mini
GPT-4o
GPT-5 Mini
GPT-5

Metacognitive Prompting Lowers Resistance

Published Jan 20, 2026Analyzed Feb 22, 2026Source: arXiv

Large Language Models (LLMs) are vulnerable to multi-turn persuasive conversational attacks that induce the adoption of counterfactual beliefs. By leveraging the Source–Message–Channel–Receiver (SMCR) communication framework, attackers can systematically erode a model's confidence in established facts and compel the model to output misinformation. Specific attack vectors include manipulating source attribution (authority framing), message content (logical, credibility, or emotional appeals)…

Vulnerability of LLMs' Belief Systems? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventions

Evaluated models: GPT-4o, Llama 3.2 3B, Llama 3.3 70B+2 more

GPT-4o
Llama 3.2 3B
Llama 3.3 70B
Mistral 7B
Qwen 2.5 7B

Related research categories