Attack Type

Hallucination Research

Exploits causing models to generate false information

8 matching entries out of 53 in this category

Hallucination research results

8 entries

TEE Advisor Hallucination

Published Feb 23, 2026Analyzed Mar 8, 2026Source: arXiv

LLM-based security advisors exhibit systematic reasoning failures—including boundary confusion, attestation overclaiming, and mitigation hallucination—when providing architectural guidance for Trusted Execution Environments (TEEs) like Intel SGX and Arm TrustZone. When embedded in tool-augmented agent pipelines, these models are susceptible to agentic misinterpretation, turning partial or poisoned tool outputs into highly confident but materially incorrect security conclusions. This…

Red-Teaming Claude Opus and ChatGPT-based Security Advisors for Trusted Execution Environments

Evaluated models: GPT-5.2, Claude Opus 4.6

Mobile Agent Visual Spoofing

Published Feb 11, 2026Analyzed Feb 22, 2026Source: arXiv

Mobile Large Language Model (LLM) agents operating under the "Screen-as-Interface" paradigm are vulnerable to visual indirect prompt injection and state desynchronization. Agents that rely on unstructured visual data (screenshots) and Accessibility Service APIs to perceive the environment lack a mechanism to distinguish between trusted system UI elements and untrusted content (e.g., web pages, emails, or malicious overlays). An attacker can inject visual cues, fake notifications, or hidden…

Blind Gods and Broken Screens: Architecting a Secure, Intent-Centric Mobile Agent Operating System

Evaluated models: Not reported

Self-Evolution Alignment Blindspots

Published Feb 10, 2026Analyzed Mar 8, 2026Source: arXiv

Closed-loop, self-evolving Large Language Model (LLM) multi-agent systems (MAS) are vulnerable to irreversible safety erosion and alignment failure. When agents recursively optimize and update their policies using only synthetic data derived from internal interactions—without continuous external human grounding—the system naturally minimizes interaction energy and optimizes for internal conversational consistency. This isolation causes a progressive drift away from initial anthropic safety…

The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies

Evaluated models: GPT-3.5 Turbo, Qwen 3 8B

Helpful Agent Default Bypass

Published Jan 14, 2026Analyzed Feb 21, 2026Source: arXiv

A vulnerability exists in the task-planning and execution logic of Large Language Model (LLM) agents, specifically within trip-planning and web-use agents. The vulnerability, identified as a "User-Mediated Attack," occurs because agents prioritize task completion and "helpfulness" over safety verification when processing content provided by the user. When a benign user forwards untrusted external content (e.g., promotional text containing phishing links or malicious instructions) to the agent…

Too Helpful to Be Safe: User-Mediated Attacks on Planning and Web-Use Agents

Evaluated models: Not reported

Distractor Inverse Scaling

Published Jan 12, 2026Analyzed Feb 21, 2026Source: arXiv

Reasoning-capable Large Language Models (LLMs) and agentic AI systems exhibit a critical vulnerability to contextual distractors, resulting in catastrophic performance degradation (up to 80% drop in accuracy) and emergent misalignment. When the input context contains noise—specifically random documents, irrelevant chat history, or task-specific "hard negative" distractors—the models fail to filter this information. Instead of ignoring the noise, the models disproportionately attend to…

Lost in the Noise: How Reasoning Models Fail with Contextual Distractors

Evaluated models: Gemini 2.5 Pro, Gemini 2.5 Flash, DeepSeek R1 0528+4 more

Gemini 2.5 Pro
Gemini 2.5 Flash
DeepSeek R1 0528
GPT-oss 120B
Qwen 3 4B Thinking 2507
DeepSeek R1 Distill Llama 8B
Qwen 3 30B-A3B Thinking 2507

CoT Detector Obfuscation Bypass

Published Dec 24, 2025Analyzed Dec 30, 2025Source: arXiv

LLM-based code agents and vulnerability detectors employing Chain-of-Thought (CoT) reasoning are susceptible to automated adversarial code obfuscation. The vulnerability exists because CoT mechanisms expose the model's decision logic, allowing reinforcement learning frameworks (such as CoTDeceptor) to iteratively refine code transformations based on the detector's own reasoning traces. By optimizing for "reasoning instability" and "hallucination" rather than just syntactic evasion, attackers…

CoTDeceptor: Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agents

Evaluated models: DeepSeek R1, GPT-5

Prompt Injection Alignment Bypass

Published Sep 4, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs

Evaluated models: Not reported

RoleBreak Character Jailbreak

Published Sep 25, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) used in role-playing systems are vulnerable to character hallucination attacks, a form of jailbreak exploiting "query sparsity" and "role-query conflict". Query sparsity occurs when prompts fall outside the model's training data distribution, causing it to generate out-of-character responses. Role-query conflict arises when the prompt contradicts the established character persona, leading to inconsistent behavior. These vulnerabilities allow attackers to elicit…

RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

Evaluated models: Claude 3 Haiku, GPT-3.5 Turbo, Llama 3 8B+1 more

Claude 3 Haiku
GPT-3.5 Turbo
Llama 3 8B
Mistral 7B Instruct v0.2

Related research categories