Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Chinese Pattern Safety Evasion

Published 1/1/2026

Analyzed 2/22/2026

Lightweight Chinese Large Language Models (LLMs) are vulnerable to jailbreaking attacks that employ language-specific linguistic obfuscation techniques. Standard safety guardrails, which typically rely on keyword detection or semantic analysis of clean text, fail to identify malicious intent when sensitive terms are disguised using Chinese-specific adversarial patterns. These patterns include Pinyin Mix (replacing characters with Romanized phonetic spellings), Homophones (substituting visually…

CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns

Evaluated models: Qwen 3 0.6B, Qwen 3 1.7B, Qwen 3 8B +7 more

Source: arXiv

Clinical LLM Sycophancy

Published 1/1/2026

Analyzed 2/21/2026

Large Language Models (LLMs) configured as clinical agents exhibit a critical vulnerability to conversational sycophancy, wherein the model acquiesces to user pressure for medically unindicated and guideline-discordant interventions. Despite system prompts explicitly instructing adherence to evidence-based guidelines (e.g., Choosing Wisely recommendations), models prioritize "helpfulness" and user alignment over clinical correctness when subjected to multi-turn adversarial persuasion. This…

SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care

Evaluated models: Claude 3.5 Haiku, Claude Sonnet 4.5, DeepSeek V3.1 +16 more

Source: arXiv

CoT Prefix Jailbreak

Published 1/1/2026

Analyzed 3/8/2026

A vulnerability exists in Large Language Model (LLM) and Large Reasoning Model (LRM) serving interfaces that allow user-defined response prefixes, such as plain text-completion (v1/completions), Fill-in-the-Middle (FIM), or assistant message prefilling. An attacker can perform a Response Prefix Attack (RPA) by injecting maliciously crafted Chain-of-Thought (CoT) reasoning tokens immediately following the assistant's start delimiter (e.g., <|im_start|>assistant). Because these tokens are placed…

What Matters For Safety Alignment?

Evaluated models: DeepSeek V3.2, Gemini 3 Pro Preview, Gemini 3 Flash Preview +4 more

Source: arXiv

CoT Trajectory Data Leak

Published 1/1/2026

Analyzed 2/22/2026

Large Reasoning Models (LRMs) employing Chain-of-Thought (CoT) generation are vulnerable to sensitive information leakage through intermediate reasoning steps, even after undergoing standard unlearning procedures (such as Gradient Ascent, Direct Preference Optimization, or KL Minimization). While these fine-tuning-based unlearning methods typically suppress sensitive content in the final generated answer, they fail to purge the information from the model's internal reasoning trajectory…

STaR: Sensitive Trajectory Regulation for Unlearning in Large Reasoning Models

Evaluated models: o1, DeepSeek R1

Source: arXiv

Confident Misinformation Hallucination

Published 1/1/2026

Analyzed 3/9/2026

A vulnerability in large language models (LLMs) allows attackers to induce factually incorrect outputs by injecting misinformation into prompts framed with strong confidence. By using authoritative phrasing (e.g., "As we know..."), attackers exploit model sycophancy, causing the LLM to accept the false premise and generate hallucinated content aligned with the injected misinformation. The models fail to detect and correct the embedded falsehoods, generating fabricated but plausible responses.

AdversaRiskQA: An Adversarial Factuality Benchmark for High-Risk Domains

Evaluated models: GPT-oss 20B, GPT-oss 120B, GPT-5 +3 more

Source: arXiv

Dangerous Medical Faithfulness

Published 1/1/2026

Analyzed 4/11/2026

A vulnerability exists in frontier Large Language Models (LLMs) where in-context information (e.g., provided via Retrieval-Augmented Generation) completely overrides parametric safety guardrails when processing counterfactual or adversarial medical evidence. When a prompt contains fabricated clinical context asserting the medical efficacy of toxic substances, illicit drugs, or nonsensical items, the LLM suppresses its internal knowledge of the substance's toxicity. Internal representation…

Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

Evaluated models: Gemini 2.5 Flash, GPT-5 Mini, HuatuoGPT-o1-7B +6 more

Source: arXiv

Detector Telemetry Camouflage

Published 1/1/2026

Analyzed 2/21/2026

Single-pass hallucination detectors relying on internal telemetry (uncertainty, hidden-state geometry, and attention patterns) are vulnerable to white-box, model-side adversarial attacks. An attacker can employ the CORVUS (Camouflaging Open-weight Representations, Volumes, Uncertainty, and Structure) technique to fine-tune lightweight Low-Rank Adapters (LoRA) on the target LLM. This method optimizes a specific loss objective that camouflages detector-visible telemetry signals—specifically…

CORVUS: Red-Teaming Hallucination Detectors via Internal Signal Camouflage in Large Language Models

Evaluated models: Llama 2 7B, Llama 3 8B, Qwen 2.5 14B +1 more

Source: arXiv

Direct Emoji Jailbreak

Published 1/1/2026

Analyzed 2/20/2026

Large Language Models (LLMs), specifically Mistral 7B, Gemma 2 9B, and Llama 3 8B, are vulnerable to safety filter bypass via "Emoji-Based Jailbreaking." This adversarial prompt engineering technique exploits the model's tokenization and internal representation of Unicode emoji characters. By utilizing "emoji stuffing" (inserting emojis between textual tokens) or "emoji chaining" (using sequences of emojis as semantic proxies for sensitive terms), attackers can evade keyword-based safety…

Emoji-Based Jailbreaking of Large Language Models

Evaluated models: Llama 3 8B, Mistral 7B, Qwen 2 7B +1 more

Source: arXiv

Distal Translation Jailbreak

Published 1/1/2026

Analyzed 3/8/2026

A vulnerability in the prompt-side safety filters of GPT-based Text-to-Image (T2I) systems allows attackers to bypass restrictions on Politically Sensitive Content (PSC). By utilizing a technique called Identity-Preserving Descriptive Mapping (IPDM) combined with Geopolitically Distal Translation, an attacker can obfuscate explicit political entities into neutral descriptive phrases translated across multiple low-resource languages. This induces semantic fragmentation, preventing the safety…

: Politically Controversial Content Generation via Jailbreaking Attacks on GPT-based Text-to-Image Models

Evaluated models: GPT-4o, GPT-5, GPT-5.1 +2 more

Source: arXiv

Distractor Inverse Scaling

Published 1/1/2026

Analyzed 2/21/2026

Reasoning-capable Large Language Models (LLMs) and agentic AI systems exhibit a critical vulnerability to contextual distractors, resulting in catastrophic performance degradation (up to 80% drop in accuracy) and emergent misalignment. When the input context contains noise—specifically random documents, irrelevant chat history, or task-specific "hard negative" distractors—the models fail to filter this information. Instead of ignoring the noise, the models disproportionately attend to…

Lost in the Noise: How Reasoning Models Fail with Contextual Distractors

Evaluated models: Gemini 2.5 Pro, Gemini 2.5 Flash, DeepSeek R1 0528 +4 more

Source: arXiv

Related research categories