Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Pattern Enhanced Multi-Turn Jailbreaking

Published 10/1/2025

Analyzed 11/1/2025

Pattern Enhanced Chain of Attack (PE-CoA) shows that safety controls can be bypassed gradually across a conversation. The paper evaluates five recurring conversational patterns and finds that resistance to one pattern does not reliably generalize to others, creating a black-box, multi-turn jailbreak risk even when individual turns appear benign.

Pattern Enhanced Multi-Turn Jailbreaking: Exploiting Structural Vulnerabilities in Large Language Models

Evaluated models: Claude 3 Haiku, DeepSeek Chat, Gemini 1.5 Flash +9 more

Source: arXiv

Personalized Disinformation Jailbreak Escalation

Published 10/1/2025

Analyzed 11/11/2025

Appending simple demographic persona details to prompts requesting policy-violating content can bypass the safety mechanisms of Large Language Models. This technique, referred to as persona-targeted prompting, adds details such as country, generation, and political orientation to a request for a harmful narrative (e.g., disinformation). This systematically increases the jailbreak rate across most tested models and languages, in some cases by over 10 percentage points, enabling the generation…

A Multilingual, Large-Scale Study of the Interplay between LLM Safeguards, Personalisation, and Disinformation

Evaluated models: Claude 3.5 Sonnet, Gemma 2 9B IT, GPT-4o +5 more

Source: arXiv

Persuasive Jailbreak Fingerprint

Published 10/1/2025

Analyzed 11/1/2025

Large Language Models (LLMs) are vulnerable to jailbreak attacks that use persuasive techniques grounded in social psychology to bypass safety alignments. Malicious instructions can be reframed using one of Cialdini's seven principles of persuasion (Authority, Reciprocity, Commitment, Social Proof, Liking, Scarcity, and Unity). These rephrased prompts, which remain human-readable and can be generated automatically, manipulate the LLM into complying with harmful requests it would otherwise…

Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks

Evaluated models: DeepSeek R1, GPT-2, Phi-4 +1 more

Source: arXiv

RL-Hammer Autonomous Jailbreak

Published 10/1/2025

Analyzed 12/9/2025

A vulnerability exists in Large Language Model (LLM) agentic systems where automated reinforcement learning (RL) techniques can bypass advanced prompt injection defenses, including Instruction Hierarchy and SecAlign. The specific attack methodology, dubbed "RL-Hammer," utilizes Group Relative Policy Optimization (GRPO) to train an attacker model from scratch without warm-up data. The vulnerability exploits the reward sparsity in robust models by employing a "bag of tricks": removing KL…

RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection

Evaluated models: Llama 3.1 8B Instruct, Meta-SecAlign 8B, Meta-SecAlign 70B +7 more

Source: arXiv

Refusal-Aware Embedding Jailbreak

Published 10/1/2025

Analyzed 12/9/2025

Large Language Models (LLMs), specifically open-weights models such as Llama-2, Mistral, and Vicuna, are vulnerable to a white-box adversarial attack framework termed RAID (Refusal-Aware and Integrated Decoding). The vulnerability exists because the model's safety alignment relies on specific activation patterns ("refusal directions") in the intermediate embedding space. Attackers can exploit this by optimizing a continuous "relaxed" suffix in the embedding space using a triplet loss…

RAID: Refusal-Aware and Integrated Decoding for Jailbreaking LLMs

Evaluated models: Llama 2 7B, Mistral 7B, Vicuna 7B

Source: arXiv

Reinforced Multi-turn Jailbreak

Published 10/1/2025

Analyzed 12/9/2025

Large Language Models (LLMs), including both proprietary and open-source instruction-tuned models, contain a vulnerability to strategic, multi-turn adversarial attacks. Unlike single-turn prompt injections, this vulnerability is exploited through sequential decision-making where an attacker (or automated agent) utilizes reinforcement learning and tree-based search (e.g., DialTree-RPO) to navigate the dialogue state space. By employing strategies such as intent laundering (framing harmful…

Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks

Evaluated models: Claude Sonnet 4, Gemini 2.0 Flash, Gemma 2 2B IT +15 more

Source: arXiv

Schema Exploitation Jailbreak

Published 10/1/2025

Analyzed 10/31/2025

A vulnerability exists in Large Language Models where their strong adherence to processing structured data schemas can be exploited to bypass safety mechanisms. The attack, named BreakFun, uses a multi-component prompt that combines an innocent framing, a Chain-of-Thought (CoT) instruction, and a core "Trojan Schema." This schema is an adversarially designed data structure (e.g., a Python class definition) that embeds a harmful user request. By instructing the model to simulate the…

BreakFun: Jailbreaking LLMs via Schema Exploitation

Evaluated models: Claude 3 Haiku, Claude 3.5 Sonnet, DeepSeek R1 +11 more

Source: arXiv

Self-Amplifying Memory Poisoning

Published 10/1/2025

Analyzed 1/14/2026

Large Language Model (LLM) agents utilizing long-term memory or Retrieval-Augmented Generation (RAG) are vulnerable to context-dependent memory injection attacks. Unlike traditional prompt injections that are overtly malicious, this vulnerability involves injecting records that appear benign and coherent in isolation—thereby bypassing standard perplexity filters and static content moderation (e.g., LlamaGuard). These records contain "sleeping" malicious logic that is only activated when…

A-memguard: A proactive defense framework for llm-based agent memory

Evaluated models: GPT-4o, Llama 3.1 8B

Source: arXiv

Self-Jailbreak Safety Override

Published 10/1/2025

Analyzed 12/9/2025

Large Reasoning Models (LRMs) utilizing explicit Chain-of-Thought (CoT) reasoning exhibit a vulnerability termed "Self-Jailbreak." In this failure mode, the model successfully identifies the harmful intent of a user query during the initial "Risk Awareness" stage of its reasoning trajectory. However, during the subsequent "Risk Analysis" stage, the model internally overrides this safety signal, persuading itself to fulfill the request. This override typically occurs through cognitive patterns…

When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models

Evaluated models: o1, DeepSeek R1, DeepSeek R1 0528 +3 more

Source: arXiv

Special Token Jailbreak

Published 10/1/2025

Analyzed 10/31/2025

Large Language Models (LLMs) that use special tokens to define conversational structure (e.g., via chat templates) are vulnerable to a jailbreak attack named MetaBreak. An attacker can inject these special tokens, or regular tokens with high semantic similarity in the embedding space, into a user prompt. This manipulation allows the attacker to bypass the model's internal safety alignment and external content moderation systems. The attack leverages four primitives: 1. Response Injection…

MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation

Evaluated models: Claude Opus 4, Gemma 2 27B IT, GPT-4.1 +9 more

Source: arXiv

Related research categories