Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Multi-Turn Question Fragmentation Jailbreak

Published 10/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a multi-turn jailbreak attack, termed "Jigsaw Puzzles" (JSP), which circumvents existing safeguards by splitting harmful questions into harmless fragments. The LLM is prompted to reconstruct and answer the complete question from these fragments, resulting in the generation of harmful responses. The attack relies on the LLM's ability to piece together seemingly benign input to form a malicious query, exploiting the model's contextual understanding…

Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Models

Evaluated models: Gemini 1.5 Pro, GPT-4, GPT-4o +2 more

Source: arXiv

Multi-turn Actor Jailbreak

Published 10/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to multi-turn adversarial attacks where malicious users obscure harmful intents across multiple queries. The ActorAttack method leverages the LLM's own knowledge base to discover semantically linked "actors" related to a harmful target. By posing seemingly innocuous questions about these actors, the attacker guides the LLM towards revealing harmful information step-by-step, accumulating knowledge until the desired malicious output is obtained, even…

Derail Yourself: Multi-turn LLM Jailbreak Attack through Self-discovered Clues

Evaluated models: Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4 +3 more

Source: arXiv

PC-Bias Jailbreak Vulnerability

Published 10/1/2024

Analyzed 7/14/2025

Large Language Models (LLMs) trained with safety mechanisms exhibit biases which disproportionately allow successful "jailbreak" attacks (circumvention of safety protocols to generate harmful content) when targeting prompts related to marginalized groups compared to privileged groups. This vulnerability stems from the unintended correlation between safety alignment techniques and demographic keywords, creating a higher success rate for malicious prompts incorporating keywords associated with…

Biasjailbreak: analyzing ethical biases and jailbreak vulnerabilities in large language models

Evaluated models: Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4 +7 more

Source: arXiv

Preference Poisoning Extrapolation

Published 10/1/2024

Analyzed 2/21/2026

Large Language Models (LLMs) undergoing alignment via preference learning (such as Reinforcement Learning from Human Feedback [RLHF] or Direct Preference Optimization [DPO]) are vulnerable to backdoor attacks through data poisoning. An attacker can inject a small percentage (e.g., 3% to 5%) of poisoned data into the preference dataset $\mathcal{D} = \{(x, y_w, y_l)\}$. The attack embeds a specific trigger string into the user query $x$.

Poisonbench: Assessing large language model vulnerability to data poisoning

Evaluated models: Llama 2 7B, Llama 3 8B, Mistral 7B +4 more

Source: arXiv

Prompt Translation Jailbreak

Published 10/1/2024

Analyzed 12/29/2024

A vulnerability in safety-aligned Large Language Models (LLMs) allows attackers to bypass safety mechanisms using adversarial prompt translation. The vulnerability stems from the ability to translate garbled adversarial prompts generated by gradient-based attacks into coherent, human-readable prompts that retain their adversarial capability. This allows for the successful transfer of attacks across different LLMs.

Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation

Evaluated models: Not reported

Source: arXiv

Robotic LLM Jailbreak

Published 10/1/2024

Analyzed 12/28/2024

Large language models (LLMs) controlling robots are vulnerable to jailbreaking attacks. The ROBOPAIR algorithm demonstrates that malicious prompts can bypass safety mechanisms, causing robots to perform harmful physical actions. This vulnerability exploits the LLM's reliance on textual prompts and its potential lack of sufficient contextual understanding to prevent unsafe commands. The attack is effective across different access levels.

Jailbreaking LLM-controlled robots

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +1 more

Source: arXiv

SMILES-Prompting LLM Jailbreak

Published 10/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) used in chemical synthesis applications are vulnerable to a novel attack vector, dubbed "SMILES-prompting," which leverages the Simplified Molecular-Input Line-Entry System (SMILES) notation to bypass safety mechanisms and elicit instructions for synthesizing hazardous substances. The attack exploits the LLM's inability to effectively filter or interpret SMILES strings representing dangerous chemicals, leading to the disclosure of synthesis procedures.

SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis

Evaluated models: GPT-4o, Llama 3 70B Instruct

Source: arXiv

Safeguard Denial-of-Service Attack

Published 10/1/2024

Analyzed 12/29/2024

A denial-of-service (DoS) vulnerability exists in certain Large Language Model (LLM) safeguard implementations due to susceptibility to adversarial prompts. Attackers can inject short, seemingly innocuous adversarial prompts into user prompt templates, causing the safeguard to incorrectly classify legitimate user requests as unsafe and reject them. This allows for a DoS attack against specific users without requiring modification of the LLM itself.

Safeguard is a Double-edged Sword: Denial-of-service Attack on Large Language Models

Evaluated models: GPT-4o Mini, Llama Guard 2 8B, Llama Guard 3 8B +2 more

Source: arXiv

Self-Tuning LLM Jailbreak

Published 10/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel iterative self-tuning attack (ADV-LLM) that crafts adversarial suffixes. This attack significantly reduces the computational cost of generating effective jailbreaks compared to prior methods, achieving near 100% success rate against various open-source LLMs and high success rates (e.g., 99% against GPT-3.5, 49% against GPT-4) against closed-source models. The attack leverages iterative self-tuning to improve the LLM's ability to generate…

Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B +4 more

Source: arXiv

Adaptive Position Jailbreak

Published 9/1/2024

Analyzed 12/29/2024

AdaPPA is a jailbreak attack that exploits the varying levels of alignment protection in LLMs at different output positions. It leverages the model's instruction-following capabilities by pre-filling the output with carefully crafted "safe" content, creating a perceived completion and lowering the model's guard before generating malicious content. The attack's effectiveness relies on the adaptive generation of both safe and harmful pre-fill content, strategically placed to exploit weaknesses…

AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs

Evaluated models: ChatGLM3 6B, GPT-4o, GPT-4o Mini +5 more

Source: arXiv

Related research categories