Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

WordGame LLM Jailbreak

Published 5/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to a novel jailbreaking attack, "WordGame," which leverages simultaneous query and response obfuscation to bypass safety mechanisms. The attack replaces malicious words with word games in the query, forcing the LLM to reason through the game before addressing the original malicious intent. This, coupled with auxiliary tasks or questions (WordGame+), creates a context absent in the LLM's safety training data, enabling the generation of harmful content.

WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response

Evaluated models: Gemini Pro, GPT-3.5 Turbo, GPT-4

Source: arXiv

Adaptive LLM Jailbreaks

Published 4/1/2024

Analyzed 12/28/2024

Leading safety-aligned Large Language Models (LLMs) are vulnerable to simple adaptive jailbreaking attacks. These attacks utilize manually crafted prompt templates, combined with random search on a suffix to maximize the log-probability of a target token indicating compliance (e.g., "Sure"). The attacks are adaptive, as the prompt template and target token are customized for specific models. Furthermore, some models are vulnerable to transfer attacks (using successful prompts from one LLM on…

Jailbreaking leading safety-aligned llms with simple adaptive attacks

Evaluated models: Claude 2.0, Claude 2.1, Claude 3 Haiku +14 more

Source: arXiv

Amplified Adversarial Suffix Generation

Published 4/1/2024

Analyzed 12/29/2024

Large language models (LLMs) are vulnerable to jailbreaking attacks using adversarially generated suffixes. The AmpleGCG attack generates a large number of diverse, effective suffixes which bypass safety mechanisms in both open and closed-source LLMs. The attack leverages the observation that low loss during suffix generation is not a reliable indicator of jailbreaking success, and generates diverse suffixes from intermediate steps of the optimization process.

Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 7B Chat +2 more

Source: arXiv

Fast Adaptive LLM Jailbreak

Published 4/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to adversarial prompting attacks, where a crafted suffix appended to an instruction causes the LLM to generate unsafe or harmful content. The AdvPrompter technique trains a separate LLM to generate these adversarial suffixes, rapidly bypassing LLM safety mechanisms. The generated suffixes are human-readable and contextually relevant, making them harder to detect than previous methods. The attack is effective against both open-source and closed-source…

Advprompter: Fast adaptive adversarial prompting for llms

Evaluated models: Falcon 7B Instruct, GPT-3.5 Turbo, GPT-4 +6 more

Source: arXiv

LLM Refusal Suppression Jailbreak

Published 4/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to jailbreaking attacks that exploit their tendency to refuse harmful requests. The "Don't Say No" (DSN) attack overcomes this refusal mechanism by optimizing prompts to suppress negative responses, increasing the likelihood of generating harmful content. This is achieved by modifying the loss function during adversarial prompt optimization, prioritizing the suppression of refusal keywords over the elicitation of affirmative responses. The attack…

Don't Say No: Jailbreaking LLM by Suppressing Refusal

Evaluated models: GPT-3.5 Turbo, GPT-4

Source: arXiv

Logic-Chain Jailbreak

Published 4/1/2024

Analyzed 1/26/2025

This vulnerability allows attackers to bypass LLM safety mechanisms and elicit malicious content by injecting a chain of benign, semantically equivalent narrations into a seemingly innocuous article. The LLM connects these scattered narrations, effectively executing the malicious intent hidden within the seemingly benign context. This differs from previous attacks which directly embed malicious prompts, making detection by both LLMs and human reviewers more difficult.

Hidden You Malicious Goal Into Benigh Narratives: Jailbreak Large Language Models through Logic Chain Injection

Evaluated models: BERT, GPT, GPT-4 +1 more

Source: arXiv

Multi-Turn Crescendo Jailbreak

Published 4/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to the "Crescendo" multi-turn jailbreak attack. This attack uses a series of benign, escalating prompts to gradually lead the LLM into generating harmful or disallowed content, bypassing built-in safety mechanisms. The attack leverages the LLM's tendency to follow conversational patterns and build upon previous responses, making it difficult to detect based solely on individual prompts.

Great, now write an article about that: The crescendo multi-turn llm jailbreak attack

Evaluated models: Claude 2, Claude 3 Opus, Claude 3.5 Sonnet +6 more

Source: arXiv

Vocabulary-Guided LLM Hijacking

Published 4/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a vocabulary attack where carefully selected words from the model's vocabulary, identified using an optimization procedure and embeddings from another LLM, are inserted into user prompts. This manipulation can cause the target LLM to generate specific undesired outputs (goal hijacking), such as offensive language or false information, even with minimal word insertions. The attack is difficult to detect because the inserted words may appear…

Vocabulary Attack to Hijack Large Language Model Applications

Evaluated models: Flan-T5 XXL, Llama 2 7B Chat, Llama 2 Chat +1 more

Source: arXiv

Diverse Jailbreak Optimization

Published 3/1/2024

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to jailbreak attacks that utilize an optimized algorithm to bypass safety mechanisms. The vulnerability stems from the redundancy in existing trigger-searching algorithms, resulting in inefficient exploration of the prompt space and allowing attackers to elicit harmful responses. The proposed DPP-based Stochastic Trigger Searching (DSTS) algorithm demonstrates a statistically significant improvement over existing optimization-based attacks.

Enhancing Jailbreak Attacks with Diversity Guidance

Evaluated models: Alpaca 7B, Gemma 7B IT, GPT-3.5 Turbo +10 more

Source: arXiv

LLM Distraction Jailbreak

Published 3/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to a novel black-box jailbreak attack, termed "Distraction-based Adversarial Prompts" (DAP). DAP leverages the distractibility and over-confidence of LLMs by concealing malicious queries within complex, unrelated prompts. A memory-reframing mechanism further redirects the LLM's attention away from the distracting context and toward the malicious query, causing the model to bypass safety mechanisms and generate harmful or unintended outputs.

Tastle: Distract large language models for automatic jailbreak attack

Evaluated models: GPT-3.5 Turbo, GPT-3.5-1106), GPT-4 +4 more

Source: arXiv

Related research categories