Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Nested-Scene LLM Jailbreak

Published 11/1/2023

Analyzed 1/26/2025

Large Language Models (LLMs) are vulnerable to a novel "DeepInception" attack that leverages the models' personification capabilities to bypass safety guardrails. The attack uses nested prompts to create a multi-layered fictional scenario, effectively hypnotizing the LLM into generating harmful content by exploiting its tendency towards obedience within the constructed narrative. This allows for continuous jailbreaks in subsequent interactions.

Deepinception: Hypnotize large language model to be jailbreaker

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +1 more

Source: arXiv

Persona-Based LLM Jailbreak

Published 11/1/2023

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to persona modulation attacks, a black-box jailbreak technique that leverages an LLM assistant to generate prompts causing the target LLM to adopt harmful personas and produce unsafe outputs. This vulnerability circumvents built-in safety mechanisms, enabling the generation of responses related to illegal activities (e.g., synthesizing drugs, building bombs, money laundering), hate speech, and other harmful content. The attack's effectiveness is…

Scalable and transferable black-box jailbreaks for language models via persona modulation

Evaluated models: Claude 2, GPT-4

Source: arXiv

RLHF Rank-Poisoning Vulnerability

Published 11/1/2023

Analyzed 12/28/2024

A vulnerability exists in the Reinforcement Learning from Human Feedback (RLHF) training process for Large Language Models (LLMs). Malicious actors can manipulate the human preference dataset used to train the reward model by strategically flipping preference labels. This allows attackers to subtly influence the LLM's behavior towards a malicious goal, such as generating longer responses (increasing computational cost) without significantly degrading its performance on the intended task (e.g…

On the exploitability of reinforcement learning with human feedback for large language models

Evaluated models: GPT-4, Llama 13B, Llama 7B +1 more

Source: arXiv

Typographic VLM Jailbreak

Published 11/1/2023

Analyzed 12/29/2024

Large Vision-Language Models (VLMs) are vulnerable to jailbreaking attacks via typographically rendered visual prompts. The vulnerability stems from the VLM's ability to process and interpret image-based text, bypassing safety mechanisms designed for text-only prompts. Malicious actors can encode harmful instructions into images, which are then processed by the VLM's visual module and subsequently interpreted by the language model, resulting in the generation of unsafe and policy-violating…

Figstep: Jailbreaking large vision-language models via typographic visual prompts

Evaluated models: Cogvlm-chat-v1.1, GPT-4V, Llava-v1.5-vicuna-v1.5-13B +4 more

Source: arXiv

Universal Jailbreak Backdoors

Published 11/1/2023

Analyzed 12/29/2024

A universal jailbreak backdoor vulnerability exists in Large Language Models (LLMs) trained using Reinforcement Learning from Human Feedback (RLHF). A malicious actor can poison the RLHF training data by introducing a trigger word into harmful prompts and labeling the harmful model outputs as preferred. This causes the LLM to generate harmful responses whenever the trigger word is included in any prompt, regardless of the prompt's content or topic. The backdoor is "universal" because it…

Universal jailbreak backdoors from poisoned human feedback

Evaluated models: Not reported

Source: arXiv

AutoDAN: Interpretable LLM Jailbreak

Published 10/1/2023

Analyzed 12/29/2024

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B +4 more

Source: arXiv

Automated LLM Jailbreak

Published 10/1/2023

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to prompt-based jailbreaks, allowing adversaries to bypass safety guardrails and elicit undesirable outputs. The Prompt Automatic Iterative Refinement (PAIR) algorithm efficiently generates these jailbreaks using a limited number of black-box queries to the target LLM. The vulnerability stems from the LLM's inability to robustly handle adversarial prompts crafted through iterative refinement, even without white-box access to its internal mechanisms.

Jailbreaking black box large language models in twenty queries

Evaluated models: Claude Instant 1.2, Claude 2.1, Gemini Pro +6 more

Source: arXiv

Automating Stealthy LLM Jailbreaks

Published 10/1/2023

Analyzed 12/28/2024

Large Language Models (LLMs) employing alignment techniques remain vulnerable to "jailbreak" attacks. The AutoDAN technique automatically generates semantically meaningful prompts that bypass safety features and elicit malicious outputs from aligned LLMs, unlike previous methods producing nonsensical prompts easily detectable by perplexity checks. These prompts exploit weaknesses in the LLM's alignment, causing it to generate responses that violate intended safety constraints.

Autodan: Generating stealthy jailbreak prompts on aligned large language models

Evaluated models: GPT-3.5 Turbo, GPT-4

Source: arXiv

Decoding-Based LLM Jailbreak

Published 10/1/2023

Analyzed 12/28/2024

Open-source Large Language Models (LLMs) are vulnerable to a generation exploitation attack that leverages variations in decoding hyperparameters and sampling methods to bypass safety mechanisms. Manipulating these parameters, even subtly, can drastically increase the likelihood of the model generating harmful or unsafe outputs, even in models previously deemed "aligned." The attack is effective even when removing only the system prompt.

Catastrophic jailbreak of open-source llms via exploiting generation

Evaluated models: GPT-3.5 Turbo

Source: arXiv

Fine-tuning Compromises LLM Safety

Published 10/1/2023

Analyzed 12/28/2024

Fine-tuning aligned Large Language Models (LLMs) on a small number of adversarially crafted examples, or even on benign datasets, can compromise their safety alignment, leading to the generation of harmful or inappropriate content. This vulnerability exploits the few-shot learning capabilities of LLMs, allowing attackers to override existing safety mechanisms with minimal effort and cost. Even unintentional fine-tuning with seemingly benign datasets can result in unintended safety degradation.

Fine-tuning aligned language models compromises safety, even when users do not intend to!

Evaluated models: GPT-3.5 Turbo, Llama 2 13B Chat, Llama 2 70B Chat +2 more

Source: arXiv

Related research categories