Impact

Integrity Research

Security research involving model-output integrity and reliability

539 matching entries

Integrity research results

539 entries

Hidden Structure Jailbreak

Published 6/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to jailbreak attacks exploiting uncommon text-encoded structures (UTES) rarely encountered during training. These UTES, such as JSON, tree representations, or LaTeX code, embedded within prompts, cause LLMs to bypass safety mechanisms and generate harmful content. The attack's success stems from the LLM's difficulty in processing and interpreting these unusual structures, coupled with the obfuscation of malicious instructions within the structured…

StructuralSleight: Automated Jailbreak Attacks on Large Language Models Utilizing Uncommon Text-Encoded Structure

Evaluated models: Claude 2, Claude 3 Opus, GPT-3.5 Turbo +3 more

Source: arXiv

Knowledge-Based LLM Jailbreak

Published 6/1/2024

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to knowledge-based jailbreaks, where an attacker provides domain-specific knowledge to elicit harmful or unintended outputs. The vulnerability stems from the LLM's ability to process and respond to knowledge inputs in a way that circumvents safety mechanisms, even if the input knowledge itself isn't inherently malicious. Attackers leverage this by constructing prompts that combine seemingly innocuous knowledge with subtly manipulative phrasing to…

Knowledge-to-jailbreak: One knowledge point worth one attack

Evaluated models: FinanceChat 7B, GPT-3.5 Turbo, GPT-4 Turbo +6 more

Source: arXiv

LLM Copyright Jailbreak

Published 6/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to prompt injection attacks that can bypass their internal copyright compliance mechanisms, causing them to generate verbatim copyrighted text. The vulnerability stems from insufficient robustness against prompt engineering techniques that manipulate the model into ignoring or circumventing its safety filters designed for copyright protection.

SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation

Evaluated models: Claude 3 Haiku, Gemini 1.5 Pro, Gemini Pro +5 more

Source: arXiv

LLM Robot Bias & Violence

Published 6/1/2024

Analyzed 4/12/2025

Large Language Models (LLMs) used to control robots exhibit biases leading to discriminatory and unsafe behaviors. When provided with personal characteristics (e.g., race, gender, disability), LLMs generate biased outputs resulting in discriminatory actions (e.g., assigning lower rescue priority to certain groups) and accept or deem feasible dangerous or unlawful instructions (e.g., removing a person's mobility aid).

Llm-driven robots risk enacting discrimination, violence, and unlawful actions

Evaluated models: GPT-3.5, GPT-3.5 Turbo, GPT-4 +1 more

Source: arXiv

LangChain Poisoning Jailbreak

Published 6/1/2024

Analyzed 12/29/2024

A vulnerability in Retrieval-Augmented Generation (RAG) systems utilizing LangChain allows for indirect jailbreaks of Large Language Models (LLMs). By poisoning the external knowledge base accessed by the LLM through LangChain, attackers can manipulate the LLM's responses, causing it to generate malicious or inappropriate content. The attack exploits the LLM's reliance on the external knowledge base and bypasses direct prompt-based jailbreak defenses.

Poisoned langchain: Jailbreak llms by langchain

Evaluated models: ChatGLM2 6B, ChatGLM3 6B, ERNIE 3.5 +3 more

Source: arXiv

Model Combination Misuse

Published 6/1/2024

Analyzed 1/26/2025

LLMs, even when individually assessed as "safe," can be combined by an adversary to achieve malicious outcomes. This vulnerability exploits the complementary strengths of multiple models—a high-capability model that refuses malicious requests and a low-capability model that does not—through task decomposition. Adversaries can either manually decompose tasks into benign (solved by the high-capability model) and easily-malicious subtasks (solved by the low-capability model) or automate the…

Adversaries can misuse combinations of safe models

Evaluated models: Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet +8 more

Source: arXiv

Obscured Prompt Jailbreak

Published 6/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to jailbreaking attacks using "obscure" input prompts. The ObscurePrompt attack iteratively transforms a base prompt containing known jailbreaking techniques into an obscured version using another LLM (e.g., GPT-4). This obfuscation weakens the LLM's safety mechanisms, causing it to bypass safety restrictions and generate harmful content.

Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o Mini +5 more

Source: arXiv

RL-Powered LLM Jailbreak

Published 6/1/2024

Analyzed 12/29/2024

RL-JACK is a reinforcement learning-based black-box attack that generates jailbreaking prompts to bypass safety mechanisms in LLMs. The attack leverages a deep reinforcement learning agent to iteratively refine prompts, maximizing the likelihood of eliciting harmful responses to unethical questions. The effectiveness stems from a novel reward function that provides continuous feedback based on cosine similarity to a reference answer from an unaligned LLM, and an action space that strategically…

RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs

Evaluated models: Falcon-40B-instruct, GPT-3.5 Turbo, Llama 2 70B Chat +3 more

Source: arXiv

Reward Misspecification Jailbreak

Published 6/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) trained with reinforcement learning from human feedback (RLHF) are vulnerable to jailbreaking attacks due to reward misspecification. The reward function used during alignment fails to accurately rank the quality of responses, particularly for adversarial prompts designed to elicit undesired behavior. This allows attackers to craft prompts that yield harmful outputs despite the model's intended safety constraints. The vulnerability manifests as a gap between the…

Jailbreaking as a Reward Misspecification Problem

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +5 more

Source: arXiv

Token Injection Jailbreak

Published 6/1/2024

Analyzed 12/29/2024

Large language models (LLMs) are vulnerable to jailbreak attacks that leverage the injection of special tokens to manipulate the model's interpretation of user input. By strategically inserting special tokens (e.g., <SEP>) that delineate user input and model output, attackers can trick the LLM into treating part of the user-provided input as its own generated content, thereby bypassing safety mechanisms and eliciting harmful responses. This allows attackers to increase the success rate of…

Virtual context: Enhancing jailbreak attacks with special token injection

Evaluated models: GPT-3.5 Turbo, GPT-4

Source: arXiv

Related research categories