Language Model Security Database

Large Language Models (LLMs) are vulnerable to jailbreaking attacks using "obscure" input prompts. The ObscurePrompt attack iteratively transforms a base prompt containing known jailbreaking techniques into an obscured version using another LLM (e.g., GPT-4). This obfuscation weakens the LLM's safety mechanisms, causing it to bypass safety restrictions and generate harmful content.

RL-JACK is a reinforcement learning-based black-box attack that generates jailbreaking prompts to bypass safety mechanisms in LLMs. The attack leverages a deep reinforcement learning agent to iteratively refine prompts, maximizing the likelihood of eliciting harmful responses to unethical questions. The effectiveness stems from a novel reward function that provides continuous feedback based on cosine similarity to a reference answer from an unaligned LLM, and an action space that strategically…

Large Language Models (LLMs) trained with reinforcement learning from human feedback (RLHF) are vulnerable to jailbreaking attacks due to reward misspecification. The reward function used during alignment fails to accurately rank the quality of responses, particularly for adversarial prompts designed to elicit undesired behavior. This allows attackers to craft prompts that yield harmful outputs despite the model's intended safety constraints. The vulnerability manifests as a gap between the…

Large language models (LLMs) are vulnerable to jailbreak attacks that leverage the injection of special tokens to manipulate the model's interpretation of user input. By strategically inserting special tokens (e.g., <SEP>) that delineate user input and model output, attackers can trick the LLM into treating part of the user-provided input as its own generated content, thereby bypassing safety mechanisms and eliciting harmful responses. This allows attackers to increase the success rate of…

A vulnerability in several open-source Large Language Models (LLMs) allows for efficient jailbreaking via Adaptive Dense-to-Sparse Constrained Optimization (ADC). This attack uses a continuous optimization method, progressively increasing sparsity to generate adversarial token sequences that bypass safety measures and elicit harmful responses. The attack is more effective and efficient than prior token-level methods.

Multimodal Large Language Models (LLMs) processing speech input are vulnerable to adversarial attacks. Imperceptible perturbations added to audio input can cause the model to generate unsafe or harmful text responses, overriding built-in safety mechanisms. The attacks are effective even with limited knowledge of the model's internal workings, demonstrating transferability across different models.

AutoBreach exploits the vulnerability of Large Language Models (LLMs) to wordplay-based adversarial prompts. By leveraging an LLM to generate diverse wordplay mapping rules and employing a two-stage optimization strategy, AutoBreach crafts prompts that bypass LLM safety mechanisms and elicit harmful or unintended responses, even without modifying system prompts. The vulnerability lies in the LLM's susceptibility to semantic manipulation through cleverly disguised inputs.

Commercial text-to-image (T2I) generative AI systems are vulnerable to automated prompt injection attacks that bypass their safety mechanisms and induce the generation of copyrighted material. An attacker can use an automated pipeline to generate prompts that precisely describe copyrighted images, effectively circumventing word-based detection filters.

A vulnerability allows attackers to bypass Large Language Model (LLM) moderation guardrails by using specially crafted prompts containing "cipher characters." These characters, strategically placed within the prompt's output, alter the LLM's response to reduce its "harm" score, enabling the generation of content that would otherwise be blocked. The attack leverages a jailbreak prefix combined with a malicious question and cipher characters to bypass both input and output level filters. This…

Large Language Models (LLMs) are vulnerable to improved optimization-based jailbreaking attacks. The vulnerability stems from the susceptibility of LLMs to crafted prompts that exploit weaknesses in their safety mechanisms, allowing them to generate harmful responses despite safety training. This vulnerability is exacerbated by the use of diverse target templates containing harmful self-suggestions and guidance within the optimization goal, accelerating the convergence of the attack.