Language Model Security Database

Large Language Models (LLMs) exhibit vulnerabilities when processing complex or ambiguous prompts containing malicious intent. The vulnerability arises from the LLMs' inability to consistently detect maliciousness when prompts are obfuscated by either splitting a single malicious query into multiple parts or by directly modifying the malicious content to increase ambiguity. This allows attackers to bypass built-in safety mechanisms and elicit harmful or restricted content.

Large Language Models (LLMs) are vulnerable to jailbreaking attacks that exploit their tendency to refuse harmful requests. The "Don't Say No" (DSN) attack overcomes this refusal mechanism by optimizing prompts to suppress negative responses, increasing the likelihood of generating harmful content. This is achieved by modifying the loss function during adversarial prompt optimization, prioritizing the suppression of refusal keywords over the elicitation of affirmative responses. The attack…

Large Language Models (LLMs) are vulnerable to adversarial prompting attacks, where a crafted suffix appended to an instruction causes the LLM to generate unsafe or harmful content. The AdvPrompter technique trains a separate LLM to generate these adversarial suffixes, rapidly bypassing LLM safety mechanisms. The generated suffixes are human-readable and contextually relevant, making them harder to detect than previous methods. The attack is effective against both open-source and closed-source…

Large language models (LLMs) are vulnerable to jailbreaking attacks using adversarially generated suffixes. The AmpleGCG attack generates a large number of diverse, effective suffixes which bypass safety mechanisms in both open and closed-source LLMs. The attack leverages the observation that low loss during suffix generation is not a reliable indicator of jailbreaking success, and generates diverse suffixes from intermediate steps of the optimization process.

This vulnerability allows attackers to bypass LLM safety mechanisms and elicit malicious content by injecting a chain of benign, semantically equivalent narrations into a seemingly innocuous article. The LLM connects these scattered narrations, effectively executing the malicious intent hidden within the seemingly benign context. This differs from previous attacks which directly embed malicious prompts, making detection by both LLMs and human reviewers more difficult.

Large Language Models (LLMs) are vulnerable to a vocabulary attack where carefully selected words from the model's vocabulary, identified using an optimization procedure and embeddings from another LLM, are inserted into user prompts. This manipulation can cause the target LLM to generate specific undesired outputs (goal hijacking), such as offensive language or false information, even with minimal word insertions. The attack is difficult to detect because the inserted words may appear…

A color-aware attack, Self Color Testing-based Substitution (SCTS), bypasses watermarking mechanisms in LLMs designed to identify AI-generated text. SCTS exploits the LLM's compliance with instructions to infer the "color" (green/red token classification) of tokens, allowing for targeted substitution of watermarked tokens with non-watermarked tokens, thus evading watermark detection. The attack is particularly effective against watermarks that utilize logit perturbation to bias token selection.

Large Language Models (LLMs) are vulnerable to a novel black-box jailbreak attack, termed "Distraction-based Adversarial Prompts" (DAP). DAP leverages the distractibility and over-confidence of LLMs by concealing malicious queries within complex, unrelated prompts. A memory-reframing mechanism further redirects the LLM's attention away from the distracting context and toward the malicious query, causing the model to bypass safety mechanisms and generate harmful or unintended outputs.

A vulnerability exists in several Large Language Models (LLMs) allowing attackers to bypass safety and ethical protocols through a novel code injection technique using personalized encryption and decryption functions. The attack leverages the LLMs' code execution capabilities to process encrypted malicious instructions, circumventing the intent security recognition mechanism.

Large Language Models (LLMs) are vulnerable to adversarial prompts generated by the Rainbow Teaming technique. Rainbow Teaming uses a quality-diversity search algorithm to create a diverse set of prompts that elicit unsafe, biased, or incorrect outputs from the target LLM, exceeding a 90% success rate across various models. The vulnerability stems from the LLMs' susceptibility to these carefully crafted prompts, bypassing existing safety mechanisms. These prompts are highly transferable across…