Prompt layer Vulnerabilities

Large language models (LLMs) are vulnerable to jailbreak attacks that leverage the injection of special tokens to manipulate the model's interpretation of user input. By strategically inserting special tokens (e.g., <SEP>) that delineate user input and model output, attackers can trick the LLM into treating part of the user-provided input as its own generated content, thereby bypassing safety mechanisms and eliciting harmful responses. This allows attackers to increase the success rate of…

AutoBreach exploits the vulnerability of Large Language Models (LLMs) to wordplay-based adversarial prompts. By leveraging an LLM to generate diverse wordplay mapping rules and employing a two-stage optimization strategy, AutoBreach crafts prompts that bypass LLM safety mechanisms and elicit harmful or unintended responses, even without modifying system prompts. The vulnerability lies in the LLM's susceptibility to semantic manipulation through cleverly disguised inputs.

A vulnerability allows attackers to bypass Large Language Model (LLM) moderation guardrails by using specially crafted prompts containing "cipher characters." These characters, strategically placed within the prompt's output, alter the LLM's response to reduce its "harm" score, enabling the generation of content that would otherwise be blocked. The attack leverages a jailbreak prefix combined with a malicious question and cipher characters to bypass both input and output level filters. This…

Large Language Models (LLMs) exhibit vulnerabilities when processing complex or ambiguous prompts containing malicious intent. The vulnerability arises from the LLMs' inability to consistently detect maliciousness when prompts are obfuscated by either splitting a single malicious query into multiple parts or by directly modifying the malicious content to increase ambiguity. This allows attackers to bypass built-in safety mechanisms and elicit harmful or restricted content.

Large Language Models (LLMs) are vulnerable to prompt extraction attacks via inversion of their normal outputs. An attacker can train a model to reconstruct the prompt used to generate multiple outputs from an LLM, even without access to internal model parameters (logits) or requiring adversarial queries. This allows extraction of both user and system prompts.

A vulnerability in large language models (LLMs) allows attackers to elicit unsafe or unethical responses through a chain of semantically relevant multi-turn prompts. The attack, termed "Chain of Attack" (CoA), exploits the model's contextual understanding and adaptive response capabilities to gradually steer the conversation towards the desired harmful output, even if single-turn prompts are rejected due to safety mechanisms. The attack leverages semantic similarity scoring (e.g., using…

A vulnerability in large language models (LLMs) allows for near-perfect jailbreaking via iterative prompt refinement and self-explanation. The attacker uses the LLM itself to iteratively refine adversarial prompts by requesting self-explanations of failed attempts, ultimately generating prompts that bypass safety mechanisms and elicit harmful content. A subsequent "Rate+Enhance" step further maximizes the harmfulness of the generated output.

Large language models (LLMs) are vulnerable to enhanced jailbreak attacks by appending multiple end-of-sentence (EOS) tokens to malicious prompts. This bypasses internal safety mechanisms, causing the LLM to respond to harmful queries that it would otherwise reject. The EOS tokens subtly shift the LLM’s internal representation of the prompt, making it appear less harmful without significantly altering the semantic meaning of the malicious content.

Large Language Models (LLMs) are vulnerable to a novel jailbreaking attack, "WordGame," which leverages simultaneous query and response obfuscation to bypass safety mechanisms. The attack replaces malicious words with word games in the query, forcing the LLM to reason through the game before addressing the original malicious intent. This, coupled with auxiliary tasks or questions (WordGame+), creates a context absent in the LLM's safety training data, enabling the generation of harmful content.

Leading safety-aligned Large Language Models (LLMs) are vulnerable to simple adaptive jailbreaking attacks. These attacks utilize manually crafted prompt templates, combined with random search on a suffix to maximize the log-probability of a target token indicating compliance (e.g., "Sure"). The attacks are adaptive, as the prompt template and target token are customized for specific models. Furthermore, some models are vulnerable to transfer attacks (using successful prompts from one LLM on…