Language Model Security Database

A vulnerability in LLM finetuning APIs allows covert malicious finetuning. Attackers can create a dataset where individual data points appear innocuous but, when used for finetuning, teach the LLM to respond to encoded harmful requests with encoded harmful responses. This bypasses existing safety checks and evaluations because the training data appears benign.

A deep reinforcement learning (DRL) based attack, termed RLbreaker, demonstrates the ability to more efficiently generate jailbreaking prompts for large language models (LLMs) than existing methods. The attack leverages a DRL agent to guide the search for effective prompt structures, bypassing safety mechanisms and eliciting undesirable responses to harmful questions. The effectiveness stems from the DRL agent's ability to strategically select prompt mutators, rather than relying on random…

A vulnerability in aligned Large Language Models (LLMs) allows circumvention of safety mechanisms through improved few-shot jailbreaking techniques. The attack leverages injection of special system tokens (e.g., [/INST]) into few-shot demonstrations and demo-level random search to optimize the probability of generating harmful responses. This bypasses defenses that rely on perplexity filtering and input perturbation.

GPT-4o exhibits vulnerability to jailbreak attacks via audio prompts, despite enhanced safety against text-based attacks. Successful jailbreaks can be achieved by converting text prompts, including those optimized for adversarial attacks against other LLMs (demonstrated using GCG, AutoDAN, PAP, and BAP methods), into audio using text-to-speech (TTS) synthesis. This circumvention allows elicitation of unsafe responses from GPT-4o that would otherwise be prevented by its safety mechanisms. The…

Large Language Models (LLMs) are vulnerable to jailbreak attacks exploiting uncommon text-encoded structures (UTES) rarely encountered during training. These UTES, such as JSON, tree representations, or LaTeX code, embedded within prompts, cause LLMs to bypass safety mechanisms and generate harmful content. The attack's success stems from the LLM's difficulty in processing and interpreting these unusual structures, coupled with the obfuscation of malicious instructions within the structured…

Large Language Models (LLMs) are vulnerable to knowledge-based jailbreaks, where an attacker provides domain-specific knowledge to elicit harmful or unintended outputs. The vulnerability stems from the LLM's ability to process and respond to knowledge inputs in a way that circumvents safety mechanisms, even if the input knowledge itself isn't inherently malicious. Attackers leverage this by constructing prompts that combine seemingly innocuous knowledge with subtly manipulative phrasing to…

Large Language Models (LLMs) are vulnerable to prompt injection attacks that can bypass their internal copyright compliance mechanisms, causing them to generate verbatim copyrighted text. The vulnerability stems from insufficient robustness against prompt engineering techniques that manipulate the model into ignoring or circumventing its safety filters designed for copyright protection.

Large Language Models (LLMs) used to control robots exhibit biases leading to discriminatory and unsafe behaviors. When provided with personal characteristics (e.g., race, gender, disability), LLMs generate biased outputs resulting in discriminatory actions (e.g., assigning lower rescue priority to certain groups) and accept or deem feasible dangerous or unlawful instructions (e.g., removing a person's mobility aid).

A vulnerability in Retrieval-Augmented Generation (RAG) systems utilizing LangChain allows for indirect jailbreaks of Large Language Models (LLMs). By poisoning the external knowledge base accessed by the LLM through LangChain, attackers can manipulate the LLM's responses, causing it to generate malicious or inappropriate content. The attack exploits the LLM's reliance on the external knowledge base and bypasses direct prompt-based jailbreak defenses.

LLMs, even when individually assessed as "safe," can be combined by an adversary to achieve malicious outcomes. This vulnerability exploits the complementary strengths of multiple models—a high-capability model that refuses malicious requests and a low-capability model that does not—through task decomposition. Adversaries can either manually decompose tasks into benign (solved by the high-capability model) and easily-malicious subtasks (solved by the low-capability model) or automate the…