Language Model Security Database

A Trojan Activation Attack (TA²) against Large Language Models (LLMs) allows injection of "trojan steering vectors" into activation layers during inference. These vectors, generated by comparing activations from a target LLM and a "teacher" (misaligned) LLM, steer the model's output towards attacker-defined misaligned behaviors (e.g., generating toxic content, biased responses, or helpful instructions for harmful activities). The attack does not require retraining or modifying model weights.

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly effective jailbreak prompts, bypassing safety restrictions and leading to undesirable outputs, including revealing personally identifiable information from images.

A vulnerability exists in several Large Language Models (LLMs) allowing attackers to bypass safety mechanisms through carefully crafted "jailbreak" prompts. The vulnerability exploits the LLMs' susceptibility to prompt rewriting and scenario nesting, allowing malicious prompts to elicit unsafe responses despite safety filters. This is achieved by modifying a harmful prompt's wording without changing its core meaning, and then embedding it within a seemingly innocuous task scenario (e.g., code…

A vulnerability in the fine-tuning API of GPT-4 allows attackers to circumvent built-in RLHF safety mechanisms by fine-tuning the model with a relatively small number of carefully crafted prompt-response pairs. This enables the generation of harmful content, including instructions for illegal activities and the creation of dangerous materials, despite the base model's refusal to generate such content.

Large Vision-Language Models (VLMs) are vulnerable to jailbreaking attacks via typographically rendered visual prompts. The vulnerability stems from the VLM's ability to process and interpret image-based text, bypassing safety mechanisms designed for text-only prompts. Malicious actors can encode harmful instructions into images, which are then processed by the VLM's visual module and subsequently interpreted by the language model, resulting in the generation of unsafe and policy-violating…

Large Language Models (LLMs) are vulnerable to a novel "DeepInception" attack that leverages the models' personification capabilities to bypass safety guardrails. The attack uses nested prompts to create a multi-layered fictional scenario, effectively hypnotizing the LLM into generating harmful content by exploiting its tendency towards obedience within the constructed narrative. This allows for continuous jailbreaks in subsequent interactions.

Large Language Models (LLMs) are vulnerable to persona modulation attacks, a black-box jailbreak technique that leverages an LLM assistant to generate prompts causing the target LLM to adopt harmful personas and produce unsafe outputs. This vulnerability circumvents built-in safety mechanisms, enabling the generation of responses related to illegal activities (e.g., synthesizing drugs, building bombs, money laundering), hate speech, and other harmful content. The attack's effectiveness is…

Large Language Models (LLMs) are vulnerable to a targeted linguistic fuzzing attack that exploits the complexity of human language to bypass safety guardrails. The attack, termed "Jade," leverages transformational-generative grammar rules to systematically increase the syntactic complexity of benign seed questions, making them increasingly difficult for LLMs to recognize as malicious. This leads to the generation of unsafe content, even when the underlying semantics remain unchanged.

Low-rank adaptation (LoRA) fine-tuning allows efficient circumvention of safety training in large language models (LLMs), such as Llama 2-Chat 70B, resulting in significantly reduced refusal rates for harmful prompts while maintaining general performance capabilities. Attackers can use LoRA with a small, synthetic dataset of harmful instructions and responses to effectively undo safety measures implemented during the model's training.

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.