Language Model Security Database

Fine-tuning Large Language Models (LLMs) on the CyberLLMInstruct dataset results in a critical degradation of safety alignment and refusal mechanisms. While the dataset comprises "pseudo-malicious" content (educational descriptions of malware, phishing, and exploits without executable payloads), the Supervised Fine-Tuning (SFT) process on this corpus causes the models to generalize this instruction-following behavior to actual malicious requests. This effectively bypasses safety guardrails…

A vulnerability exists in large language models (LLMs) where the model's internal representations (activations) in specific latent subspaces can be manipulated to trigger jailbreak responses. By calculating a perturbation vector based on the difference between the mean activations of "safe" and "jailbroken" states, an attacker can introduce a targeted perturbation to the model's activations, causing it to generate unsafe outputs even when presented with a safe prompt. This manipulates the…

Multimodal Large Language Models (MLLMs) are vulnerable to Jailbreak-Probability-based Attacks (JPA). JPA leverages a Jailbreak Probability Prediction Network (JPPN) to identify and optimize adversarial perturbations in input images, maximizing the probability of eliciting harmful responses from the MLLM, even with small perturbation bounds and few iterations. The attack operates by modifying the input image's hidden states within the MLLM to increase the predicted jailbreak probability.

Predictive Large Language Model (LLM) routers, specifically those utilizing Deep Neural Network (DNN) and Matrix Factorization (MF) architectures, are vulnerable to adversarial manipulation and backdoor poisoning. These routers are designed to optimize cost and latency by dynamically directing simple queries to "weak" (cheap) models and complex queries to "strong" (expensive) models. Attackers can exploit this mechanism in two ways: 1. Inference-time Attacks: By appending specific adversarial…

Multimodal Large Language Models (MLLMs) are vulnerable to coupled cross-modal jailbreak attacks that combine continuous visual perturbations with discrete textual manipulations. Because standard alignment and single-modality defenses (such as text-only safety tuning or isolated vision-encoder adversarial training) fail to secure the cross-modal interaction, attackers can simultaneously apply gradient-based noise (e.g., PGD) to input images and adversarial suffixes (e.g., GCG) to text prompts…

A vulnerability exists in Large Language Models (LLMs) that allows for efficient jailbreaking by selectively fine-tuning only the lower layers of the model with a toxic dataset. This "Freeze Training" method, as described in the research paper, concentrates the fine-tuning on layers identified as being highly sensitive to the generation of harmful content. This approach significantly reduces training duration and GPU memory consumption while maintaining a high jailbreak success rate.

Large Language Models (LLMs) are vulnerable to one-shot steering vector optimization attacks. By applying gradient descent to a single training example, an attacker can generate steering vectors that induce or suppress specific behaviors across multiple inputs, even those unseen during the optimization process. This allows malicious actors to manipulate the model's output in a generalized way, bypassing safety mechanisms designed to prevent harmful responses.

Large Language Models (LLMs) employing gradient-based optimization for jailbreaking defense are vulnerable to enhanced transferability attacks due to superfluous constraints in their objective functions. Specifically, the "response pattern constraint" (forcing a specific initial response phrase) and the "token tail constraint" (penalizing variations in the response beyond a fixed prefix) limit the search space and reduce the effectiveness of attacks across different models. Removing these…

Large Language Models (LLMs), specifically Llama 2, Llama 3, Gemma, and Vicuna, are vulnerable to an adaptive, distributional adversarial attack methodology termed "REINFORCE." Existing gradient-based jailbreak attacks (such as Greedy Coordinate Gradient - GCG) typically optimize adversarial suffixes to maximize the likelihood of a fixed affirmative response (e.g., "Sure, here is how"). The REINFORCE method circumvents this by treating the LLM as a probabilistic policy and using Reinforcement…

A vulnerability in large language models (LLMs) allows attackers to bypass safety-alignment mechanisms by manipulating the model's internal attention weights. The attack, termed "Attention Eclipse," modifies the attention scores between specific tokens within a prompt, either amplifying or suppressing attention to selectively strengthen or weaken the influence of certain parts of the prompt on the model's output. This allows injection of malicious content while appearing benign to the model's…