Language Model Security Database

Large Language Models (LLMs) exhibit a significantly lower safety threshold when prompted in low-resource languages, such as Singlish, Malay, and Tamil, compared to high-resource languages like English. This vulnerability allows for the generation of toxic, biased, and hateful content through simple prompts. The models are susceptible to "toxicity jailbreaks" where providing a few toxic examples in-context (few-shot prompting) causes a substantial increase in the generation of harmful outputs…

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Aligned Large Language Models (LLMs) utilizing Transformer architectures are vulnerable to representation-level attacks targeting safety-knowledge neurons within the Multi-Layer Perceptron (MLP) layers. Research indicates that safety decision-making (Rejection vs. Conformity) is localized to specific neurons in middle-to-late layers (layers 10-30). An attacker with white-box access can calculate a "Conformity" direction vector based on the activation differences between benign and harmful…

Safety alignment degradation occurs in instruction-tuned Large Language Models (LLMs), specifically Llama-2-7B, Llama-3.2-1B, Qwen2.5, and Phi-3, during the fine-tuning process on benign downstream datasets (e.g., Dolly, Alpaca). This vulnerability results from suboptimal optimization configurations—specifically aggressive learning rates, small batch sizes, and insufficient gradient accumulation—which cause the model parameters to diverge from the pre-trained safety optimization landscape (the…

The Magic-Token-Guided Co-Training (MTC) framework for Large Language Models (LLMs) introduces a mechanism where distinct behavioral modes are activated via hardcoded system-level strings known as "magic tokens." A specific vulnerability exists in the implementation of the "negative" (neg) behavior mode, which is explicitly trained to generate unfiltered, risk-prone, and harmful content for internal red-teaming. The framework relies on the secrecy of the magic token (e.g., a random string like…

A vulnerability exists in LLM-based Multi-Agent Systems (LLM-MAS) where an attacker with control over the communication network can perform a multi-round, adaptive, and stealthy message tampering attack. By intercepting and subtly modifying inter-agent messages over multiple conversational turns, an attacker can manipulate the system's collective reasoning process. The attack (named MAST in the reference paper) uses a fine-tuned policy model to generate a sequence of small, context-aware…

A vulnerability exists in the fine-tuning lifecycle of Vision-Language Models (VLMs) derived from open-source base models, termed the "grey-box threat." Adversaries with white-box access to a public base model (e.g., Qwen2-VL) can generate universal adversarial images that successfully bypass safety guardrails in proprietary, fine-tuned downstream variants. This is achieved via the Simulated Ensemble Attack (SEA), which combines two techniques: Fine-tuning Trajectory Simulation (FTS), where…

Large Language Model (LLM) systems integrated with private enterprise data, such as those using Retrieval-Augmented Generation (RAG), are vulnerable to multi-stage prompt inference attacks. An attacker can use a sequence of individually benign-looking queries to incrementally extract confidential information from the LLM's context. Each query appears innocuous in isolation, bypassing safety filters designed to block single malicious prompts. By chaining these queries, the attacker can…

Adversarial Activation Patching enables the induction of emergent deceptive behaviors in safety-aligned transformer-based Large Language Models (LLMs). By extracting intermediate activations ($A_{d}$) generated during the processing of a deceptive or harmful prompt and injecting them into the forward pass of a benign target prompt ($x_{t}$) at specific layers (specifically mid-layers, e.g., 5-10 in 32-layer architectures), an attacker can manipulate the model's internal reasoning circuits…

A contextual priming vulnerability, termed "Response Attack," exists in certain multimodal and large language models. The vulnerability allows an attacker to bypass safety alignments by crafting a dialogue history where a prior, fabricated model response contains mildly harmful or scaffolding content. This primes the model to generate policy-violating content in response to a subsequent trigger prompt. The model's safety mechanisms, which primarily evaluate the user's current prompt, are…