Language Model Security Database

A vulnerability exists in the communication mechanisms of Large Language Model (LLM)-based Multi-Agent Systems (LLM-MAS) enabling an Agent-in-the-Middle (AiTM) attack. An attacker can intercept and manipulate messages between agents, causing the victim agent to produce malicious outputs. The attack does not require compromising individual agents directly; instead, it leverages contextual manipulation of inter-agent communications.

A vulnerability exists in large language models (LLMs) where insufficient sanitization of system prompts allows attackers to extract sensitive information embedded within those prompts. Attackers can use an agentic approach, employing multiple interacting LLMs (as demonstrated in the referenced research), to iteratively refine prompts and elicit confidential data from the target LLM's responses. The vulnerability is exacerbated by the LLM's ability to infer context from seemingly innocuous…

A vulnerability in large language models (LLMs) allows attackers to bypass safety-alignment mechanisms by manipulating the model's internal attention weights. The attack, termed "Attention Eclipse," modifies the attention scores between specific tokens within a prompt, either amplifying or suppressing attention to selectively strengthen or weaken the influence of certain parts of the prompt on the model's output. This allows injection of malicious content while appearing benign to the model's…

CRI (Compliance Refusal Initialization) initializes jailbreak attacks by leveraging pre-trained jailbreak prompts, effectively guiding the optimization process towards the compliance subspace of harmful prompts. This significantly enhances the success rate and reduces the computational overhead of attacks, often requiring only a single optimization step to bypass safety mechanisms. Attacks utilizing CRI demonstrate significantly improved ASR (Adversarial Success Rate) and reduced median steps…

Large Language Models (LLMs) employing safety alignment strategies are vulnerable to jailbreak attacks. These attacks manipulate the LLM's internal representation by activating "jailbreak concepts" in addition to "toxic concepts," causing the model to bypass safety guardrails and generate unsafe outputs despite recognizing the harmful nature of the input. The vulnerability stems from the insufficient mitigation of the influence of the activated jailbreak concepts on model output.

A context-coherent jailbreak attack (CCJA) allows bypassing safety mechanisms in aligned large language models (LLMs) by optimizing perturbations in the continuous word embedding space of a masked language model (MLM). The attack leverages the MLM's ability to reconstruct text from hidden states to generate semantically coherent yet malicious prompts that induce the target LLM to produce unsafe outputs, even with strong safety alignment. The attack's effectiveness is enhanced by using a seed…

The Knowledge-Distilled Attacker (KDA) model, when used to generate prompts for large language models (LLMs), can bypass LLM safety mechanisms resulting in the generation of harmful, inappropriate, or misaligned content. KDA's effectiveness stems from its ability to generate diverse and coherent attack prompts efficiently, surpassing existing methods in attack success rate and speed. The vulnerability lies in the LLMs' insufficient defenses against the diverse prompt generation strategies…

FC-Attack leverages automatically generated flowcharts containing step-by-step descriptions derived or rephrased from harmful queries, combined with a benign textual prompt, to jailbreak Large Vision-Language Models (LVLMs). The vulnerability lies in the model's susceptibility to visual prompts containing harmful information within the flowcharts, thus bypassing safety alignment mechanisms.

Autoregressive Large Language Models (LLMs) utilizing In-Context Learning (ICL) are vulnerable to demonstration permutation attacks due to inherent sensitivity to the ordering of input examples. This vulnerability arises from the limitations of unidirectional attention mechanisms and standard Empirical Risk Minimization (ERM) training, which fails to account for worst-case input permutations. An attacker can exploit this by permuting the order of valid, semantically correct few-shot…

A policy compliance vulnerability exists in the OpenAI GPT Store ecosystem affecting Custom GPTs. The vulnerability stems from the inheritance of safety alignment weaknesses from foundational models (GPT-4 and GPT-4o) and the insufficient enforcement of usage policies during the customization and review process. Custom GPTs can be trivially manipulated to violate safety guidelines—specifically regarding Cybersecurity (malware generation), Academic Integrity (ghostwriting), and Romantic…