Architecture Components

Embedding Research

Security concerns in embedding models and vectors

71 matching entries

Embedding research results

71 entries

Zero-Shot Embedding Leak

Published 4/1/2025

Analyzed 12/30/2025

A Universal Zero-shot Embedding Inversion vulnerability exists in vector databases and embedding-based retrieval systems. The flaw allows an attacker to reconstruct original plaintext documents from their vector embeddings without requiring access to the original training data or training an embedding-specific inversion model. The attack, identified as "ZSinvert," leverages a multi-stage adversarial decoding process: (1) a cosine-similarity guided beam search using a Large Language Model (LLM)…

Universal Zero-shot Embedding Inversion

Evaluated models: Qwen 2 5B

Source: arXiv

Embedding Inversion Privacy Leak

Published 3/1/2025

Analyzed 12/30/2025

Retrieval-Augmented Generation (RAG) systems employing standard dense embedding models (e.g., Sentence-T5, SimCSE-BERT, RoBERTa, MPNet) for End-Cloud collaboration are vulnerable to Embedding Inversion Attacks (EIA). While embeddings are vector representations designed to be human-unrecognizable, they retain sufficient semantic information to allow an attacker with access to the vectors (e.g., a malicious or compromised cloud provider) to reconstruct the original sensitive plaintext input.

Safeguarding LLM Embeddings in End-Cloud Collaboration via Entropy-Driven Perturbation

Evaluated models: Not reported

Source: arXiv

MLM Adaptive RAG Poisoning

Published 3/1/2025

Analyzed 12/30/2025

A vulnerability exists in Retrieval-Augmented Generation (RAG) systems that allows for black-box adversarial attacks known as "CtrlRAG." This flaw allows an attacker to manipulate the generation of Large Language Models (LLMs) by injecting maliciously crafted inputs into the system's knowledge base. Unlike traditional injection attacks that rely on direct concatenation, CtrlRAG utilizes a Masked Language Model (MLM) to iteratively replace words in the malicious text. This optimization ensures…

CtrlRAG: Black-box Document Poisoning Attacks for Retrieval-Augmented Generation of Large Language Models

Evaluated models: GPT-4 Turbo, GPT-4o, Claude 3.5 Sonnet +2 more

Source: arXiv

Adversarial VLM Jailbreak

Published 2/1/2025

Analyzed 12/9/2025

Vision-Language Models (VLMs), specifically the LLaVA-1.5 and LLaVA-1.6 series, are vulnerable to optimization-based white-box jailbreak attacks despite standard safety alignment measures like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Attackers can craft adversarial perturbations in the image space (imperceptible noise) or latent space using Projected Gradient Descent (PGD) to manipulate the model's internal representations. These perturbations maximize the…

Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training

Evaluated models: LLaVA 1.5 7B, LLaVA 1.6 7B

Source: arXiv

Single Image Universal Jailbreak

Published 2/1/2025

Analyzed 12/9/2025

Multimodal Large Language Models (MLLMs) are vulnerable to a universal adversarial attack where a single, optimized image can bypass safety alignment mechanisms across diverse textual queries. By employing gradient-based optimization on the input image pixels while propagating gradients through the vision encoder and language model, an attacker can craft a visual perturbation that coerces the model into a compliant state. When this adversarial image is present in the context, the model’s…

Universal Adversarial Attack on Multimodal Aligned LLMs

Evaluated models: LLaVA 1.5 7B, Llama 3.2 11B Vision Instruct, Phi-3.5 Vision Instruct +1 more

Source: arXiv

Topic-Flip RAG Poisoning

Published 2/1/2025

Analyzed 12/30/2025

Retrieval-Augmented Generation (RAG) systems utilizing dense retrieval mechanisms are vulnerable to topic-oriented adversarial corpus poisoning, specifically via the "Topic-FlipRAG" attack method. This vulnerability allows an attacker to manipulate the opinion or stance of the LLM's output across a broad cluster of related queries, rather than a single specific prompt. The attack leverages a two-stage pipeline: (1) Knowledge-Guided Attack, where an LLM is used to edit a target document to…

Topic-fliprag: Topic-orientated adversarial opinion manipulation attacks to retrieval-augmented generation models

Evaluated models: GPT-4o, Llama 3.1 8B, Qwen 2.5 7B +1 more

Source: arXiv

Weight Tampering Unlearning Bypass

Published 2/1/2025

Analyzed 12/9/2025

State-of-the-art machine unlearning and safety fine-tuning methods for Large Language Models (LLMs) fail to robustly remove hazardous capabilities or refusal mechanisms from model weights. While these methods suppress model outputs during standard input-output interactions, the underlying capabilities remain latent in the parameter space. An attacker with access to model weights (e.g., via open releases or leaked weights) can restore "unlearned" knowledge (such as dual-use biology hazards) or…

Model tampering attacks enable more rigorous evaluations of llm capabilities

Evaluated models: Llama 3 8B

Source: arXiv

AD Black-Box Cascading Disruption

Published 1/1/2025

Analyzed 12/9/2025

Vision Language Models (VLMs) integrated into autonomous driving (AD) systems are vulnerable to a black-box adversarial attack method termed Cascading Adversarial Disruption (CAD). The vulnerability stems from the model's susceptibility to optimized visual perturbations that disrupt the decision-making reasoning chain (perception, prediction, and planning). Attackers can generate adversarial images or physical patches by aligning visual noise with deceptive textual semantics in the model's…

Black-box adversarial attack on vision language models for autonomous driving

Evaluated models: GPT-4, GPT-4o, InstructBLIP

Source: arXiv

Confounder Gadgets Reroute LLMs

Published 1/1/2025

Analyzed 1/14/2026

A vulnerability exists in Large Language Model (LLM) routing systems (control planes) that allows for the manipulation of inference flow via adversarial input sequences. LLM routers, which dynamically direct user queries to either "weak" (cheaper) or "strong" (expensive) models based on predicted query complexity, can be bypassed by appending specific, pre-optimized token sequences known as "confounder gadgets." These gadgets artificially inflate the router's complexity score for an input…

Rerouting llm routers

Evaluated models: Not reported

Source: arXiv

Universal Magic Word Jailbreak

Published 1/1/2025

Analyzed 2/2/2025

A vulnerability exists in text embedding models used as safeguards for Large Language Models (LLMs). Due to a biased distribution of text embeddings, universal "magic words" (adversarial suffixes) can be appended to input or output text, manipulating the similarity scores calculated by the embedding model and thus bypassing the safeguard. This allows attackers to inject malicious prompts or responses undetected.

Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models

Evaluated models: E5 Base v2, Jina Embeddings v2, Nomic Embed +2 more

Source: arXiv

Related research categories