Architecture Components

Embedding Research

Security concerns in embedding models and vectors

15 matching entries out of 71 in this category

Embedding research results

15 entries

Discrete Token Subversion

Published Feb 20, 2026Analyzed Mar 9, 2026Source: arXiv

Discrete image tokenizers are vulnerable to unsupervised embedding-space adversarial attacks. Attackers can apply $\ell_p$-bounded perturbations to an input image to maximize the $\ell_2$ distance of the pre-quantization continuous embeddings produced by the tokenizer's vision encoder. This forces the vector quantizer to cross discrete cell boundaries and assign incorrect codebook vectors, fundamentally altering the resulting token sequence. Because the attack targets the pre-quantization…

On the Adversarial Robustness of Discrete Image Tokenizers

Evaluated models: Llama 2 7B

RAG Knowledgebase Exfiltration

Published Feb 10, 2026Analyzed Feb 21, 2026Source: arXiv

Retrieval-Augmented Generation (RAG) systems are vulnerable to iterative knowledge-extraction attacks designed to reconstruct the underlying private knowledge base. The vulnerability exists due to the decoupled optimization of the retrieval and generation phases. Attackers can craft adversarial queries consisting of two distinct components: an "Information" component (optimized via gradient descent or random sampling to steer embeddings toward specific, diverse regions of the vector space) and…

Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation

Evaluated models: GPT-4o, Llama 3 8B, Qwen 2.5 7B

VLM Split-Image Blindspot

Published Feb 8, 2026Analyzed Feb 22, 2026Source: arXiv

Vision Language Models (VLMs) utilizing independent vision encoders (e.g., ViT) and Large Language Model (LLM) decoders are vulnerable to Split-Image Visual Jailbreak Attacks (SIVA). The vulnerability arises from an architectural and alignment discrepancy: while the vision encoder processes image fragments (splits) in isolation via constrained attention or block-diagonal masks, the LLM decoder aggregates these features via cross-attention to reconstruct the semantic content. Current safety…

Robustness of Vision Language Models Against Split-Image Harmful Input Attacks

Evaluated models: Llama 3.2 11B

Unified Robustness Gap

Published Jan 31, 2026Analyzed Mar 9, 2026Source: arXiv

Text scoring models, including dense retrievers, rerankers, and reward models, are vulnerable to score manipulation attacks via search-based discrete perturbations and content injection. An attacker can systematically modify candidate texts using rudimentary string manipulations, gradient-guided token swaps (e.g., HotFlip), masked language modeling (MLM) swaps, or query/sentence injections to spuriously increase model scores. This structural failure condition allows an irrelevant passage or a…

Unifying Adversarial Robustness and Training Across Text Scoring Models

Evaluated models: E5 BERT-base, Qwen 3 0.6B, Llama 3.2 3B Instruct+2 more

E5 BERT-base
Qwen 3 0.6B
Llama 3.2 3B Instruct
Llama 3.1 8B Instruct
Skywork Reward V2

Attention Breaks Concept Resistance

Published Jan 30, 2026Analyzed Feb 21, 2026Source: arXiv

A vulnerability exists in the internal representation mechanisms of Transformer-based Large Language Models (LLMs), specifically Llama-3 and Qwen series models. The vulnerability allows for high-accuracy "steering" of model outputs, effectively bypassing safety guardrails and refusal mechanisms (jailbreaking) without modifying model weights. By exploiting attention-guided feature learning, an attacker can extract a precise "concept vector" representing refusal behaviors. This is achieved by…

Efficient and accurate steering of Large Language Models through attention-guided feature learning

Evaluated models: Llama 3.1 8B, Llama 3.3 70B, Qwen 2.5 14B

Point Cloud Adversarial Attack

Published Jan 10, 2026Analyzed Feb 22, 2026Source: arXiv

Point-based 3D Vision-Language Models (VLMs), specifically PointLLM and GPT4Point, are vulnerable to white-box, gradient-based adversarial attacks. The vulnerability exists in the model's processing of 3D point cloud data, where an attacker can optimize imperceptible geometric perturbations ($\delta$) on the input point cloud ($x$) to manipulate the model's textual output. The paper identifies two specific attack vectors: 1. Vision Attack: Directly perturbs the high-dimensional visual token…

On the Adversarial Robustness of 3D Large Vision-Language Models

Evaluated models: Vicuna 7B

Pretrained Leak Jailbreak

Published Dec 14, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Models (LLMs) finetuned from open-weight pretrained sources inherit adversarial vulnerabilities encoded in the pretrained model's internal representations. An attacker with white-box access to a pretrained model (e.g., Llama-2, Llama-3) can identify linearly separable features in the hidden states that correlate with "transferable" jailbreak prompts. By exploiting these features using a Probe-Guided Projection (PGP) attack, the attacker can optimize adversarial suffixes on the…

One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

Evaluated models: Llama 2 7B Chat, Llama 3 8B Instruct, DeepSeek LLM 7B Chat+5 more

Llama 2 7B Chat
Llama 3 8B Instruct
DeepSeek LLM 7B Chat
Gemma 7B IT
Llama 2 13B
Qwen 7B
Vicuna 7B v1.5
Mistral 7B Instruct v0.2

Refusal-Aware Embedding Jailbreak

Published Oct 14, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), specifically open-weights models such as Llama-2, Mistral, and Vicuna, are vulnerable to a white-box adversarial attack framework termed RAID (Refusal-Aware and Integrated Decoding). The vulnerability exists because the model's safety alignment relies on specific activation patterns ("refusal directions") in the intermediate embedding space. Attackers can exploit this by optimizing a continuous "relaxed" suffix in the embedding space using a triplet loss…

RAID: Refusal-Aware and Integrated Decoding for Jailbreaking LLMs

Evaluated models: Llama 2 7B, Mistral 7B, Vicuna 7B

Universal Transferable LLM Jailbreak

Published Aug 20, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), including Llama 2, Mistral, and Vicuna, are susceptible to a white-box adversarial attack that circumvents safety alignment mechanisms (such as RLHF). The vulnerability exists due to the models' susceptibility to intrinsic optimization of adversarial suffixes using Exponentiated Gradient Descent (EGD). Unlike previous methods that rely on inefficient discrete token searches (e.g., Greedy Coordinate Gradient) or standard projected gradient descent, this attack…

Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent

Evaluated models: GPT-3.5, GPT-4o, Llama 2 7B+3 more

GPT-3.5
GPT-4o
Llama 2 7B
Mistral 7B
Falcon 7B
Vicuna 7B

Pretraining Modality Gap Jailbreak

Published May 30, 2025Analyzed Dec 30, 2025Source: arXiv

Large Vision-Language Models (LVLMs) that utilize a projection layer (adapter) to bridge a vision encoder and a Large Language Model (LLM) contain a vulnerability stemming from the "Modality Gap"—a distributional distance between image and text token embeddings. This gap allows the visual modality to bypass the safety alignment (RLHF/instruction tuning) of the backbone LLM. Attackers can trigger harmful, toxic, or illegal responses to queries that would be refused in text-only contexts by…

Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap

Evaluated models: LLaVA 7B, Vicuna 7B

Related research categories