Updated 2/22/2026
Large Vision-Language Models (VLMs) are vulnerable to a transferable targeted adversarial attack known as SGHA-Attack (Semantic-Guided Hierarchical Alignment). This vulnerability arises from the susceptibility of visual encoders (specifically Vision Transformers) to intermediate-layer feature manipulation optimized on a surrogate model (e.g., CLIP). An attacker can craft adversarial images by injecting imperceptible perturbations that enforce semantic consistency with a target text prompt…
SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models
Affects: UniDiffuser, BLIP-2 ViT-g/14, InstructBLIP Vicuna 13B +4 more
Source: arXiv