Information-Overloading Jailbreaks in Vision-Language Models

Description

The paper describes a reproducible black-box multimodal jailbreak evaluation, INFER/INFER+, in which dense image typography, nested cross-modal references, recursive visual layouts, and entropy-guided search increase processing complexity and weaken refusal behavior in large vision-language models. The authors report average ASRs of 88.6% on open-source models and 84.0% on commercial models; these are paper-reported measurements, not independently verified facts. For safe defensive reproduction, evaluate the layout and filtering behavior only with benign policy-boundary prompts or established harmless red-team surrogates rather than operational harmful payloads.

Examples

See the primary source for the paper's evaluation and defensive reproduction context: https://arxiv.org/abs/2607.02961 (opens in a new tab)

Impact

Successful exploitation can cause safety-aligned vision-language models to bypass refusals and produce unsafe responses from adversarial image-text inputs. The reported cross-model transferability makes the issue relevant to black-box commercial deployments, including document analysis, browsing, assistant, and embodied-agent workflows that accept complex visual inputs.

Affected Systems

• Large vision-language model applications that accept user-supplied images and text
• Multimodal document-analysis, web-browsing, personal-assistant, and embodied-agent systems

Mitigation Steps

• Apply multimodal input screening that jointly analyzes OCR-extracted typography, surrounding image content, and accompanying text before model invocation.
• Detect or constrain unusually dense, recursive, repeated, or deeply nested image-typography layouts; route suspicious inputs to stricter review or refusal paths.
• Perform safety classification after cross-modal semantic reconstruction, not only on raw text or isolated image regions.
• Adversarially train and regression-test safety behavior on complex multimodal layouts, typography injection, nested references, and cross-model transfer cases.
• Use layered output safety checks and least-privilege controls for downstream tools, because the paper reports that input guards and prompt defenses reduced but did not eliminate attack success.
• Monitor refusal-confidence and anomalous multimodal complexity signals where available, while treating these as supplementary indicators rather than sole defenses.

Evidence

• Primary source and publication metadata: https://arxiv.org/abs/2607.02961 (opens in a new tab) (submitted 2026-07-03; abstract reports information overloading through recursive image-typography layouts and average ASRs of 88.6% open-source and 84.0% commercial).
• Method description: https://arxiv.org/html/2607.02961#S4 (opens in a new tab) (Sections 4.1–4.3 describe INFER/INFER+, nested image-text-typography references, recursive layouts, and entropy-guided random search).
• Evaluation scope: https://arxiv.org/html/2607.02961#S5 (opens in a new tab) (Section 5 identifies MM-SafetyBench and HADES, the eight tested models, Llama Guard 3-8B judging, and ASR methodology).
• Reported model results: https://arxiv.org/html/2607.02961#S6.SS1 (opens in a new tab) (Section 6.1, Table 1 reports per-model ASRs, including INFER+ results for open and commercial models; these results were not independently verified).
• Reported transferability: https://arxiv.org/html/2607.02961#S6.SS2 (opens in a new tab) (Section 6.2, Table 2 reports transfer across open-source surrogate and target LVLMs).
• Defense evaluation: https://arxiv.org/html/2607.02961#S6.SS3 (opens in a new tab) (Section 6.3, Table 4 reports that Input Guard, TVPI Defense, and FigStep Defense reduce but do not eliminate INFER+ ASR).
• Mechanism and limitations context: https://arxiv.org/html/2607.02961#S3.SS2 (opens in a new tab) and https://arxiv.org/html/2607.02961#S7 (opens in a new tab) (Sections 3.2 and 7 attribute reduced refusal confidence to increased multimodal complexity and discuss risks to real-world LVLM applications).