Multimodal LLM Jailbreak

Description

A hybrid multimodal jailbreaking attack, dubbed JMLLM, exploits vulnerabilities in 13 popular large language models (LLMs) across text, image, and speech modalities. The attack leverages alternating translation, word encryption, feature collapse in images, and harmful text injection to bypass safety mechanisms and elicit harmful responses. Success rates vary across LLMs and modalities, with some models exhibiting significantly higher vulnerability than others.

Examples

Specific examples of successful attacks are detailed in the paper, including crafting prompts that use a combination of techniques like alternating translation into low-resource languages and word encryption to evade detection. See arXiv:2412.16555 (opens in a new tab).

Impact

Successful exploitation allows attackers to bypass safety and ethical constraints implemented in LLMs, leading to the generation of harmful content including hate speech, misinformation, threats, and instructions for illegal activities. The impact's severity varies depending on the specific LLM, its deployment context, and the nature of the elicited harmful content.

Affected Systems

The vulnerability affects the 13 named LLMs detailed in Table 2: GPT-3.5 Turbo, GPT-4, GPT-4o, GPT-4o Mini, Ernie 3.5 Turbo, Qwen 2.5 72B, Qwen VL Max, Llama 2 7B, Llama 3 8B, Llama 3 70B, Llama 3.1 405B, Claude 1, and Claude 2. The vulnerability may also be present in other LLMs employing similar architectures and safety mechanisms.

The paper proposes a partial "Harmful Separator" defense that separates instructions from examples and independently analyzes the example for malicious content; it offers incomplete protection and should be combined with additional controls.

Mitigation Steps

• Evaluate each input modality and their combination with bounded, inert perturbations; use modality-aware policy checks and reject inputs whose provenance or consistency cannot be established.
• Reassess the full input and conversation intent before responding or invoking tools, combine model-level alignment with independent input and output policy checks, and avoid relying on a single signature or refusal heuristic.
• Add a targeted regression using inert data and actions, measure both safety and utility regressions, and monitor production for repeated or adaptive attempts.