Combined Malicious Code Jailbreak

Description

Mainstream Large Language Models (LLMs), including DeepSeek-R1, GPT-4o, Llama-3.3-70B-Instruct, and Qwen-Coder, are vulnerable to black-box jailbreak attacks that bypass safety alignment mechanisms to generate functional malicious code. The vulnerability is exploited through specific prompt engineering techniques, most notably "Benign Expression" (substituting malicious keywords with harmless synonyms) and "DRA" (Decomposed Requirement Attack), which conceal malicious intent within seemingly harmless contexts or fragmented character sequences. While standard safety filters often reject direct requests for malware, these adversarial mutations reduce rejection rates significantly (dropping from ~60-70% to ~40%), inducing the models to generate "Level 4" quality responses: comprehensive, detailed, and accurate code for malware categories such as Trojans, Ransomware, Denial of Service (DoS) scripts, and System Destruction tools. High-reasoning models (e.g., OpenAI-o1, DeepSeek-R1) are particularly prone to generating high-sophistication malware once the initial refusal barrier is breached.

Examples

Specific jailbreak prompts and the corresponding malicious code outputs are available in the MalwareBench dataset.

• Benign Expression Attack: The attacker replaces specific triggers (e.g., "encrypt files," "delete system32") with semantic equivalents that do not trigger safety filters, requesting code that performs the same logic under the guise of system administration or testing.
• See repository: https://github.com/MAIL-Tele-AI/MalwareBench (opens in a new tab)
• DRA (Decomposed Requirement Attack): The malicious requirement is broken down into individual letters or phonemes and reconstructed within the context of a benign sentence, bypassing string-matching safety guardrails.
• See dataset: MalwareBench (specifically the "System Destruction" and "Information Theft" categories).

Impact

• Malware Generation: Attackers can generate functional code for 29 malware categories, including Keyloggers, DDoS scripts, and file wipers.
• Lowered Barrier to Entry: Enables low-skill actors ("script kiddies") to acquire sophisticated malware code by leveraging the advanced reasoning capabilities of state-of-the-art LLMs.
• Evasion of Static Analysis: Code generated by LLMs may differ syntactically from known malware signatures, potentially evading traditional antivirus detection.
• Operational Risk: Models deployed in automated coding assistants could be manipulated to inject malicious logic into legitimate software projects.

Affected Systems

• Closed-source models: Claude-3.5-Sonnet-20240620; GPT-4o-preview-20240801; GPT-4o-mini-20240718; GPT-4o-nosafe-preview-20240801; OpenAI-o1-preview-20240912; Qwen-Coder-Turbo, Qwen-Max, Qwen-Plus, and Qwen-Turbo (20240919); SparkDesk-v4.0.
• Open-source models: CodeGen-Multi-350M; StarCoder2-3B; CodeGeeX2-6B; CodeGen25-Instruct-7B; CodeLlama-Instruct-7B and -70B; Qwen-2.5-Coder-Instruct-7B and -32B; Llama3-Instruct-8B; StarCoder2-15B; WizardCoder-v1-15B; StarCoder-15.5B; DeepSeek-Coder-v2-Lite-16B; Wizard-v1.1-33B; Llama-3.3-70B-Instruct; Mistral-Large-Instruct-2407-123B; DeepSeek-Chat-v2-236B; DeepSeek-Coder-v2-Instruct-0724-236B; DeepSeek-R1-671B.

Mitigation Steps

• Adversarial Training: Incorporate the MalwareBench dataset (3,520 jailbreaking prompts) into the model's safety alignment training (RLHF) to improve rejection of "Benign Expression" and "DRA" attacks.
• Intrinsic Safety Alignment: Adopt training methodologies similar to CodeLlama’s "RLHF V5," which integrates diverse "safety" and "helpfulness" data specifically for code, rather than relying solely on external guardrails (like Llama Guard) or generic text safety alignment.
• Context-Aware Filtering: Implement input/output filtering that analyzes code logic and intent rather than relying solely on keyword matching, to detect malicious logic disguised by benign variable names or comments.
• Reasoning Chain Inspection: For Chain-of-Thought (CoT) models (like o1 and R1), monitor the reasoning tokens for alignment violations before the final code is generated.