Scientific Language Jailbreak

Description

Large Language Models (LLMs) are vulnerable to malicious prompts disguised as summaries of scientific papers, even when those papers are fabricated by the attacker. This allows attackers to manipulate LLMs into generating responses exhibiting significantly increased stereotypical bias and toxicity. The vulnerability is exacerbated by multi-turn interactions, where bias scores tend to increase with each subsequent response. The inclusion of author names and publication venues in the fabricated summaries enhances the effectiveness of the attack.

Examples

See the paper "LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language (opens in a new tab)". The paper includes specific examples demonstrating the attack against GPT-4o, GPT-4o Mini, GPT-4, Llama 3.1 405B Instruct, Llama 3.1 70B Instruct, Command R+ (Cohere), and an unspecified Gemini endpoint using both real and fabricated scientific papers as input. The examples show how neutral prompts are used as a springboard to elicit biased and toxic outputs.

Impact

The vulnerability allows attackers to manipulate LLMs into generating harmful content, including but not limited to:

• Reinforcement of harmful societal stereotypes.
• Generation of toxic and offensive language.
• Dissemination of misinformation disguised as scientific fact.

The impact is increased in multi-turn scenarios, potentially leading to the propagation of increasingly harmful content. The vulnerability poses a potential risk to the trustworthiness of LLM-powered applications and systems.

Affected Systems

Various LLMs evaluated in the paper include GPT-4o, GPT-4o Mini, GPT-4, Llama 3.1 405B Instruct, Llama 3.1 70B Instruct, Command R+ (Cohere), and Gemini. The paper does not report a Gemini checkpoint identifier, so that family alias is excluded from model facets. The vulnerability may also be present in other LLMs.

Mitigation Steps

• Develop improved prompt filtering mechanisms to detect and block malicious prompts disguised as scientific statements.
• Enhance LLM training data to reduce biases and improve resistance to manipulation.
• Implement multi-turn dialogue safety measures to monitor and limit the escalation of bias in conversational interactions.
• Develop robust methods for distinguishing between genuine scientific evidence and fabricated or misinterpreted data within prompts.