Skip to main content
LLM Security Database
Skip to research details
Back to research findings
LMVD-ID: 886657fa
Paper published May 1, 2025
Entry analyzed May 31, 2025
Paper-reported evidence
Confidence: Source-linked

The LMVD-ID is an internal research identifier, not an official CVE identifier.

LLM Judge Prompt Injection

Large Language Models (LLMs) used for evaluating text quality (LLM-as-a-Judge architectures) are vulnerable to prompt-injection attacks. Maliciously crafted suffixes appended to input text can manipulate the LLM's…

Prompt layer
Injection
Application layer
Black-box
Integrity
Safety
BibTeX citation

Paper-evaluated models(2)

  • Falcon 3 3B Instruct
  • Qwen 2.5 3B Instruct

Description

Large Language Models (LLMs) used for evaluating text quality (LLM-as-a-Judge architectures) are vulnerable to prompt-injection attacks. Maliciously crafted suffixes appended to input text can manipulate the LLM's judgment, causing it to incorrectly favor a predetermined response even if another response is objectively superior. Two attack vectors are identified: Comparative Undermining Attack (CUA), directly targeting the final decision, and Justification Manipulation Attack (JMA), altering the model's generated reasoning.

Examples

See arXiv:2505.13348 (opens in a new tab) for specific examples of adversarial suffixes generated using the Greedy Coordinate Gradient (GCG) optimization method. Examples show successful attacks resulting in over 30% Attack Success Rate (ASR) for CUA and 15-17% for JMA.

Impact

Compromised evaluation leading to inaccurate assessments of generated text quality. This could impact multiple applications that rely on LLMs for evaluation such as Reinforcement Learning from Human Feedback (RLHF), automated validation in crowdsourcing, and intelligent search and retrieval systems. Attackers can manipulate the LLM's judgment to promote inferior content or suppress superior content.

Affected Systems

Systems employing open-source instruction-tuned LLMs (such as Qwen2.5-3B-Instruct and Falcon3-3B-Instruct) in LLM-as-a-Judge architectures, or similar models vulnerable to prompt injection.

Mitigation Steps

  • Develop and implement robust defense mechanisms against prompt injection attacks. The paper suggests exploring techniques like goal prioritization and attention tracking.
  • Utilize adversarial training techniques to improve the robustness of LLM-as-a-Judge models to prompt injection.
  • Employ input sanitization and filtering techniques to identify and remove potentially malicious suffixes.
  • Regularly audit and update LLM models to address vulnerabilities as they are discovered.
  • Implement verification methods to independently validate the evaluations produced by the LLM-as-a-Judge system.

Research context and confidence

Evidence and verification
Paper-reported; independent reproduction is not documented.
Primary research source linked.
Severity
Not rated by this catalog.
Source and publication type
arXiv · Research preprint.
Peer-review status is not provided by this source.
Author and publication status
Author metadata is not stored; see the primary paper.
Threat model and attacker access
Black-box model, service, or application access.
Related deployment categories
No related deployment category is classified.
Taxonomy labels only; paper-specific deployment prerequisites are not inferred.
Affected systems
Systems employing open-source instruction-tuned LLMs (such as Qwen2.5-3B-Instruct and Falcon3-3B-Instruct) in LLM-as-a-Judge architectures, or similar models vulnerable to prompt injection.

Research Paper

Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks

Primary source: arXiv. Findings are reported by the cited research and have not been independently verified.

View Paper

Evidence

This entry is based on a primary research source. Its findings are paper-reported; independent reproduction and verification are not claimed.

https://arxiv.org/abs/2505.13348

Related research