RLHF Preference Data Poisoning

Description

An untrusted reinforcement-learning-from-human-feedback (RLHF) platform can selectively manipulate preference samples associated with an attacker's target domain. The corrupted preference data trains a compromised reward model and then steers the fine-tuned language model toward undesirable behavior, creating a model-supply-chain risk before deployment.

Examples

A defensive reproduction can introduce clearly labeled, inert preference inversions into a held-out target category and compare the resulting reward-model and policy behavior with a verified baseline. The primary source describes selective manipulation of preference data in publicly available RLHF tooling: arXiv:2503.03039 (opens in a new tab).

Impact

Targeted preference poisoning can persist through reward modeling and policy optimization, causing a deployed model to behave unsafely in selected domains while appearing normal on broad utility evaluations.

Affected Systems

• RLHF platforms and fine-tuning pipelines that accept, transform, or store preference datasets without independent provenance and integrity checks.
• Downstream models trained from a reward model or preference dataset controlled by an untrusted service or operator.

Mitigation Steps

• Cryptographically version and audit preference data, sampling transformations, reward-model checkpoints, and training configuration; restrict who can alter each stage.
• Maintain trusted, domain-specific holdout evaluations and compare reward-model and policy behavior before accepting a fine-tuned checkpoint.
• Isolate third-party training services, require reproducible training artifacts, and investigate targeted regressions even when aggregate utility remains stable.

Evidence

Primary source: LLM Misalignment via Adversarial RLHF Platforms (opens in a new tab).