Impact

Safety Research

Security concerns affecting AI safety and alignment

7 matching entries out of 798 in this category

Safety research results

7 entries

Trajectory Poisoning in Self-Evolving Agent Skills

Published Aug 6, 2026Analyzed Aug 13, 2026Source: arXiv

Self-evolving agent-skill systems may promote recurring, attacker-controlled execution records into persistent reusable instructions. The paper evaluates whether poisoned but task-plausible trajectories survive aggregation and appear in generated skill artifacts, using inert canary behaviors and two structurally different skill-evolution pipelines.

When Experience Becomes Instruction: Trajectory Poisoning in Self-Evolving Agent Skill Systems

Evaluated models: GPT-5.4, MiniMax M2.5, DeepSeek V3.2+3 more

GPT-5.4
MiniMax M2.5
DeepSeek V3.2
DeepSeek V4-Pro
Qwen3.5-35B-A3B
Qwen3.5-122B-A10B

Dynamic Security Testing for Third-Party Agent Skills

Published Aug 4, 2026Analyzed Aug 13, 2026Source: arXiv

SkillSentry evaluates third-party agent skills by constructing source-grounded decoy environments and comparing matched executions with and without the tested skill. The method requires completed, observable, skill-attributed side effects rather than treating suspicious text, ordinary privileged operations, or unexecuted paths as proven malicious behavior.

SkillSentry: Adaptive Honey Worlds for Dynamic Safety Testing of Agent Skills

Evaluated models: DeepSeek V4-Pro

Backdoored PEFT adapters and limits of weight-only detection

Published Aug 3, 2026Analyzed Sep 9, 2026Source: arXiv

Third-party fine-tuning adapters may contain backdoors. Z-PEFT screens adapter weights using spectral features, evaluated on PADBench's 13,300 adapters.

Z-PEFT: Zero-shot Backdoor Detection in Parameter-Efficient Fine-Tuning via Canonical Spectral Signatures

Evaluated models: Llama 2 7B, Llama 2 13B, Qwen 1.5 7B+2 more

Llama 2 7B
Llama 2 13B
Qwen 1.5 7B
Flan-T5 XL
RoBERTa Base

Malicious GitHub Issue Injection Against Coding Agents

Published Jul 22, 2026Analyzed Aug 13, 2026Source: arXiv

IssueTrojanBench studies indirect prompt injection when a coding agent processes an apparently ordinary software-development issue or related artifact. Starting with six legitimate seed issues from two Python repositories, the authors construct 696 adversarial issue variants spanning four unsafe-action families and six delivery formats, then execute those variants across six agent-model configurations.

IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests

Evaluated models: GPT-5.3 Codex, GPT-5.4, Claude Sonnet 4.6

Third-Party Agent Skill Supply-Chain Execution Risks

Published Jul 22, 2026Analyzed Aug 13, 2026Source: arXiv

OpenSkillRisk evaluates whether agent harnesses safely handle third-party skills that introduce risky behavior through otherwise plausible, benign tasks. The benchmark assembles 263 risky skills from public agent-skill ecosystems and tests three CLI-agent harnesses against seven risk categories using isolated task workspaces, mocked external services, and execution-level evidence.

OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

Evaluated models: GPT-5.1 Codex Mini, GPT-5.3 Codex, GPT-5.4+10 more

GPT-5.1 Codex Mini
GPT-5.3 Codex
GPT-5.4
Gemini 2.5 Pro
Gemini 3 Flash
Gemini 3.1 Pro
Claude Haiku 4.5
Claude Sonnet 4.6
Claude Opus 4.6
DeepSeek V3.2
DeepSeek V4-Pro
Kimi-K2.6
GLM-5.1

Lifecycle poisoning of reusable agent skills

Published Jul 15, 2026Analyzed Jul 21, 2026Source: arXiv

The paper presents SkillSec-Eval, a controlled evaluation of attacks against reusable agent skills across repository admission, semantic retrieval, planner selection, runtime execution, and updates. It reports that malicious metadata, retrieval manipulation, unsafe workflow composition, and poisoned updates can cause agents to retrieve, select, or execute unintended skills. These are paper-reported benchmark results, not independently verified vulnerabilities in a named production product.

Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation

Evaluated models: all-MiniLM-L6-v2, Gemini 3.1 Pro, Gemini 1.5 Flash

RLHF Preference Data Poisoning

Published Mar 4, 2025Analyzed Dec 9, 2025Source: arXiv

An untrusted reinforcement-learning-from-human-feedback (RLHF) platform can selectively manipulate preference samples associated with an attacker's target domain. The corrupted preference data trains a compromised reward model and then steers the fine-tuned language model toward undesirable behavior, creating a model-supply-chain risk before deployment.

LLM Misalignment via Adversarial RLHF Platforms

Evaluated models: Not reported

Related research categories