Third-Party Agent Skill Supply-Chain Execution Risks

Description

OpenSkillRisk evaluates whether agent harnesses safely handle third-party skills that introduce risky behavior through otherwise plausible, benign tasks. The benchmark assembles 263 risky skills from public agent-skill ecosystems and tests three CLI-agent harnesses against seven risk categories using isolated task workspaces, mocked external services, and execution-level evidence.

Examples

Use sandboxed copies of benchmark tasks, fictional credentials, local service stubs, and inert action logging to compare skill-enabled behavior with the authorized task; see https://arxiv.org/html/2607.20121 (opens in a new tab).

Impact

Across the authors’ evaluated harness-model combinations, the lowest reported unsafe-action rate is 17.11% for Claude Code with Claude Opus 4.6. An actively loaded guard skill reduces Claude Haiku 4.5 from 20.53% to 7.22% in the reported experiment, but the same configuration has substantial over-defense; mock services, task selection, judge decisions, and harness-specific setup limit deployment claims.

Affected Systems

• Codex, Gemini CLI, and Claude Code agent harnesses under their evaluated configurations
• Third-party agent-skill repositories, installation flows, and skill-enabled coding workflows
• Sandboxed file, credential, configuration, API, database, and deployment interactions

Mitigation Steps

• Verify skill provenance, integrity, requested capabilities, and update history before installation or activation.
• Restrict each skill to the minimum required filesystem, network, credential, and tool permissions.
• Compare skill-enabled actions against the user-authorized task and block out-of-scope side effects before execution.
• Evaluate guard skills against both unsafe execution and over-refusal, rather than relying only on awareness or warning messages.
• Exercise real skill packages only in isolated environments with synthetic credentials and mocked external services.

Evidence

• https://arxiv.org/html/2607.20121 (opens in a new tab) — Benchmark construction describes 263 risky third-party skills, seven risk categories, task validation, fictional sensitive fields, and isolated service stubs.
• https://arxiv.org/html/2607.20121 (opens in a new tab) — Table 1 identifies Codex, Gemini CLI, Claude Code, and all directly evaluated model configurations; the lowest listed attack-success rate is 17.11%.
• https://arxiv.org/html/2607.20121 (opens in a new tab) — Table 4 reports the Claude Haiku 4.5 unsafe-action rate falling from 20.53% to 7.22% when a guard skill is actively loaded.
• https://arxiv.org/html/2607.20121 (opens in a new tab) — Appendix results report over-defense for active guard-skill loading, including the tested Claude Haiku 4.5 configuration.