Repository-Level Coding Agent Vulnerability Detection Gaps
VulnGym measures whether coding agents can locate and explain repository-level security vulnerabilities from realistic advisory and source-code context. The benchmark contains 184 reviewed advisories, 408 line-annotated vulnerability entries, and 23 repositories, with separate end-to-end detection and oracle-conditioned localization tasks.
Evaluated models: DeepSeek V4 Flash, GLM 5.2, MiniMax-M3+4 more
- DeepSeek V4 Flash
- GLM 5.2
- MiniMax-M3
- Qwen3.5-2B
- Qwen3.5-4B
- Qwen3.5-9B
- Qwen3.5-27B