Long-Context Enterprise Agent Policy Adherence Failures

Description

HANDBOOK.md measures whether an agent can apply detailed organizational rules while completing realistic, multi-step enterprise tasks. The vendor-authored benchmark includes 65 resettable MCP-backed workflows, policy documents of 20 to 124 pages, and 824 deterministic rubric checks covering required decisions, prohibited actions, and final environment state.

Examples

Reproduce defensively in a resettable synthetic company environment with fictional email, chat, calendar, document, and project-board data, and independently check every task criterion; see https://arxiv.org/html/2607.25398 (opens in a new tab).

Impact

Across 30 configurations of 20 models, the highest strict all-criteria pass rate is 36.2% for Claude Fable 5 at its maximum setting; GPT-5.6 Sol reaches 23.5% at its maximum setting. Strict failure means at least one benchmark criterion was missed, not necessarily an exploit, data breach, or real production incident. Synthetic organizations, a shared harness, strict conjunctive grading, and four trials per task constrain generalization.

Affected Systems

• OpenHands-based enterprise-agent harness connected to resettable MCP-backed workplace services
• Agent workflows using long policy manuals, email, chat, calendars, documents, spreadsheets, and project-management tools
• Enterprise automation requiring policy adherence, authorization checks, prohibited-action prevention, and auditable state changes

Mitigation Steps

• Keep standing organizational policies in authoritative structured stores and bind relevant constraints to each proposed action.
• Apply deterministic authorization and policy checks at tool-execution boundaries instead of relying on handbook text remaining in model context.
• Require confirmation or human escalation when an action has external effects, ambiguous authority, or conflicting policy requirements.
• Evaluate full final environment state and each independent policy criterion, not only task completion or model-generated explanations.
• Track missed rules separately from unsafe side effects and validate across representative real policies before claiming deployment effectiveness.

Evidence

• https://arxiv.org/html/2607.25398 (opens in a new tab) — Benchmark design specifies 65 tasks, policy handbooks spanning 20–124 pages, MCP-backed environments, and 824 deterministic grading criteria.
• https://arxiv.org/html/2607.25398 (opens in a new tab) — Section 4 states that each of 30 configurations runs every task four times in a shared OpenHands-based harness.
• https://arxiv.org/html/2607.25398 (opens in a new tab) — Table 2 reports 36.2% strict pass for Claude Fable 5 and 23.5% for GPT-5.6 Sol in their highest listed reasoning configurations.
• https://arxiv.org/html/2607.25398 (opens in a new tab) — Sections 4 and 5 define strict conjunctive task grading and explain that the benchmark measures simulated enterprise policy compliance rather than confirmed production compromise.