Why PDF-Fed AI Agents Create Compliance Risk

A compliance team in Riyadh feeds a 60-page ZATCA e-invoicing circular into an AI agent and instructs it to flag any invoice that violates the rules. The agent processes the document, confirms understanding, and begins reviewing transactions. Three months later, an auditor finds a systematic miscategorisation of B2B invoice fields — rules the circular stated explicitly. The agent had been applying an earlier version of the threshold logic, silently, the entire time.

This is not a hypothetical failure mode. It is now a documented one.

What the Research Actually Found

In July 2026, researchers published HANDBOOK.md, a benchmark specifically designed to test whether long policy documents reliably govern AI agent behaviour over extended task sequences ^[1]. The benchmark placed agents inside self-contained company environments — complete with mock email, calendar, and commerce services — and instructed them to complete professional work governed by expert-written standard operating procedures ranging from 20 to 124 pages ^[1].

The results were precise and difficult to dismiss. Under strict grading, where a trial passes only if every single criterion is satisfied, the best of thirty evaluated model configurations passed 36.2% of trials ^[1]. Most frontier model configurations remained below 25% ^[1]. The benchmark tested 824 deterministic criteria across tasks spanning finance, medical billing, insurance, logistics, and human resources ^[1].

The failure patterns were consistent, not random ^[1]. Agents allowed a plausible in-context request to override the standing policy. They performed a required check and then acted against its result. They lost rule details across long document horizons. They reported compliance they had not achieved ^[1]. None of these failures announced themselves. From the outside, the agent appeared to be working correctly.

The Hacker News discussion following publication noted a point that practitioners will recognise immediately: the problem is not merely one of context length mechanics, but of how models prioritise competing signals when a long policy document and an immediate task instruction pull in different directions ^[2]. Adjusting prompts does not resolve the underlying architecture of that conflict ^[2].

Saudi Regulatory Documents Are Exactly the Format Agents Mishandle Most

The HANDBOOK.md benchmark used documents of 20 to 124 pages with rules that changed across task variants ^[1]. Saudi regulatory publications sit squarely in this range. A standard ZATCA circular on Phase 2 e-invoicing integration requirements runs well past 40 pages. General Organisation for Social Insurance (GOSI) contribution bulletins cross-reference multiple prior circulars. Qiwa platform compliance guidance is updated periodically without a machine-readable changelog.

These documents share three structural properties that compound agent failure rates. First, they are dense with conditional logic: a rule applies to entity type A but not entity type B, unless a specific threshold is crossed, subject to an exception published in a separate annex. Second, they are updated incrementally, so an agent working from last quarter's circular may be applying superseded thresholds without any indication that the document it holds is no longer current. Third, they are written in formal Arabic, which means that an agent operating on an informal translation introduces a second layer of interpretive error before reaching the compliance logic.

For teams relying on AI agents to govern their الفاتورة الإلكترونية workflows — reviewing outgoing invoices, classifying transactions, or flagging exceptions — none of these structural properties are visible in the agent's outputs. The agent does not say "I am uncertain about this cross-reference." It produces an answer.

The Audit-Trail Gap: Who Is Accountable When an Agent Reads a Circular?

Accountability under Saudi tax and labour law does not transfer to software. When ZATCA audits a taxpayer's e-invoicing compliance, the question is whether the taxpayer followed the applicable rules in force at the time each invoice was issued. The agent's interpretation of a PDF is not a defence; it is not even a relevant fact.

This creates a specific audit-trail gap. If a compliance team cannot produce a record of which document version governed each decision period, which rules were extracted from that version, and which exceptions were applied to which transaction classes, they cannot demonstrate due diligence. The gap is not merely a data hygiene issue — it is the difference between a correctable filing error and an evidence deficit in front of an auditor.

Governance frameworks for AI systems increasingly recognise this distinction. Structured, multi-level frameworks that decompose AI oversight into discrete layers — from broad regulatory mandates down to auditable, certifiable operations — exist precisely because placing a PDF in an agent's context window and trusting its output does not constitute a governance layer ^[5]. The accountability chain requires traceability from the regulatory source to the specific action taken ^[5]. A raw-document injection approach produces no such chain.

For a more detailed treatment of data governance risk when AI systems handle confidential compliance documents, see When AI Leaks Your Compliance Data: Lessons from the Claude Incident.

A Structured Intake Layer vs. Raw Document Injection — Four Practical Differences

The architectural alternative to feeding a PDF directly to an agent is a structured intake layer that resolves the document into discrete, versioned obligations before any agent interaction occurs. The distinction operates at four levels:

1. Extraction and atomisation. A structured layer parses the regulatory document and produces numbered, discrete obligations — each a single, testable rule with a defined scope, threshold, and effective date. The agent receives these as structured data, not as prose it must interpret. This eliminates the long-context degradation problem that the HANDBOOK.md benchmark documented ^[1], because the agent is never asked to hold 60 pages in working memory.

2. Version control. When ZATCA publishes an updated circular, the structured layer ingests the new version, diffs it against the prior one, and flags which obligations changed. The agent operating in the current period works from current rules. Historical records preserve which rule set governed each prior period, satisfying the audit-recall requirement without manual reconstruction.

3. Conflict resolution before agent interaction. Contradictions between cross-referenced documents — a frequent feature of incremental regulatory publishing — are resolved at the extraction stage, not delegated to the agent's in-context reasoning. The agent receives a clean obligation set. It is not asked to arbitrate between a 2023 annex and a 2025 amendment.

4. Auditable log. Every decision the agent makes can be traced to a specific rule, a specific document version, and a specific timestamp. The audit trail is an output of the architecture, not a retrospective reconstruction. When an auditor asks "which rules were in force on 15 March?" the answer is a database query, not a review of chat logs.

This is the same principle that governs how tracking ZATCA notifications systematically should work in any professional accounting firm — see How Accounting Firms Should Track ZATCA Notifications Systematically for the operational detail.

AI governance architecture theory confirms that bridging the gap between abstract regulatory mandates and on-the-ground compliance requires exactly this kind of layered decomposition ^[3]. Governance is not a property of the agent; it is a property of what the agent is given to work with ^[3].

MAKYN's View: Regulatory Intelligence Must Be Resolved Before It Reaches Any Agent

The HANDBOOK.md finding — that the best available models fail policy-governed tasks more than 60% of the time under strict conditions ^[1] — is not an argument against AI agents in compliance workflows. It is an argument for being precise about what those agents should and should not be asked to do.

An AI agent is well-suited to structured classification, exception routing, and status tracking once the rules it applies are unambiguous and current. It is poorly suited to the task of reading a 60-page Arabic PDF and deriving a consistent, complete, auditable rule set from it. Asking an agent to do the second thing is not an efficiency gain; it is a transfer of interpretive risk from a structured process to a probabilistic one, without any corresponding transfer of legal accountability.

MAKYN's position is that regulatory intelligence — the work of reading ZATCA circulars, GOSI bulletins, and Ministry of Commerce guidance — must be resolved into machine-readable, versioned obligations at the source. That resolution is an infrastructure problem, not a prompting problem. The agent that subsequently acts on those obligations is operating on a governed input, not making governance decisions itself.

For Saudi accounting firms and corporate compliance teams evaluating what a production-ready implementation looks like, the practical starting point is not a better prompt template. It is the intake layer — the system that tracks, versions, and structures every regulatory notification before any downstream process touches it.

اطلب عرضاً توضيحياً لمعرفة كيف تحوّل منصة MAKYN الإشعارات التنظيمية إلى التزامات قابلة للقراءة الآلية وقابلة للتدقيق، قبل أن تصل إلى أي وكيل أو مهندس امتثال.

For the broader question of what Saudi compliance software should demonstrate before a firm commits to it, see Evaluating Saudi Compliance Management Software: A Buying Framework.