Policy enforcement — evals

~3 min read · two-layer corpus, injection on rule inputs, and the authoring-time compiler

The runtime is deterministic, so the testing surface is two layers, not a model sweep. Vitest unit tests cover the rule engine cell-by-cell; the golden corpus then drives 22 end-to-end runtime cases — 6 allow, 9 flag, 7 deny/refuse, 3 foreign-exchange edges, 3 prompt-injection probes — against the same FinanceEvent → verdict path the operator queue uses. A model-agnostic skill still has to prove it does not silently miscode, which is why injection cases live in the runtime corpus and not just the policy-compiler one.

The natural-language policy-compiler is the separate authoring-time surface: 10 boundary cases at evals/policy-compiler.jsonl test that English policy text compiles to the same typed bundle the engine evaluates at runtime. Compiler refusal — "I cannot encode this clause" — is preferred over a silently-wrong rule row, because a wrong rule row corrupts every later verdict.

Coverage

Policy enforcement · 22 runtime + 10 boundary6 allow, 9 flag, 7 deny / refuse, 3 foreign-exchange edge, 3 injection. 10 natural-language policy-compiler cases as authoring-time boundary.

22runtime golden cases

10policy-compiler boundary cases

3prompt-injection cases on rule inputs

0 modelsin the hot path · model-agnostic

Reproduce with pnpm eval -- --skill policy-enforcement for the runtime corpus and pnpm eval:policy-compiler for the authoring-time compiler. Verdict payloads stamp the policy bundle version, so any case can be replayed against a later bundle to detect drift.

Slices worth growing next

• Verdict accuracy across allow / flag / deny over merchant-category-code, geography, amount-cap, after-hours, and receipt rules.
• Foreign-exchange edge cases: amounts straddling the United States dollar cap after conversion from Hong Kong dollar, Japanese yen, and Vietnamese dong.
• Adversarial: prompt-injection inside vendor.rawName and lineItems[].description. The policy fast-path is deterministic so injection should not change verdicts — this eval guarantees that and will catch the day someone wires the large-language-model ambiguity classifier in.
• Policy-compiler boundary set (evals/policy-compiler.jsonl, 10 cases) covers all seven rule kinds: amount cap, merchant-category-code block, geography block, receipt required, after-hours, vendor blocklist, and structuring. Includes refusal cases for ambiguous prose and injection-shaped text, plus an empty-input error path that must not call the model. Hermetic contract gate: tests/policy-compiler-corpus.test.ts.

Read next

• Overview — the deterministic stance and rule taxonomy.
• Worked example — the Singapore Airlines row, end to end.
• Eval harness — the cross-skill harness (in Operations).