Auto-tagging — worked example

A single live row walked end-to-end, plus the architecture, decision policy, learning loop, design patterns, and production seams behind it. Auto-tagging is the one skill where the large language model is load-bearing, so I made the model small: deterministic shortcuts first, tenant evidence second, typed model call third, refusal and decision policy last.

Core flow

Seven stages, read top to bottom. The first three are deliberately free or near-free; the model only enters at stage five, and only when the cheaper stages cannot answer. Every stage past stage one can short-circuit the rest by returning a decision — the chain stops as soon as someone is confident enough to answer.

1

Reap transaction feedCard authorizations, stablecoin payouts, and bill events enter the same FinanceEvent shape.

↓

2

NormalizeAmounts, currency, merchant identity, merchant category code, tenant, and base currency are made explicit before any model call.

↓

3

Vendor-rule short-circuitA learned per-tenant vendor rule bypasses the large language model and emits a deterministic high-confidence decision.

↓ miss

4

Evidence packTenant chart of accounts, vendor hints, merchant-category-code hints, and historical labels constrain the candidate answer space.

↓ low support

5

Zod-typed model classifierOpenRouter + Vercel Artificial Intelligence Software Development Kit generateObject returns a schema-checked general-ledger code, confidence, evidence, or refusal reason.

↓

6

Decision policyConfidence and chart-of-accounts validity map to autoApply, suggest, or refuse before the autonomy gate runs.

↓

7

Decision log + review loopThe row carries prompt/model/chart-of-accounts versions, evidence, idempotency, override capture, and eval replay hooks.

Stages 1–2 are pure normalisation: take whatever shape Reap's transaction feed delivers and turn it into a FinanceEvent with explicit amount, currency, vendor identity, MCC, tenant, and base currency. Most production bugs in a tagging pipeline live here, in the silent unit-conversion or currency-handling mistakes — making them explicit costs nothing and prevents them everywhere downstream.

Stage 3 is where most of the volume actually exits the pipeline: in steady state, the majority of transactions come from vendors a controller has already coded once. Stage 4 only matters when the model is going to run — it gathers the tenant's chart of accounts, vendor hints, and merchant-category hints into the prompt context. Stage 5 is the only place the model executes, and even there the output shape is locked. Stages 6 and 7 turn the result into an outcome and log it for audit and replay.

Branch outcomes

There are exactly three terminal states for any transaction. The branch chosen depends on what evidence is available at the time the event arrives — the same event can move between branches as the tenant's vendor memory grows.

rule hitFast deterministic pathNo model cost, no model risk, and no prompt dependency.

model pathTyped constrained cognitionThe model can only return the schema; off-chart-of-accounts output is refused.

review pathHuman correction becomes memoryOverride can supersede vendor rules and append a golden eval case.

Rule hits are the steady-state goal: every controller override teaches the system to handle that vendor for free next time, so the population of model-path transactions shrinks as the tenant matures. The model path is where the system earns its keep on first-encounter vendors, but it is also the most expensive and the most risky — which is why every output from it has to clear schema validation and CoA membership before becoming a decision. The review path is the safety valve: when the model is not confident enough to suggest, the row goes to the operator with the candidates and rationale visible, and the operator's correction becomes a rule for the next time the same vendor shows up.

Calibration chart

Confidence bands and what each one looks like in the queue. The widths reflect roughly where each band sits in the demo corpus — vendor-rule hits dominate, off-CoA refusals are rare but always visible.

Vendor ruleauto-apply eligible at high autonomy

High model confidencesuggest by default, auto only after tenant evidence

Medium supportqueue with alternatives and evidence

Off-chart-of-accounts / missingstructured refusal beats a plausible wrong coding

The two middle bands are where most of the operator's time actually goes. "High model confidence" is the band where the suggestion is almost always right but autonomy is held back until the tenant has accumulated enough overrides for the vendor to graduate to the rule path. "Medium support" is the band where the queue shows the suggestion next to the alternatives the model considered — the operator's job is to pick, not to type. The off-CoA band is structurally tiny because the membership check rejects almost all of those cases at the engine boundary; the cases that do show up there are the ones worth investigating.

Decision policy & learning

Confidence is not a single number. The skill combines retrieval support, vendor history, chart-of-accounts validity, schema conformance, and model confidence into a calibrated probability of correctness. The autonomy gate consumes that probability; I do not let the model decide its own authority.

The two thresholds in skill.ts are:

Threshold	Value	Effect
AUTO_APPLY_CONFIDENCE	0.85	At or above, returns autoApply (subject to tenant autonomy rung).
SUGGEST_CONFIDENCE	0.5	Between this and AUTO_APPLY, returns suggest to the operator queue.
Below SUGGEST_CONFIDENCE	—	Returns refuse("insufficient_evidence") — a structured refusal rather than a plausible guess.

Vendor-rule hits bypass these entirely and emit autoApply at 0.99 confidence with modelId: "deterministic/vendor-rule". That hard-coded 0.99 is intentional: a rule that survived an operator override is higher-confidence than any single model call, because it carries social proof from the previous reviewer.

Why 0.99 not 1.0

The reserved 1% acknowledges the rare case where a tenant changes its chart of accounts and an old vendor rule points to a code that no longer exists. The CoA-validity check still runs and will refuse rather than auto-apply.

The model's schema constrains its answers

AutoTagLLMOutput (in schema.ts) is the only shape the LLM can return. The interesting fields:

{
  glCode: string,                    // must exist in the CoA we provided
  confidence: number,                // 0..1, calibrated by the prompt
  rationale: string,                 // max 280 chars, one sentence
  candidatesConsidered: Array<{      // max 3, why-ruled-out per alternative
    glCode: string,
    reason: string
  }>,
  refusalReason: null
    | "insufficient_evidence"
    | "ambiguous_match"
    | "missing_input"
    | "out_of_distribution"
}

Two enforcement layers sit on top of the schema:

1. Schema validation. AI SDK's generateObject retries on schema failure; if every retry fails, the skill returns refuse("model_unsafe") with telemetry that the model couldn't produce a valid object (skill.ts:104-117).
2. Membership check. After the schema passes, the skill verifies the returned glCode is in the set of accounts it offered the model. If not, the decision becomes refuse("out_of_distribution") regardless of how confident the model was. The model cannot invent GL codes (skill.ts:75-86).

The model never decides its own authority

Confidence comes from the model, but the outcome (autoApply / suggest / refuse) is decided by code thresholds. A model that returns confidence 0.99 on an out-of-CoA GL code still gets refused. There is no path by which an over-confident model talks the system into auto-applying a wrong coding.

The four refusal families

Each refusal reason maps to a distinct failure mode. Colour reflects who acts on it: gold rows go to the operator queue, red rows are engine-level safety nets that should rarely fire in steady state.

insufficient_evidenceConfidence below 0.5 — model did not have enough to decide.

ambiguous_matchMultiple plausible accounts; reviewer should pick. The Anthropic live row below.

missing_inputRequired event field absent (e.g. missing vendor identity).

out_of_distribution / model_unsafeModel returned a GL code not in the CoA, or failed schema validation after retries.

The learning loop

Every operator override is a free addition to the deterministic path. The loop is the structural reason the system gets cheaper and safer as it is used — not as a function of better prompts, but as a function of accumulated reviewer judgement.

SUGModel suggestsConfidence in suggest band; row lands in queue with candidates and rationale.

→

OVROperator overridesReviewer picks the correct GL code; selection is recorded with reason text.

→

RULEVendor rule writtenTenant-scoped rule supersedes any prior rule for this vendor; stamped with the override decision id.

→

FREENext charge: no model callSame vendor next time short-circuits stage 5 entirely and emits 0.99 autoApply.

Reviewer effort compounds into future automation coverage. The override loop is the only learning mechanism in the skill — there is no online fine-tuning, no prompt optimisation, no feedback to the model itself. The system gets better because the cache of overrides grows, not because the model does.

Live row walkthrough

A USD 266.03 Anthropic card charge from employee Jia. MCC 5734 (computer software stores) suggests Subscriptions or AI Tools, but the vendor's prior history is sparse and the amount sits in the ambiguous zone where two GL accounts are equally defensible. The skill refuses rather than commit to a guess.

Anthropic card authorization: United States dollar 266.03, event 5e015729-dd15-413c-99ef-bbb1d8802d61.

The /queue surface — auto-tagging decisions land here alongside policy holds and AP recommendations. The "Agent call" column tells the operator the verdict in one glance ("Awaiting you", "Needs approval"); clicking through opens the decision row with the full evidence pack.

Layer	Detail
What the user sees	The row refuses rather than guessing. It says the transaction could map to the vendor's normal software account or a related category, and asks for a receipt or purchase order.
Event row	events.kind = "card_auth", source = "reap-card", vendor Anthropic, merchant category code 5734, employee e-jia.
Decision row	decisions.skill = "auto-tagging", outcome_kind = "refuse", reason code ambiguous_match, chart-of-accounts version 2026-Q2.
Evidence	Vendor memory says 6010, but the decision policy treats the support as insufficient. The demo row also stamps synthetic/demo-seed; the production path would stamp the real model identifier.
Reap implication	Reap can safely say the agent is helping close, not silently posting uncertain software spend into the ledger. A visible refusal is better than a later reversal.

The "What the user sees" line is the load-bearing one. In a less careful tagging product this row would have silently posted to 6010 based on the weak vendor-memory signal, the controller would have noticed at month-end (or not), and an incorrect P&L line would have been the result. Here the row goes to the queue with both candidate codes (6010 Subscriptions, 6015 AI Tools) and a one-line rationale; the operator picks one, the choice writes a vendor rule, and the next Anthropic charge from this tenant skips the model entirely.

A visible refusal beats a plausible wrong coding

Silent miscoding is the worst outcome in this domain — it corrupts the P&L and VAT/GST filings and is expensive to unwind months later. Every refusal in this skill carries a structured reason code so it's eval-visible and operator-actionable.

The production version of this exact path would replace the synthetic demo decision with a schema-checked model call, then keep the same decision log, autonomy gate, and override flow.

What the telemetry row looks like

Beside the decision payload, the skill writes a telemetry record with the inputs needed for replay:

{
  modelId: "anthropic/claude-sonnet-4-5",   // or "deterministic/vendor-rule" on a hit
  promptVersion: "auto-tagging-v1.2",       // bumped on prompt changes
  coaVersion: "2026-Q2",                    // chart-of-accounts pinned at evaluation
  tokensIn: 1284, tokensOut: 86,
  costMicrocents: 412,
  rationale: "Anthropic is a software vendor; could be Subscriptions (6010) or AI Tools (6015).",
  evidence: [
    { kind: "vendor_typical_gl", glCode: "6010" },
    { kind: "candidate", glCode: "6010", reason: "Vendor historically coded here." },
    { kind: "candidate", glCode: "6015", reason: "Some tenants split AI tooling." }
  ]
}

The coaVersion is what makes replay reproducible — re-running this event against today's chart of accounts uses today's CoA, but the original decision shows which CoA judged it. Prompt and model identifiers are stamped for the same reason: a regression report can group decisions by (promptVersion, modelId, coaVersion) and isolate which axis moved.

Production seams

The MVP ships with seams chosen so that the production version requires no architectural change — only the swap of demo stamps for real ones and the wiring of override capture into the queue. The four cards below are the seams I deliberately kept clean.

providerOpenRouter + Vercel Artificial Intelligence Software Development KitOne environment flip swaps providers; Zod schema is the output contract.

versionsPrompt / model / chart-of-accounts stampsEvery decision can be replayed against the exact context that produced it.

costVendor shortcut coverageWidening deterministic coverage is the main way the product gets cheaper and safer.

safetyReason-coded refusalmissing_input, out_of_distribution, ambiguous_match, and model_unsafe are eval-visible.

The provider seam matters because LLM pricing and capability change every quarter; tying the system to a specific model is technical debt the day after a better one ships. The version stamps seam is what makes regression analysis possible — without (promptVersion, modelId, coaVersion) on every decision, "did accuracy drop after we changed prompts?" is unanswerable. The cost seam reflects how the unit economics improve: cost per transaction trends toward zero as the vendor-rule cache grows, not as a function of better prompts. The safety seam keeps refusals in the eval surface: each reason code maps to a class of failure that has its own golden cases, so a regression in any refusal family shows up as a failing test rather than as a wrong coding posted to the ledger.

Design patterns

Those defensibility properties did not appear by accident — each one is a software-engineering pattern doing visible work in the codebase. Naming the patterns makes it easier to compare this skill against the other three (policy enforcement and the daily pulse instantiate some of the same ones).

Chain of ResponsibilityEach pipeline stage (vendor rule → evidence pack → model classifier → decision policy) either handles the transaction or passes it to the next stage. No stage second-guesses a prior handler's decision.Separation of concerns across the decision chain.

StrategyThe classifier strategy is selected at runtime: deterministic vendor-rule path when evidence exists, typed LLM call when it does not. Both strategies emit the same output contract.Provider and rule swaps without touching downstream logic.

Cache-aside / MemoizationThe vendor-rule short-circuit acts as a memoized result: once an override is written, the model is never called again for that vendor-tenant pair.Cost and latency drop as the rule cache grows.

Event SourcingFinanceEvents are immutable facts; decisions are derived state written to a separate append-only log. Replaying events against any prompt or chart-of-accounts version is a first-class operation.Audit, rollback, and eval replay all read the same log.

Feedback loopOperator overrides write corrected codings back into vendor rules and the eval golden set. The system's accuracy improves as a side effect of normal review work.Reviewer effort compounds into future automation coverage.

Four of those five patterns (Chain of Responsibility, Strategy, Cache-aside, Event Sourcing) are generic — they would apply to any pipeline that mixes deterministic and learned components. The fifth, Feedback loop, is the one that matters most for finance: it is the structural reason the system gets cheaper and safer as it is used, instead of accumulating prompt-engineering debt.