Operating in production

~5 min read · SLOs, observability, what would page someone

TAKE-HOME BRIEF

The production-ready axis specifically names "how would you ship and operate this — observability, SLOs, on-call shape, rollback." This page consolidates the operating answer. The MVP runs locally; this is what would change when it goes to a tenant.

Trust in a finance agent is not earned by the happy path. It is earned by what happens when the schema fails, the model drifts, the corridor spread blows out, or someone tries to inject "code this to retained earnings" into a vendor name.

Service-level objectives

One row per skill. Each SLO is something a future on-call would actually be paged on.

Auto-taggingp95 latency < 2.5s end-to-end · top-1 accuracy ≥ 85% on the head set · refusal precision ≥ 99% · schema-conformance ≥ 99% on first try · cost per decision < $0.002 amortised across head and tail.SLO

Policy enforcementp95 latency < 50ms (no model in the path) · verdict stability — same event ⇒ same verdict, pinned by hermetic test · zero injection-driven verdict flips, pinned by the prompt-injection eval.SLO

AP agentZero unattended money movement (architectural invariant, not a metric) · dual-control gate fires on every spend ≥ $10k USD-equivalent · scheduler proposes paymentDate ≤ dueAt − 1 banking day on 100% of non-overdue bills.SLO

Daily pulseBrief lands before 09:00 tenant-local on a deterministic schedule · cash-floor breach alerts within one scheduler tick of crossing the threshold · narration never alters a number, pinned by structural test.SLO

The headline production metric across all four is the north-star reversal rate — auto-applied decisions later overridden. Token cost is plotted on the same axis so accuracy improvements that just buy themselves with bigger models are visible.

What gets logged

Every skill writes one row to the decision log. Reading the schema is the fastest way to see what observability comes for free.

Decision row shape — the only thing that needs to be queryable for ops

// src/db/schema.ts — decisions table (abbreviated)
{
  id: string,
  tenantId: string,                  // every query filters on this
  skillId: SkillId,
  eventId: string,                   // the FinanceEvent that triggered the skill
  outcome: "auto_apply" | "suggest" | "refuse" | "escalate",
  payload: jsonb,                    // skill-specific; validated by payloadSchema
  confidence: number | null,         // null on refuse / escalate
  reasonCode: RefusalReasonCode | null,
  rationale: string | null,

  // Versioning — answers "why did Jan differ from Mar?"
  promptVersion: string | null,
  modelId: string | null,
  rulesVersion: string | null,
  coaVersion: string | null,

  // Cost / latency telemetry
  tokensIn: number | null,
  tokensOut: number | null,
  costMicrocents: number | null,
  latencyMs: number,

  // Replay safety
  idempotencyKey: string,            // sha256 of (eventId × version tuple)

  createdAt: timestamp,
  executedAt: timestamp | null,      // null until the autonomy gate + adapter ratify
}

This row is the source of truth for every dashboard, every alert, and every eval replay. Nothing else needs to be wired up to start operating this — every chart is a GROUP BY over decisions:

-- cost-per-decision by skill × model
SELECT skillId, modelId, AVG(costMicrocents)
FROM decisions
GROUP BY skillId, modelId;

-- refusal mix (join to golden set for refusal precision)
SELECT reasonCode, COUNT(*)
FROM decisions
WHERE outcome = 'refuse'
GROUP BY reasonCode;

What would page someone

Five alerts that would actually fire, ranked by severity. Each maps to a query over the decision log plus a runbook entry.

AP money-movement invariant breachAny decision row where skillId = 'ap-agent' and outcome = 'auto_apply' exists. Should be impossible by type signature; if it fires, the type signature is wrong.Sev-1

Schema-conformance regressionAuto-tagging schema OK rate drops below 99% over a rolling 100 calls. Usually means a model deprecation or a prompt edit that didn't go through the eval gate.Sev-2

Refusal-precision dropRefusals re-graded against the must-refuse golden set drop below 99% for any model in the sweep. Means the model is now refusing things it should accept — silent loss of throughput.Sev-2

Cost-per-decision spikeSonnet auto-tagging cost-per-decision > 2× the 30-day rolling mean. Usually a prompt bloat regression; sometimes a provider price change.Sev-3

Cash-floor breachDaily pulse writes a scheduled_pulse decision with cash.breachWithinDays ≤ 3. Page the tenant's controller, not the on-call.Sev-2 (tenant)

One runbook entry

The shape every runbook entry would follow. This is the schema-conformance regression — the one most likely to actually fire.

Alert. auto_tagging.schema_conformance < 0.99 over rolling 100 calls.

First check. Run:

SELECT modelId, COUNT(*)
FROM decisions
WHERE skillId = 'auto-tagging'
  AND reasonCode = 'model_unsafe'
  AND createdAt > now() - interval '1 hour'
GROUP BY modelId;

If one model dominates, you have a provider issue. If it's spread evenly, you have a prompt issue.

Mitigation. Set the env flag AUTO_TAGGING_MODEL to the last-known-good model in the sweep (the eval report's prompt × model matrix tells you which). The skill keeps running; latency may move; refusal rate temporarily rises while you debug.

Resolution. Re-run the eval against the candidate fix:

pnpm eval -- --skill auto-tagging

The gate is schema OK ≥ 99% on the v1.0 golden set plus no regression on refusal precision. PR the prompt-version bump; the version stamp on the decision row makes the rollback boundary obvious.

What would change at tenant 1

These are the deltas from today's MVP to a production deployment of the same four skills. The skill code does not change — these are runtime concerns.

Layer	Today	Tenant 1
Runtime	Synchronous in-process	Inngest activities with retries, replay, scheduled work
Database	SQLite via libsql	Postgres + row-level security on tenantId
Retrieval	Deterministic vendor / MCC lookups	pgvector or sqlite-vec; tenant-partitioned indexes
Secrets	.env.local	Vercel env per tenant, scoped to the function
Observability	Decision log + eval reports	Same decision log + a dashboard over it (Grafana / Vercel observability)
Alerts	None	The five above, wired to PagerDuty / Slack
Cost cap	Soft	Per-tenant monthly cap on tokensIn + tokensOut; circuit-break to refuse on breach
Adapters	Decision payloads only	Xero/QBO/NetSuite for ledger; Reap Pay rails for AP

None of these change the skill interface. That is the architectural payoff of ADR-005 — the runtime can move without rewriting cognition.

Eval harness — what feeds the alerts

The alerts above are only as honest as the harness that gates them. The eval surface is inspectable, not hand-wavy: a real corpus, a real multi-provider sweep, and a markdown report that pastes into a pull-request description.

AccuracyTop-1 + payload matchAuto-tagging classification and deterministic skill payloads are checked against golden expectations.

RefusalPrecision and recallSilent errors are worse than refusal, so must-refuse cases are tracked separately.

OperationsLatency, cost, schemaThe harness reports practical deployment signals, not just model leaderboard scores.

How the harness runs

CASES

Golden corpusJSONL files per skill: 92 runtime cases across four skills, plus 18 boundary cases for the policy compiler and the AP optimizer.

↓

SWEEP

Multi-provider sweepOne prompt version is swept across Anthropic Sonnet 4.5, Anthropic Haiku 4.5, OpenAI GPT-4o-mini, and Google Gemini 2.5 Flash through OpenRouter. One env flip.

↓

SCORE

Multi-dimensionalTop-1 accuracy, refusal precision & recall, mean confidence, expected calibration error, latency p50/p95, schema conformance, token volume, cost.

↓

REPORT

Markdown + JSONLMarkdown report paste-able into a PR description; per-case JSONL for diffing prompt versions side-by-side.

Corpus totals

37auto-tagging

22policy enforcement

18accounts payable

15daily pulse

Plus 18 boundary cases: 10 policy-compiler + 8 AP-optimizer. Per-skill breakdowns: auto-tagging · policy enforcement · accounts payable · daily pulse.

The 18 boundary cases sit beside the runtime harness because they test authoring and optimization seams, not a hot-path skill: 10 natural-language policy-compiler cases (evals/policy-compiler.jsonl) and 8 accounts-payable optimizer cases (evals/ap-optimizer.jsonl). Deterministic skills (policy, accounts-payable scheduler, daily-pulse analyzer) run model-agnostic — the sweep only varies the auto-tagging model.

Reproduce: pnpm eval. Slice with pnpm eval -- --tag injection or --skill daily-pulse. Run the policy compiler corpus with pnpm eval:policy-compiler. Accounts-payable optimizer cases run in Vitest via tests/ap-optimizer-eval.test.ts. Full reports write to evals/results/<timestamp>.md.

What the harness gives me

• A model-selection tool, not a leaderboard: one env flip swaps providers and re-runs the same golden set.
• Per-case JSONL output for diffing prompt versions side-by-side.
• A markdown report you can paste into a pull-request description.

What it does not measure yet

• Confidence drift over time — needs production telemetry; the decision log captures the inputs.
• Integration-backed evidence — OCR extraction, live ledger posts, real payment rails, production cash backtests, role-based-access-control identity, and privacy redaction now have named entries in evals/coverage-gaps.json, but they are intentionally marked specified/blocked until those adapters or datasets exist.

What the harness gained in v1.1

• Slicing — every golden case carries a tags?: string[] field. Run a slice with pnpm eval -- --tag tail (or injection, fx-edge, refuse, …). Tags don't need a registry; the runner just filters.
• Calibration through expected calibration error — computed over non-refuse outcomes, 10-bucket binning, weighted by bucket population. Reported per model alongside the existing four scores, with a collapsible per-bucket breakdown (n, mean confidence, accuracy, gap). Bar: expected calibration error < 0.03 once buckets have ≥10 cases each — small slices will be noisy.
• Latency p50/p95 — per-case wall-clock around skill.run(). Surfaced in the report and stdout. Treat as relative-not-absolute on cold runs; the deterministic skills clock in at <1 ms.
• Cost — provider-reported cost when available, with a client-side per-model price-table fallback otherwise. Deterministic skills cost $0.00 by construction; treat the dollar figure as directional, not invoice-grade.
• Schema conformance — schema OK column tracks the % of LLM calls that produced a Zod-valid object on first try after the Vercel AI SDK's internal retries. Reported only for skills that actually called a model; deterministic skills show —. A model_unsafe refusal is the structured signal the harness keys off.
• Multi-seed determinism — pnpm eval -- --seeds N runs each case N times and reports agree(Nseed) = fraction of cases where every seed produced the same outcome fingerprint. Hosted models drift even at temperature: 0; this surfaces the noise floor. Cost scales N×; the run prints a warning.
• Reversal-rate replay — pnpm eval -- --replay-overrides joins overrides ⋈ decisions ⋈ events and re-runs each captured override through the current prompt × model × chart of accounts × vendor_rules. Reports averted vs still reversed, with averted via learned rule broken out separately so a vendor-rule short-circuit is not conflated with the model getting smarter. This is the only eval that maps directly to the north-star.
• Verbose mode — pnpm eval -- --verbose prints per-case pass/fail with expected vs actual and confidence; useful for diffing prompt regressions before the markdown report writes.
• Escalate-aware scoring — the AP overdue path produces an escalate outcome with a proposed payload; the runner now matches that payload against expected.payload.

Coverage manifest

evals/coverage-gaps.json tracks the 12 eval families still needed before claiming production-grade autonomy. It distinguishes executable repo gates from specified evals blocked on real adapters, telemetry, identity, or consented datasets — so a reviewer can tell "not shipped" from "not yet measurable."

Cross-cutting roadmap

• Prompt-injection suite — vendor names, memo fields, OCR-derived receipts containing "ignore prior instructions" / "code this to retained earnings". Cheap to run on every prompt change; the only eval that grows in importance with autonomy.
• Schema-conformance rate — % of LLM outputs that pass Zod validation on first try, per model. Regression signal for prompt + model changes that does not need labels.
• LLM-as-judge spot check — weekly, ~50 sampled auto-posted decisions graded by a different model family against a rubric. Catches drift without a labeled set.
• Latency / p95 per skill — production SLO, not accuracy.
• Multi-seed determinism check — same case × same model × temperature: 0 × N seeds. Surfaces silent provider drift.

Dangers & tradeoffs

• Golden-set overfitting. 15 cases is a smoke test. Iterate the prompt against it more than a handful of times and you're memorizing. Hold out a frozen slice that only runs before release.
• Refusal as a free lunch. Refusal precision is gameable by never refusing; recall is gameable by always refusing. Always report both, against a fixed must-refuse set.
• LLM-as-judge circularity. If Sonnet writes the answer and Sonnet grades it, the eval measures self-consistency, not correctness. Use a different family as judge, or pin to deterministic ground truth.
• Cost of full sweep on every PR. Four providers × N cases × every prompt iteration gets expensive. Gate the sweep behind a label or run nightly; on PR run cheapest-only.
• Provider non-determinism. Even with temperature: 0, hosted models drift between runs. Average over ≥3 seeds before publishing a number, or accept a ~1–2 point noise floor.
• Adversarial cases that are too easy. One of the refusal cases has a literal ambiguous_match keyword cue — the model isn't reasoning, it's pattern-matching the phrasing. Adversarial cases should be hard because of context, not vocabulary.
• Tenant-data leakage. The moment evals touch real tenant data, the golden set is a privacy surface. Synthesize, or sign a DPA before importing.
• Calibration ≠ accuracy. Measure calibration before raising the auto-post threshold. The 5% you get wrong is more dangerous than the 95% you get right.
• Drift detection lag. Weekly cadence is fine for vendor-pattern drift, too slow for a chart-of-accounts change or a provider silent update. Add a daily 20-case canary that fails loud on regression.
• Auto-post precision is the only metric that matters for trust. Every other number is a means to that end. Resist leaderboard-culture creep around top-1 accuracy.

Read next

• Cross-cutting decisions — the six ADRs that drive these SLOs.
• Security & threat model — what gets attacked and what defends it.
• Per-skill evals: auto-tagging · policy enforcement · accounts payable · daily pulse.