Auto-tagging — evals

~4 min read · model sweep, refusal metrics, and the slices that load-bear the result

This is the only skill where a language model is in the hot path, so the eval surface is the largest of the four. The harness sweeps one prompt across four providers — Anthropic Sonnet 4.5, Anthropic Haiku 4.5, OpenAI GPT-4o-mini, Google Gemini 2.5 Flash — through OpenRouter with one env flip.

Golden corpus

Auto-tagging · 37 cases20 head, 8 tail, 4 prompt-injection, 3 foreign-exchange edge, 3 cross-tenant. 9 cases expect refusal.

Latest run

86.7%top-1 accuracy · Sonnet 4.5

100%refusal precision · Sonnet / Haiku / GPT-4o-mini

33%refusal recall · load-bearing weakness

0.94mean confidence on accepted decisions

The 33% refusal recall is the weakness I would not paper over. Three providers refuse correctly when they do refuse (100% precision), but the model still answers confidently on cases where it should have folded. That is exactly where the deterministic safety net earns its place — schema validation, vendor-rule short-circuit, and the autonomy gate catch what the model misses. Gemini Flash sits at 66.7% top-1 and 25% refusal precision (hallucinated refusals), which is why the production default is Sonnet for reasoning and Haiku for the cheap 80%, not a single cheapest provider. Prompt-injection cases (4 of 37) verify the typed-output boundary holds even when the vendor name itself is hostile.

Each case also reports schema conformance, latency p50/p95, token volume, and cost — the harness is built to compare prompt versions, not to chase a leaderboard. Reproduce with pnpm eval -- --skill auto-tagging; reports write to evals/results/<timestamp>.md and per-case JSONL for diffing prompt versions side-by-side.

Multi-model sweep — the full table

Eval report — 2026-05-13T02:44:23.581Z

auto-tagging

model	prompt	n	top-1 acc	refusal P	refusal R	mean conf	tok in	tok out
anthropic/claude-sonnet-4.5	v1.0	15	86.7%	100.0%	33.3%	0.94	24499	3174
anthropic/claude-haiku-4.5	v1.0	15	80.0%	100.0%	33.3%	0.91	24499	2981
openai/gpt-4o-mini	v1.0	15	80.0%	100.0%	33.3%	0.87	12282	1791
google/gemini-2.5-flash	v1.0	15	66.7%	25.0%	33.3%	0.95	9169	605

How I read the results

• Sonnet 4.5 is the right reasoning default for production in this setup — 7 points of accuracy over the cost-cheaper alternatives.
• Refusal precision is perfect across the Anthropic + OpenAI models: when they refuse, they're right to. Gemini Flash hallucinates refusals (25% precision) and is over-confident on the cases it accepts.
• Refusal recall caps at 33% across all four — the adversarial cases require context the prompt does not surface yet. The next major prompt version should move that lever.
• Token spend on Sonnet is ~2× Gemini for ~20 points of accuracy. Use Haiku for the easy 80%, Sonnet for the hard 20% — Reap's eventual cost model writes itself off this curve.

Slices worth growing next

• Calibration through expected calibration error — harness now reports expected calibration error per model; bar is < 0.03 once buckets have ≥10 cases. Next is growing the golden set to make the per-bucket numbers meaningful.
• Tail-vendor slice — seeded 4 cases tagged tail (Korean printing, unknown Software-as-a-Service, Cebu handicraft, Lalamove courier). Three expect refusal because there is no clean chart-of-accounts hint — the right behaviour for the tail. Bar: ≥80% top-1 on the textbook subset; the refusal cases load-bear refusal recall.
• Cross-tenant adversarial slice — two tenants carry contradicting vendor rules for the same vendor; the slice asserts each tenant resolves to its own rule via the per-tenant vendor-rule short-circuit, with no large-language-model call. The isolation claim is also pinned by a unit test so it is a hard continuous-integration gate, not a passing-eval observation.
• Reversal-rate replay — ✅ shipped as pnpm eval -- --replay-overrides. The demo seed creates two overrides (auto-tagging → sponsorship correction); both are averted on replay via the learned vendor_rules short-circuit. The metric splits averted via learned rule from "the model actually got better" so the two signals aren't conflated.
• Refusal-recall focused set — current 33% is the load-bearing weakness. Build 10–20 adversarial cases targeting the specific reason codes (missing_input, out_of_distribution, ambiguous_match) so each can be regressed independently. The new tail cases already contribute three out_of_distribution cases.

Read next

• Overview — how the skill is wired and why the model earns its place.
• Worked example — the Anthropic row end-to-end.
• Eval harness — the cross-skill harness, coverage manifest, and roadmap (in Operations).