A backtest against real bookings (scripts/backtest-categorize.ts, read-only)
showed the selector reporting 0.95 on pure category guesses, so "säker" was a
lie: high-confidence picks were only ~52% accurate.
Confidence is now driven by DETERMINISTIC BACKING — the confidence of a
candidate that independently points at the chosen account — not the model's
verbalized confidence (which the backtest showed is ~always "high"):
- a BACKED pick takes the candidate's confidence, reduced only when the model
itself is unsure;
- an UNBACKED pick (a category guess no candidate agreed with) is capped at 0.7,
below the säker band (0.8) — a guess is never "säker", however sure the model
claims to be.
Re-running the backtest: säker (conf ≥0.8) accuracy 52% → 73%, and it now fires
only on template-backed picks. Still not auto-book-grade (want ~95%), so
auto-book stays off until isotonic calibration on real approvals — but the band
is now honest, which is what makes the whole UX trustworthy.
Also adds the read-only backtest harness so we can re-measure after any change.
37 categorize tests green; lint + guards clean.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The highest-leverage quality lever for real users. A prod read showed the
majority are cold-start (365 companies, 32.7k unbooked transactions, median 0
counterparty templates), so the LLM selector carries them — and it was only
seeing the bank line (merchant + amount), never the receipt.
- lib/agent/categorize/underlag.ts: gathers the matched receipt/invoice text
for a transaction (receipts.matched_transaction_id + invoice_inbox_items
.matched_transaction_id + the transaction's own attached document) and renders
it as bounded Swedish text — supplier, date, total, moms, line items. Same
sources the categorization intent reads, as a string not a tool loop. Core
queries the tables directly (no @/extensions import). Best-effort: '' on any
failure.
- POST /api/agent/categorize gathers it server-side when the caller didn't
supply `underlag`, so the model reasons over the actual supplier + line items.
Server-side only, no client change. 31 categorize tests green; lint + guards +
scoped typecheck clean.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Turns the selector's raw confidence into a score that means what it says.
- lib/agent/categorize/calibration.ts: the engine. Isotonic regression
(pool-adjacent-violators, distribution-free + monotonic) over
(confidence, was_correct) samples → a calibrator; plus reliabilityByBucket,
ECE, and bandFor(). bandFor NEVER returns 'auto' without a fitted calibrator
(no silent booking on an unproven score) and never auto-books above an amount
cap. 12 engine tests (overconfidence pulled down, underconfidence lifted,
monotonicity, ECE, band gating).
- Measurement loop: migration categorize_calibration_samples (append-only,
company-scoped RLS, confidence CHECK [0,1]) + POST /api/agent/categorize/
outcome logging one sample (proposed vs actually booked) fire-and-forget from
QuickReviewDialog on a successful book (sandbox skipped). AiCategorizeProposal
surfaces the proposal metadata via onProposal.
- scripts/fit-categorize-calibration.ts (read-only): prints the reliability
diagram + ECE + fitted calibrator once data has accumulated.
Fitting needs a few hundred real outcomes, so nothing calibrates today — the
loop starts collecting, and "säker" stays uncalibrated (no auto-book) until the
data proves it. 131 unit tests green; RLS covered by a pg-real test.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The auto-booking cascade end to end (retrieval → selector), minus the write.
- lib/agent/categorize/candidates.ts (Tier 1): assembles the deterministic
candidate slate for a transaction — the learned counterparty template
(strongest, carries its own VAT) plus mapping rules / patterns / per-merchant
history via the same engine gnubok_suggest_categories uses. No model call.
Deduped by account (highest confidence wins), capped; suggestions get the
category's default VAT treatment derived.
- POST /api/agent/categorize: loads the transaction + company VAT context,
runs Tier 1 → Tier 2 selectAccount, returns the proposed account + VAT +
confidence + reasoning + the candidate slate. Never posts anything — the
caller renders an approval card. Gated on configured (any provider incl.
local), same gates as /api/agent/ask.
12 tests: candidate merge/dedupe/VAT-derivation, and the route (401/429/400/
403/404/503 + happy path threading entity type, VAT, underlag, samples).
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The core of the "optimal" RIP-4 categorizer, built to the researched 2026
architecture (retrieve → SELECT → escalate). Given a transaction, its underlag,
and the deterministic candidate accounts the engine already retrieved, the
model reasons and then CHOOSES from a closed set:
- a retrieved candidate account (the known path), or
- a standard business category → deterministic BAS account (the novel path,
a first-time vendor with no candidate), or
- needs_review (routed to a human, never auto-applied).
Because it picks from a closed enum, the model can't invent an account; the
account + VAT resolution stays deterministic and validated (the model chooses,
code resolves the numbers). It runs on any backend via getAiService()
.generateStructured — Bedrock or a local model.
Founder chose the optimal path (the model selects on every transaction, LLM
calls are fine), so confidence uses self-consistency: N samples (default 3),
majority vote, agreement fraction, combined with the model's stated confidence
and floored by the winning candidate's deterministic confidence — never the
model's verbalized confidence alone (systematically overconfident). reasoning
precedes choice in the schema (reason-before-choice); an unknown/hallucinated
choice degrades to needs_review.
13 unit tests (candidate/category/needs_review resolution, reverse-charge gating,
self-consistency majority + agreement + candidate floor, prompt/schema shape).
Not yet wired: Tier 1 candidate gathering + a route + the ApprovalCard UI.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Jakob Wennberg