From 5fe3ae71a10caffcf64c431771a5f3a8b445395c Mon Sep 17 00:00:00 2001 From: Jakob Wennberg Date: Thu, 27 Aug 2026 18:00:27 +0200 Subject: [PATCH] feat(mcp): surface documents that are attached to nothing on the attention resource (#1979) A document_attachments row is reachable from eight places. A row referenced by none of them is stored, retained for seven years under BFL, and connected to no bookkeeping at all. Nothing surfaced those, so they accumulated: 4 497 across 210 companies, 481 of them in the preceding week. The naive predicate is a trap. Without a mime filter the same query returns 15 806 rows, and 11 309 of those are archived PSD2 bank-API responses that are unlinked by design. Putting them on an orientation surface would hand an agent eleven thousand items of work it must not do, which is worse than showing nothing. So the rule is an allow-list of the mime types an underlag can actually be. Measured on production: application/json was 11 309 of 11 309 PSD2 archive, and pdf/png/jpeg/heic were 0 of 4 495. The split is clean, and an allow-list keeps the next machine-payload format out by default rather than after someone notices it leaking. Two passes, mirroring fetchPurchasesWithoutUnderlag: the indexed column filter first, then eight reference lookups that run only when candidates exist, so a company with none costs exactly one query. The scan cap is set by URL length rather than table size, because every candidate id is echoed back through those eight .in() lookups. Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) --- DECISIONS.md | 2 + .../mcp-server/__tests__/attention.test.ts | 48 +++++- .../general/mcp-server/resources/attention.ts | 39 ++++- .../__tests__/unlinked-documents.test.ts | 113 +++++++++++++ lib/documents/unlinked-documents.ts | 158 ++++++++++++++++++ 5 files changed, 358 insertions(+), 2 deletions(-) create mode 100644 lib/documents/__tests__/unlinked-documents.test.ts create mode 100644 lib/documents/unlinked-documents.ts diff --git a/DECISIONS.md b/DECISIONS.md index 41e738d6..60bd8307 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1295,5 +1295,7 @@ One line per decision: `[YYYY-MM-DD] : `. Appended by agents and [2026-08-27] gnubok_call_tool is implemented as a REWRITE in the tools/call dispatcher, not as a tool whose execute() forwards to another tool. Server-side every tool was always callable (the dispatcher resolves the name against the whole `tools` array; isDefaultCatalogTool gates only what tools/list SHOWS), so the failure was purely client-side and is what forced gnubok_reconcile_match back into the default catalog on 2026-08-26 ("a search-only tool is uncallable on Claude.ai"). A forwarding wrapper would have run the inner tool's execute() directly and thereby skipped the scope check, the unknown-argument guard, company routing + membership check, the test-key write block, the staging _meta and telemetry, all of which live between resolution and execute. Rewriting {tool, arguments} into a direct call BEFORE resolution makes every one of them apply to the real target for free; the wrapper's own execute() throws, and a test asserts that, so the rewrite cannot be silently removed. Restricted to annotations.readOnlyHint === true: a write must be named directly so the client sees its own annotations and its approval contract rather than a generic wrapper's (this refuses 13 of the 18 search-only tools and unlocks the 5 read ones). Doubly closed to anonymous callers: the pre-auth gate keys on the OUTER name and gnubok_call_tool is deliberately absent from PUBLIC_TOOLS, and the in-dispatcher re-check then keys on the INNER name; nothing is lost because all three public tools are in the default catalog. Telemetry gained errorKind 'bridge_refused' in BOTH lib/events/types.ts and the server's local payload type: widening only the local one type-checks under vitest (which does not typecheck) and fails `npm run build`, which is how it was caught. [2026-08-27] gnubok_get_agent_briefing's outputSchema condensed 7 743 -> 4 565 chars by replacing four sub-schema interiors (ledger_context, dimensions, skatteverket_connection, recommended_tools) with a permissive `{type:'object'}` plus a fuller description, keeping the property declared so the top-level `additionalProperties: false` still holds. Deleting the properties was not an option for that reason, and adding `additionalProperties:false` to the condensed forms would have rejected the real payload. Safe because agent-briefing.test.ts pins the RUNTIME shape of all four blocks, so the contract stays guarded while the schema stops carrying 3 178 chars of documentation into every tools/list. Kept intact: `company` (its accounting_method prose drives the settlement posting), `atoms` and `memory` (they tell the agent to fetch bodies via gnubok_load_skill). Net with the new tool: guarded (accounted) projection 63 491 -> 62 942 tokens, and the payload ceiling TIGHTENS 63.6K -> 63.1K, the first downward move in that ledger. [2026-08-27] Cockpit auto-landing gated to byrå owner/admin (isCockpitLandingRole; landing route + '/' bounce), superseding the 2026-08-05 all-members widening: plain members land like regular users and open the cockpit from the nav; the middleware zero-company steer stays ungated because a member with no client companies has nowhere else to land. Allowlist over role!=='member' so future roles default to the regular landing. +[2026-08-27] New `unlinked_documents` category on the Accounted://attention resource, backed by lib/documents/unlinked-documents.ts. The whole design is the mime ALLOW-LIST, and the naive predicate is a trap: "current version, no journal_entry_id, referenced by none of the eight linking tables" returns 15 806 rows on prod, of which 11 309 are application/json and every single one is named psd2-response__pN.json, the archived PSD2 bank-API responses the integration stores as evidence of each fetch. Those are unlinked BY DESIGN; surfacing them would hand an agent 11 309 items of work it must not action, which is worse than showing nothing. Measured 2026-08-27: application/json was 11 309 of 11 309 psd2, and pdf/png/jpeg/heic were 0 of 4 495, so the split is clean. Chose an allow-list of underlag-shaped mime types over excluding known-bad filenames, so a future machine-payload format (XML, CSV, an audit bundle) stays out by default instead of leaking until someone notices. Real remaining surface: 4 497 documents across 210 companies, median 3 per company, 481 in the preceding week, and NOT agent-specific (2 374 upload_source=api vs 1 623 file_upload from the web UI). Two-pass fetch mirroring fetchPurchasesWithoutUnderlag: indexed column filter, then eight reference lookups that run only when candidates exist, so the common case costs one query. Scan cap is 300 and is set by URL LENGTH, not table size: each candidate id is echoed through eight .in(column, ids) lookups at ~38 bytes per UUID, and a cap in the thousands would exceed the gateway limit, fail the lookups, and the "claims nothing" fallback would turn every candidate into a false positive. A failing lookup is deliberately treated as "claims nothing" (can only ADD a row) rather than dropping the category, so one misbehaving table cannot hide real work. UnlinkedDocument is a type alias not an interface: the resource assigns it into samples: Record[] and an interface has no implicit index signature; vitest does not typecheck so this only fails in npm run build. +[2026-08-27] NOT fixed, and recorded so the next person does not act on an inflated number: the agent-facing readers (resources/attention.ts, resources/recent-activity.ts) still test booked-ness with a raw journal_entry_id null check instead of the canonical isTransactionBooked, which misses the bulk-book (transaction_voucher_links) and multi-allocation (invoice_payments / supplier_invoice_payments) cases. Real scale measured on prod 2026-08-27: 4 transactions, in 1 company, out of 567 column-filtered unbooked, all 4 via transaction_voucher_links and 0 via either payments table. Worth fixing as hygiene, but it is a 4-row problem and doing it properly in attention.ts needs the same two-pass treatment plus a decision about count semantics for a tenant with thousands of unbooked rows, so it does not belong bolted onto this change. [2026-08-27] Klarmarkera (markPeriodClosedExternally) gets an undo, reopenExternallyClosedPeriod, allowed only while the closed state still comes from klarmarkera (closed_externally set, no closing entry): that close was a person's control decision without a bokslutsverifikat, so reversing it strands nothing, whereas a closePeriod close keeps its closing entry and stays irreversible here. The reopen clears the lock too, because the reason to reopen is to change the period's contents (Forsslund Systems 2026-08-27: five imported years klarmarkerade, then the prior-year SIE turned out wrong; replace refused the closed year, unlock refused the closed state, no way back). Audit_log row plus period.unlocked event; the MCP staged-op surface (lock/unlock) does not get a reopen op yet, follow-up. [2026-08-27] Added `npm run check:types`, a typecheck ratchet (scripts/checks/no-new-type-errors.mjs + typecheck-baseline.json), wired into the core-build `checks` job next to check:lint. Reason: `npm test` does NOT typecheck. Vitest transpiles and discards types, so a type error passes all 18 000 tests and only surfaces in `npm run build` minutes later; that happened TWICE on 2026-08-27 (a widened errorKind union in the MCP server that lib/events/types.ts still contradicted, and an `interface` that would not assign into `Record[]` because interfaces have no implicit index signature). It is not merely a faster copy of the build job: `tsc --noEmit` also covers `__tests__` files, which the Next.js build never compiles, and that is where all 539 baseline errors live. Baseline is keyed per FILE, deliberately unlike the per-RULE lint ratchet: the legacy errors are concentrated in a handful of old test files and TS2322 is common enough that a code-keyed budget would silently absorb a real regression somewhere else, whereas per-file trips the moment a previously-clean file gains an error. Verified the gate actually fires by introducing a deliberate `const x: number = 'str'` and watching it fail with the exact location, then restoring. Cost measured: 36 s cold (what CI pays, since tsconfig.tsbuildinfo is gitignored) and 4.4 s warm locally via the existing `incremental: true`. The script sets NODE_OPTIONS=--max-old-space-size=8192 because a bare tsc dies with "Ineffective mark-compacts near heap limit" on this graph after about two minutes, which reads like a hang rather than a misconfiguration; it also detects that OOM string and exits 2 with a "raise HEAP_MB" message rather than silently reporting zero errors. NOT changed: Definition of Done item 1 still says only lint + test. CI enforcement is the stronger mechanism and does not need the policy edit; adding it to DoD is a founder call. diff --git a/extensions/general/mcp-server/__tests__/attention.test.ts b/extensions/general/mcp-server/__tests__/attention.test.ts index c2dbde97..1a85bf1a 100644 --- a/extensions/general/mcp-server/__tests__/attention.test.ts +++ b/extensions/general/mcp-server/__tests__/attention.test.ts @@ -22,7 +22,7 @@ const ctx = (supabase: ReturnType['supabase']) }) /** - * Enqueues 14 baseline empty results in the order the resource consumes them. + * Enqueues 15 baseline empty results in the order the resource consumes them. * Tests can override individual slots before invoking by enqueueing in advance. */ function enqueueEmpty(enqueue: (r: { data?: unknown; error?: unknown; count?: number | null }) => void) { @@ -54,6 +54,10 @@ function enqueueEmpty(enqueue: (r: { data?: unknown; error?: unknown; count?: nu enqueue({ data: null }) // 14. companySettingsRow enqueue({ data: null }) + // 15. unlinked-document candidates. fetchUnlinkedDocuments returns early on + // an empty candidate set, so it consumes exactly this one slot; with + // candidates it consumes eight more, one per referencing table. + enqueue({ data: [] }) } describe('Accounted://attention', () => { @@ -188,6 +192,47 @@ describe('Accounted://attention', () => { expect(result.summary.critical).toBe(1) }) + it('surfaces documents that nothing references, pointing at the link tool', async () => { + const { supabase, enqueue } = createQueuedMockSupabase() + // Slots 1-14 empty, then this test's own candidate set in slot 15 and the + // eight reference lookups behind it. + for (let i = 0; i < 14; i += 1) enqueue({ data: i === 12 || i === 13 ? null : [], count: 0 }) + enqueue({ + data: [ + { id: 'doc-new', file_name: 'kvitto.pdf', mime_type: 'application/pdf', file_size_bytes: 900, upload_source: 'api', created_at: '2026-08-20T00:00:00.000Z' }, + { id: 'doc-old', file_name: 'faktura.pdf', mime_type: 'application/pdf', file_size_bytes: 900, upload_source: 'file_upload', created_at: '2026-08-01T00:00:00.000Z' }, + ], + }) + for (let i = 0; i < 8; i += 1) enqueue({ data: [] }) + + const result = (await attentionResource.read(ctx(supabase))) as AttentionResponse + const cat = result.categories.find((c) => c.key === 'unlinked_documents') + + expect(cat).toBeDefined() + expect(cat?.count).toBe(2) + expect(cat?.severity).toBe('warning') + expect(cat?.next?.tool).toBe('gnubok_link_document_to_voucher') + // Oldest first, matching every other category's "start with the stalest" hint. + expect(cat?.next?.args).toEqual({ document_id: 'doc-old' }) + }) + + it('omits the unlinked-documents category when every candidate is claimed', async () => { + const { supabase, enqueue } = createQueuedMockSupabase() + for (let i = 0; i < 14; i += 1) enqueue({ data: i === 12 || i === 13 ? null : [], count: 0 }) + enqueue({ + data: [ + { id: 'doc-1', file_name: 'kvitto.pdf', mime_type: 'application/pdf', file_size_bytes: 900, upload_source: 'api', created_at: '2026-08-20T00:00:00.000Z' }, + ], + }) + // The first referencing table claims it; the remaining seven return empty. + enqueue({ data: [{ document_id: 'doc-1' }] }) + for (let i = 0; i < 7; i += 1) enqueue({ data: [] }) + + const result = (await attentionResource.read(ctx(supabase))) as AttentionResponse + + expect(result.categories.find((c) => c.key === 'unlinked_documents')).toBeUndefined() + }) + it('flags voucher gaps as critical and includes next tool args', async () => { const { supabase, enqueue } = createQueuedMockSupabase() const seriesRows = [{ voucher_series: 'A', fiscal_period_id: 'fp-1' }] @@ -206,6 +251,7 @@ describe('Accounted://attention', () => { enqueue({ data: [] }) // bankConnRows enqueue({ data: null }) // activePeriodRow enqueue({ data: null }) // companySettingsRow + enqueue({ data: [] }) // unlinked-document candidates // Loop body for series 'A': enqueue({ data: [{ gap_start: 5, gap_end: 7 }] }) // detect_voucher_gaps RPC enqueue({ data: [] }) // voucher_gap_explanations follow-up diff --git a/extensions/general/mcp-server/resources/attention.ts b/extensions/general/mcp-server/resources/attention.ts index db9fc980..a8ff1ab8 100644 --- a/extensions/general/mcp-server/resources/attention.ts +++ b/extensions/general/mcp-server/resources/attention.ts @@ -1,5 +1,9 @@ import type { McpResource } from './types' import { ACTION_NEEDED_THRESHOLD_DAYS } from '@/lib/deadlines/status-engine' +import { + fetchUnlinkedDocuments, + UNLINKED_DOCUMENT_SCAN_CAP, +} from '@/lib/documents/unlinked-documents' import { countReconciliationDue } from '@/lib/worklist/categories' type Severity = 'critical' | 'warning' | 'info' @@ -29,7 +33,7 @@ export const attentionResource: McpResource = { uri: 'Accounted://attention', name: 'What Needs Attention', description: - 'One-shot summary of outstanding work for the active company: unbooked transactions, overdue invoices, pending approvals, voucher gaps, upcoming deadlines, bank consent expiry, and period-lock alerts. Each category includes a count, up to 5 sample rows, and a suggested next tool call. Use this at session start to orient before chaining read tools.', + 'One-shot summary of outstanding work for the active company: unbooked transactions, overdue invoices, pending approvals, documents linked to no verifikat, voucher gaps, upcoming deadlines, bank consent expiry, and period-lock alerts. Each category includes a count, up to 5 sample rows, and a suggested next tool call. Use this at session start to orient before chaining read tools.', mimeType: 'application/json', read: async ({ supabase, companyId }) => { const now = new Date() @@ -52,6 +56,7 @@ export const attentionResource: McpResource = { bankConnRows, activePeriodRow, companySettingsRow, + unlinkedDocuments, ] = await Promise.all([ supabase .from('transactions') @@ -143,6 +148,7 @@ export const attentionResource: McpResource = { .select('bookkeeping_locked_through, auto_lock_period_days') .eq('company_id', companyId) .maybeSingle(), + fetchUnlinkedDocuments(supabase, companyId), ]) const categories: AttentionCategory[] = [] @@ -242,6 +248,37 @@ export const attentionResource: McpResource = { }) } + // ── Documents attached to nothing ────────────────────────────── + // + // Underlag-shaped files only: the same query without a mime allow-list + // returns 11 309 archived PSD2 bank-API responses on production, which are + // unlinked by design and must never be presented as work. See + // lib/documents/unlinked-documents.ts. + if (unlinkedDocuments.count > 0) { + const oldest = unlinkedDocuments.documents[unlinkedDocuments.documents.length - 1] + categories.push({ + key: 'unlinked_documents', + label_sv: 'Dokument utan koppling till verifikat eller transaktion', + severity: 'warning', + count: unlinkedDocuments.count, + samples: unlinkedDocuments.documents.slice(0, SAMPLE_LIMIT), + next: { + // Two legitimate destinations, and the tool pointer can only name + // one. A document that arrived after the fact is linked to the + // posted verifikat; one whose affärshändelse was never booked + // belongs to a new verifikat as its underlag (BFL 5 kap. 6 §), which + // is what gnubok_link_document_to_voucher's own description says to + // prefer. The prose carries the choice, the pointer carries the + // common case, and journal_entry_id is the agent's to resolve. + description: unlinkedDocuments.capped + ? `Koppla dokumentet till rätt verifikat, eller bokför affärshändelsen med dokumentet som underlag om den inte är bokförd än. Minst ${unlinkedDocuments.count} dokument saknar koppling (avsökningen stannade vid ${UNLINKED_DOCUMENT_SCAN_CAP} kandidater).` + : 'Koppla dokumentet till rätt verifikat, eller bokför affärshändelsen med dokumentet som underlag om den inte är bokförd än.', + tool: 'gnubok_link_document_to_voucher', + args: oldest ? { document_id: oldest.id } : undefined, + }, + }) + } + // ── Voucher gaps without explanations ────────────────────────── const seriesRows = (voucherSeriesRows.data ?? []) as Array<{ voucher_series: string; fiscal_period_id: string }> const allGaps: Array<{ series: string; gap_start: number; gap_end: number; fiscal_period_id: string }> = [] diff --git a/lib/documents/__tests__/unlinked-documents.test.ts b/lib/documents/__tests__/unlinked-documents.test.ts new file mode 100644 index 00000000..5d0e7dbe --- /dev/null +++ b/lib/documents/__tests__/unlinked-documents.test.ts @@ -0,0 +1,113 @@ +/** + * The unlinked-document predicate, and above all the mime allow-list, which is + * the part that decides whether this surface is useful or actively harmful. + */ +import { describe, it, expect } from 'vitest' +import { createQueuedMockSupabase } from '@/tests/helpers' +import { + fetchUnlinkedDocuments, + UNDERLAG_MIME_TYPES, + UNLINKED_DOCUMENT_SCAN_CAP, +} from '../unlinked-documents' + +type Enqueue = (r: { data?: unknown; error?: unknown; count?: number | null }) => void + +/** One empty result per referencing table, so nothing claims any candidate. */ +function enqueueNoReferences(enqueue: Enqueue) { + for (let i = 0; i < 8; i += 1) enqueue({ data: [] }) +} + +const doc = (id: string, overrides: Record = {}) => ({ + id, + file_name: `${id}.pdf`, + mime_type: 'application/pdf', + file_size_bytes: 1024, + upload_source: 'api', + created_at: '2026-08-01T00:00:00.000Z', + ...overrides, +}) + +describe('UNDERLAG_MIME_TYPES', () => { + it('excludes application/json, which on this surface is only the PSD2 archive', () => { + // Measured on production 2026-08-27: of the document rows referenced by + // nothing, application/json was 11 309 of 11 309 archived PSD2 bank-API + // responses, and 0 of 4 495 pdf/png/jpeg/heic rows were. Those archives are + // unlinked BY DESIGN. Admitting them here would hand an agent 11 309 items + // of work it must not do, which is worse than showing nothing at all. + expect(UNDERLAG_MIME_TYPES).not.toContain('application/json') + }) + + it('is an allow-list, so a future machine payload format stays out by default', () => { + // The alternative, excluding known-bad filenames, leaks every new archive + // format until someone notices and adds another exclusion. + for (const mime of UNDERLAG_MIME_TYPES) { + expect(mime === 'application/pdf' || mime.startsWith('image/')).toBe(true) + } + }) +}) + +describe('fetchUnlinkedDocuments', () => { + it('costs exactly one query when the company has no candidates', async () => { + const { supabase, enqueue } = createQueuedMockSupabase() + enqueue({ data: [] }) + // Deliberately NOT enqueueing the eight reference lookups: if the early + // return regressed, the mock would starve and this test would fail. + + const result = await fetchUnlinkedDocuments(supabase as never, 'company-1') + + expect(result).toEqual({ documents: [], count: 0, capped: false }) + }) + + it('returns candidates that no table references', async () => { + const { supabase, enqueue } = createQueuedMockSupabase() + enqueue({ data: [doc('d-1'), doc('d-2')] }) + enqueueNoReferences(enqueue) + + const result = await fetchUnlinkedDocuments(supabase as never, 'company-1') + + expect(result.count).toBe(2) + expect(result.documents.map((d) => d.id)).toEqual(['d-1', 'd-2']) + expect(result.capped).toBe(false) + }) + + it('drops a candidate claimed by any one of the eight referencing tables', async () => { + // Walk the tables one at a time: a document claimed only by the Nth table + // must still be excluded, which is what catches a missing entry in the list. + for (let table = 0; table < 8; table += 1) { + const { supabase, enqueue } = createQueuedMockSupabase() + enqueue({ data: [doc('d-claimed'), doc('d-free')] }) + for (let i = 0; i < 8; i += 1) { + // Every referencing query selects its own column name, and the + // implementation reads whichever key the row carries. + enqueue({ data: i === table ? [{ document_id: 'd-claimed', document_attachment_id: 'd-claimed', xml_document_id: 'd-claimed', dokument_id: 'd-claimed', file_document_id: 'd-claimed' }] : [] }) + } + + const result = await fetchUnlinkedDocuments(supabase as never, 'company-1') + + expect(result.documents.map((d) => d.id), `table index ${table}`).toEqual(['d-free']) + } + }) + + it('reports capped when the candidate scan hits the limit', async () => { + const { supabase, enqueue } = createQueuedMockSupabase() + const many = Array.from({ length: 3 }, (_, i) => doc(`d-${i}`)) + enqueue({ data: many }) + enqueueNoReferences(enqueue) + + const result = await fetchUnlinkedDocuments(supabase as never, 'company-1', { limit: 3 }) + + // count is a floor, not a total, and the caller says so in its prose. + expect(result.capped).toBe(true) + expect(result.count).toBe(3) + }) + + it('keeps the scan cap small enough that the eight .in() lookups stay valid URLs', () => { + // Each candidate id is echoed back through eight .in(column, ids) queries, + // and a UUID costs ~38 bytes in a PostgREST query string. Past a few + // hundred the URL exceeds what the gateway accepts, the lookups fail, and + // the "claims nothing" fallback turns every candidate into a false + // positive. This is the constraint that sets the cap, not table size. + const bytesPerLookup = UNLINKED_DOCUMENT_SCAN_CAP * 38 + expect(bytesPerLookup).toBeLessThan(16_000) + }) +}) diff --git a/lib/documents/unlinked-documents.ts b/lib/documents/unlinked-documents.ts new file mode 100644 index 00000000..843cbddb --- /dev/null +++ b/lib/documents/unlinked-documents.ts @@ -0,0 +1,158 @@ +/** + * Documents in the archive that are attached to nothing. + * + * A `document_attachments` row is reachable from eight places: a journal + * entry (its own `journal_entry_id`), a bank transaction, a receipt, an + * inbox item, a supplier invoice, an invoice delivery, an inbound Peppol + * document, an årsredovisning submission, and a ROT/RUT payout request. A + * row referenced by none of them is stored, retained for seven years under + * BFL, and connected to no bookkeeping at all. Nothing surfaces those today, + * so they accumulate silently: 4 497 of them across 210 companies as of + * 2026-08-27, 481 of them created in the preceding week. + * + * ## Why the mime allow-list is the whole design + * + * The obvious predicate, "current version with no journal_entry_id and no + * referencing row", returns 15 806 rows on production. 11 309 of those are + * `application/json` and every single one is named `psd2-response__pN.json`: + * the archived bank-API responses the PSD2 integration stores as evidence of + * each fetch. They are SUPPOSED to have no verifikat and no transaction. Put + * them on an attention surface and an agent is handed 11 309 items of busywork + * that it must not action, which is worse than showing nothing. + * + * Measured on production 2026-08-27: of the unreferenced rows, + * `application/json` was 11 309 of 11 309 PSD2 archive, and pdf / png / jpeg / + * heic were 0 of 4 495. The split is clean, so the rule is an allow-list of + * the mime types an underlag can actually be, not a filename exclusion. A new + * machine-payload format (XML, CSV, an audit bundle) stays out by default + * rather than needing another exclusion added after it starts leaking. + */ +import type { SupabaseClient } from '@supabase/supabase-js' + +/** + * Mime types a bookkeeping underlag can plausibly be: something a human could + * open and read as a receipt or an invoice. Everything else on this surface is + * a machine payload that is unlinked by design. + */ +export const UNDERLAG_MIME_TYPES = [ + 'application/pdf', + 'image/png', + 'image/jpeg', + 'image/heic', + 'image/webp', + 'image/tiff', +] as const + +/** + * The candidate scan is bounded, and the bound is set by URL length rather + * than by table size. + * + * Every candidate id is sent back through eight `.in(column, ids)` lookups, + * and a UUID costs ~38 bytes in a PostgREST query string, so 300 ids is + * already a ~12 KB URL per lookup. A cap in the thousands would exceed what + * the gateway accepts and the eight lookups would start failing, which the + * error handling below would read as "nothing claims these" and would turn + * every candidate into a false positive. Small and honest beats large and + * silently wrong. + * + * 300 covers all but one company on production (median 3, worst 1 298 as of + * 2026-08-27). When the cap is hit, `capped` says so and `count` becomes a + * floor rather than a total, because the caller renders a number either way + * and a silently truncated one would read as "nearly done". + */ +export const UNLINKED_DOCUMENT_SCAN_CAP = 300 + +/** + * A type alias, not an interface, on purpose: the attention resource assigns + * these straight into its `samples: Record[]` field, and an + * interface has no implicit index signature so that assignment does not + * type-check. Vitest does not typecheck, so this only ever shows up in + * `npm run build`. + */ +export type UnlinkedDocument = { + id: string + file_name: string + mime_type: string | null + file_size_bytes: number | null + upload_source: string | null + created_at: string +} + +export interface UnlinkedDocumentsResult { + documents: UnlinkedDocument[] + count: number + capped: boolean +} + +const COLUMNS = 'id, file_name, mime_type, file_size_bytes, upload_source, created_at' + +/** + * The eight tables that can claim a document, as [table, column] pairs. + * + * `journal_entry_id` is deliberately absent: it lives on the document row + * itself and is already excluded by the column filter below. + */ +const REFERENCING_COLUMNS: ReadonlyArray = [ + ['transactions', 'document_id'], + ['receipts', 'document_id'], + ['invoice_inbox_items', 'document_id'], + ['supplier_invoices', 'document_id'], + ['invoice_deliveries', 'document_attachment_id'], + ['peppol_inbound_documents', 'xml_document_id'], + ['arsredovisning_submissions', 'dokument_id'], + ['rot_rut_payout_requests', 'file_document_id'], +] + +/** + * Underlag-shaped documents in this company's archive that nothing references. + * + * Two passes, the same shape as `fetchPurchasesWithoutUnderlag`: the column + * filter is the cheap part the database can index, and the eight reference + * lookups settle it afterwards for the candidates that survived. They run only + * when there are candidates, so the common case (a company with none) costs + * exactly one query. + */ +export async function fetchUnlinkedDocuments( + supabase: SupabaseClient, + companyId: string, + options?: { limit?: number }, +): Promise { + const cap = options?.limit ?? UNLINKED_DOCUMENT_SCAN_CAP + + const { data: candidates } = await supabase + .from('document_attachments') + .select(COLUMNS) + .eq('company_id', companyId) + .is('journal_entry_id', null) + .not('is_current_version', 'is', false) + .in('mime_type', UNDERLAG_MIME_TYPES as unknown as string[]) + .order('created_at', { ascending: false }) + .limit(cap) + + const rows = (candidates ?? []) as UnlinkedDocument[] + if (rows.length === 0) return { documents: [], count: 0, capped: false } + + const ids = rows.map((r) => r.id) + const referenced = new Set() + + // Fetched once for the whole candidate set rather than per row. A failing + // lookup is treated as "claims nothing", which can only ever ADD a row to + // the list; the alternative (dropping the whole category on one bad table) + // would hide real work because an unrelated feature's table misbehaved. + await Promise.all( + REFERENCING_COLUMNS.map(async ([table, column]) => { + const { data } = await supabase.from(table).select(column).in(column, ids) + // The column name is resolved at runtime, so the client cannot infer a + // row type here; `unknown` first because the inferred error union does + // not overlap the record shape. + const referencingRows = (data ?? []) as unknown as Array> + for (const row of referencingRows) { + const value = row[column] + if (value) referenced.add(value) + } + }), + ) + + const documents = rows.filter((r) => !referenced.has(r.id)) + return { documents, count: documents.length, capped: rows.length >= cap } +}