Files
0pt/docs/12-edge-capability-api.md
T
adminandClaude Sonnet 5 2902a26bac docs: full accuracy overhaul, grounded claim-by-claim in real code (#4)
* docs: full accuracy overhaul, grounded claim-by-claim in real code

Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was
audited and rewritten by a multi-pass process: independent rewrite per
cluster, adversarial re-verification of that rewrite against the actual
code, then a final cross-doc consistency pass — each stage re-checking
claims against the repo directly rather than trusting the previous stage.

Two corrections matter most:

1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and
   docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts,
   quality-report-service.ts, src/data-platform/services/catalog-service.ts,
   and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code.
   git log --all across every branch shows zero commits for any of these
   paths, ever. 0pt's data-quality/QA enforcement does not exist in code.
   Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added
   as an explicit target-spec (not-yet-built) replacement.

2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are
   two diverged remotes sharing part of the same history (services/api/
   workers/adaptive_router.py is byte-identical between them). Gitea's main
   independently accumulated a WAF-bypass/proxy-rotation scraper stack in
   three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main
   never had it. Which remote is canonical/deployed is unresolved — flagged
   consistently across the corpus, not decided here.

Also newly and consistently documented: the tenant_id/project_id-read-from-
client-body gap on the P0 lakehouse/rate-policy/source-query routes (no
auth/session system exists anywhere in this repo yet); that every CL0UD/
N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env
reads for any of their *_URL vars); that two P0 mutation routes
(lakehouse/mappings, sources/rate-policies) always fail over HTTP today
because their prerequisite entity can never be created via any route; a
schema bug in sql/004 (FK references to opt_datasets/opt_sources, which
don't exist); and that 0pt.app's live deployment was independently
confirmed (curled this pass) to be running this exact codebase with
DATABASE_URL/REDIS_URL unset in production.

New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript
capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously
undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files) — docs-only change, no source touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes

Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs
land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the
lakehouse mapping routes and added proper status-code mapping to the
document extraction route. This doc was written against the pre-fix
state in the same pass that added it, so it needs the same correction.

Does not claim the lakehouse write path fully works now -- the deeper
dataset-registration gap (no route ever calls DatasetService.registerDataset)
is unchanged and still blocks it end-to-end; only the shared-instance half
of the bug is fixed.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 18:49:01 +02:00

23 KiB
Raw Blame History

12 · Edge-native P0 capability API (Next.js / TypeScript)

This doc did not exist before this pass. It covers the edge-native TypeScript API surface — Next.js Route Handlers under app/api/0pt/*, plus app/api/screen/route.ts and app/api/ai/parse/route.ts — as a contract distinct from docs/04-api.md's FastAPI engine (services/api/orchestrator/main.py, runs on a VPS). Everything here runs on Vercel. The two layers are not the same service, and as of this pass are only partially wired together (docs/04-api.md documents one confirmed case where the edge layer calls an engine route that doesn't exist).

Every route below was read directly from source this pass (app/api/0pt/**/route.ts, app/api/screen/route.ts, app/api/ai/parse/route.ts, and their backing src/services/*.ts / src/types/*.ts). Where a claim below is about runtime behavior rather than just the shape of the code (e.g. "this call always fails"), it's derived by tracing the actual code path, not by running it — flagged as such.

Capability-registry status (verified against packages/capability-registry/src/registry.ts in sax3l/cl0ud)

14 distinct 0pt.* capabilities are registered today. Every single one has surfaces containing only 'api' (plus 'job' for the two heaviest writes), and mcpEnabled/cliEnabled are false on all 14, with no exception. Any doc, comment, or assumption that a siax 0pt ... CLI or an MCP tool for 0pt already exists is describing a target, not current reality.

Capability Surfaces Route(s) below Notes
0pt.query_source api POST /api/0pt/sources/{sourceId}/query riskLevel L2, piiLevel high
0pt.document_ocr api POST/GET /api/0pt/documents/capture, POST/GET /api/0pt/documents/{documentId}/extraction riskLevel L2, piiLevel high
0pt.write_external_record job (not api) none — no HTTP route exists see "Capabilities with no route" below
0pt.lakehouse_mapping_manage api POST/GET /api/0pt/lakehouse/mappings, GET .../{mappingId} riskLevel L2
0pt.lakehouse_write api, job POST /api/0pt/lakehouse/mappings/{mappingId}/writes riskLevel L3
0pt.source_rate_policy_manage api POST/GET /api/0pt/sources/rate-policies, GET .../{id} riskLevel L1
0pt.screen_sanctions api GET /api/screen riskLevel L2
0pt.write_iceberg_table api, job (invoked internally by 0pt.lakehouse_mapping_manage/0pt.lakehouse_write routes via IcebergWriteService) riskLevel L3, piiLevel high
0pt.estimate_compute, 0pt.vehicle_lookup, 0pt.catalog_list, 0pt.source_registry_list, 0pt.order_create, 0pt.order_payment api app/api/estimates, app/api/vehicles, app/api/catalog, app/api/sources, app/api/orders, app/api/payments/* not under app/api/0pt/ — out of this doc's assigned scope, not read in detail this pass

POST /api/ai/parse has no entry at all in the capability registry (grep-verified: no ai/parse or ai_parse or ai.parse string anywhere in registry.ts). It is real, working code, but sits entirely outside the capability model — no declared risk level, no audit requirement, no policy gate.

Routes

0pt.document_ocr — document capture + extraction

POST /api/0pt/documents/capture — app/api/0pt/documents/capture/route.ts → DocumentCaptureService.captureFromMailbox

body: {
  mailbox: { tenant_id, client_id, client_secret, mailbox_address, folder },  -- Microsoft Graph / Entra creds for the mailbox itself
  since_iso?, top?, max_pages?,
  tenant_id, project_id, actor_id, correlation_id                            -- SIAX platform scoping (unrelated to mailbox.tenant_id above)
}
-> 201 { scanned_messages, captured_documents: DocumentRecord[], errors: string[] }
-> 400 { error } on missing/invalid mailbox creds or missing tenant_id/project_id/actor_id

Note the naming collision: body.mailbox.tenant_id is the Microsoft 365 / Entra tenant the mailbox lives in; body.tenant_id is the SIAX platform tenant. Same field name, two unrelated identifiers, both required, both plain strings in the request body.

Polls the mailbox via Microsoft Graph (app-only), downloads PDF attachments, writes bytes to LocalBlobStore (src/adapters/local-blob-store.ts — an explicitly-marked interim local-disk store, because st0re-client.ts only registers artifact metadata today and has no real bytes-upload endpoint yet), registers the artifact with ST0RE (mock client — see "Mock-only platform clients" below), and enqueues an N0D job placeholder for extraction.

GET /api/0pt/documents/capture — same file → DocumentCaptureService.listDocuments

query: ?tenant_id=&project_id=  (both optional filters)
-> 200 DocumentRecord[]

POST /api/0pt/documents/{documentId}/extraction — app/api/0pt/documents/[documentId]/extraction/route.ts → DocumentExtractionService.extractDocument

body: { tenant_id, project_id, actor_id, correlation_id, model_id }
-> 201 DocumentExtractionRecord
-> 501 { error } if Document Intelligence is not configured (missing DOC_INTELLIGENCE_ENDPOINT/DOC_INTELLIGENCE_KEY) — "refusing to fabricate a result"
-> 403 { error } if CL0UD policy denies
-> 409 { error } if CL0UD policy requires approval
-> 404 { error } if the document doesn't exist
-> 400 { error } for any other failure (e.g. missing model_id)

Corrected in fix/0pt-full-bug-hunt (PR #5, merged separately): previously every one of the above collapsed into a blanket 400, which made "not configured" indistinguishable from "bad request" from the caller's side. Now status-mapped, matching the convention already used by app/api/0pt/sources/[sourceId]/query/route.ts. Every field in the response is evidence-based: a field is only populated when Azure Document Intelligence actually returned it, always carrying DI's own confidence; not-found fields are {value: null, confidence: ..., found: false}, never guessed. This is enforced twice (adapter + service layer) per the file's own doc comment.

GET /api/0pt/documents/{documentId}/extraction — same file → DocumentExtractionService.listExtractionsForDocument

-> 200 DocumentExtractionRecord[]
-> 404 { error } on any thrown error (including errors unrelated to "not found" — the route maps every exception here to 404)

This pair actually works end-to-end, unlike the routes below — documentCaptureService and documentExtractionService are both exported as module-level singletons (src/services/document-capture-service.ts / document-extraction-service.ts) specifically so the extraction route can see documents the capture route created in the same process. This is the one place in app/api/0pt/* that got the shared-instance pattern right.

0pt.lakehouse_mapping_manage / 0pt.lakehouse_write — Iceberg write path

POST /api/0pt/lakehouse/mappings — app/api/0pt/lakehouse/mappings/route.ts → IcebergWriteService.mapDatasetToIcebergTable

body: { tenant_id, project_id, dataset_id, dataset_iceberg_mapping_id, namespace, table_name, storage_uri, catalog_ref, table_format?, actor_id, correlation_id }
-> 201 { mapping: DatasetIcebergMapping, iceberg_table_id, policy_decision_id }
-> 400 { error }

GET /api/0pt/lakehouse/mappings — same file → DatasetIcebergMappingService.listMappings

query: ?tenant_id=&project_id=&dataset_id=
-> 200 DatasetIcebergMapping[]

GET /api/0pt/lakehouse/mappings/{mappingId} — separate file, .../[mappingId]/route.ts → DatasetIcebergMappingService.getMapping

-> 200 DatasetIcebergMapping
-> 404 { error }

POST /api/0pt/lakehouse/mappings/{mappingId}/writes — separate file, .../[mappingId]/writes/route.ts → IcebergWriteService.requestIcebergWrite

body: { actor_id, correlation_id, snapshot_type?, schema_version, manifest_uri, record_count?, size_bytes?, heavy? }
-> 202 { snapshot_id, iceberg_table_id, policy_decision_id, n0d_job_id? }
-> 400 { error }

Confirmed this pass — the POST above still always fails, though one of its two root causes is now fixed: mapDatasetToIcebergTable's first line is await this.datasetService.getDataset(input.dataset_id), which throws Dataset {id} not found unless the dataset was already registered. DatasetService.registerDataset is never called from any route in this repo (grep-verified across app/ and src/) — there is no POST /api/0pt/datasets route, despite app/api/0pt/route.ts's own index claiming /api/0pt/datasets exists (it doesn't — see "The /api/0pt index is wrong" below). So every call to POST /api/0pt/lakehouse/mappings throws Dataset {id} not found, unconditionally, regardless of input, because no HTTP path can ever get a dataset into the registry first. This half is still open — whether to add POST /api/0pt/datasets/POST /api/0pt/sources routes is a scope decision, not a mechanical fix.

Fixed in fix/0pt-full-bug-hunt (PR #5, merged separately): the three route files above previously each constructed their own separate instance of the relevant service — .../[mappingId]/route.ts had a second, independent DatasetIcebergMappingService, and .../[mappingId]/writes/route.ts had a third IcebergWriteService with its own fresh internal DatasetIcebergMappingService, so a mapping created via POST /api/0pt/lakehouse/mappings was invisible to both sibling routes even setting the dataset-registration issue aside. All three route files now import shared datasetService/mappingService/writeService singletons from src/services/lakehouse-mapping-registry.ts instead. This alone does not make the write path succeed end-to-end — the dataset-registration gap above still blocks it — but it removes a second, independent failure mode.

0pt.source_rate_policy_manage — source rate policies

POST /api/0pt/sources/rate-policies — app/api/0pt/sources/rate-policies/route.ts → SourceRatePolicyService.createSourceRatePolicy

body: CreateSourceRatePolicyInput { source_rate_policy_id, tenant_id, project_id, source_id, rate_limit_per_minute, crawl_budget, compliance_notes?, domain?, robots_ok?, access?, status?, correlation_id?, created_by? }
-> 201 SourceRatePolicy
-> 400 { error }

GET /api/0pt/sources/rate-policies — same file → SourceRatePolicyService.listSourceRatePolicies

query: ?tenant_id=&project_id=&source_id=
-> 200 SourceRatePolicy[]

GET /api/0pt/sources/rate-policies/{sourceRatePolicyId} — separate file → SourceRatePolicyService.getSourceRatePolicy

-> 200 SourceRatePolicy
-> 404 { error }

Confirmed this pass — the POST above always fails too, same root cause as lakehouse mappings: createSourceRatePolicy checks sourceService.getSource(input.source_id) before creating the policy, and throws Source {id} not found otherwise. SourceService.createSource is never called from any route in this repo — there is no POST /api/0pt/sources route (see next section). Every call to POST /api/0pt/sources/rate-policies throws Source {id} not found, unconditionally. The .../{sourceRatePolicyId} GET route also constructs its own separate SourceRatePolicyService instance, independent of .../rate-policies/route.ts's — so even a hypothetically-successful create wouldn't be visible there either.

0pt.query_source — bounded synchronous query

POST /api/0pt/sources/{sourceId}/query — app/api/0pt/sources/[sourceId]/query/route.ts → SourceQueryService.querySource

body: { query (required, non-empty string), params?, tenant_id (required), project_id (required), actor_id?, correlation_id?, max_rows?, timeout_ms?, connection_config (required — FabricOneLakeConnectionConfig: {credentials:{tenantId,clientId,clientSecret}, sqlEndpoint, database, scope?, requestTimeoutMs?}) }
-> 200 { source_id, rows, row_count, truncated, duration_ms, bounds:{max_rows,timeout_ms} }
-> 400 { error } (missing query/tenant_id/project_id/connection_config)
-> 404 { error } (source not found)
-> 422 { error } (SourceQueryUnsupportedError — wrong source_type or adapter_id)
-> 504 { error } (SourceQueryTimeoutError)

The route's own inline comment already documents the in-memory-isolation gap for this route: "a source registered via POST /api/0pt/sources will not be resolvable here until SourceService is backed by shared persistence."

Two things worth adding to what that comment says:

  1. There is no POST /api/0pt/sources route in this repo at all. The comment describes a route that does not exist as a file anywhere under app/api/ (verified — only sources/rate-policies* and sources/[sourceId]/query exist under app/api/0pt/sources/; the non-0pt-prefixed app/api/sources/route.ts is a completely different, GET-only, hardcoded 9-entry list — the dim_source transparency-moat catalog described in docs/03-data-model.md and docs/04-api.md, not a per-tenant Source registry). So this isn't "not yet resolvable across processes" — it's "there is currently no way, via any HTTP endpoint, to create a Source at all." SourceService.createSource is only ever called from tests.
  2. Even with a Source registered directly in code, the only supported adapter (adapter_id: "fabric_onelake_sql") depends on the mssql npm package, which is not in package.json as of this pass (src/adapters/fabric-onelake-adapter.ts's own header comment: deliberately not added yet to avoid colliding with a separate in-flight session's package.json changes). query() throws a clear, explicit error until mssql is installed — by design, not silently.

Net effect: POST /api/0pt/sources/{sourceId}/query cannot succeed against any HTTP-only workflow today, for two independent, stacked reasons — no way to register a Source, and no query driver installed even if one existed.

0pt.write_external_record — no route exists

src/services/external-writeback-service.ts is real, tested TypeScript (tests/p0/external-writeback-service.test.ts) implementing "POST a validated, already-approved structured record to a registered external-system adapter" (worked example: Odoo Vendor Bill via src/adapters/odoo-bills-adapter.ts). The capability registry itself documents this plainly: surfaces: ['job'], apiEnabled: false, with a comment noting it's "invoked today only via direct import, not yet wired to a live surface." There is no app/api/0pt/* route for it. Not part of this doc's route list because there is no route to document.

The /api/0pt index is materially wrong

GET /api/0pt (app/api/0pt/route.ts) returns a static endpoints array as self-documentation:

["/api/0pt/sources", "/api/0pt/datasets", "/api/0pt/data-contracts",
 "/api/0pt/quality/checks", "/api/0pt/quality/reports", "/api/0pt/features/views",
 "/api/0pt/documents/capture", "/api/0pt/documents/{documentId}/extraction"]

Of these 8 listed paths, only 2 exist (documents/capture, documents/{documentId}/extraction). The other 6 — sources, datasets, data-contracts, quality/checks, quality/reports, features/views — have no corresponding route file anywhere in this repo. quality/checks and quality/reports in particular line up exactly with the confirmed-fabricated quality-check-service work described elsewhere in this doc corpus (CLAUDE.md → README "Known issues") — this index was apparently written against a planned surface, not the real one. Conversely, 6 routes that do exist are missing from this list entirely: lakehouse/mappings (×3) and sources/rate-policies (×2), plus sources/{sourceId}/query. This file is outside this doc's assigned scope to edit (not one of docs/03-data-model.md/docs/04-api.md) — flagging it here since it's the most visible, single-request way a caller would discover this API's shape, and it's currently misleading.

0pt.screen_sanctions — GET /api/screen

GET /api/screen?q=<name>
-> 200 { matches: [] }                                    if q is empty
-> 200 (engine response, verbatim)                          if ENGINE_URL is set and reachable and engine returns ok
-> 200 { source: "opensanctions", matches: [{name, score, schema}] }   direct OpenSanctions fallback
-> 200 { matches: [], source: "opensanctions" }             on fetch failure or non-2xx

Calls engineFetch('/screen?q=...') first (services/api/orchestrator/main.py's real GET /screen route — see docs/04-api.md), falls back to api.opensanctions.org directly otherwise. Unlike /api/ai/parse below, this fallback path is a working reconciliation between the edge and engine layers, not a silent bug — both paths return the same shape. Comment in the route itself: "VERIFY commercial terms before production volume" — not verified by this pass either.

POST /api/ai/parse — no capability mapping, engine call silently no-ops

body: { text }
-> 200 (engine response, verbatim)          if ENGINE_URL set, reachable, and engine returns ok
-> 200 { config: {...}, rationale, heuristic: true }   local keyword-heuristic fallback otherwise
-> 400 { error } if text is empty

As documented in docs/04-api.md: services/api/orchestrator/main.py has no /ai/parse route at all, so engineFetch("/ai/parse", ...) always 404s and this route always falls through to its local heuristic (MARKETS/SERVICE/FIELDS/DEST regex tables in the route file itself), even when the engine is up and configured. Not a crash — just a permanently-dark code path (if (eng && eng.ok) is never true for this one route). Has no entry in the capability registry (see top of this doc) — no declared risk level, audit requirement, or policy gate, unlike every other route documented above.

Cross-cutting gap 1: tenant_id read from the request body, not a session

Hard platform rule (per this doc corpus's platform-wide conventions): tenant_id must be server-bound to an authenticated session, never read from client request body/query/header. This is violated, today, in:

  • app/api/0pt/lakehouse/mappings/route.ts (POST body tenant_id/project_id)
  • app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts (indirectly — actor_id/correlation_id from body feed into a call keyed off a mapping whose own tenant_id was set by the equally-unauthenticated POST above)
  • app/api/0pt/sources/rate-policies/route.ts (POST body tenant_id/project_id)
  • app/api/0pt/sources/[sourceId]/query/route.ts (POST body tenant_id/project_id, explicitly required — 400 if missing)
  • src/services/document-capture-service.ts (captureFromMailbox's ctx.tenant_id comes straight from app/api/0pt/documents/capture/route.ts's request body; the sibling GET route also takes tenant_id/project_id as query-string filters, which is a softer version of the same problem — it doesn't enforce isolation, it just optionally filters an already-unscoped in-memory list)

This is not a deliberate two-tier design choice — it exists because no authentication or session system exists anywhere in this repo yet (see app/api/orders/route.ts's own GET handler, which has the literal comment "org scoping added with auth" — i.e., not yet; and docs/09-build-plan.md Phase 3, which is where auth is planned, not built). Building that auth design is explicitly Simon's call — this doc doesn't invent one. Until it lands, any caller can claim to be any tenant_id on every route above.

Cross-cutting gap 2: in-memory-only persistence, and worse

Every service backing app/api/0pt/* (SourceService, DatasetService, DatasetIcebergMappingService, SourceRatePolicyService, DocumentCaptureService, DocumentExtractionService) stores state in a plain in-memory Map — nothing survives a process restart, and (per the findings above) in most cases nothing is even shared across route files within the same running process, because each route file constructs its own fresh service instance rather than importing a shared singleton. The one exception is documentCaptureService/documentExtractionService, which are correctly exported as module-level singletons for exactly this reason.

Two SQL tables exist for two of these seven types (opt_dataset_iceberg_mappings, opt_source_rate_policies — sql/004_iceberg_write_boundary.sql), but nothing in app/api/0pt/* reads or writes through Postgres — the routes only ever touch the in-memory Maps. See docs/03-data-model.md for the full SQL-side picture, including a confirmed broken foreign-key reference in that same migration file.

Mock-only platform clients

SourceService, DocumentCaptureService, DocumentExtractionService, IcebergWriteService, and DatasetIcebergMappingService all construct CL0UD/N0D/AUD0/ST0RE clients (src/clients/{cl0ud,n0d,aud0,st0re,st0re-lakehouse}-client.ts) via createXClient(baseUrl?) with no baseUrl ever passed by any route in this repo. createXClient returns a deterministic in-memory mock whenever baseUrl is falsy (verified in each client file), and no route or service anywhere reads CL0UD_URL/N0D_URL/AUD0_URL/ST0RE_URL from process.env (grep-verified this pass, zero hits outside comments and the it.todo in tests/p0/iceberg-write-service.test.ts — this corrects an earlier version of this section, which claimed ST0RE_URL is read by the two ST0RE clients; it is not, matching README's "Known issues" #2, docs/02-architecture.md's "Platform clients" section, docs/08-infrastructure.md's "Platform-client URLs" section, and docs/ICEBERG_WRITE_BOUNDARY.md's "Verification status" section, all independently verified against the same grep). None of the five clients has an env fallback wired at all — every one requires an explicit baseUrl/st0reBaseUrl constructor argument that nothing in this repo ever supplies. Practically: every policy check (cl0ud.evaluatePolicy) always returns whatever the mock's canned decision is, and every audit event (aud0.ingestEvent) is a no-op that never leaves the process. This is real, typechecked, tested code — not a stub file — but it has never been exercised against a live CL0UD/N0D/AUD0/ST0RE instance from this repo.

Open questions for Simon

  • Should app/api/0pt/route.ts's index be fixed to list real routes, or is a stale/aspirational index acceptable short-term? (Not this doc's file to edit.)
  • Should a POST /api/0pt/sources (and POST /api/0pt/datasets) route be added so Source/Dataset registration is reachable at all over HTTP — this is the blocking prerequisite for 0pt.query_source, 0pt.source_rate_policy_manage, and 0pt.lakehouse_mapping_manage to ever succeed via the API as currently wired?
  • Should the in-memory registry services (SourceService, DatasetService, etc.) move to shared singletons (like documentCaptureService) as an interim fix, or wait for real Postgres-backed persistence?
  • Whether/when mssql gets added to package.json so the fabric_onelake_sql adapter can actually run.