0pt/docs/02-architecture.md

02 · Architecture

Known issues affecting this doc, in brief — full detail + evidence in README.md's "Known issues" section: (1) no auth/session system exists yet, so several newer routes trust client-supplied tenant_id (see "Known gaps" below); (2) the CL0UD/N0D/AUD0/ST0RE platform clients are real code but mock-only in this repo; (3) docs/ICEBERG_WRITE_BOUNDARY.md / docs/11-lakehouse-write-plan.md / docs/P0_QUERY_SOURCE_FLOW.md reference several src/services/quality-* and sql/002/sql/003 files that were never actually committed to this repo (out of this doc's file scope to fix — flagged here so it isn't silently repeated); (4) two git remotes (github.com/sax3l/0pt and git.siax.io/sax3l/0pt) have diverged and it is unresolved which is authoritative — Simon's call, not assumed here.

Principle: one database, many lenses

Postgres is the source of truth. Everything else is a lens on it. A tool enters the stack only if it (a) reads that Postgres and (b) sits behind the same auth. This is the difference between an app and a pile of separate logins.

This principle describes — and is largely realized by — the original edge + engine plane: marketing, catalog, order lifecycle, and the FastAPI engine. It does not describe the newer P0 data-platform surface (app/api/0pt/*), which is real, additive, and deliberately does not go through this same path. See "Two API surfaces" below.

                 ┌──────────────────────────────────────────────┐
   COLLECTION    │                  POSTGRES                     │   LENSES (customer + admin)
  (legal engine) │  dim_company · dim_contact · dim_vehicle      │
 ─────────────── │  fact_financials · dim_ubo                    │ ─────────────────────────────
  Official APIs  │  dim_source · bridge_provenance               │   Next.js portal (Vercel)
  /bulk  ───────▶│  orders · order_items · jobs · deliveries     │◀──  Catalog + order lifecycle
  Authorized web │  subscriptions · users · orgs · files         │   NocoDB  (embedded "Min data")
  Licensed data  │  db_grants · audit_log · pgvector embeddings  │   Metabase (embedded dashboards)
  Customer upload│                                               │   Export: CSV/Parquet/JSON/API/FiveCRM
                 └───────────────────▲──────────────────────────┘
                                     │
                        services/api — FastAPI orchestrator + workers
              (estimates, orders, jobs, AI-assist, webhooks, pipeline)

Components & responsibilities

• Next.js (Vercel) — all customer/admin UI: marketing site, catalog, configurator, checkout, production tracker, validation, portal, admin. It also hosts two distinct families of API route handlers under app/api/: the original order/catalog/estimate surface (some of which defer to the FastAPI engine), and the newer P0 data-platform surface under app/api/0pt/* (does not defer to the engine at all — see below).
• FastAPI orchestrator (services/api/orchestrator/main.py) — REST contract in docs/04-api.md: estimates, orders, sources, vehicle lookup, sanctions screening, job run/status. Not on Vercel (serverless timeouts kill long jobs). Correction: earlier drafts of this doc pointed at src/orchestrator; that was the pre-migration location. docs/09-build-plan.md Phase 0 called for moving it into services/api, and that move has happened — the code lives at services/api/orchestrator today.
• Workers (services/api/workers/) — adaptive router (adaptive_router.py, verified: chooses only among compliant methods — official API → bulk → authorized web → licensed, no bypass/stealth/proxy-rotation tier exists in it, by design) + pipeline stages (pipeline.py, queue.py, resolve.py, run.py). See docs/06-data-engine.md.
• Postgres — backbone (docs/03-data-model.md) incl. pgvector for matching/embeddings, for the original edge + engine plane. Wired via lib/db.ts (env-guarded: DATABASE_URL/POSTGRES_URL; falls back to an in-memory, non-durable store when unset, e.g. lib/orders-store.ts).
• NocoDB / Metabase — both are defined as services in infra/docker-compose.yml; embedding them into the portal as "Min data" / customer dashboards was the original target, and neither is confirmed wired into a portal page. Correction 2026-09-06: Metabase is no longer the canonical BI lens — Apache Superset is (see "Stack decisions" below). Metabase may stay as an optional App Catalog app; it must not be treated as the platform default.
• Cloudflare R2 — file storage (uploads + deliverables) via presigned URLs. Env-guarded (app/api/uploads/route.ts checks R2_ACCESS_KEY_ID/R2_BUCKET/R2_ENDPOINT and returns 501 until set); the presign call itself is still a stub (put_url: null) even when the env vars are present — not wired end-to-end yet.
• Stripe — deposit capture + escrow balance release. This one is wired: app/api/payments/[order_id]/deposit/route.ts and .../release/route.ts are real Stripe Checkout calls (env-guarded on STRIPE_SECRET_KEY, 501 until configured). See the correction in docs/01-product.md's business model section for exactly which escrow mechanism is implemented.
• Resend — documented as the transactional-email provider (docs/08-infrastructure.md) but has no call site anywhere in this repo yet. Treat "+ email" as not-yet-implemented, not merely unconfigured.
• Auth.js (NextAuth) with a Postgres adapter — this is the target, listed here as a stack decision, not a built component. What exists today is a lightweight signed-session scaffold (lib/auth.ts: HMAC-SHA256 via Web Crypto, no external deps) behind a guarded dev-only login gate (DEV_LOGIN_KEY). There is no real session-to-org/tenant binding anywhere in this repo. Full Auth.js + OAuth is Phase 3 of docs/09-build-plan.md and has not been built. This matters directly for the tenant-scoping gap described below.

Two API surfaces

This repo currently contains two real, distinct API surfaces, not one. Earlier versions of this doc described the edge as a uniform thin proxy to the FastAPI engine — that was true for part of the original surface and remains true for that part, but a second, newer surface was added that is a deliberate, ADR-documented exception to it. Conflating the two is the single biggest way to misread this codebase's architecture.

1. The original edge → engine surface

A subset of the original app/api/* route handlers defer to the FastAPI engine (services/api) via lib/engine.ts's engineFetch()/hasEngine() bridge when ENGINE_URL is set, and fall back to local computation when it isn't:

• app/api/estimates/route.ts — defers to POST {ENGINE_URL}/estimates, falls back to lib/data.ts's computeEstimate.
• app/api/vehicles/route.ts, app/api/screen/route.ts, app/api/ai/parse/route.ts — same pattern.

Other original routes are not engine-backed at all today: app/api/orders/* reads/writes lib/orders-store.ts directly (Postgres when configured, else in-memory); app/api/catalog/route.ts returns a hardcoded static array. Both of these are entirely edge-local — there is no engine call in either path currently, not "thin proxy with a local fallback."

This surface is documented in full in docs/04-api.md and matches the "one Postgres, many lenses" principle above (once DATABASE_URL is set).

2. The P0 data-platform surface (app/api/0pt/*)

A second, newer surface implements dataset registration, Iceberg lakehouse table mapping/writes, per-source rate-policy management, bounded synchronous source querying, and mailbox-based document capture — directly in the edge app, backed by src/services/* and src/clients/*. It does not call lib/engine.ts/ENGINE_URL and does not go through the FastAPI engine at all. This is deliberate: it is the concrete implementation of ADR-0030 (Apache Iceberg as SIAX's canonical lakehouse table format, authored in the n0d repo) and the ownership split it defines. See docs/10-lakehouse-decision.md (the pointer to the ecosystem decision), docs/11-lakehouse-write-plan.md (the plan), and docs/ICEBERG_WRITE_BOUNDARY.md (the implementation record) for the full boundary — this doc only summarizes it.

Real, existing routes today (verified against the filesystem, not against app/api/0pt/route.ts's own advertised list — see the note below):

Route	Backed by
GET /api/0pt	static index (see note below)
POST/GET /api/0pt/documents/capture	src/services/document-capture-service.ts
.../documents/{documentId}/extraction	src/services/document-extraction-service.ts
POST/GET /api/0pt/lakehouse/mappings	src/services/dataset-iceberg-mapping-service.ts, iceberg-write-service.ts
.../lakehouse/mappings/{mappingId}	same
POST .../lakehouse/mappings/{mappingId}/writes	src/services/iceberg-write-service.ts#requestIcebergWrite
POST/GET /api/0pt/sources/rate-policies	src/services/source-rate-policy-service.ts
.../sources/rate-policies/{sourceRatePolicyId}	same
POST /api/0pt/sources/{sourceId}/query	src/services/source-query-service.ts (bounded, timeout-guarded ad-hoc SQL against a registered database source — see docs/P0_QUERY_SOURCE_FLOW.md)

Code-level inconsistency worth knowing about: app/api/0pt/route.ts's own GET handler advertises an endpoints list including /api/0pt/sources, /api/0pt/datasets, /api/0pt/data-contracts, /api/0pt/quality/checks, /api/0pt/quality/reports, /api/0pt/features/views — none of these exist as routes (confirmed by listing app/api/0pt/ on disk); the lakehouse-mapping and rate-policy routes that do exist aren't listed there either. This is the same "described but never built" pattern as the quality-check-service.ts fabrication flagged below — this doc's route table above reflects what is actually on disk, not what the index endpoint claims.

In-memory scope: every P0 service (DatasetService, SourceService, DatasetIcebergMappingService, SourceRatePolicyService, DocumentCaptureService, …) holds its state in a plain Map inside the Next.js route module that instantiates it. Most route modules construct a fresh instance per module, so a record created via one route is not visible to a different route module in the same running process (each file's own header comment documents this; document-capture-service.ts worked around it for its own two routes with a shared singleton export). A Postgres migration exists (sql/004_iceberg_write_boundary.sql) but is not wired into any route handler yet. So the "one database, many lenses" principle above does not yet hold for this surface.

Six-surface capability model & 0pt's current coverage

SIAX Cloud's target model (defined in cl0ud, docs/architecture/SIAX_CLOUD_UNIFIED_ARCHITECTURE.md) is that one capability can expose up to six surfaces: REST api, cli (siax 0pt ...), mcp (an agent tool), ui (a page/panel), job (an async N0D job type), and sdk (a generated client method). The capability registry's own Surface type (cl0ud packages/capability-registry/src/types.ts) currently encodes five of those six as a declarative field ('api' | 'cli' | 'mcp' | 'ui' | 'job'); sdk is generated separately, not yet a registry field.

0pt's actual coverage today (verified against cl0ud packages/capability-registry/src/registry.ts): 14 distinct 0pt.* capabilities are registered —

0pt.query_source, 0pt.document_ocr, 0pt.write_external_record, 0pt.lakehouse_mapping_manage, 0pt.lakehouse_write, 0pt.source_rate_policy_manage, 0pt.screen_sanctions, 0pt.estimate_compute, 0pt.vehicle_lookup, 0pt.catalog_list, 0pt.source_registry_list, 0pt.order_create, 0pt.order_payment, 0pt.write_iceberg_table

— and, checked entry by entry:

• 11 declare surfaces: ['api'] only.
• 2 (0pt.lakehouse_write, 0pt.write_iceberg_table) declare surfaces: ['api', 'job'].
• 1 (0pt.write_external_record) declares surfaces: ['job'] only — apiEnabled: false, and its own comment states it has "NO dedicated HTTP route in the 0pt app yet"; it exists as a tested TS service (src/services/external-writeback-service.ts) invoked only via direct import today, not a live callable path.
• Every single one has cliEnabled: false and mcpEnabled: false. None has a ui surface entry either.

In plain terms: 0pt is API-only today (plus two job-backed writes, and one job-only capability with no live route at all). If any doc — inside or outside this repo — describes a live siax 0pt ... CLI command or an 0pt.* MCP tool, it is describing a target, not current reality; correct it on sight.

Platform clients: CL0UD / N0D / AUD0 / ST0RE

src/clients/{cl0ud,n0d,aud0,st0re,st0re-lakehouse}-client.ts are real, typechecked TypeScript with accompanying tests (tests/p0/*), each following the same shape: a thin interface, an HTTP implementation written against the target service's real, committed route contract, and a deterministic in-memory mock used whenever no baseUrl is supplied to the constructor.

SUPERSEDED 2026-08-31 (SIAX Platforms P0 wave 2) — the paragraph that stood here ("every one of these clients runs against its in-memory mock today") was accurate when written and is now fixed, twice over:

1. every factory reads its *_URL from process.env when no constructor baseUrl is given, and under NODE_ENV=production an unset URL is a hard boot failure (SiaxError DEPENDENCY_UNAVAILABLE, src/platform/dependency-mode.ts) — production can no longer fall back to a mock (Master-DoD §3/§10);
2. the HTTP wire shapes were verified against the sibling repos' actual main branches and THREE of four were wrong (cl0ud path never existed, st0re body was snake_case against a camelCase contract, aud0 lineage path never existed). All rewritten and pinned by tests/contracts/*.contract.test.ts.

Current per-capability status lives in STATUS.md — that file, not this doc, is the status source of truth (§56).

What each client is for, concretely:

• CL0UD — policy evaluation (evaluatePolicy). iceberg-write-service.ts and external-writeback-service.ts call it before acting; the mock always returns "allow".
• N0D — job requests (createJob). document-capture-service.ts requests document-extraction jobs; iceberg-write-service.ts requests heavy-write/compaction jobs. 0pt only requests N0D jobs — it never executes them, per docs/ICEBERG_WRITE_BOUNDARY.md's "N0D executes" boundary. No evidence supports a claim that N0D has a dedicated-node ("t_0pt") auth pattern already built for this: n0d's repository was searched in full (all branches, all commits, all files) and contains zero references to t_0pt. Treat any such claim as unconfirmed until proven otherwise.
• AUD0 — audit/lineage events (ingestEvent, ingestLineageEvent). Every mutating P0 service emits these; the mock is a true no-op, matching AUD0's own documented stance that it has no live ingestion endpoint yet.
• ST0RE / ST0RE-lakehouse — artifact-metadata registration and Iceberg table/namespace/snapshot registration, respectively. st0re-lakehouse-client.ts's HTTP path is written against sax3l/st0re's actual shipped route contract (commit 9ea0abb, read directly during a prior pass) — real code, but not executed against a running ST0RE in this repo's test environment either.

Known gaps

tenant_id is not server-bound anywhere in the P0 surface

The platform-wide rule is that tenant_id must be bound to an authenticated session server-side and never trusted from client input. This repo violates that rule today, and it is not a deliberate two-tier design — it's a direct consequence of there being no auth/session system at all yet (see the Auth.js note above; app/api/orders/route.ts's own GET handler literally comments // org scoping added with auth, i.e. not yet).

Concretely: tenant_id/project_id are accepted straight from the client-supplied request body (or query string) at every P0 mutation entry point —

• app/api/0pt/lakehouse/mappings/route.ts (POST)
• app/api/0pt/sources/rate-policies/route.ts (POST)
• app/api/0pt/sources/[sourceId]/query/route.ts (POST)
• app/api/0pt/documents/capture/route.ts → src/services/document-capture-service.ts#captureFromMailbox

app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts doesn't read tenant_id from its own body — but it resolves the tenant by looking up mapping.tenant_id from a mapping that was itself created via one of the unauthenticated routes above, so the trust boundary problem is inherited, not avoided.

Do not expose any of these routes to untrusted callers before Phase 3 of docs/09-build-plan.md (Auth.js + org/session binding) lands. This is Simon's call on sequencing, not something to silently patch in a docs pass.

0pt's data-quality/QA stage is a fabrication in sibling docs, not built

Flagged so it is not repeated: src/services/quality-check-service.ts, src/services/quality-report-service.ts, src/types/quality-check.ts, src/types/quality-report.ts, src/data-platform/services/catalog-service.ts, sql/002_p0_dataset_quality.sql, and sql/003_*.sql were at one point referenced in docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and docs/P0_QUERY_SOURCE_FLOW.md as if already built and tested. Verified this pass: git log --all for every one of those paths returns zero commits, on any branch, ever — sql/ on disk today contains only 001_init.sql and 004_iceberg_write_boundary.sql. Those files have never existed in this repository. 0pt's data-quality/QA enforcement is spec/target only — a real, open build gap, not a doc typo. All three of the docs above have since been corrected (verified this pass by reading each in full) — none currently asserts those files as real; each carries its own explicit "fabricated, not built" callout, and docs/P0_DATASET_QUALITY_FLOW.md is the honest target-spec replacement.

Two diverged git remotes

origin in local checkouts of this repo is github.com/sax3l/0pt. A separate Gitea remote, git.siax.io/sax3l/0pt, shares at least part of the same history (services/api/workers/adaptive_router.py is byte-for-byte identical between GitHub's current main and a Gitea post-fix PR branch) but independently accumulated commits GitHub's main never received — including, at one point, a WAF-bypass/proxy-rotation/fake-identity scraper stack that directly violated the CLAUDE.md §1 sourcing rule (since removed on Gitea via a separate PR). Which remote is authoritative, which is actually deployed to 0pt.app, and whether the two should be kept in sync is unresolved and is Simon's decision — do not assume origin/GitHub is the sole copy of this codebase, here or in any other doc.

Stack decisions — IN

Concern	Choice	Why
Frontend	Next.js App Router + TS + Tailwind + shadcn/ui	Vercel-native, matches existing skills
Backend/jobs	FastAPI + workers (services/api)	already built; long-running jobs
DB	Postgres (+ pgvector)	backbone; Neon free tier or on-VPS
Airtable layer	NocoDB	sits on existing tables, no duplication
BI	Apache Superset	canonical generic BI for SIAX Cloud: Apache-2.0 with an official embed SDK + guest tokens, so embedding never needs a commercial tier. Already defined as a service in services/infra/docker-compose.data-stack.yml and inventoried in docs/compliance/OSS-INVENTORY.md. Corrected 2026-09-06 — this row previously said Metabase; see the correction under "OUT" below
Files	Cloudflare R2 + presigned URLs	cheapest S3-compatible
Payments	Stripe Checkout + webhook → Next.js API	~1 day, no SaaS boilerplate needed
Auth	Auth.js + Postgres adapter	one system — target, not yet built (see Components above)
Email	Resend	best Next.js integration, free tier — not yet wired (no call site in this repo)
Quality	Great Expectations / checks inside workers	scores shipped with each order — not yet built for the P0 surface (see "Known gaps" above)

Stack decisions — OUT (do not add)

CORRECTED 2026-09-06 — two rows in this table were wrong and are withdrawn below. This section was written when 0pt was planned as a standalone SaaS product. Under the SIAX Cloud analytics decision, 0pt is the canonical Analytics/Data/Semantic owner — there is no BI0/Analytics0 repo, no separate Superset platform, and no product-specific BI backend acting as a parallel capability owner. Third-party BI tools are interchangeable providers or App Catalog apps. Two consequences contradict the table as it stood:

• Apache Superset is canonical, not rejected (Apache-2.0, official embed SDK + guest tokens). Metabase moves to the optional App Catalog.
• OpenMetadata is the canonical catalog (default before DataHub), not rejected. DataHub remains optional.

This is not a paper change — this repository already shipped both, which is what makes the old rows self-contradicting rather than merely out of date:

• src/metadata/openmetadata-adapter.ts + tests/contracts/metadata-features.contract.test.ts (OpenMetadata projection; STATUS.md classifies it IMPLEMENTED, projection-never-canonical),
• services/infra/docker-compose.data-stack.yml defines a superset service (apache/superset:latest), and docs/compliance/OSS-INVENTORY.md already inventories superset | Apache-2.0 | services/infra/data-stack.

The governing principle behind the change is one definition of every metric: the 0pt Metric Catalog feeds MetricFlow, which feeds Superset / Cube / the native API / INF0. Point a product at a BI engine directly and each surface invents its own "average price".

Rejected	Reason
Superset	WITHDRAWN 2026-09-06 — see the correction note above. Superset is now the canonical BI engine, not a rejected one
DataHub / OpenMetadata	WITHDRAWN 2026-09-06 — see the correction note above. OpenMetadata is the canonical catalog; only DataHub stays non-canonical (optional App Catalog)
Metabase / Lightdash / Rill / Grafana / DataHub as canonical BI or catalog	optional App Catalog apps only, never the platform default. Metabase's OSS build is AGPL and its full app-/SSO-embedding sits behind Pro/Enterprise, which does not work for SIAX as a platform provider; Lightdash gates embedding/caching behind an enterprise self-host licence; Rill's documented deployment path is bound to Rill Cloud (breaks zero-SaaS); Grafana (AGPLv3) is the right tool for operational analysis and is AUD0-owned, not business BI
Baserow / Teable	NocoDB chosen — not three Airtable clones
Meilisearch (v1)	Postgres full-text suffices to a few thousand rows; add later
Delta Sharing (v1)	unnecessary; customer pull via scoped role + exports
Trigger.dev	the existing queue already schedules jobs; Resend covers mail
FlareSolverr / stealth browsers / undetected-* / proxy-fingerprint rotation / fake-account creation	prohibited — see CLAUDE.md §1. Replace with compliant fetching. (The Gitea-only scraper stack described under "Two diverged git remotes" above is exactly this — removed there, never present on GitHub main.)

Hosting (cheapest coherent) — SUPERSEDED (BACKLOG P2-5)

SUPERSEDED 2026-08-11. This section describes the original Vercel+Neon hosting plan. siax.config.json flipped deploy.target from "vercel" to "coolify" on 2026-08-09 to match every other SIAX Cloud OS repo — that decision supersedes the Vercel prescription below. The live instance is still actually running on Vercel today (no Coolify app has been provisioned yet — see README.md's "Deploy" section and docs/architecture/ARCHITECTURE.md §5/§8 for the current, verified state); this section is kept for history, not as the current target. Do not read "Vercel (Hobby/free)" below as still-current guidance.

CORRECTED 2026-09-06 — the struck sentence above is no longer true. The Coolify deploy hook is live: repo secret COOLIFY_TOKEN exists (created 2026-08-31T11:34:34Z) and repo Actions variable COOLIFY_APP_ID holds a real 24-character UUID, not the TODO-no-coolify-app-provisioned-yet literal, so both guards in .gitea/workflows/cd.yml pass. On the push to main at 1ff34410 (2026-09-06) the Deploy to Coolify / deploy job finished success with coolify deploy HTTP 200. Every merge into main fires a real redeploy against cloud.siax.io.

Probed 2026-09-06 08:00 UTC: 0pt.app still answers from Vercel (server: Vercel) with HTTP 402 DEPLOYMENT_DISABLED on both / and /api/health. The public domain is therefore down, DNS has not been cut over, and the Coolify app serves no 0pt.app traffic. Still unestablished: whether the Coolify app builds and starts at all — an accepted deploy request is not a healthy container.

• Vercel (Hobby/free) — Next.js + custom domain 0pt.app.
• One small VPS (e.g. Hetzner ~€5/mo) running docker-compose (infra/docker-compose.yml): FastAPI, workers, NocoDB, Metabase, (Postgres or use Neon), optional Meilisearch later. Per-service managed platforms (Railway) get expensive once you have 4–5 always-on services; a single VPS + Vercel + R2 lands ≈ <€10/mo to start.
• Cloudflare R2 — object storage (env-guarded, presign not yet wired — see Components above).
• Postgres — on the VPS, or Neon free tier if you want it managed.

Full config, env, security, CI/CD: docs/08-infrastructure.md. Env var inventory: .env.example (present in this repo today — the linked GitHub PR that originally tracked it has long since merged; removed the stale cross-repo link per BACKLOG P3-5).