0pt/docs/08-infrastructure.md

08 · Infrastructure

Verified against actual repo state 2026-07-19 (branch main, plus .env.example read from feature/0pt-env-example). Where this doc previously described a target that doesn't match what's committed, that's called out explicitly rather than silently corrected — see the repo's honesty convention in docs/ICEBERG_WRITE_BOUNDARY.md.

Repo structure (actual, not the original apps/web sketch)

The monorepo did not converge on the apps/web/ + packages/shared/ layout this doc originally sketched. What's actually on main today:

0pt/
├─ CLAUDE.md
├─ README.md · SECURITY.md
├─ docs/                      # this documentation set
├─ app/                       # Next.js App Router (Vercel) — the "edge" plane, AT THE REPO ROOT
│  ├─ api/                    # route handlers: orders, auth, admin, catalog, connections,
│  │                          #   vehicles, subscriptions, sources, estimates, health, uploads,
│  │                          #   payments, 0pt/*, screen, webhooks, ai
│  ├─ bestall/ katalog/ kallor/ order/ portal/ admin/ tjanster/   # pages
├─ lib/                       # edge-side helpers: db.ts, engine.ts, auth.ts, stripe.ts,
│  │                          #   orders-store.ts, subscriptions-store.ts, data.ts, services.ts
├─ src/                       # SIAX-platform-facing TS layer (NOT under app/ or lib/)
│  ├─ clients/                # cl0ud-, n0d-, aud0-, st0re-, st0re-lakehouse-client.ts
│  ├─ services/                # dataset-, source-, source-query-, iceberg-write-,
│  │                          #   document-capture-, document-extraction-, external-writeback-,
│  │                          #   source-rate-policy-, dataset-iceberg-mapping-service.ts
│  ├─ adapters/                # local-blob-store.ts, odoo-bills-adapter.ts, graph-mailbox-adapter, ...
│  └─ types/
├─ components/ hooks/ styles/  # design system in code (docs/05) — data-artifact.tsx,
│  │                          #   provenance-badge.tsx, embed.tsx, use-reveal.ts, globals.css
├─ packages/schemas/           # JSON Schemas for 0pt.* event/entity contracts (not a pnpm workspace
│  │                          #   package today — pnpm-workspace.yaml has no `packages:` list)
├─ services/api/               # FastAPI orchestrator + workers (the "engine" plane, on a VPS)
│  ├─ orchestrator/main.py
│  ├─ workers/                 # adaptive_router.py, pipeline.py, resolve.py, queue.py, run.py
│  ├─ sources/                 # sources.yaml + connectors.py (docs/06)
│  └─ Dockerfile · requirements.txt
├─ sql/                        # migrations — see "Schema state" below
├─ infra/
│  ├─ docker-compose.yml       # real, committed (not a sketch — see below)
│  └─ Caddyfile                # real, committed
├─ tests/p0/                   # vitest, TS-side (services/clients/adapters)
└─ (no .github/workflows/ directory exists — see "CI/CD" below)

Known open question (not resolved this pass, Simon's call): CLAUDE.md and README.md both now note that github.com/sax3l/0pt (this checkout's origin) and git.siax.io/sax3l/0pt (Gitea) are two remotes that share part of the same history but have independently diverged — Gitea's main picked up a WAF-bypass/proxy-rotation scraper stack (since removed via a Gitea PR) that never touched GitHub's main. Which remote is authoritative/deployed, and whether they should be kept in sync, is undecided. This doc describes what's on this checkout's main only.

Schema state (sql/)

Only two migration files exist in the repo's history, on any branch: sql/001_init.sql (the core schema — dim_company, dim_contact, fact_financials, orders, etc., per docs/03-data-model.md) and sql/004_iceberg_write_boundary.sql (opt_dataset_iceberg_mappings + source-rate-policy tables, per docs/ICEBERG_WRITE_BOUNDARY.md). There is no sql/002_* or sql/003_* anywhere in this repo's git history (git log --all for those paths returns zero commits) — despite docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and docs/P0_QUERY_SOURCE_FLOW.md citing sql/002_p0_dataset_quality.sql and a sql/003_* as if they were already-applied migrations. They are not real. See "Security & compliance" below for what this means for QA/quality-check claims.

Hosting topology

• Vercel — app/ (this repo's root, not a separate apps/web) → 0pt.app. Verified this pass (network check performed): https://0pt.app returns HTTP 200, serves a Next.js/Turbopack app titled "0pt — your data" with SIAX branding, and GET https://0pt.app/api/health returns {"ok":true,"service":"0pt-web","db":"not_configured","queue":"not_configured",...} — an exact shape match for this repo's own app/api/health/route.ts. So README's "Live: https://0pt.app" is confirmed accurate, and the live deployment confirms DATABASE_URL/REDIS_URL are not set in production — i.e. the live site is running on the in-memory order/subscription stores described throughout this doc, not against a real Postgres. This does not confirm which git remote/commit is deployed, or that every route behaves as documented — only that the deployment exists and matches this repo's shape.
• One VPS (Hetzner, per docs/02) — infra/docker-compose.yml (see below) runs FastAPI, worker, NocoDB, Metabase, Postgres, Redis, Caddy. This compose file and its Caddyfile are real, committed files — not a sketch — but this pass found no evidence (no CI/CD, no deploy log, no accessible host) that they have ever actually been run against a live VPS. Treat as "written, unexecuted."
• Cloudflare R2 — documented target for file storage. app/api/uploads/route.ts checks for R2_ACCESS_KEY_ID/R2_BUCKET/R2_ENDPOINT and returns 501 storage_not_configured when absent; when "present" it still returns a hardcoded put_url: null — there is no actual presigned-URL generation implemented yet. R2 upload is an env-guarded stub, not a working integration.
• Postgres — lib/db.ts resolves DATABASE_URL || POSTGRES_URL || POSTGRES_PRISMA_URL (Vercel Postgres integration sets the latter two automatically) and falls back to null (in-memory stores) when none is set. This part is real and wired.

docker-compose (infra/docker-compose.yml — real file, reproduced here for reference)

services:
  postgres:
    image: pgvector/pgvector:pg16
    environment: { POSTGRES_USER: opt, POSTGRES_PASSWORD: ${PG_PASSWORD}, POSTGRES_DB: opt }
    volumes: ["pgdata:/var/lib/postgresql/data", "../sql:/docker-entrypoint-initdb.d:ro"]
  redis:
    image: redis:7
  api:
    build: ../services/api
    command: uvicorn orchestrator.main:app --host 0.0.0.0 --port 8000
    environment: { DATABASE_URL: "postgres://opt:${PG_PASSWORD}@postgres:5432/opt", COMPANIES_HOUSE_API_KEY: "${COMPANIES_HOUSE_API_KEY:-}", INTERNAL_TOKEN: "${INTERNAL_TOKEN}" }
    depends_on: [postgres, redis]
  worker:
    build: ../services/api
    command: python -m workers.run
    environment: { DATABASE_URL: "postgres://opt:${PG_PASSWORD}@postgres:5432/opt", COMPANIES_HOUSE_API_KEY: "${COMPANIES_HOUSE_API_KEY:-}" }
    depends_on: [postgres, redis]
  nocodb:
    image: nocodb/nocodb:latest
    environment: { NC_DB: "pg://postgres:5432?u=opt&p=${PG_PASSWORD}&d=opt", NC_PUBLIC_URL: "https://data.0pt.app" }
    depends_on: [postgres]
  metabase:
    image: metabase/metabase:latest
    environment: { MB_DB_TYPE: postgres, MB_DB_HOST: postgres, MB_DB_PORT: "5432", MB_DB_USER: opt, MB_DB_PASS: "${PG_PASSWORD}", MB_DB_DBNAME: opt }
    depends_on: [postgres]
  caddy:
    image: caddy:2
    ports: ["80:80", "443:443"]
    volumes: ["./Caddyfile:/etc/caddy/Caddyfile", "caddydata:/data"]
    depends_on: [api, nocodb, metabase]
volumes: { pgdata: {}, caddydata: {} }

Note vs. an earlier draft of this doc: the api/worker containers here only receive DATABASE_URL, COMPANIES_HOUSE_API_KEY, and INTERNAL_TOKEN — not STRIPE_*, RESEND_API_KEY, or R2_*. That's correct as written: Stripe, R2, and (a documented but unbuilt) Resend integration are all called from app/api/* route handlers, which run on Vercel, not inside this compose stack.

Caddyfile (real, committed):

api.0pt.app { reverse_proxy api:8000 }
data.0pt.app { reverse_proxy nocodb:8080 }
dash.0pt.app { reverse_proxy metabase:3000 }

Environment variables

.env.example now exists and is merged to main (BACKLOG P3-5: the stale link to the unmerged github.com/sax3l/0pt/pull/3 that used to sit here has been removed — that PR landed a while ago). It was previously impossible to commit: .gitignore's blanket .env* rule silently matched .env.example too, so it was never added despite docs/09-build-plan.md Phase 0 requiring it since day one; .gitignore now has !.env.example to fix that. README's "Env inventory: .env.example" link is a real, current reference.

The table below is grep-verified against main this pass, not transcribed from .env.example's own comments — in one case (ST0RE_URL, see note) that turned up a discrepancy worth flagging.

Var	Status on main	Where
DATABASE_URL / POSTGRES_URL / POSTGRES_PRISMA_URL	Wired — priority-ordered fallback	lib/db.ts
ENGINE_URL	Wired — edge defers to FastAPI engine when set, else computes locally	lib/engine.ts
INTERNAL_TOKEN	Wired — sent as X-Internal-Token on every engineFetch() call	lib/engine.ts
AUTH_SECRET	Wired, but only for a lightweight HMAC session scaffold — not Auth.js. lib/auth.ts's own comment: "Production auth = Auth.js + Postgres adapter... wired once the database is provisioned"	lib/auth.ts, app/api/auth/login/route.ts
DEV_LOGIN_KEY	Wired — gates the dev-only login route (403 without a match); not a real auth system	app/api/auth/login/route.ts
AUTH_URL	Documented target — no read found	—
STRIPE_SECRET_KEY	Wired — real Stripe Checkout session creation (test or live key both work; not test-mode-locked in code)	app/api/payments/[order_id]/deposit/route.ts
STRIPE_WEBHOOK_SECRET	Wired — real signature verification, advances order status	app/api/webhooks/stripe/route.ts
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY	Documented target — no read found	—
R2_ACCESS_KEY_ID, R2_BUCKET, R2_ENDPOINT	Presence-checked only — gates a 501 vs stub 200; no actual R2 client/presign call exists yet	app/api/uploads/route.ts
R2_ACCOUNT_ID, R2_SECRET_ACCESS_KEY	Documented target — no read found (even the stub above doesn't check these)	—
OPT_LOCAL_BLOB_DIR	Wired — local-disk fallback artifact store, used when no object storage is configured	src/adapters/local-blob-store.ts
RESEND_API_KEY	Documented target — no Resend call site anywhere in this repo; not "unconfigured", genuinely not implemented	—
METABASE_SITE_URL, METABASE_SECRET_KEY, NOCODB_BASE_URL, NOCODB_API_TOKEN	Documented target — no read found. components/embed.tsx renders an iframe from a src prop the caller must supply; no server-side signed-URL generator exists to produce that src	—
DOC_INTELLIGENCE_ENDPOINT, DOC_INTELLIGENCE_KEY, DOC_INTELLIGENCE_API_VERSION	Wired	src/services/document-extraction-service.ts
GRANT_ENC_KEY	Presence-checked only — gates 501 vs a stub 201 that returns { grant_id, status: "pending" }. No AES-GCM encryption, no read-only validation, no actual storage happens yet despite the route's own comment describing that flow	app/api/connections/db-grant/route.ts
NEXT_PUBLIC_APP_URL	Wired — used as a fallback origin for building Stripe redirect URLs	app/api/payments/[order_id]/deposit/route.ts
NEXT_PUBLIC_API_URL	Documented target — no read found	—
REDIS_URL	Presence-checked only, in /api/health — no queue code in this repo reads it (Celery/queue usage, if any, is services/api's own concern, out of this repo's TS surface)	app/api/health/route.ts
COMPANIES_HOUSE_API_KEY	Real key referenced by services/api (Python engine, sources.yaml/connectors.py) and passed through by infra/docker-compose.yml — not part of the Next.js edge app's own env surface	services/api/

Platform-client URLs — corrected from a stronger claim

CL0UD_URL, N0D_URL, AUD0_URL, and ST0RE_URL are all documented targets only — none of the four is read via process.env.* anywhere in this repo's committed code, grep-verified this pass (grep -rn "process\.env\.\(ST0RE_URL\|CL0UD_URL\|N0D_URL\|AUD0_URL\)" returns zero matches outside comments and one it.todo(...) placeholder in tests/p0/iceberg-write-service.test.ts).

This corrects a claim that appears in .env.example itself (on feature/0pt-env-example) and was carried into an earlier version of this doc: that comment states "ST0RE_URL IS read (src/clients/st0re-client.ts, src/clients/st0re-lakehouse-client.ts)". That is not accurate as of this verification. Both of those client files — like cl0ud-client.ts, n0d-client.ts, and aud0-client.ts — only export a create*Client(baseUrl?: string) factory that falls back to an in-memory mock when no baseUrl argument is passed. Every call site in this repo (app/api/0pt/** routes, src/services/*.ts) either passes no baseUrl at all or forwards an opts?.st0reBaseUrl-style constructor option that nothing upstream ever populates from process.env.ST0RE_URL. This matches docs/ICEBERG_WRITE_BOUNDARY.md's own "Verification status" section, which lists wiring ST0RE_URL → IcebergWriteService({ st0reBaseUrl: process.env.ST0RE_URL }) as future work, not something already done.

Net effect: setting any of CL0UD_URL/N0D_URL/AUD0_URL/ST0RE_URL in a deployed environment today does nothing by itself. A future pass has to wire each route/service to actually read its corresponding env var before the value has any effect. Until then, every 0pt→platform call in this repo runs against the deterministic in-memory mock built into each client file — real, tested TS, but never exercised against a live CL0UD/N0D/AUD0/ST0RE instance from this codebase.

Security & compliance — corrected against actual code, not the target design

The items below were previously described as already-built behavior. Grep/read-verified this pass, several are env-guarded stubs that return a plausible-shaped response without doing the described work. Stating this plainly, per this repo's own honesty convention:

• Customer DB grants — app/api/connections/db-grant/route.ts returns 501 when GRANT_ENC_KEY is unset, and a stub 201 { grant_id, status: "pending" } when it is set. There is no AES-GCM encryption, no read-only-role validation, and no persistence implemented yet — the route's own comment describes the target behavior ("validate read-only, AES-GCM encrypt conn_string, store, audit") but the code doesn't do it. This is a real gap, not a doc typo.

• File delivery / presigned R2 URLs — same pattern: app/api/uploads/route.ts is a 501 stub with a hardcoded put_url: null. No presigned URL is ever actually generated in this repo today.

• Tenant/org isolation — OPEN SECURITY GAP, not a deliberate two-tier design. Several app/api/0pt/* routes read tenant_id/project_id directly from the request body instead of from a server-bound session:

  • app/api/0pt/lakehouse/mappings/route.ts
  • app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts
  • app/api/0pt/sources/rate-policies/route.ts
  • app/api/0pt/sources/[sourceId]/query/route.ts
  • src/services/document-capture-service.ts

  This is not a defensible internal-vs-external boundary — it's because no auth/session system exists anywhere in this repo yet. app/api/orders/route.ts's own GET handler carries the literal comment // org scoping added with auth — i.e., not yet. lib/auth.ts is a lightweight HMAC session scaffold gated by a dev-only login key, unconnected to any of the 0pt/* routes above. Any caller can currently claim any tenant_id for these endpoints. Fixing this is Phase 3 of docs/09-build-plan.md (Auth & portal shell) — it is Simon's call how that auth design binds tenant_id server-side; it is not built anywhere in this repo today.

• PII (contacts/UBO) / DPA — no code found in this pass implementing lawful-basis recording, deletion-on-request, or a published DPA. Treat as target design, per CLAUDE.md §5, not shipped.

• Secrets rotation / least privilege — no rotation tooling found in this repo (expected — that's an operational practice, not application code — but noting it's unverified either way).

• Quality/QA gate on delivered data — src/services/quality-check-service.ts, src/services/quality-report-service.ts, src/types/quality-check.ts, src/types/quality-report.ts, src/data-platform/services/catalog-service.ts, and sql/002_p0_dataset_quality.sql/sql/003_*.sql were at one point referenced in docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and docs/P0_QUERY_SOURCE_FLOW.md as if built and tested. None of these files has ever existed in this repo's git history, on any branch (git log --all confirms zero commits for every one of those paths). All three of those docs have since been corrected (verified this pass) — each now carries its own "fabricated, not built" callout and none currently asserts those files as real; this section restates the underlying fact because it directly bears on the "Security & compliance" claims below: there is no lakehouse-side data-quality/QA enforcement built in this repo today. (The Python engine's services/api/workers/pipeline.py does have a real, working confidence-threshold drop — delivered = [r for r in deduped if r["confidence"] >= threshold] — but that's a separate, narrower mechanism inside the FastAPI pipeline, not the TS/lakehouse quality-check layer those three docs describe.)

Embedding NocoDB / Metabase

components/embed.tsx (EmbeddedView) is real, committed code — a sandboxed <iframe> that renders whatever src URL its caller passes, with a placeholder state when src is null. What's missing: the server-side code that generates a signed Metabase/NocoDB embed URL from METABASE_SECRET_KEY / NOCODB_API_TOKEN. No such generator was found in this pass. So the embed shell exists; the signed-URL plumbing that would make it show real customer data does not yet.

CI/CD

There is no .github/workflows/ directory anywhere in this repo. No lint/typecheck/build/test pipeline runs in CI today. On the TS side, pnpm lint, pnpm typecheck, and pnpm test (vitest, tests/p0/*.test.ts) exist as local scripts in package.json, but nothing invokes them automatically on push/PR. On the Python side there is no lint/test tooling at all — grep-verified this pass: ruff and pytest appear nowhere in services/api/requirements.txt or anywhere else in this repo (no pyproject.toml, no pytest.ini, no test files). services/api/README.md's own "Run locally" section only lists pip install -r requirements.txt + uvicorn/python -m workers.run — no lint or test step. An earlier version of this doc claimed ruff/pytest "exist as local scripts" for services/api; that was unverified and is corrected here. This is a real gap against docs/09-build-plan.md Phase 0's "CI skeleton" deliverable — not yet built, not a doc omission.

Monitoring

• Vercel Analytics — genuinely wired: app/layout.tsx imports and renders <Analytics /> from @vercel/analytics/next (package.json dependency @vercel/analytics@1.3.1).
• Sentry — not found. No sentry package in package.json, no Sentry init/config file anywhere in the repo. This is a documented target only, not shipped.
• Uptime ping on /health — app/api/health/route.ts exists and returns { ok, db, queue, ts } based on env presence checks; no external uptime monitor wired to it was found in this repo (that would live outside this codebase regardless, e.g. an external pinger — unverified either way).
• Worker/job health in admin — app/admin/jobs/page.tsx exists as a page; this pass did not trace whether it reads real job state or placeholder data (out of this doc's grounding scope — flagged for whoever owns docs/01-product.md/admin-portal docs to verify).