0pt/docs/08-infrastructure.md

# 08 · Infrastructure

> Verified against actual repo state 2026-07-19 (branch `main`, plus `.env.example` read from
> `feature/0pt-env-example`). Where this doc previously described a target that doesn't match
> what's committed, that's called out explicitly rather than silently corrected — see the repo's
> honesty convention in `docs/ICEBERG_WRITE_BOUNDARY.md`.

## Repo structure (actual, not the original `apps/web` sketch)

The monorepo did **not** converge on the `apps/web/` + `packages/shared/` layout this doc originally
sketched. What's actually on `main` today:

```
0pt/
├─ CLAUDE.md
├─ README.md · SECURITY.md
├─ docs/                      # this documentation set
├─ app/                       # Next.js App Router (Vercel) — the "edge" plane, AT THE REPO ROOT
│  ├─ api/                    # route handlers: orders, auth, admin, catalog, connections,
│  │                          #   vehicles, subscriptions, sources, estimates, health, uploads,
│  │                          #   payments, 0pt/*, screen, webhooks, ai
│  ├─ bestall/ katalog/ kallor/ order/ portal/ admin/ tjanster/   # pages
├─ lib/                       # edge-side helpers: db.ts, engine.ts, auth.ts, stripe.ts,
│  │                          #   orders-store.ts, subscriptions-store.ts, data.ts, services.ts
├─ src/                       # SIAX-platform-facing TS layer (NOT under app/ or lib/)
│  ├─ clients/                # cl0ud-, n0d-, aud0-, st0re-, st0re-lakehouse-client.ts
│  ├─ services/                # dataset-, source-, source-query-, iceberg-write-,
│  │                          #   document-capture-, document-extraction-, external-writeback-,
│  │                          #   source-rate-policy-, dataset-iceberg-mapping-service.ts
│  ├─ adapters/                # local-blob-store.ts, odoo-bills-adapter.ts, graph-mailbox-adapter, ...
│  └─ types/
├─ components/ hooks/ styles/  # design system in code (docs/05) — data-artifact.tsx,
│  │                          #   provenance-badge.tsx, embed.tsx, use-reveal.ts, globals.css
├─ packages/schemas/           # JSON Schemas for 0pt.* event/entity contracts (not a pnpm workspace
│  │                          #   package today — pnpm-workspace.yaml has no `packages:` list)
├─ services/api/               # FastAPI orchestrator + workers (the "engine" plane, on a VPS)
│  ├─ orchestrator/main.py
│  ├─ workers/                 # adaptive_router.py, pipeline.py, resolve.py, queue.py, run.py
│  ├─ sources/                 # sources.yaml + connectors.py (docs/06)
│  └─ Dockerfile · requirements.txt
├─ sql/                        # migrations — see "Schema state" below
├─ infra/
│  ├─ docker-compose.yml       # real, committed (not a sketch — see below)
│  └─ Caddyfile                # real, committed
├─ tests/p0/                   # vitest, TS-side (services/clients/adapters)
└─ (no .github/workflows/ directory exists — see "CI/CD" below)
```

**Known open question (not resolved this pass, Simon's call):** `CLAUDE.md` and `README.md` both
now note that `github.com/sax3l/0pt` (this checkout's `origin`) and `git.siax.io/sax3l/0pt` (Gitea)
are two remotes that share part of the same history but have independently diverged — Gitea's `main`
picked up a WAF-bypass/proxy-rotation scraper stack (since removed via a Gitea PR) that never touched
GitHub's `main`. Which remote is authoritative/deployed, and whether they should be kept in sync, is
undecided. This doc describes what's on **this** checkout's `main` only.

### Schema state (sql/)

Only two migration files exist in the repo's history, on any branch: `sql/001_init.sql` (the core
schema — `dim_company`, `dim_contact`, `fact_financials`, `orders`, etc., per `docs/03-data-model.md`)
and `sql/004_iceberg_write_boundary.sql` (`opt_dataset_iceberg_mappings` + source-rate-policy tables,
per `docs/ICEBERG_WRITE_BOUNDARY.md`). **There is no `sql/002_*` or `sql/003_*` anywhere in this
repo's git history** (`git log --all` for those paths returns zero commits) — despite
`docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, and `docs/P0_QUERY_SOURCE_FLOW.md`
citing `sql/002_p0_dataset_quality.sql` and a `sql/003_*` as if they were already-applied migrations.
They are not real. See "Security & compliance" below for what this means for QA/quality-check claims.

## Hosting topology

- **Vercel** — `app/` (this repo's root, not a separate `apps/web`) → `0pt.app`. **Verified this pass**
  (network check performed): `https://0pt.app` returns HTTP 200, serves a Next.js/Turbopack app titled
  "0pt — your data" with SIAX branding, and `GET https://0pt.app/api/health` returns
  `{"ok":true,"service":"0pt-web","db":"not_configured","queue":"not_configured",...}` — an exact shape
  match for this repo's own `app/api/health/route.ts`. So README's "Live: https://0pt.app" is confirmed
  accurate, and the live deployment confirms `DATABASE_URL`/`REDIS_URL` are **not** set in production —
  i.e. the live site is running on the in-memory order/subscription stores described throughout this
  doc, not against a real Postgres. This does not confirm which git remote/commit is deployed, or that
  every route behaves as documented — only that the deployment exists and matches this repo's shape.
- **One VPS** (Hetzner, per `docs/02`) — `infra/docker-compose.yml` (see below) runs FastAPI, worker,
  NocoDB, Metabase, Postgres, Redis, Caddy. This compose file and its `Caddyfile` are real, committed
  files — not a sketch — but this pass found no evidence (no CI/CD, no deploy log, no accessible host)
  that they have ever actually been run against a live VPS. Treat as "written, unexecuted."
- **Cloudflare R2** — documented target for file storage. `app/api/uploads/route.ts` checks for
  `R2_ACCESS_KEY_ID`/`R2_BUCKET`/`R2_ENDPOINT` and returns `501 storage_not_configured` when absent;
  when "present" it still returns a **hardcoded `put_url: null`** — there is no actual presigned-URL
  generation implemented yet. R2 upload is an env-guarded stub, not a working integration.
- **Postgres** — `lib/db.ts` resolves `DATABASE_URL || POSTGRES_URL || POSTGRES_PRISMA_URL` (Vercel
  Postgres integration sets the latter two automatically) and falls back to `null` (in-memory stores)
  when none is set. This part is real and wired.

## docker-compose (`infra/docker-compose.yml` — real file, reproduced here for reference)

```yaml
services:
  postgres:
    image: pgvector/pgvector:pg16
    environment: { POSTGRES_USER: opt, POSTGRES_PASSWORD: ${PG_PASSWORD}, POSTGRES_DB: opt }
    volumes: ["pgdata:/var/lib/postgresql/data", "../sql:/docker-entrypoint-initdb.d:ro"]
  redis:
    image: redis:7
  api:
    build: ../services/api
    command: uvicorn orchestrator.main:app --host 0.0.0.0 --port 8000
    environment: { DATABASE_URL: "postgres://opt:${PG_PASSWORD}@postgres:5432/opt", COMPANIES_HOUSE_API_KEY: "${COMPANIES_HOUSE_API_KEY:-}", INTERNAL_TOKEN: "${INTERNAL_TOKEN}" }
    depends_on: [postgres, redis]
  worker:
    build: ../services/api
    command: python -m workers.run
    environment: { DATABASE_URL: "postgres://opt:${PG_PASSWORD}@postgres:5432/opt", COMPANIES_HOUSE_API_KEY: "${COMPANIES_HOUSE_API_KEY:-}" }
    depends_on: [postgres, redis]
  nocodb:
    image: nocodb/nocodb:latest
    environment: { NC_DB: "pg://postgres:5432?u=opt&p=${PG_PASSWORD}&d=opt", NC_PUBLIC_URL: "https://data.0pt.app" }
    depends_on: [postgres]
  metabase:
    image: metabase/metabase:latest
    environment: { MB_DB_TYPE: postgres, MB_DB_HOST: postgres, MB_DB_PORT: "5432", MB_DB_USER: opt, MB_DB_PASS: "${PG_PASSWORD}", MB_DB_DBNAME: opt }
    depends_on: [postgres]
  caddy:
    image: caddy:2
    ports: ["80:80", "443:443"]
    volumes: ["./Caddyfile:/etc/caddy/Caddyfile", "caddydata:/data"]
    depends_on: [api, nocodb, metabase]
volumes: { pgdata: {}, caddydata: {} }
```

Note vs. an earlier draft of this doc: the `api`/`worker` containers here only receive
`DATABASE_URL`, `COMPANIES_HOUSE_API_KEY`, and `INTERNAL_TOKEN` — **not** `STRIPE_*`, `RESEND_API_KEY`,
or `R2_*`. That's correct as written: Stripe, R2, and (a documented but unbuilt) Resend integration
are all called from `app/api/*` route handlers, which run on Vercel, not inside this compose stack.

**`Caddyfile` (real, committed):**
```
api.0pt.app { reverse_proxy api:8000 }
data.0pt.app { reverse_proxy nocodb:8080 }
dash.0pt.app { reverse_proxy metabase:3000 }
```

## Environment variables

**`.env.example` now exists and is merged to `main`** (BACKLOG P3-5: the stale
link to the unmerged `github.com/sax3l/0pt/pull/3` that used to sit here has
been removed — that PR landed a while ago). It was previously impossible to
commit: `.gitignore`'s blanket `.env*` rule silently matched `.env.example`
too, so it was never added despite `docs/09-build-plan.md` Phase 0 requiring
it since day one; `.gitignore` now has `!.env.example` to fix that. README's
"Env inventory: `.env.example`" link is a real, current reference.

The table below is grep-verified against `main` this pass, not transcribed from `.env.example`'s own
comments — in one case (`ST0RE_URL`, see note) that turned up a discrepancy worth flagging.

| Var | Status on `main` | Where |
|---|---|---|
| `DATABASE_URL` / `POSTGRES_URL` / `POSTGRES_PRISMA_URL` | **Wired** — priority-ordered fallback | `lib/db.ts` |
| `ENGINE_URL` | **Wired** — edge defers to FastAPI engine when set, else computes locally | `lib/engine.ts` |
| `INTERNAL_TOKEN` | **Wired** — sent as `X-Internal-Token` on every `engineFetch()` call | `lib/engine.ts` |
| `AUTH_SECRET` | **Wired**, but only for a lightweight HMAC session scaffold — **not Auth.js**. `lib/auth.ts`'s own comment: "Production auth = Auth.js + Postgres adapter... wired once the database is provisioned" | `lib/auth.ts`, `app/api/auth/login/route.ts` |
| `DEV_LOGIN_KEY` | **Wired** — gates the dev-only login route (403 without a match); not a real auth system | `app/api/auth/login/route.ts` |
| `AUTH_URL` | Documented target — no read found | — |
| `STRIPE_SECRET_KEY` | **Wired** — real Stripe Checkout session creation (test or live key both work; not test-mode-locked in code) | `app/api/payments/[order_id]/deposit/route.ts` |
| `STRIPE_WEBHOOK_SECRET` | **Wired** — real signature verification, advances order status | `app/api/webhooks/stripe/route.ts` |
| `NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY` | Documented target — no read found | — |
| `R2_ACCESS_KEY_ID`, `R2_BUCKET`, `R2_ENDPOINT` | **Presence-checked only** — gates a `501` vs stub `200`; no actual R2 client/presign call exists yet | `app/api/uploads/route.ts` |
| `R2_ACCOUNT_ID`, `R2_SECRET_ACCESS_KEY` | Documented target — no read found (even the stub above doesn't check these) | — |
| `OPT_LOCAL_BLOB_DIR` | **Wired** — local-disk fallback artifact store, used when no object storage is configured | `src/adapters/local-blob-store.ts` |
| `RESEND_API_KEY` | Documented target — **no Resend call site anywhere in this repo**; not "unconfigured", genuinely not implemented | — |
| `METABASE_SITE_URL`, `METABASE_SECRET_KEY`, `NOCODB_BASE_URL`, `NOCODB_API_TOKEN` | Documented target — no read found. `components/embed.tsx` renders an iframe from a `src` prop the *caller* must supply; no server-side signed-URL generator exists to produce that `src` | — |
| `DOC_INTELLIGENCE_ENDPOINT`, `DOC_INTELLIGENCE_KEY`, `DOC_INTELLIGENCE_API_VERSION` | **Wired** | `src/services/document-extraction-service.ts` |
| `GRANT_ENC_KEY` | **Presence-checked only** — gates `501` vs a stub `201` that returns `{ grant_id, status: "pending" }`. No AES-GCM encryption, no read-only validation, no actual storage happens yet despite the route's own comment describing that flow | `app/api/connections/db-grant/route.ts` |
| `NEXT_PUBLIC_APP_URL` | **Wired** — used as a fallback origin for building Stripe redirect URLs | `app/api/payments/[order_id]/deposit/route.ts` |
| `NEXT_PUBLIC_API_URL` | Documented target — no read found | — |
| `REDIS_URL` | **Presence-checked only**, in `/api/health` — no queue code in this repo reads it (Celery/queue usage, if any, is `services/api`'s own concern, out of this repo's TS surface) | `app/api/health/route.ts` |
| `COMPANIES_HOUSE_API_KEY` | Real key referenced by `services/api` (Python engine, `sources.yaml`/`connectors.py`) and passed through by `infra/docker-compose.yml` — not part of the Next.js edge app's own env surface | `services/api/` |

### Platform-client URLs — corrected from a stronger claim

`CL0UD_URL`, `N0D_URL`, `AUD0_URL`, and **`ST0RE_URL`** are all **documented targets only — none of
the four is read via `process.env.*` anywhere in this repo's committed code**, grep-verified this
pass (`grep -rn "process\.env\.\(ST0RE_URL\|CL0UD_URL\|N0D_URL\|AUD0_URL\)"` returns zero matches
outside comments and one `it.todo(...)` placeholder in `tests/p0/iceberg-write-service.test.ts`).

This corrects a claim that appears in `.env.example` itself (on `feature/0pt-env-example`) and was
carried into an earlier version of this doc: that comment states "`ST0RE_URL` **IS** read
(`src/clients/st0re-client.ts`, `src/clients/st0re-lakehouse-client.ts`)". That is not accurate as of
this verification. Both of those client files — like `cl0ud-client.ts`, `n0d-client.ts`, and
`aud0-client.ts` — only export a `create*Client(baseUrl?: string)` factory that falls back to an
in-memory mock when no `baseUrl` argument is passed. Every call site in this repo (`app/api/0pt/**`
routes, `src/services/*.ts`) either passes no `baseUrl` at all or forwards an `opts?.st0reBaseUrl`-style
constructor option that nothing upstream ever populates from `process.env.ST0RE_URL`. This matches
`docs/ICEBERG_WRITE_BOUNDARY.md`'s own "Verification status" section, which lists wiring
`ST0RE_URL` → `IcebergWriteService({ st0reBaseUrl: process.env.ST0RE_URL })` as **future work**, not
something already done.

**Net effect:** setting any of `CL0UD_URL`/`N0D_URL`/`AUD0_URL`/`ST0RE_URL` in a deployed environment
today does nothing by itself. A future pass has to wire each route/service to actually read its
corresponding env var before the value has any effect. Until then, every 0pt→platform call in this
repo runs against the deterministic in-memory mock built into each client file — real, tested TS, but
never exercised against a live CL0UD/N0D/AUD0/ST0RE instance from this codebase.

## Security & compliance — corrected against actual code, not the target design

The items below were previously described as already-built behavior. Grep/read-verified this pass,
several are **env-guarded stubs that return a plausible-shaped response without doing the described
work**. Stating this plainly, per this repo's own honesty convention:

- **Customer DB grants** — `app/api/connections/db-grant/route.ts` returns `501` when `GRANT_ENC_KEY`
  is unset, and a stub `201 { grant_id, status: "pending" }` when it is set. **There is no AES-GCM
  encryption, no read-only-role validation, and no persistence implemented yet** — the route's own
  comment describes the target behavior ("validate read-only, AES-GCM encrypt conn_string, store,
  audit") but the code doesn't do it. This is a real gap, not a doc typo.
- **File delivery / presigned R2 URLs** — same pattern: `app/api/uploads/route.ts` is a `501` stub
  with a hardcoded `put_url: null`. No presigned URL is ever actually generated in this repo today.
- **Tenant/org isolation — OPEN SECURITY GAP, not a deliberate two-tier design.** Several `app/api/0pt/*`
  routes read `tenant_id`/`project_id` directly from the request body instead of from a server-bound
  session:
  - `app/api/0pt/lakehouse/mappings/route.ts`
  - `app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts`
  - `app/api/0pt/sources/rate-policies/route.ts`
  - `app/api/0pt/sources/[sourceId]/query/route.ts`
  - `src/services/document-capture-service.ts`

  This is not a defensible internal-vs-external boundary — **it's because no auth/session system
  exists anywhere in this repo yet.** `app/api/orders/route.ts`'s own `GET` handler carries the
  literal comment `// org scoping added with auth` — i.e., not yet. `lib/auth.ts` is a lightweight
  HMAC session scaffold gated by a dev-only login key, unconnected to any of the `0pt/*` routes above.
  Any caller can currently claim any `tenant_id` for these endpoints. Fixing this is Phase 3 of
  `docs/09-build-plan.md` (Auth & portal shell) — it is Simon's call how that auth design binds
  `tenant_id` server-side; it is not built anywhere in this repo today.
- **PII (contacts/UBO) / DPA** — no code found in this pass implementing lawful-basis recording,
  deletion-on-request, or a published DPA. Treat as target design, per `CLAUDE.md` §5, not shipped.
- **Secrets rotation / least privilege** — no rotation tooling found in this repo (expected — that's
  an operational practice, not application code — but noting it's unverified either way).
- **Quality/QA gate on delivered data** — `src/services/quality-check-service.ts`,
  `src/services/quality-report-service.ts`, `src/types/quality-check.ts`, `src/types/quality-report.ts`,
  `src/data-platform/services/catalog-service.ts`, and `sql/002_p0_dataset_quality.sql`/`sql/003_*.sql`
  were at one point referenced in `docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`,
  and `docs/P0_QUERY_SOURCE_FLOW.md` as if built and tested. **None of these files has ever existed in
  this repo's git history, on any branch** (`git log --all` confirms zero commits for every one of
  those paths). **All three of those docs have since been corrected** (verified this pass) — each now
  carries its own "fabricated, not built" callout and none currently asserts those files as real; this
  section restates the underlying fact because it directly bears on the "Security & compliance" claims
  below: **there is no lakehouse-side data-quality/QA enforcement built in this repo today**. (The
  Python engine's `services/api/workers/pipeline.py` does have a real, working confidence-threshold
  drop — `delivered = [r for r in deduped if r["confidence"] >= threshold]` — but that's a separate,
  narrower mechanism inside the FastAPI pipeline, not the TS/lakehouse quality-check layer those three
  docs describe.)

## Embedding NocoDB / Metabase

`components/embed.tsx` (`EmbeddedView`) is real, committed code — a sandboxed `<iframe>` that renders
whatever `src` URL its caller passes, with a placeholder state when `src` is `null`. **What's missing:
the server-side code that generates a signed Metabase/NocoDB embed URL from `METABASE_SECRET_KEY` /
`NOCODB_API_TOKEN`.** No such generator was found in this pass. So the embed *shell* exists; the
signed-URL *plumbing* that would make it show real customer data does not yet.

## CI/CD

**There is no `.github/workflows/` directory anywhere in this repo.** No lint/typecheck/build/test
pipeline runs in CI today. On the TS side, `pnpm lint`, `pnpm typecheck`, and `pnpm test` (vitest,
`tests/p0/*.test.ts`) exist as local scripts in `package.json`, but nothing invokes them automatically
on push/PR. **On the Python side there is no lint/test tooling at all** — grep-verified this pass:
`ruff` and `pytest` appear nowhere in `services/api/requirements.txt` or anywhere else in this repo
(no `pyproject.toml`, no `pytest.ini`, no test files). `services/api/README.md`'s own "Run locally"
section only lists `pip install -r requirements.txt` + `uvicorn`/`python -m workers.run` — no lint or
test step. An earlier version of this doc claimed `ruff`/`pytest` "exist as local scripts" for
`services/api`; that was unverified and is corrected here. This is a real gap against
`docs/09-build-plan.md` Phase 0's "CI skeleton" deliverable — not yet built, not a doc omission.

## Monitoring

- **Vercel Analytics** — genuinely wired: `app/layout.tsx` imports and renders `<Analytics />` from
  `@vercel/analytics/next` (`package.json` dependency `@vercel/analytics@1.3.1`).
- **Sentry** — not found. No `sentry` package in `package.json`, no Sentry init/config file anywhere
  in the repo. This is a documented target only, not shipped.
- **Uptime ping on `/health`** — `app/api/health/route.ts` exists and returns `{ ok, db, queue, ts }`
  based on env presence checks; no external uptime monitor wired to it was found in this repo (that
  would live outside this codebase regardless, e.g. an external pinger — unverified either way).
- **Worker/job health in admin** — `app/admin/jobs/page.tsx` exists as a page; this pass did not trace
  whether it reads real job state or placeholder data (out of this doc's grounding scope — flagged for
  whoever owns `docs/01-product.md`/admin-portal docs to verify).