diff --git a/.env.example b/.env.example index 61e712b..be9397f 100644 --- a/.env.example +++ b/.env.example @@ -1,38 +1,64 @@ # ============================================================================= -# 0pt web console -- example environment +# 0pt (edge plane) -- environment inventory / example environment # ============================================================================= -# Copy to `.env` (gitignored) and fill in real values. Every key below is -# actually referenced by the code (derived from `process.env.*` usage); nothing -# here is invented. All secrets are placeholders -- do NOT commit real values. +# Copy to `.env` (gitignored) and fill in real values. Never commit real values. +# Vercel env for the edge app; VPS .env (git-ignored) for infra/docker-compose.yml. +# See docs/08-infrastructure.md for the canonical description of each integration. # -# The app is designed to build and boot with these UNSET: lib/db.ts and -# lib/stripe.ts are lazy + env-guarded, and the storage/payment/doc routes -# return HTTP 501 ("not configured") until the relevant keys are provided. +# Two provenance markers are used below and they mean different things: +# * "wired in code today" -- read via a `process.env.*` somewhere in +# this repo right now (grep-verified). +# * "documented target, not yet read" -- named in docs/08-infrastructure.md / +# docs/02-architecture.md but with no +# `process.env.*` read yet. Filling it in +# unblocks the integration; it does not by +# itself change behaviour. +# +# The app is designed to build and boot with all of these UNSET: lib/db.ts and +# lib/stripe.ts are lazy + env-guarded, and the storage/payment/doc routes return +# HTTP 501 ("not configured") until the relevant keys are provided. -# ---- Core ------------------------------------------------------------------- +# --------------------------------------------------------------------------- +# Core / public +# --------------------------------------------------------------------------- # Public base URL of this app (used for absolute links / callbacks). NEXT_PUBLIC_APP_URL=http://localhost:3000 +# Documented target (docs/08); no call site found reading it in this pass. +NEXT_PUBLIC_API_URL=https://api.0pt.app -# ---- Database (Postgres) ---------------------------------------------------- -# First one set wins: DATABASE_URL -> POSTGRES_URL -> POSTGRES_PRISMA_URL. +# --------------------------------------------------------------------------- +# Database (Postgres) -- wired in code today: lib/db.ts, app/api/health +# --------------------------------------------------------------------------- +# Vercel Postgres sets POSTGRES_URL / POSTGRES_PRISMA_URL automatically when the +# integration is attached. First one set wins, in this order (lib/db.ts): +# DATABASE_URL -> POSTGRES_URL -> POSTGRES_PRISMA_URL. DATABASE_URL=postgres://user:password@localhost:5432/opt POSTGRES_URL= POSTGRES_PRISMA_URL= -# ---- Queue / cache (Redis) -------------------------------------------------- +# --------------------------------------------------------------------------- +# Cache / queue (Redis) -- app/api/health only checks presence; no other call +# site found reading it in this pass (worker queue usage, if any, lives in +# services/api, outside this repo's scope). +# --------------------------------------------------------------------------- REDIS_URL=redis://localhost:6379 -# ---- Auth ------------------------------------------------------------------- +# --------------------------------------------------------------------------- +# Auth -- session / dev-login / service-token secrets (wired in code today) +# --------------------------------------------------------------------------- # Session-signing secret (generate a long random string). AUTH_SECRET=change-me-long-random-string -# Optional dev-only login shortcut key (leave unset in production). +# AUTH_URL is documented (docs/08) but not yet read by any route in this pass. +AUTH_URL=https://0pt.app +# Optional dev-only login shortcut key (leave UNSET in production). DEV_LOGIN_KEY= -# Encryption key for stored DB-connection grants (/api/connections/db-grant). -GRANT_ENC_KEY=change-me-32-byte-key -# Shared token for internal/service-to-service calls. +# Shared token for internal/service-to-service calls; sent as X-Internal-Token +# on every engineFetch() call (lib/engine.ts). INTERNAL_TOKEN=change-me-internal-token -# ---- Zitadel / OIDC Bearer verification (Wave 2 hardening) ------------------ +# --------------------------------------------------------------------------- +# Zitadel / OIDC Bearer verification (Wave 2 hardening) -- wired in code today +# --------------------------------------------------------------------------- # 0pt verifies inbound Bearer tokens against the fleet Zitadel IdP # (src/auth/zitadel.ts, wired via middleware.ts). JWKS are discovered from # /.well-known/openid-configuration, falling back to /oauth/v2/keys. @@ -51,30 +77,98 @@ AUTH_ENFORCE=true # urn:zitadel:iam:org:id -> urn:zitadel:iam:user:resourceowner:id -> tenant_id -> tenant. ZITADEL_TENANT_CLAIM= -# ---- Payments (Stripe) ------------------------------------------------------ +# --------------------------------------------------------------------------- +# Data engine bridge -- wired in code today: lib/engine.ts +# --------------------------------------------------------------------------- +# When set, edge route handlers defer to the FastAPI engine (services/api) as the +# single source of truth (orders/estimates/catalog/ai-parse). When unset, the edge +# computes locally / falls back to an honest heuristic -- see lib/engine.ts and +# app/api/ai/parse. +ENGINE_URL=http://localhost:8000 + +# --------------------------------------------------------------------------- +# SIAX platform clients +# --------------------------------------------------------------------------- +# src/clients/{cl0ud,n0d,aud0}-client.ts currently only accept an optional +# constructor baseUrl and fall back to an in-memory mock / no-op -- none of these +# three vars is read via process.env yet (grep-verified). Setting them does +# nothing until those clients are wired to read them; a future pass should add +# that. See docs/ICEBERG_WRITE_BOUNDARY.md. +CL0UD_URL= +N0D_URL= +AUD0_URL= +# ST0RE_URL is the convention name for src/clients/st0re-client.ts / +# st0re-lakehouse-client.ts and is read by the real-infra test suite +# (tests/p0/*.realinfra.test.ts). The lakehouse write path was live-verified +# against ST0RE on 2026-07-13 with this set; unset, the clients fall back to +# their deterministic in-memory mocks. +ST0RE_URL= + +# --------------------------------------------------------------------------- +# Payments (Stripe) -- wired in code today: app/api/payments/*, +# app/api/webhooks/stripe +# --------------------------------------------------------------------------- STRIPE_SECRET_KEY=sk_test_xxx STRIPE_WEBHOOK_SECRET=whsec_xxx +# Documented target (docs/08); not yet read by any route in this pass. +NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY= -# ---- Object storage (Cloudflare R2 / S3-compatible) ------------------------- -# All three required to enable /api/uploads; otherwise it returns 501. +# --------------------------------------------------------------------------- +# Object storage -- Cloudflare R2 / S3-compatible +# --------------------------------------------------------------------------- +# R2_ENDPOINT / R2_BUCKET / R2_ACCESS_KEY_ID are read in code today; all three +# are required to enable /api/uploads, otherwise it returns 501. +# R2_ACCOUNT_ID and R2_SECRET_ACCESS_KEY are documented in docs/08 but were NOT +# found via a process.env.* read in this pass -- verify before assuming the R2 +# client is fully wired. R2_ENDPOINT=https://.r2.cloudflarestorage.com R2_BUCKET=opt-uploads R2_ACCESS_KEY_ID=change-me -# Optional local blob dir fallback used when R2 is not configured. +R2_ACCOUNT_ID= +R2_SECRET_ACCESS_KEY= +# Fallback local artifact store used when no object storage is configured +# (src/adapters/local-blob-store.ts). Leave unset in production. OPT_LOCAL_BLOB_DIR=./.opt-blobs -# ---- Document intelligence (Azure) ------------------------------------------ -# Powers document extraction routes; unset => extraction "not configured". +# --------------------------------------------------------------------------- +# Document intelligence / OCR (Azure) -- wired in code today: +# src/services/document-extraction-service.ts. Unset => extraction +# "not configured". +# --------------------------------------------------------------------------- DOC_INTELLIGENCE_ENDPOINT=https://.cognitiveservices.azure.com DOC_INTELLIGENCE_KEY=change-me DOC_INTELLIGENCE_API_VERSION=2024-11-30 -# ---- Engine / integrations -------------------------------------------------- -# URL of the 0pt data engine (services/api orchestrator). -ENGINE_URL=http://localhost:8000 +# --------------------------------------------------------------------------- +# Customer DB-grant encryption -- wired in code today: +# /api/connections/db-grant +# --------------------------------------------------------------------------- +# 32-byte key, AES-GCM. Encrypts customer-supplied read-only connection strings +# at rest. +GRANT_ENC_KEY=change-me-32-byte-key + +# --------------------------------------------------------------------------- +# External write-back targets +# --------------------------------------------------------------------------- # Odoo base URL for the bills/write-back adapter. ODOO_BASE_URL=https:// -# ---- Test-only (tests/p0 real-infra suite; not needed to run the app) -------- -# ST0RE_URL= +# --------------------------------------------------------------------------- +# Email -- Resend (documented target, docs/08; no Resend call site found in +# this pass -- treat as not-yet-implemented, not just unconfigured) +# --------------------------------------------------------------------------- +RESEND_API_KEY= + +# --------------------------------------------------------------------------- +# Embedded analytics (documented target, docs/08; no call site found in this +# pass) +# --------------------------------------------------------------------------- +METABASE_SITE_URL= +METABASE_SECRET_KEY= +NOCODB_BASE_URL= +NOCODB_API_TOKEN= + +# --------------------------------------------------------------------------- +# Test-only (tests/p0 real-infra suite; not needed to run the app) +# --------------------------------------------------------------------------- # ST0RE_TEST_TENANT_ID= diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..73a27fd --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,59 @@ +name: CI + +# Quality gate: typecheck + test on every PR and non-main push. +# 0pt previously relied solely on the Vercel build with no pre-merge gate. +# Build is intentionally NOT run here because it needs runtime secrets +# (POSTGRES_URL, ENGINE_URL, STRIPE, RESEND) — Vercel owns the build; CI owns +# the static gate. +# +# Updated by the GitHub<->Gitea unification merge (2026-07-28). The note that +# used to sit here — "lint is not a step because this repo has no eslint +# dependency or config wired" — is no longer true: the Gitea line's Wave 2 +# hardening added eslint + typescript-eslint + a flat config and repointed the +# "lint" script at `eslint src tests`. Lint now runs, ADVISORY +# (continue-on-error), matching .gitea/workflows/ci.yml: the v0-generated +# legacy tree carries pre-existing lint noise, so it must not block merges. +# +# `typecheck` (BLOCKING) is tsc over tsconfig.check.json = src/ + tests/. +# `typecheck:app` (ADVISORY) is the whole-repo tsc this workflow used to run as +# its blocking step. It was green on the GitHub line alone but reports 9 +# pre-existing errors on the unified tree (app/api/0pt/* route handlers, where +# the Wave 2 Zod schemas type fields as plain `string` while the services want +# narrower unions). It is kept running, and kept visible, rather than deleted — +# see package.json's "//scripts" note. Flip it back to blocking once those 9 +# are cleared. +# +# Python engine (services/api) is gated separately if/when a Python workflow +# is added. +on: + pull_request: + push: + branches-ignore: [main] + +concurrency: + group: ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + gate: + name: Typecheck + test + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '22' + - name: Enable corepack (pnpm) + run: corepack enable + - name: Install (frozen lockfile) + run: pnpm install --frozen-lockfile + - name: Typecheck (tsc over src/ + tests/ -- blocking) + run: pnpm run typecheck + - name: Test (vitest -- blocking) + run: pnpm run test + - name: Typecheck whole repo (advisory -- 9 pre-existing app/ errors) + continue-on-error: true + run: pnpm run typecheck:app + - name: Lint (eslint -- advisory; legacy tree has pre-existing noise) + continue-on-error: true + run: pnpm run lint diff --git a/.gitignore b/.gitignore index 963d8ee..fc6a65c 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,7 @@ yarn-error.log* # env files .env* +!.env.example # vercel .vercel diff --git a/CLAUDE.md b/CLAUDE.md index 10b8e42..f454df6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,7 +16,7 @@ Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, Veh 1. **Legal sourcing only.** The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect `robots.txt` + rate limits) → licensed partner data → otherwise the field is delivered as **"not available / licensed only"**. We never invent values. - **Do NOT build, keep, or extend**: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.). - - If existing code (e.g. `src/workers/browser_worker.py`) contains a FlareSolverr/bypass path, **remove or disable it** and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat. + - **Path corrected this pass:** no `src/workers/browser_worker.py` exists anywhere in this checkout (verified via full-repo search) — the engine's actual workers live under `services/api/workers/` (`adaptive_router.py`, `pipeline.py`, `queue.py`, `resolve.py`, `run.py`), none of which contain a FlareSolverr/bypass path today. The precedent for this concern is real and, **as of this GitHub<->Gitea unification merge, it is in this tree**: Gitea's `git.siax.io/sax3l/0pt` `main` independently accumulated a WAF-bypass/proxy-rotation/fake-identity scraper stack (`services/scrapers/*` — `curl_cffi`, `cloudscraper`, Tor circuit rotation) in three 2026-07-13 commits, and this branch inherits it because it branches from Gitea `main`. **The deletion PR (`feature/0pt-remove-noncompliant-scrapers`) is open, NOT merged** — an earlier version of this paragraph claimed it was "since removed", which was false. That removal is deliberately left to its own PR and was *not* folded into this merge; nothing in this merge resurrects, extends, or re-wires any of it. If new code (from either remote) reintroduces a bypass path anywhere, **remove or disable it** and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat. 2. **Postgres is the backbone.** One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the *same* DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth. 3. **Provenance per field.** Every fact carries source, freshness, confidence (`bridge_provenance`). This drives the UI signature too. 4. **One tool per job.** Do not add overlapping tools. The approved stack is in `docs/02-architecture.md` and `docs/07-dependencies.md`. Anything not listed needs a documented reason. @@ -25,7 +25,11 @@ Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, Veh ## Build on existing code -A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestrator/main.py`, 25+ endpoints), workers (`src/workers/`: adaptive_router, browser_worker, ocr_worker, db), and country data for 20 markets. **Reconcile with it, don't blindly replace.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing. +A working engine repo already exists. **Corrected this pass — the previous description of it was wrong on every specific:** the FastAPI orchestrator lives at `services/api/orchestrator/main.py` (not `src/orchestrator/main.py`) and registers **12** route handlers (not "25+"); the workers directory is `services/api/workers/` and contains `adaptive_router.py`, `pipeline.py`, `queue.py`, `resolve.py`, and `run.py` (not `browser_worker`/`ocr_worker`/`db` — none of those three files exist anywhere in this repo); and `services/api/sources/sources.yaml` currently covers **7** markets (`SE`, `NO`, `DK`, `FI`, `GB`, `NL`, plus `global`), not 20. `services/api/` holds 11 Python source files total (excluding `__pycache__`). **Reconcile with the code as it actually is, not this description's prior claims.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing. + +**Two remotes claim to be this repo, and it's unresolved which is authoritative.** `origin` here is `github.com/sax3l/0pt`. A separate Gitea remote, `git.siax.io/sax3l/0pt`, shares at least part of the same history but has independently diverged (see README "Known issues" for specifics). Do not assume `origin` is the only copy of this codebase, and do not silently resolve the question of which is canonical — that is Simon's decision. + +**Before trusting any status claim in this doc corpus, read README's "Known issues" section.** It covers the auth/tenant-scoping gap, the platform-client mock-only status, and the underlying fact that 0pt's quality-check/QA enforcement code was never built despite once being described as real. **Corrected this pass:** `docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, and `docs/P0_QUERY_SOURCE_FLOW.md` are now all consistently corrected — each carries its own "fabricated, not built" callout and none asserts `quality-check-service.ts`/`quality-report-service.ts`/etc. as real or tested anywhere in its current body (verified this pass by reading all three in full). `docs/P0_DATASET_QUALITY_FLOW.md` remains the explicit target-spec replacement for what that stage should look like once built. Treat any *other* doc claiming the QA stage already exists as wrong. ## How to work @@ -49,11 +53,14 @@ A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestra | `docs/07-dependencies.md` | Repos/libraries to import (+ explicit exclusions) | | `docs/08-infrastructure.md` | Hosting, docker-compose, env, security, CI/CD | | `docs/09-build-plan.md` | Phased milestones with acceptance criteria | -| `docs/design/0pt-v0-build-brief.md` | Per-component v0 prompts (design reference) | -| `docs/design/0pt-design-coverage-and-visuals.md` | Full page/component inventory + visual system | -| `docs/10-lakehouse-decision.md` | Pointer: Apache Iceberg = canonical SIAX lakehouse format (ADR-0030, `n0d` repo); 0pt = write-path owner | -| `docs/11-lakehouse-write-plan.md` | 0pt's concrete lakehouse write to-do list, schema/event names for a future implementation session | -| `docs/ICEBERG_WRITE_BOUNDARY.md` | Implementation record: dataset<->Iceberg-table mapping + source-rate-policy contracts, ST0RE lakehouse client, live-verified write path (real namespace/table/snapshot IDs), what's real vs. contract-only | +| `docs/10-lakehouse-decision.md` | Pointer to an ecosystem-wide lakehouse ADR made outside this repo (Apache Iceberg = canonical SIAX lakehouse format, ADR-0030 in the `n0d` repo; 0pt = write-path owner). Does not change `docs/02`/`docs/03` | +| `docs/11-lakehouse-write-plan.md` | 0pt's own follow-up task list for the Iceberg write boundary — schema/event names for the implementation. **Corrected**: item 4 (quality rules) is explicitly labeled NOT BUILT, and the old fabricated `quality-check-service.ts`/`quality-report-service.ts` claim is quoted only inside its own "Correction" callout, never asserted as fact | +| `docs/12-edge-capability-api.md` | Edge-native TypeScript P0 capability API contract (`app/api/0pt/*`, `/api/screen`, `/api/ai/parse`) — distinct from `docs/04-api.md`'s FastAPI-engine contract; documents several confirmed always-fails-over-HTTP routes and the tenant_id/in-memory gaps | +| `docs/ICEBERG_WRITE_BOUNDARY.md` | Implementation record + "real vs. contract-only" verification status: dataset<->Iceberg-table mapping and source-rate-policy contracts, ST0RE lakehouse client, and the 2026-07-13 **live-verified** write path (real namespace/table/snapshot IDs). **Corrected**: its "Data quality: not yet built" section explicitly quotes and retracts the old fabricated claim rather than repeating it as fact | +| `docs/P0_QUERY_SOURCE_FLOW.md` | `0pt.query_source` bounded source-query flow — **corrected**: no longer claims `quality-check-service.ts` exists, now points at the real precedent (`iceberg-write-service.ts`) instead | +| `docs/P0_DATASET_QUALITY_FLOW.md` | An explicit **target-spec-only, not-yet-built** replacement for the fabricated quality-check/QA claims — read it as "should", never "does" | + +**Corrected:** this table previously listed `docs/design/0pt-v0-build-brief.md` and `docs/design/0pt-design-coverage-and-visuals.md` as if they existed. They do not — confirmed via a full-repo search, including after this unification merge; there is no `docs/design/` directory at all. No `docs/` design-brief content is fabricated here to fill the gap. Either write those two docs or stop referencing them; until one of those happens, treat any prior claim that they exist as false. ## Definition of done (every screen, before real data) diff --git a/SECURITY.md b/SECURITY.md index 0fd7efc..b5a4bb7 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -42,9 +42,37 @@ Out of scope: - Automated scanning that generates excessive load against production systems without prior coordination. -Some known findings (for example authentication gaps on specific internal -API routes) are already tracked internally and pending an intentional, -reviewed fix; reporting them is still welcome and helps confirm priority. +Note: `github.com/sax3l/0pt` (this checkout's `origin`) and +`git.siax.io/sax3l/0pt` (Gitea) are two diverged remotes that share part of +this codebase's history — see README "Known issues" for what's known and +what isn't about the relationship between them. Which one is deployed to +0pt.app is currently unconfirmed; if your finding is remote-specific, say +which remote/branch you tested against. + +## Known, already-tracked gaps + +The following are **not** news to us — they are open, named gaps as of this +pass, not secrets, and reporting them again mainly helps confirm priority +rather than surfacing something new: + +- **No authentication or tenant-isolation system exists in this repo yet.** + The only gate in front of any route is a `DEV_LOGIN_KEY` dev-login scaffold + (`app/api/auth/login/route.ts`) — there is no session-to-tenant binding. + `app/api/orders/route.ts`'s `GET` handler carries the literal comment + `// GET /api/orders — list (org scoping added with auth).`, confirming this + directly. As a consequence, the P0 data-platform routes under `/api/0pt/*` + (`lakehouse/mappings`, `sources/rate-policies`, `sources/[sourceId]/query`) + and `src/services/document-capture-service.ts` accept `tenant_id`/ + `project_id` **from the client-supplied request body or query string** + rather than from a server-bound session — a violation of this platform's + own tenant-isolation rule (tenant_id must never be trusted from client + input). This is an open, prioritized gap pending real auth (Phase 3, + `docs/09-build-plan.md`), not an intentional design. Do not treat this repo + as tenant-isolated until that lands. +- Beyond that specific item, other authentication gaps on internal API + routes may still exist and are tracked internally pending an intentional, + reviewed fix; reporting any you find is still welcome and helps confirm + priority. ## Disclosure diff --git a/app/admin/page.tsx b/app/admin/page.tsx index 540423f..370497c 100644 --- a/app/admin/page.tsx +++ b/app/admin/page.tsx @@ -42,10 +42,13 @@ export default function AdminPage() { const load = useCallback(async () => { setError(false); try { - const [o, m] = await Promise.all([ - fetch("/api/orders", { cache: "no-store" }).then((r) => r.json()), - fetch("/api/admin/metrics", { cache: "no-store" }).then((r) => r.json()), + const [oRes, mRes] = await Promise.all([ + fetch("/api/orders", { cache: "no-store" }), + fetch("/api/admin/metrics", { cache: "no-store" }), ]); + if (!oRes.ok || !mRes.ok) throw new Error("fetch failed"); + const [o, m] = await Promise.all([oRes.json(), mRes.json()]); + if (!m || typeof m.total !== "number") throw new Error("bad metrics shape"); setOrders(mergeOrders(Array.isArray(o) ? o : []) as Order[]); setMetrics(m); } catch { diff --git a/app/api/0pt/documents/[documentId]/extraction/route.ts b/app/api/0pt/documents/[documentId]/extraction/route.ts index 2d6a9b1..17d8573 100644 --- a/app/api/0pt/documents/[documentId]/extraction/route.ts +++ b/app/api/0pt/documents/[documentId]/extraction/route.ts @@ -21,7 +21,17 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{ return NextResponse.json(result, { status: 201 }); } catch (error: unknown) { const message = error instanceof Error ? error.message : "Unknown error"; - return NextResponse.json({ error: message }, { status: 400 }); + if (/not configured/i.test(message)) { + return NextResponse.json({ error: message }, { status: 501 }); + } + if (/^Policy denied/i.test(message)) { + return NextResponse.json({ error: message }, { status: 403 }); + } + if (/^Policy requires approval/i.test(message)) { + return NextResponse.json({ error: message }, { status: 409 }); + } + const status = /not found/i.test(message) ? 404 : 400; + return NextResponse.json({ error: message }, { status }); } } diff --git a/app/api/0pt/lakehouse/mappings/[mappingId]/route.ts b/app/api/0pt/lakehouse/mappings/[mappingId]/route.ts index 5e76ca2..6010236 100644 --- a/app/api/0pt/lakehouse/mappings/[mappingId]/route.ts +++ b/app/api/0pt/lakehouse/mappings/[mappingId]/route.ts @@ -1,7 +1,5 @@ import { NextResponse } from "next/server"; -import { DatasetIcebergMappingService } from "@/src/services/dataset-iceberg-mapping-service"; - -const mappingService = new DatasetIcebergMappingService(); +import { mappingService } from "@/src/services/lakehouse-mapping-registry"; export async function GET(_request: Request, { params }: { params: Promise<{ mappingId: string }> }) { try { diff --git a/app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts b/app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts index 2113bc7..e6815b9 100644 --- a/app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts +++ b/app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts @@ -1,11 +1,18 @@ import { NextRequest, NextResponse } from "next/server"; import { parseBody, requestIcebergWriteSchema } from "@/src/validation/0pt-schemas"; -import { IcebergWriteService } from "@/src/services/iceberg-write-service"; - -const writeService = new IcebergWriteService(); +import { writeService } from "@/src/services/lakehouse-mapping-registry"; // Registers a new Iceberg snapshot (a write) through ST0RE's table registry // for an already-mapped dataset. See docs/ICEBERG_WRITE_BOUNDARY.md. +// +// `writeService` is the SHARED instance from +// src/services/lakehouse-mapping-registry.ts -- this route used to construct +// its own `new IcebergWriteService()`, which held a private in-memory mapping +// registry, so a mapping created via POST /lakehouse/mappings was never +// visible here and every write failed with "mapping not found". +// +// The body is validated against requestIcebergWriteSchema before it reaches +// the service (Wave 2 hardening). export async function POST(request: NextRequest, { params }: { params: Promise<{ mappingId: string }> }) { try { const { mappingId } = await params; diff --git a/app/api/0pt/lakehouse/mappings/route.ts b/app/api/0pt/lakehouse/mappings/route.ts index c217cd1..b601175 100644 --- a/app/api/0pt/lakehouse/mappings/route.ts +++ b/app/api/0pt/lakehouse/mappings/route.ts @@ -1,20 +1,25 @@ import { NextRequest, NextResponse } from "next/server"; import { parseBody, mapDatasetToIcebergTableSchema } from "@/src/validation/0pt-schemas"; -import { IcebergWriteService } from "@/src/services/iceberg-write-service"; -import { DatasetIcebergMappingService } from "@/src/services/dataset-iceberg-mapping-service"; +import { mappingService, writeService } from "@/src/services/lakehouse-mapping-registry"; -// Shared instances so a mapping created via POST is visible to a later GET -// within the same server process, matching the in-memory-per-module -// convention already used by app/api/0pt/datasets and app/api/0pt/quality -// (Postgres persistence exists in sql/ but is not wired into route handlers -// in this pass -- same limitation those routes already have). -const mappingService = new DatasetIcebergMappingService(); -const writeService = new IcebergWriteService({ mappingService }); +// Shared instances come from src/services/lakehouse-mapping-registry.ts so a +// mapping created via POST is visible to a later GET, AND to the sibling +// /{mappingId}/writes route, within the same server process. (Each route file +// previously constructed its own DatasetIcebergMappingService/ +// IcebergWriteService, so a mapping created here was invisible to the writes +// route -- every write failed with "mapping not found".) This still matches the +// in-memory-per-module convention app/api/0pt/datasets and app/api/0pt/quality +// use; Postgres persistence exists in sql/ but is not wired into route handlers +// in this pass -- same limitation those routes already have. // POST registers a NEW Iceberg table in ST0RE's table registry (via // IcebergWriteService.mapDatasetToIcebergTable) and records the // dataset_iceberg_mapping row that points at it -- this is 0pt's write-path // entrypoint, see docs/ICEBERG_WRITE_BOUNDARY.md. +// +// The body is validated against mapDatasetToIcebergTableSchema BEFORE it +// reaches the service (Wave 2 hardening): an unvalidated request body used to +// be passed straight into the capability call. export async function POST(request: NextRequest) { try { const __raw = await request.json(); diff --git a/app/api/0pt/sources/[sourceId]/query/route.ts b/app/api/0pt/sources/[sourceId]/query/route.ts index 038098f..9d97140 100644 --- a/app/api/0pt/sources/[sourceId]/query/route.ts +++ b/app/api/0pt/sources/[sourceId]/query/route.ts @@ -10,8 +10,13 @@ * is not shared across route modules yet — see the "Singleton note" in * src/services/document-capture-service.ts for prior art on this same repo * limitation). A source registered via POST /api/0pt/sources will not be - * resolvable here until SourceService is backed by shared persistence - * (sql/002_p0_dataset_quality.sql already defines `opt_sources` for that). + * resolvable here until SourceService is backed by shared persistence. + * Correction: this comment previously claimed `sql/002_p0_dataset_quality.sql` + * already defined an `opt_sources` table for that. That file does not exist + * and never has (git log --all: zero commits, any branch, ever). sql/ today + * has only 001_init.sql and 004_iceberg_write_boundary.sql. The real fix — + * backing SourceService with real persistence — is still open; no migration + * for it exists yet. * Details: docs/P0_QUERY_SOURCE_FLOW.md. */ import { NextRequest, NextResponse } from "next/server"; diff --git a/app/api/ai/parse/route.ts b/app/api/ai/parse/route.ts index e4f94e6..aebfdec 100644 --- a/app/api/ai/parse/route.ts +++ b/app/api/ai/parse/route.ts @@ -24,7 +24,9 @@ const DEST: [RegExp, string][] = [ ]; export async function POST(req: NextRequest) { - const { text } = await req.json().catch(() => ({ text: "" })); + const parsed = await req.json().catch(() => ({ text: "" })); + const body = parsed && typeof parsed === "object" ? parsed : { text: "" }; + const { text } = body as { text?: string }; if (!text) return NextResponse.json({ error: { code: "bad_request", message: "text krävs" } }, { status: 400 }); const eng = await engineFetch("/ai/parse", { method: "POST", body: JSON.stringify({ text }) }); diff --git a/app/api/auth/login/route.ts b/app/api/auth/login/route.ts index 7893d4b..8e1929e 100644 --- a/app/api/auth/login/route.ts +++ b/app/api/auth/login/route.ts @@ -1,3 +1,4 @@ +import { timingSafeEqual } from "node:crypto"; import { NextRequest, NextResponse } from "next/server"; import { signSession, SESSION_COOKIE } from "@/lib/auth"; @@ -9,7 +10,13 @@ export async function POST(req: NextRequest) { return NextResponse.json({ error: { code: "auth_not_configured", message: "AUTH_SECRET saknas" } }, { status: 501 }); } const devKey = process.env.DEV_LOGIN_KEY; - if (!devKey || key !== devKey) { + // Constant-time compare: a plain !== leaks the number of matching leading + // bytes via response timing, letting an attacker recover DEV_LOGIN_KEY + // byte-by-byte instead of it being an all-or-nothing guess. + const keyBuf = Buffer.from(typeof key === "string" ? key : ""); + const devKeyBuf = Buffer.from(devKey ?? ""); + const keysMatch = Boolean(devKey) && keyBuf.length === devKeyBuf.length && timingSafeEqual(keyBuf, devKeyBuf); + if (!keysMatch) { return NextResponse.json({ error: { code: "forbidden", message: "Dev-login kräver giltig nyckel" } }, { status: 403 }); } if (!email) { diff --git a/app/api/estimates/route.ts b/app/api/estimates/route.ts index d2709fa..0028688 100644 --- a/app/api/estimates/route.ts +++ b/app/api/estimates/route.ts @@ -11,6 +11,9 @@ export async function POST(req: NextRequest) { } catch { return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 }); } + if (!body || typeof body !== "object") { + return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 }); + } // Defer to the FastAPI data engine when it's running; fall back to local math. const eng = await engineFetch("/estimates", { method: "POST", body: JSON.stringify(body) }); diff --git a/app/api/orders/[id]/route.ts b/app/api/orders/[id]/route.ts index b59c063..2d6a248 100644 --- a/app/api/orders/[id]/route.ts +++ b/app/api/orders/[id]/route.ts @@ -19,6 +19,9 @@ export async function PATCH(req: NextRequest, ctx: { params: Promise<{ id: strin } catch { return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON" } }, { status: 400 }); } + if (!body || typeof body !== "object") { + return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 }); + } const allowed = ["draft", "quoted", "deposit_pending", "in_production", "review", "revision", "accepted", "delivered"]; if (!body.status || !allowed.includes(body.status)) { return NextResponse.json({ error: { code: "bad_status", message: "Ogiltig status" } }, { status: 400 }); diff --git a/app/api/orders/route.ts b/app/api/orders/route.ts index c0c1183..a4cf033 100644 --- a/app/api/orders/route.ts +++ b/app/api/orders/route.ts @@ -9,6 +9,9 @@ export async function POST(req: NextRequest) { } catch { return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON" } }, { status: 400 }); } + if (!body || typeof body !== "object") { + return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 }); + } const flow = body.flow || "custom"; const config = body.config || {}; const order = await createOrder(flow, config); diff --git a/app/api/payments/[order_id]/deposit/route.ts b/app/api/payments/[order_id]/deposit/route.ts index 5d9843e..b732790 100644 --- a/app/api/payments/[order_id]/deposit/route.ts +++ b/app/api/payments/[order_id]/deposit/route.ts @@ -1,4 +1,5 @@ import { NextRequest, NextResponse } from "next/server"; +import { getOrder } from "@/lib/orders-store"; // POST /api/payments/[order_id]/deposit // Returns { url } for Stripe Checkout. Returns 501 when STRIPE_SECRET_KEY is absent. @@ -13,19 +14,15 @@ export async function POST( return NextResponse.json({ error: "payment_not_configured" }, { status: 501 }); } - const origin = - request.headers.get("origin") ?? - process.env.NEXT_PUBLIC_APP_URL ?? - "http://localhost:3000"; + // Never derive from the client-supplied Origin header (SSRF + open-redirect vector) — + // request.nextUrl.origin reflects the actual request URL, not attacker-controlled input. + const origin = request.nextUrl.origin; - let depositOre = 44750; - try { - const orderRes = await fetch(`${origin}/api/orders/${order_id}`); - if (orderRes.ok) { - const order = (await orderRes.json()) as { deposit_ore?: number; total_ore?: number }; - depositOre = order.deposit_ore ?? Math.round((order.total_ore ?? 89500) / 2); - } - } catch {} + const order = await getOrder(order_id); + if (!order) { + return NextResponse.json({ error: { code: "not_found", message: "Order saknas" } }, { status: 404 }); + } + const depositOre = order.deposit_ore ?? Math.round((order.total_ore ?? 89500) / 2); const body = new URLSearchParams({ "payment_method_types[0]": "card", diff --git a/app/api/subscriptions/route.ts b/app/api/subscriptions/route.ts index ffab6f4..2846da1 100644 --- a/app/api/subscriptions/route.ts +++ b/app/api/subscriptions/route.ts @@ -7,7 +7,8 @@ export async function GET() { } export async function POST(req: NextRequest) { - const body = await req.json().catch(() => ({})); + const parsed = await req.json().catch(() => ({})); + const body = parsed && typeof parsed === "object" ? parsed : {}; if (!body.order_id) { return NextResponse.json({ error: { code: "bad_request", message: "order_id krävs" } }, { status: 400 }); } diff --git a/app/api/uploads/route.ts b/app/api/uploads/route.ts index e960d21..ce533f8 100644 --- a/app/api/uploads/route.ts +++ b/app/api/uploads/route.ts @@ -1,4 +1,4 @@ -import { NextRequest, NextResponse } from "next/server"; +import { NextResponse } from "next/server"; // GET /api/uploads — list uploaded files. Returns 501 until storage is configured. export async function GET() { @@ -10,9 +10,10 @@ export async function GET() { } // POST /api/uploads (docs/04) — issue a presigned R2 PUT URL. Env-guarded: -// returns 501 until R2 storage is configured. Client then PUTs the file directly. -export async function POST(req: NextRequest) { - const body = await req.json().catch(() => ({})); +// returns 501 until R2 storage is configured, and until presign generation +// is implemented — a configured-but-unimplemented state must not report +// success. Client then PUTs the file directly. +export async function POST() { const haveR2 = process.env.R2_ACCESS_KEY_ID && process.env.R2_BUCKET && process.env.R2_ENDPOINT; if (!haveR2) { return NextResponse.json( @@ -20,6 +21,10 @@ export async function POST(req: NextRequest) { { status: 501 }, ); } - // With R2 configured: build a presigned PUT (S3-compatible) and return it. - return NextResponse.json({ file_id: crypto.randomUUID(), put_url: null, name: body?.name ?? null }); + // TODO: presigned PUT generation not implemented yet — keep returning 501 + // instead of a fake 200/null success shape until real R2 presign wiring exists. + return NextResponse.json( + { error: { code: "storage_not_configured", message: "Presignerad uppladdning är inte implementerad än" } }, + { status: 501 }, + ); } diff --git a/app/api/vehicles/route.ts b/app/api/vehicles/route.ts index ee0dc1a..be3d78e 100644 --- a/app/api/vehicles/route.ts +++ b/app/api/vehicles/route.ts @@ -4,14 +4,14 @@ import { engineFetch } from "@/lib/engine"; // GET /api/vehicles?plate=XX-99-YY — RDW (NL) open vehicle data (technical/fleet // only, no owner PII). Defers to the engine when running; else calls RDW directly. export async function GET(req: NextRequest) { - const plate = (req.nextUrl.searchParams.get("plate") ?? "").toUpperCase().replace(/[-\s]/g, ""); + const plate = (req.nextUrl.searchParams.get("plate") ?? "").toUpperCase().replace(/[-\s]/g, "").replace(/[^A-Z0-9]/g, ""); if (!plate) return NextResponse.json({ error: { code: "bad_request", message: "plate krävs" } }, { status: 400 }); const eng = await engineFetch(`/vehicles/${plate}`); if (eng && eng.ok) return NextResponse.json(await eng.json()); try { - const r = await fetch(`https://opendata.rdw.nl/resource/m9d7-ebf2.json?kenteken=${plate}`, { + const r = await fetch(`https://opendata.rdw.nl/resource/m9d7-ebf2.json?kenteken=${encodeURIComponent(plate)}`, { headers: { "User-Agent": "0pt/0.1 (+https://0pt.app; SIAX)" }, cache: "no-store", }); diff --git a/app/order/[id]/page.tsx b/app/order/[id]/page.tsx index 29cdcb2..bdc0b12 100644 --- a/app/order/[id]/page.tsx +++ b/app/order/[id]/page.tsx @@ -65,6 +65,10 @@ export default function ProductionPage() { }, [id]); const finish = () => { + if (timer.current) { + clearInterval(timer.current); + timer.current = null; + } setProgress([100, 100, 100, 100, 100]); setMarkets([100, 100, 100, 100]); setConf(94); diff --git a/docs/01-product.md b/docs/01-product.md index 5274344..4b62110 100644 --- a/docs/01-product.md +++ b/docs/01-product.md @@ -6,7 +6,9 @@ ## The three flows (everything reduces to these) -Every service maps to one of three intake/delivery flows. The nine "categories" (scraping, structuring, datasets, ICP/leads, BI, ML, documents, subscriptions, custom) are **tags/filters within these flows**, not separate machinery. +Every service maps to one of three intake/delivery flows (`Flow` in `lib/services.ts`: `catalog` | `custom` | `subscription`). + +*Correction: this section previously said "nine categories (scraping, structuring, datasets, ICP/leads, BI, ML, documents, subscriptions, custom)" — that list doesn't match `lib/services.ts`'s actual catalog and never did (no `structuring`, `datasets`, `subscriptions`, or `custom` service id exists there). The real catalog today (`lib/services.ts`, `SERVICES`) has 11 entries: data delivery, web scraping, mapping & matching, normalization, consolidation → CRM/ERP, documents & OCR, enrichment, ICP & leads, ML jobs, BI & market analysis, feeds & monitoring.* These are **tags/filters within the three flows**, not separate machinery — each entry carries a `flow` field pointing back to `catalog` / `custom` / `subscription`. 1. **Catalog product** — ready-made, browse → sample → buy → deliver. Exists in Postgres or produced fast from the pipeline. Margin + scale. 2. **Custom job** — does not exist yet; must be scoped. `configure (AI-assist) → estimate → manual scope/quote confirm → produce → QA → validate → deliver`. The hero flow. Consulting in a portal. @@ -29,12 +31,14 @@ DRAFT → QUOTED → DEPOSIT_PENDING → IN_PRODUCTION → REVIEW - **DELIVERED**: exports + destinations unlocked; provenance log + quality report attached. - **SUBSCRIPTION_ACTIVE**: if converted to a feed. +This is real, working code today, not just a spec: `app/api/orders/[id]/route.ts`'s `PATCH` handler enforces exactly this sequence (lowercase: `draft`, `quoted`, `deposit_pending`, `in_production`, `review`, `revision`, `accepted`, `delivered`) as its allowed-status list. One gap: `SUBSCRIPTION_ACTIVE` is **not** in that allowed list yet — feed conversion (Phase 7, `docs/09-build-plan.md`) isn't wired into the order status machine. + ## Business / escrow model - **50 / 50.** Deposit (50 %) captured to start production. Balance (50 %) **held** and only captured/released on **ACCEPTED**. Revision never moves the held amount. - Pricing models: pay-per-job, volume, subscription (monthly), custom quote. - Estimate is a function of: selected field weights × markets × volume × entity multiplier × confidence-target multiplier + enrichment. Custom jobs get a manual scope/quote gate before DEPOSIT_PENDING. -- Stripe: deposit = capture now; balance = either a held authorization released on accept, or a second invoice on accept (implement as a second Checkout/invoice keyed to the order; see `docs/04-api.md`). +- Stripe: deposit = capture now (`app/api/payments/[order_id]/deposit/route.ts`); balance = **a second Stripe Checkout session created on accept**, keyed to the order (`app/api/payments/[order_id]/release/route.ts`). *Correction: earlier drafts of this doc left the balance mechanism open ("either a held authorization … or a second invoice"). The second-Checkout-session approach is what's actually implemented — there is no held-PaymentIntent-capture path in the code.* Both routes are env-guarded on `STRIPE_SECRET_KEY` and return `501` until it's configured (`lib/stripe.ts`). ## Customer DB access (designed, not a checkbox) @@ -46,4 +50,4 @@ SaaS-Martin (RevOps, wants clean ICP/leads), Bilhandlar-Bengt (vehicle + company ## First-launch scope (de-risked) -Do **not** ship "20 markets × 9 categories." Ship the thinnest sellable slice: **Nordic company data** (the open stuff — GLEIF + Brreg + verified sources) + **vehicle data** (VehIQ synergy) as a couple of catalog datasets, **plus the custom-job intake**, Stripe, and delivery to download + into FiveCRM/Data Loader. First customers are Protosell internally + existing dealer relationships — services you already deliver, productized. Then expand markets as the sourcing map fills in. +Do **not** ship "20 markets × 11 categories." Ship the thinnest sellable slice: **Nordic company data** (the open stuff — GLEIF + Brreg + verified sources) + **vehicle data** (VehIQ synergy) as a couple of catalog datasets, **plus the custom-job intake**, Stripe, and delivery to download + into FiveCRM/Data Loader. First customers are Protosell internally + existing dealer relationships — services you already deliver, productized. Then expand markets as the sourcing map fills in. diff --git a/docs/02-architecture.md b/docs/02-architecture.md index 9fd1da1..278e3ff 100644 --- a/docs/02-architecture.md +++ b/docs/02-architecture.md @@ -1,9 +1,20 @@ # 02 · Architecture +> **Known issues affecting this doc, in brief** — full detail + evidence in [`README.md`](../README.md)'s +> "Known issues" section: (1) no auth/session system exists yet, so several newer routes trust +> client-supplied `tenant_id` (see "Known gaps" below); (2) the CL0UD/N0D/AUD0/ST0RE platform clients are +> real code but mock-only in this repo; (3) `docs/ICEBERG_WRITE_BOUNDARY.md` / `docs/11-lakehouse-write-plan.md` +> / `docs/P0_QUERY_SOURCE_FLOW.md` reference several `src/services/quality-*` and `sql/002`/`sql/003` files +> that were never actually committed to this repo (out of this doc's file scope to fix — flagged here so it +> isn't silently repeated); (4) two git remotes (`github.com/sax3l/0pt` and `git.siax.io/sax3l/0pt`) have +> diverged and it is unresolved which is authoritative — Simon's call, not assumed here. + ## Principle: one database, many lenses Postgres is the source of truth. Everything else is a **lens** on it. A tool enters the stack only if it (a) reads that Postgres and (b) sits behind the same auth. This is the difference between an app and a pile of separate logins. +This principle describes — and is largely realized by — the **original edge + engine plane**: marketing, catalog, order lifecycle, and the FastAPI engine. It does **not** describe the newer **P0 data-platform surface** (`app/api/0pt/*`), which is real, additive, and deliberately does not go through this same path. See "Two API surfaces" below. + ``` ┌──────────────────────────────────────────────┐ COLLECTION │ POSTGRES │ LENSES (customer + admin) @@ -16,37 +27,128 @@ Postgres is the source of truth. Everything else is a **lens** on it. A tool ent Customer upload│ │ Export: CSV/Parquet/JSON/API/FiveCRM └───────────────────▲──────────────────────────┘ │ - FastAPI orchestrator + workers + services/api — FastAPI orchestrator + workers (estimates, orders, jobs, AI-assist, webhooks, pipeline) ``` ## Components & responsibilities -- **Next.js (Vercel)** — all customer/admin UI: marketing site, catalog, configurator, checkout, production tracker, validation, portal, admin. Calls FastAPI over an internal token. Embeds NocoDB/Metabase via signed URLs (no separate login shown). -- **FastAPI orchestrator** (`src/orchestrator`) — REST contract in `docs/04-api.md`: estimates, orders, AI-assist, uploads, deliveries, webhooks, catalog, sources, job status/SSE. **Not on Vercel** (serverless timeouts kill long jobs). -- **Workers** (`src/workers`) — adaptive router + pipeline stages (sourcing → extraction/OCR → structuring → enrichment/fusion → QA). Write to Postgres with provenance. See `docs/06-data-engine.md`. -- **Postgres** — backbone (`docs/03-data-model.md`) incl. `pgvector` for matching/embeddings. -- **NocoDB** — Airtable-style views over existing tables (no data duplication). Embedded into portal as "Min data". -- **Metabase** — customer-facing dashboards. Embedded. -- **Cloudflare R2** — file storage (uploads + deliverables) via presigned URLs. -- **Stripe** — deposit capture + escrow balance release. -- **Resend** — transactional email (order confirmation, "din data är klar", feedback). -- **Auth.js** (NextAuth) with a Postgres adapter — single auth for the whole app; NocoDB/Metabase hidden behind it. +- **Next.js (Vercel)** — all customer/admin UI: marketing site, catalog, configurator, checkout, production tracker, validation, portal, admin. It also hosts **two distinct families** of API route handlers under `app/api/`: the original order/catalog/estimate surface (some of which defer to the FastAPI engine), and the newer P0 data-platform surface under `app/api/0pt/*` (does not defer to the engine at all — see below). +- **FastAPI orchestrator** (`services/api/orchestrator/main.py`) — REST contract in `docs/04-api.md`: estimates, orders, sources, vehicle lookup, sanctions screening, job run/status. **Not on Vercel** (serverless timeouts kill long jobs). *Correction: earlier drafts of this doc pointed at `src/orchestrator`; that was the pre-migration location. `docs/09-build-plan.md` Phase 0 called for moving it into `services/api`, and that move has happened — the code lives at `services/api/orchestrator` today.* +- **Workers** (`services/api/workers/`) — adaptive router (`adaptive_router.py`, verified: chooses only among compliant methods — official API → bulk → authorized web → licensed, no bypass/stealth/proxy-rotation tier exists in it, by design) + pipeline stages (`pipeline.py`, `queue.py`, `resolve.py`, `run.py`). See `docs/06-data-engine.md`. +- **Postgres** — backbone (`docs/03-data-model.md`) incl. `pgvector` for matching/embeddings, for the original edge + engine plane. Wired via `lib/db.ts` (env-guarded: `DATABASE_URL`/`POSTGRES_URL`; falls back to an in-memory, non-durable store when unset, e.g. `lib/orders-store.ts`). +- **NocoDB / Metabase** — both are defined as services in `infra/docker-compose.yml`; embedding them into the portal as "Min data" / customer dashboards is the target, not yet confirmed wired into a portal page in this pass. +- **Cloudflare R2** — file storage (uploads + deliverables) via presigned URLs. Env-guarded (`app/api/uploads/route.ts` checks `R2_ACCESS_KEY_ID`/`R2_BUCKET`/`R2_ENDPOINT` and returns `501` until set); the presign call itself is still a stub (`put_url: null`) even when the env vars are present — not wired end-to-end yet. +- **Stripe** — deposit capture + escrow balance release. This one **is** wired: `app/api/payments/[order_id]/deposit/route.ts` and `.../release/route.ts` are real Stripe Checkout calls (env-guarded on `STRIPE_SECRET_KEY`, `501` until configured). See the correction in `docs/01-product.md`'s business model section for exactly which escrow mechanism is implemented. +- **Resend** — documented as the transactional-email provider (`docs/08-infrastructure.md`) but has no call site anywhere in this repo yet. Treat "+ email" as not-yet-implemented, not merely unconfigured. +- **Auth.js (NextAuth) with a Postgres adapter** — this is the **target**, listed here as a stack decision, not a built component. What exists today is a lightweight signed-session scaffold (`lib/auth.ts`: HMAC-SHA256 via Web Crypto, no external deps) behind a guarded dev-only login gate (`DEV_LOGIN_KEY`). There is **no** real session-to-org/tenant binding anywhere in this repo. Full Auth.js + OAuth is Phase 3 of `docs/09-build-plan.md` and has not been built. This matters directly for the tenant-scoping gap described below. + +## Two API surfaces + +This repo currently contains **two real, distinct API surfaces**, not one. Earlier versions of this doc described the edge as a uniform thin proxy to the FastAPI engine — that was true for part of the original surface and remains true for that part, but a second, newer surface was added that is a deliberate, ADR-documented exception to it. Conflating the two is the single biggest way to misread this codebase's architecture. + +### 1. The original edge → engine surface + +A subset of the original `app/api/*` route handlers defer to the FastAPI engine (`services/api`) via `lib/engine.ts`'s `engineFetch()`/`hasEngine()` bridge when `ENGINE_URL` is set, and fall back to local computation when it isn't: + +- `app/api/estimates/route.ts` — defers to `POST {ENGINE_URL}/estimates`, falls back to `lib/data.ts`'s `computeEstimate`. +- `app/api/vehicles/route.ts`, `app/api/screen/route.ts`, `app/api/ai/parse/route.ts` — same pattern. + +Other original routes are **not** engine-backed at all today: `app/api/orders/*` reads/writes `lib/orders-store.ts` directly (Postgres when configured, else in-memory); `app/api/catalog/route.ts` returns a hardcoded static array. Both of these are entirely edge-local — there is no engine call in either path currently, not "thin proxy with a local fallback." + +This surface is documented in full in `docs/04-api.md` and matches the "one Postgres, many lenses" principle above (once `DATABASE_URL` is set). + +### 2. The P0 data-platform surface (`app/api/0pt/*`) + +A second, newer surface implements dataset registration, Iceberg lakehouse table mapping/writes, per-source rate-policy management, bounded synchronous source querying, and mailbox-based document capture — **directly in the edge app**, backed by `src/services/*` and `src/clients/*`. It does **not** call `lib/engine.ts`/`ENGINE_URL` and does **not** go through the FastAPI engine at all. This is deliberate: it is the concrete implementation of **ADR-0030** (Apache Iceberg as SIAX's canonical lakehouse table format, authored in the `n0d` repo) and the ownership split it defines. See `docs/10-lakehouse-decision.md` (the pointer to the ecosystem decision), `docs/11-lakehouse-write-plan.md` (the plan), and `docs/ICEBERG_WRITE_BOUNDARY.md` (the implementation record) for the full boundary — this doc only summarizes it. + +Real, existing routes today (verified against the filesystem, not against `app/api/0pt/route.ts`'s own advertised list — see the note below): + +| Route | Backed by | +|---|---| +| `GET /api/0pt` | static index (see note below) | +| `POST/GET /api/0pt/documents/capture` | `src/services/document-capture-service.ts` | +| `.../documents/{documentId}/extraction` | `src/services/document-extraction-service.ts` | +| `POST/GET /api/0pt/lakehouse/mappings` | `src/services/dataset-iceberg-mapping-service.ts`, `iceberg-write-service.ts` | +| `.../lakehouse/mappings/{mappingId}` | same | +| `POST .../lakehouse/mappings/{mappingId}/writes` | `src/services/iceberg-write-service.ts#requestIcebergWrite` | +| `POST/GET /api/0pt/sources/rate-policies` | `src/services/source-rate-policy-service.ts` | +| `.../sources/rate-policies/{sourceRatePolicyId}` | same | +| `POST /api/0pt/sources/{sourceId}/query` | `src/services/source-query-service.ts` (bounded, timeout-guarded ad-hoc SQL against a registered `database` source — see `docs/P0_QUERY_SOURCE_FLOW.md`) | + +**Code-level inconsistency worth knowing about**: `app/api/0pt/route.ts`'s own `GET` handler advertises an `endpoints` list including `/api/0pt/sources`, `/api/0pt/datasets`, `/api/0pt/data-contracts`, `/api/0pt/quality/checks`, `/api/0pt/quality/reports`, `/api/0pt/features/views` — **none of these exist as routes** (confirmed by listing `app/api/0pt/` on disk); the lakehouse-mapping and rate-policy routes that *do* exist aren't listed there either. This is the same "described but never built" pattern as the `quality-check-service.ts` fabrication flagged below — this doc's route table above reflects what is actually on disk, not what the index endpoint claims. + +**In-memory scope**: every P0 service (`DatasetService`, `SourceService`, `DatasetIcebergMappingService`, `SourceRatePolicyService`, `DocumentCaptureService`, …) holds its state in a plain `Map` inside the Next.js route module that instantiates it. Most route modules construct a fresh instance per module, so a record created via one route is not visible to a different route module in the same running process (each file's own header comment documents this; `document-capture-service.ts` worked around it for its own two routes with a shared singleton export). A Postgres migration exists (`sql/004_iceberg_write_boundary.sql`) but is not wired into any route handler yet. So the "one database, many lenses" principle above does not yet hold for this surface. + +## Six-surface capability model & 0pt's current coverage + +SIAX Cloud's target model (defined in `cl0ud`, `docs/architecture/SIAX_CLOUD_UNIFIED_ARCHITECTURE.md`) is that one capability can expose up to **six surfaces**: REST **api**, **cli** (`siax 0pt ...`), **mcp** (an agent tool), **ui** (a page/panel), **job** (an async N0D job type), and **sdk** (a generated client method). The capability registry's own `Surface` type (`cl0ud` `packages/capability-registry/src/types.ts`) currently encodes five of those six as a declarative field (`'api' | 'cli' | 'mcp' | 'ui' | 'job'`); `sdk` is generated separately, not yet a registry field. + +**0pt's actual coverage today** (verified against `cl0ud packages/capability-registry/src/registry.ts`): 14 distinct `0pt.*` capabilities are registered — + +`0pt.query_source`, `0pt.document_ocr`, `0pt.write_external_record`, `0pt.lakehouse_mapping_manage`, `0pt.lakehouse_write`, `0pt.source_rate_policy_manage`, `0pt.screen_sanctions`, `0pt.estimate_compute`, `0pt.vehicle_lookup`, `0pt.catalog_list`, `0pt.source_registry_list`, `0pt.order_create`, `0pt.order_payment`, `0pt.write_iceberg_table` + +— and, checked entry by entry: + +- **11** declare `surfaces: ['api']` only. +- **2** (`0pt.lakehouse_write`, `0pt.write_iceberg_table`) declare `surfaces: ['api', 'job']`. +- **1** (`0pt.write_external_record`) declares `surfaces: ['job']` **only** — `apiEnabled: false`, and its own comment states it has "NO dedicated HTTP route in the 0pt app yet"; it exists as a tested TS service (`src/services/external-writeback-service.ts`) invoked only via direct import today, not a live callable path. +- **Every single one** has `cliEnabled: false` and `mcpEnabled: false`. None has a `ui` surface entry either. + +In plain terms: **0pt is API-only today** (plus two job-backed writes, and one job-only capability with no live route at all). If any doc — inside or outside this repo — describes a live `siax 0pt ...` CLI command or an `0pt.*` MCP tool, it is describing a target, not current reality; correct it on sight. + +## Platform clients: CL0UD / N0D / AUD0 / ST0RE + +`src/clients/{cl0ud,n0d,aud0,st0re,st0re-lakehouse}-client.ts` are real, typechecked TypeScript with accompanying tests (`tests/p0/*`), each following the same shape: a thin interface, an HTTP implementation written against the target service's real, committed route contract, and a deterministic in-memory mock used whenever no `baseUrl` is supplied to the constructor. + +**Verified this pass**: none of the five client files, and none of their call sites in `src/services/*`, read `CL0UD_URL`, `N0D_URL`, `AUD0_URL`, or `ST0RE_URL` from `process.env` anywhere (`grep -rn "process.env" src/clients src/services` turns up nothing for any of those four names outside comments/TODOs). Every constructor call in this repo (`createCl0udClient(opts?.cl0udBaseUrl)`, etc.) is invoked with no `baseUrl` argument, so **every one of these clients runs against its in-memory mock today, in every code path that exists in this repo.** Read this as "real code, mock-only, not live-verified" — not "integrated" or "connected" — wherever platform-client status comes up. + +What each client is for, concretely: + +- **CL0UD** — policy evaluation (`evaluatePolicy`). `iceberg-write-service.ts` and `external-writeback-service.ts` call it before acting; the mock always returns `"allow"`. +- **N0D** — job requests (`createJob`). `document-capture-service.ts` requests document-extraction jobs; `iceberg-write-service.ts` requests heavy-write/compaction jobs. 0pt only *requests* N0D jobs — it never executes them, per `docs/ICEBERG_WRITE_BOUNDARY.md`'s "N0D executes" boundary. **No evidence supports a claim that N0D has a dedicated-node ("`t_0pt`") auth pattern already built for this**: `n0d`'s repository was searched in full (all branches, all commits, all files) and contains zero references to `t_0pt`. Treat any such claim as unconfirmed until proven otherwise. +- **AUD0** — audit/lineage events (`ingestEvent`, `ingestLineageEvent`). Every mutating P0 service emits these; the mock is a true no-op, matching AUD0's own documented stance that it has no live ingestion endpoint yet. +- **ST0RE / ST0RE-lakehouse** — artifact-metadata registration and Iceberg table/namespace/snapshot registration, respectively. `st0re-lakehouse-client.ts`'s HTTP path is written against `sax3l/st0re`'s actual shipped route contract (commit `9ea0abb`, read directly during a prior pass) — real code, but not executed against a running ST0RE in this repo's test environment either. + +## Known gaps + +### tenant_id is not server-bound anywhere in the P0 surface + +The platform-wide rule is that `tenant_id` must be bound to an authenticated session server-side and never trusted from client input. This repo violates that rule today, and it is **not** a deliberate two-tier design — it's a direct consequence of there being no auth/session system at all yet (see the Auth.js note above; `app/api/orders/route.ts`'s own `GET` handler literally comments `// org scoping added with auth`, i.e. not yet). + +Concretely: `tenant_id`/`project_id` are accepted straight from the client-supplied request body (or query string) at every P0 mutation entry point — + +- `app/api/0pt/lakehouse/mappings/route.ts` (POST) +- `app/api/0pt/sources/rate-policies/route.ts` (POST) +- `app/api/0pt/sources/[sourceId]/query/route.ts` (POST) +- `app/api/0pt/documents/capture/route.ts` → `src/services/document-capture-service.ts#captureFromMailbox` + +`app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts` doesn't read `tenant_id` from its own body — but it resolves the tenant by looking up `mapping.tenant_id` from a mapping that was itself created via one of the unauthenticated routes above, so the trust boundary problem is inherited, not avoided. + +Do not expose any of these routes to untrusted callers before Phase 3 of `docs/09-build-plan.md` (Auth.js + org/session binding) lands. This is Simon's call on sequencing, not something to silently patch in a docs pass. + +### 0pt's data-quality/QA stage is a fabrication in sibling docs, not built + +Flagged so it is not repeated: `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`, `src/types/quality-check.ts`, `src/types/quality-report.ts`, `src/data-platform/services/catalog-service.ts`, `sql/002_p0_dataset_quality.sql`, and `sql/003_*.sql` were at one point referenced in `docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, and `docs/P0_QUERY_SOURCE_FLOW.md` as if already built and tested. Verified this pass: `git log --all` for every one of those paths returns zero commits, on any branch, ever — `sql/` on disk today contains only `001_init.sql` and `004_iceberg_write_boundary.sql`. Those files have never existed in this repository. 0pt's data-quality/QA enforcement is **spec/target only** — a real, open build gap, not a doc typo. **All three of the docs above have since been corrected** (verified this pass by reading each in full) — none currently asserts those files as real; each carries its own explicit "fabricated, not built" callout, and `docs/P0_DATASET_QUALITY_FLOW.md` is the honest target-spec replacement. + +### Two diverged git remotes + +`origin` in local checkouts of this repo is `github.com/sax3l/0pt`. A separate Gitea remote, `git.siax.io/sax3l/0pt`, shares at least part of the same history (`services/api/workers/adaptive_router.py` is byte-for-byte identical between GitHub's current `main` and a Gitea post-fix PR branch) but independently accumulated commits GitHub's `main` never received — including, at one point, a WAF-bypass/proxy-rotation/fake-identity scraper stack that directly violated the CLAUDE.md §1 sourcing rule (since removed on Gitea via a separate PR). **Which remote is authoritative, which is actually deployed to `0pt.app`, and whether the two should be kept in sync is unresolved and is Simon's decision** — do not assume `origin`/GitHub is the sole copy of this codebase, here or in any other doc. ## Stack decisions — IN | Concern | Choice | Why | |---|---|---| | Frontend | Next.js App Router + TS + Tailwind + shadcn/ui | Vercel-native, matches existing skills | -| Backend/jobs | FastAPI + workers (existing) | already built; long-running jobs | +| Backend/jobs | FastAPI + workers (`services/api`) | already built; long-running jobs | | DB | Postgres (+ pgvector) | backbone; Neon free tier or on-VPS | | Airtable layer | **NocoDB** | sits on existing tables, no duplication | | BI | **Metabase** | customer-facing embedding, simple | | Files | Cloudflare R2 + presigned URLs | cheapest S3-compatible | -| Payments | Stripe Checkout + webhook → FastAPI | ~1 day, no SaaS boilerplate needed | -| Auth | Auth.js + Postgres adapter | one system | -| Email | Resend | best Next.js integration, free tier | -| Quality | Great Expectations / checks **inside workers** | scores shipped with each order | +| Payments | Stripe Checkout + webhook → Next.js API | ~1 day, no SaaS boilerplate needed | +| Auth | Auth.js + Postgres adapter | one system — **target, not yet built** (see Components above) | +| Email | Resend | best Next.js integration, free tier — **not yet wired** (no call site in this repo) | +| Quality | Great Expectations / checks **inside workers** | scores shipped with each order — **not yet built** for the P0 surface (see "Known gaps" above) | ## Stack decisions — OUT (do not add) @@ -58,13 +160,13 @@ Postgres is the source of truth. Everything else is a **lens** on it. A tool ent | Meilisearch (v1) | Postgres full-text suffices to a few thousand rows; add later | | Delta Sharing (v1) | unnecessary; customer pull via scoped role + exports | | Trigger.dev | the existing queue already schedules jobs; Resend covers mail | -| FlareSolverr / stealth browsers / undetected-* / proxy-fingerprint rotation / fake-account creation | **prohibited** — see CLAUDE.md §1. Replace with compliant fetching | +| FlareSolverr / stealth browsers / undetected-* / proxy-fingerprint rotation / fake-account creation | **prohibited** — see CLAUDE.md §1. Replace with compliant fetching. (The Gitea-only scraper stack described under "Two diverged git remotes" above is exactly this — removed there, never present on GitHub `main`.) | ## Hosting (cheapest coherent) - **Vercel** (Hobby/free) — Next.js + custom domain `0pt.app`. -- **One small VPS** (e.g. Hetzner ~€5/mo) running `docker-compose`: FastAPI, workers, NocoDB, Metabase, (Postgres or use Neon), optional Meilisearch later. Per-service managed platforms (Railway) get expensive once you have 4–5 always-on services; a single VPS + Vercel + R2 lands ≈ <€10/mo to start. -- **Cloudflare R2** — object storage. +- **One small VPS** (e.g. Hetzner ~€5/mo) running `docker-compose` (`infra/docker-compose.yml`): FastAPI, workers, NocoDB, Metabase, (Postgres or use Neon), optional Meilisearch later. Per-service managed platforms (Railway) get expensive once you have 4–5 always-on services; a single VPS + Vercel + R2 lands ≈ <€10/mo to start. +- **Cloudflare R2** — object storage (env-guarded, presign not yet wired — see Components above). - **Postgres** — on the VPS, or **Neon** free tier if you want it managed. -Full config, env, security, CI/CD: `docs/08-infrastructure.md`. +Full config, env, security, CI/CD: `docs/08-infrastructure.md`. Env var inventory: `.env.example` exists on branch/PR [`feature/0pt-env-example`](https://github.com/sax3l/0pt/pull/3) (unmerged as of this pass) — until it merges to `main`, `docs/08-infrastructure.md` is the source of truth for env var names. diff --git a/docs/03-data-model.md b/docs/03-data-model.md index 25c3da5..361a7d3 100644 --- a/docs/03-data-model.md +++ b/docs/03-data-model.md @@ -1,6 +1,10 @@ # 03 · Data model -Postgres is the backbone. Two halves: **data entities** (what we sell) and **platform tables** (orders, users, jobs). Every fact links to `bridge_provenance`. Migrations via Alembic (Python) or `sql/` migrations checked into the repo — pick one and be consistent; align with existing `src/workers/db.py`. +Postgres is the backbone. Two halves: **data entities** (what we sell) and **platform tables** (orders, users, jobs). Every fact links to `bridge_provenance`. + +**Migrations are plain SQL files in `sql/`, applied in filename order.** There is no Alembic (or any other Python migration tool) anywhere in this repo, and no `src/workers/db.py` — that file does not exist (the Python engine's workers live in `services/api/workers/` — `adaptive_router.py`, `pipeline.py`, `queue.py`, `resolve.py`, `run.py` — none of them named `db.py`, and there is no TypeScript `src/workers/` directory either; `src/` only has `adapters/`, `clients/`, `services/`, `types/`). `infra/docker-compose.yml` mounts `../sql` as `/docker-entrypoint-initdb.d` on the `postgres` service, so Postgres auto-applies every `.sql` file in that directory, in name order, the first time that container initializes its data volume. + +**Only two migration files exist in this repo, ever:** `sql/001_init.sql` and `sql/004_iceberg_write_boundary.sql` (verified via `find sql/` — no `002` or `003`). Three other docs in this repo (`docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, `docs/P0_QUERY_SOURCE_FLOW.md`) mention `sql/002_p0_dataset_quality.sql` and `sql/003_*.sql` by name — they don't exist, on any branch, ever (git history checked) — but all three have been corrected to say exactly that (each carries its own "never existed" callout, none claims the files are real as of this pass). This file only asserts what's true of `sql/001` and `sql/004`, which are real. ## Conventions @@ -11,6 +15,8 @@ Postgres is the backbone. Two halves: **data entities** (what we sell) and **pla ## Data entities +Mirrors `sql/001_init.sql` (verified against the file this pass; the migration additionally wraps every statement in `CREATE TABLE IF NOT EXISTS` for idempotent re-runs, and enables `pgcrypto` alongside `vector` — needed for `gen_random_uuid()`): + ```sql -- A company (the core registry entity) CREATE TABLE dim_company ( @@ -98,8 +104,12 @@ CREATE INDEX ON bridge_provenance (entity_type, entity_key); > Rule: **no value ships without a `bridge_provenance` row.** Level 0 = delivered as "not available / licensed", never a guessed value. +`dim_source` is a **different, older entity** than the per-tenant `Source` registry described below (`src/types/source.ts`) that `0pt.query_source` acts against — same name-ish concept, two separate schemas. `src/types/source.ts`'s own doc comment calls this out, but its "~9-row list of national registries served today by `GET /api/sources`" phrasing is easy to misread: `dim_source` itself, as a Postgres table, has **zero rows** — `sql/001_init.sql` only creates the table, there is no seed/`INSERT` statement anywhere in this repo's SQL or Python (grep-verified for `INSERT INTO dim_source`). The actual 9-entry national-registry list that comment is pointing at lives in a third, distinct artifact: the edge-native `GET /api/sources` route (`app/api/sources/route.ts`), a hardcoded array with a `confidence` field. That's different again from the FastAPI engine's `sources.yaml` (8 entries, no `confidence` field — see `docs/04-api.md`'s Sources section). So there are three separate "source list" artifacts in this repo — the empty `dim_source` SQL table, `sources.yaml` (8), and `app/api/sources/route.ts` (9) — none of which currently share data with each other; see "TypeScript-side P0 entities" below for the fourth, per-tenant `Source` type. + ## Platform tables +Mirrors `sql/001_init.sql` (verified this pass — same tables, same columns): + ```sql CREATE TABLE orgs ( org_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), @@ -191,11 +201,71 @@ CREATE TABLE payments ( ); ``` +> **Gap, verified this pass:** `sql/001_init.sql` does **not** contain `order_items`. It is documented here (and was already documented before this pass) as part of the target platform-table set, but there is no `CREATE TABLE order_items` in the migration file — check before relying on it existing in any real database. + +> **Gap, verified this pass:** this schema is real and (if `infra/docker-compose.yml`'s `postgres` service is actually run) gets created — but the FastAPI engine's own code never connects to it. `services/api/orchestrator/main.py` has no `asyncpg`/`psycopg`/`sqlalchemy` import anywhere (grep-verified across `services/api/`), and its `orders`/`jobs` state lives entirely in two module-level Python dicts (`_ORDERS`, `_JOBS`) that reset on every process restart and are never written to Postgres. The one `DATABASE_URL` reference in `main.py` (`GET /health`) only reports whether the env var is *set* — it never opens a connection. See `docs/04-api.md`'s implementation-status section for the full picture. Until that wiring lands, this table set is schema-only, not a live system of record. + +## P0 registry tables (`sql/004_iceberg_write_boundary.sql`) + +A second, newer migration adds two tables for the lakehouse write-path and crawl-compliance capabilities (`0pt.write_iceberg_table`, `0pt.source_rate_policy_manage` — see the new edge-API doc, `docs/12-edge-capability-api.md`). Verified against the file this pass: + +```sql +CREATE TABLE opt_dataset_iceberg_mappings ( + dataset_iceberg_mapping_id TEXT PRIMARY KEY, + tenant_id TEXT NOT NULL, + project_id TEXT NOT NULL, + dataset_id TEXT NOT NULL REFERENCES opt_datasets (dataset_id), + iceberg_table_id TEXT NOT NULL, -- reference into ST0RE's table registry, NOT a foreign key + namespace TEXT, + table_name TEXT, + mapping_status TEXT NOT NULL DEFAULT 'active', + mapped_by TEXT, + correlation_id TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ +); + +CREATE TABLE opt_source_rate_policies ( + source_rate_policy_id TEXT PRIMARY KEY, + tenant_id TEXT NOT NULL, + project_id TEXT NOT NULL, + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + rate_limit_per_minute NUMERIC NOT NULL, + crawl_budget NUMERIC NOT NULL, + compliance_notes TEXT, + domain TEXT, + robots_ok BOOLEAN, + access TEXT, + status TEXT NOT NULL DEFAULT 'active', + correlation_id TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ, + created_by TEXT +); +``` + +`iceberg_table_id` is deliberately **not** a foreign key — it references ST0RE's own `st0re_iceberg_tables.table_id`, a different service's database. 0pt holds only the reference, never a copy of ST0RE's table/catalog/snapshot state (see the module doc comment on `src/clients/st0re-lakehouse-client.ts`, and `docs/ICEBERG_WRITE_BOUNDARY.md` for the full ownership boundary — that doc is outside this file's scope to otherwise correct). + +> **Confirmed bug, found this pass — flagging plainly, not silently fixing it:** both `CREATE TABLE` statements above have a `REFERENCES` clause pointing at tables that **do not exist anywhere in this repo's SQL**: `opt_dataset_iceberg_mappings.dataset_id REFERENCES opt_datasets (dataset_id)` and `opt_source_rate_policies.source_id REFERENCES opt_sources (source_id)`. Neither `opt_datasets` nor `opt_sources` is created by `sql/001_init.sql`, `sql/004_iceberg_write_boundary.sql`, or any other file in `sql/` (grep-verified — there are only two migration files, full stop). If `sql/004_iceberg_write_boundary.sql` is ever actually run against a fresh database (e.g. via `infra/docker-compose.yml`'s `docker-entrypoint-initdb.d` mount), **it will fail** with `relation "opt_datasets" does not exist`. This is not a naming coincidence: `src/types/dataset.ts` and `src/types/source.ts` define exactly the `Dataset`/`Source` shapes those table names imply, and `DatasetService`/`SourceService` (`src/services/`) are their in-memory-only implementations — but no migration ever creates the SQL tables to back them. **Open question for Simon:** either author a migration that creates `opt_datasets`/`opt_sources` before `sql/004` (so the FK resolves), or drop those two `REFERENCES` clauses and enforce the dataset/source-exists check purely in the TypeScript service layer (which already does the check today, redundantly, in `DatasetIcebergMappingService.createMapping` and `SourceRatePolicyService.createSourceRatePolicy`) — this file does not choose between those for you. + +## TypeScript-side P0 entities without SQL backing + +Newer P0 capabilities (`docs/12-edge-capability-api.md`) introduced their own typed entities under `src/types/`, each with an in-memory-only service under `src/services/`. Only two of these (above) have a real migration. The rest have **no SQL table at all** — every instance of the owning service starts with an empty in-memory `Map` and loses all state on restart (or even between two `new XService()` calls in the same process — see `docs/12-edge-capability-api.md`'s "in-memory isolation" findings for exactly how badly this bites in practice today): + +| Type | File | Service (in-memory only) | SQL table | +|---|---|---|---| +| `Source` | `src/types/source.ts` | `src/services/source-service.ts` | none | +| `Dataset` | `src/types/dataset.ts` | `src/services/dataset-service.ts` | none | +| `DocumentRecord` | `src/types/document.ts` | `src/services/document-capture-service.ts` | none | +| `DocumentExtractionRecord` | `src/types/document-extraction.ts` | `src/services/document-extraction-service.ts` | none | +| `ExternalWriteRecord` | `src/types/external-writeback.ts` | `src/services/external-writeback-service.ts` | none | +| `DatasetIcebergMapping` | `src/types/dataset-iceberg-mapping.ts` | `src/services/dataset-iceberg-mapping-service.ts` | `opt_dataset_iceberg_mappings` (`sql/004`) | +| `SourceRatePolicy` | `src/types/source-rate-policy.ts` | `src/services/source-rate-policy-service.ts` | `opt_source_rate_policies` (`sql/004`) | + ## pgvector -`CREATE EXTENSION vector;` Use for entity matching/dedup and for the AI-assist (semantic match of NL request → catalog/config). Store embeddings in a `embeddings` table keyed by `(entity_type, entity_key)` or alongside catalog rows. +`CREATE EXTENSION vector;` — this is real (`sql/001_init.sql` line 3, along with `pgcrypto`). **Everything past the extension is target only:** there is no `embeddings` table, or any other table using the `vector` type, anywhere in `sql/001_init.sql` or `sql/004_iceberg_write_boundary.sql`. The intended use (entity matching/dedup, AI-assist semantic match of NL request → catalog/config) is a real design goal, not yet a real table. ## Row-level security & catalog -- Catalog products: a `catalog` table (or view) over curated datasets with sample rows, counts, coverage, price. -- Per-customer data isolation enforced by `org_id` filtering in the API and (where exposed via NocoDB/Metabase) by scoped views/connection params, plus Postgres RLS where practical. +**Target only — none of this exists in `sql/` today**, verified by grep for `ROW LEVEL SECURITY`, `CREATE POLICY`, and `catalog` across both migration files (no matches beyond a comment). There is no `catalog` table or view, and no Postgres RLS policy anywhere in this repo's SQL. Per-customer isolation is intended to be `org_id` filtering in the API plus RLS where practical — but as of this pass the FastAPI engine (`services/api/orchestrator/main.py`) has **no `org_id` concept in any request or response at all** (see `docs/04-api.md`'s implementation-status section: `GET /orders` returns every order in the process, unfiltered). This is the same platform-wide "no auth/session system exists yet" gap noted across this doc corpus — not a data-model-specific problem to solve here, but the schema above cannot be considered tenant-isolated until it is. diff --git a/docs/04-api.md b/docs/04-api.md index 2547565..4346085 100644 --- a/docs/04-api.md +++ b/docs/04-api.md @@ -1,25 +1,46 @@ # 04 · API contract (FastAPI) -Canonical REST contract. Reconcile the existing 25+ endpoints in `src/orchestrator/main.py` toward this. JSON in/out, `snake_case`. All amounts in **öre**. Auth: Next.js → FastAPI over a shared internal token (`X-Internal-Token`); customer identity passed as `org_id`/`user_id` resolved from the Auth.js session server-side. +Canonical REST contract for the **Python data engine** — `services/api/orchestrator/main.py`, run on a VPS/worker host, not on Vercel. JSON in/out, `snake_case`. All amounts in **öre**. -## Conventions -- Errors: `{ "error": { "code", "message" } }`, proper HTTP status. Interface-voice messages (Swedish) are produced in the frontend, not here. -- Idempotency: accept `Idempotency-Key` on POSTs that create orders/payments. -- Pagination: `?limit=&cursor=`. +**Scope note:** this doc covers *only* the FastAPI engine. The newer edge-native TypeScript P0 capability surface (`app/api/0pt/*`, `/api/screen`, `/api/ai/parse` — Next.js Route Handlers that run on Vercel) is a separate, additive layer with its own contract: see `docs/12-edge-capability-api.md`. The two are not the same service and, as of this pass, are not fully wired together (see "Existing-code reconciliation" below for a concrete case where the edge layer expects an engine route that doesn't exist). + +## Implementation status (verified this pass — read before trusting any endpoint below) + +The engine file is `services/api/orchestrator/main.py` (162 lines) — not `src/orchestrator/main.py`, which does not exist in this repo (older doc text, including CLAUDE.md's "Build on existing code" section, still uses that stale path). It implements **exactly 12 endpoints**, all listed below — not "25+" as CLAUDE.md and the previous version of this doc claimed. That number does not match anything in the current codebase; flagging it here since it's this doc's own error to fix (CLAUDE.md's copy of the same claim is outside this file's scope). + +Endpoints that exist in `main.py` today: `GET /health`, `GET /sources`, `GET /sources/{market}`, `GET /vehicles/{plate}`, `GET /screen`, `POST /estimates`, `POST /orders`, `GET /orders`, `GET /orders/{oid}`, `PATCH /orders/{oid}`, `POST /orders/{oid}/run`, `GET /orders/{oid}/status`. + +Everything else in this document — payments/escrow, uploads, customer DB-grant, deliveries, subscriptions, outbound webhooks, `/orders/{id}/events` (SSE), `/orders/{id}/scope|accept|revise` — is a **target contract for the engine**, not yet implemented in `main.py`. Some of it already exists as a *separate* Next.js edge route instead (payments, orders, catalog, estimates, vehicles, sources all have edge-layer counterparts under `app/api/` — not read in detail for this pass, since they're outside this doc's FastAPI scope; noted only so you don't assume they don't exist anywhere). + +Three structural gaps apply to **every** endpoint below that main.py does implement: + +- **No persistence.** `main.py` has no `asyncpg`/`psycopg`/`sqlalchemy` import anywhere. `_ORDERS` and `_JOBS` are plain module-level Python `dict`s — state is lost on every restart, and never touches the `orders`/`jobs` tables `docs/03-data-model.md` documents. `GET /health`'s `db` field only reports whether the `DATABASE_URL` env var is *set* (`bool(os.getenv(...))`) — it never opens a connection, so it does not indicate the database is actually reachable or in use. +- **No authentication, no tenant scoping.** `main.py` never reads any request header (no `X-Internal-Token` check, despite the edge layer sending one — see `lib/engine.ts`), and no request/response model anywhere carries `org_id` or `user_id`. Concretely: **`GET /orders` returns every order held by the process, for every org, with no filtering at all.** The "Auth: … customer identity passed as `org_id`/`user_id` resolved from the Auth.js session server-side" framing below is the target design, not current behavior — this is the same platform-wide "no auth/session system exists yet" gap noted across this doc corpus (see `docs/09-build-plan.md` Phase 3), not something specific to fix in this file. +- **No idempotency, no pagination.** Neither is implemented by any endpoint in `main.py` today, despite being listed as target conventions below. + +## Conventions (target contract — see gaps above for current reality) + +- Errors: `{ "error": { "code", "message" } }`, proper HTTP status. *(Not implemented in `main.py`: it raises plain FastAPI `HTTPException(status, "message")`, which serializes as `{"detail": "message"}`, not the `{error:{code,message}}` shape below.)* Interface-voice messages (Swedish) are produced in the frontend, not here. +- Idempotency: accept `Idempotency-Key` on POSTs that create orders/payments. *(Target only — not implemented.)* +- Pagination: `?limit=&cursor=`. *(Target only — not implemented; `GET /orders` returns the full unfiltered list.)* +- Auth: Next.js → FastAPI over a shared internal token (`X-Internal-Token`, `lib/engine.ts` sends it via `INTERNAL_TOKEN`) — the edge side of this is real, but `main.py` never validates the header, so today it has no effect. Customer identity as `org_id`/`user_id` resolved from an Auth.js session is target design; no such session system exists in this repo yet. ## Catalog + ``` GET /catalog -> [{id,title,markets,entity_type,count,updated,price_ore,confidence}] GET /catalog/{id} -> full dataset (overview, coverage, schema, quality summary) GET /catalog/{id}/sample?n=10 -> sample rows WITH per-field provenance ``` +**Status: target only.** No `/catalog` route exists in `main.py`. A separate `GET /api/catalog` exists at the edge (`app/api/catalog/route.ts`, registered in CL0UD's capability registry as `0pt.catalog_list`) — not read in detail for this pass; out of scope here. ## Estimate (configurator → price/time/confidence) ``` POST /estimates - body: { flow, markets[], entity_type, fields[], enrichment[], volume, frequency, confidence_target, formats[] } - -> { total_ore, deposit_ore, balance_ore, est_days, est_confidence, breakdown[] } + body: { markets[], entity_type, fields[], enrichment[], volume, frequency, confidence_target } + -> { total_ore, deposit_ore, balance_ore, est_days, est_confidence, recurring } ``` +**Status: implemented**, but the shape differs from the previous version of this doc: the request body has **no `flow` field**, and the response has **no `breakdown[]` field** — `main.py`'s `EstimateIn` model is exactly `{markets, entity_type, fields, enrichment, volume, frequency, confidence_target}` (defaults: `entity_type="Bolag"`, `volume=1000`, `frequency="Engångs"`, `confidence_target=92`), and `compute_estimate()` returns exactly `{total_ore, deposit_ore, balance_ore, est_days, est_confidence, recurring}` — `recurring` is a bool (`frequency != "Engångs"`), not a `breakdown[]` array. There is also no `formats[]` field on the request. ## AI-assist (NL → config) ``` @@ -27,16 +48,18 @@ POST /ai/parse body: { text } -- "5000 svenska SaaS-bolag med e-post och omsättning" -> { config: , rationale } -- prefilled configurator; uses pgvector match to catalog/fields ``` +**Status: not implemented — confirmed contract mismatch.** `main.py` has no `/ai/parse` route at all (grep-verified: only the 12 endpoints listed in "Implementation status" exist). This matters beyond "target only": the edge route `app/api/ai/parse/route.ts` **actively calls** `engineFetch("/ai/parse", {method:"POST", ...})` expecting the engine to have this route, checks `if (eng && eng.ok)`, and — because the engine 404s — silently falls through to a local keyword-heuristic parser every time, even when `ENGINE_URL` is configured and the engine is reachable. See `docs/12-edge-capability-api.md` for the edge side of this route. There is also no pgvector usage anywhere in `main.py` (no `embeddings` table exists either — see `docs/03-data-model.md`). ## Orders (lifecycle) ``` -POST /orders body: { flow, config } -> { order_id, status:'draft'|'quoted' } -GET /orders -> [orders for org] -GET /orders/{id} -> order + current job + delivery (if any) -POST /orders/{id}/scope (custom flow) body:{ notes } -> status:'quoted' (manual quote step) -POST /orders/{id}/accept customer approves delivery -> triggers balance release; status:'accepted' -POST /orders/{id}/revise body:{ notes } -> status:'revision'; escrow untouched +POST /orders body: { flow, config } -> { order_id, status:'draft', deposit_ore, total_ore } +GET /orders -> [orders] -- NOT filtered by org; returns every order in the process (no auth) +GET /orders/{id} -> { ...order, job } -- job is null if none exists yet; no "delivery" field (deliveries aren't implemented) +PATCH /orders/{id} body: { status } -> updates order.status to any caller-supplied string, no validation against a lifecycle enum +POST /orders/{id}/run -> runs the pipeline SYNCHRONOUSLY (awaits workers/pipeline.py:run_job), sets status "in_production" then "review", returns the job record +GET /orders/{id}/status -> the job record, or {stage:"queued", pct:0, state:"queued"} if none exists (never 404s) ``` +**Status: partially implemented, but shaped differently from the target below.** What actually exists: `POST /orders`, `GET /orders`, `GET /orders/{oid}`, `PATCH /orders/{oid}`, `POST /orders/{oid}/run`, `GET /orders/{oid}/status` — six endpoints, listed above with their real behavior. What the original target contract described — `POST /orders/{id}/scope` (manual quote step), `POST /orders/{id}/accept` (triggers balance release), `POST /orders/{id}/revise` — **do not exist in `main.py` at all.** Order-status transitions today happen only through the generic `PATCH /orders/{oid}` (accepts any string, no state-machine enforcement) or implicitly through `POST /orders/{oid}/run`. `POST /orders/{oid}/run` itself is undocumented in any prior version of this doc despite being the only way to actually trigger the sourcing pipeline — it has no payment/escrow gate of any kind (see Payments below). ## Payments (50/50 escrow) ``` @@ -44,19 +67,21 @@ POST /payments/{order_id}/deposit -> { checkout_url } (Stripe Checkout for 50 POST /payments/{order_id}/release -> captures/charges the held balance (called on accept) POST /webhooks/stripe (Stripe → here) verify signature; update payments + order status ``` -> Implement balance as either a manual capture on a held PaymentIntent, or a second Checkout/invoice keyed to `order_id`. Never charge balance before `accept`. +**Status: not implemented in the engine at all.** There is no Stripe reference anywhere in `services/api/` (grep-verified). `POST /orders/{oid}/run` — the engine's only way to start production — has no payment dependency whatsoever; nothing in `main.py` checks that a deposit was paid before running a job. If Stripe/escrow exists anywhere in this repo, it's at the edge (`app/api/payments/[order_id]/deposit`, `app/api/payments/[order_id]/release`, `app/api/webhooks/stripe` — registered in CL0UD's capability registry as `0pt.order_payment`, evidence comment notes both routes 501 when `STRIPE_SECRET_KEY` is unset) — not read in detail this pass, out of scope here. **Open question:** whether escrow/payment gating is meant to live in the FastAPI engine (as this doc's original framing implies) or stay purely at the edge is unresolved; this doc doesn't decide it for you. ## Jobs / live status ``` GET /orders/{id}/status -> { stage, pct, per_market[], confidence, eta, rows } GET /orders/{id}/events -> Server-Sent Events stream (preferred for the production tracker) ``` +**Status: `GET /orders/{id}/status` exists but returns a different, smaller shape** — the actual dict returned by `workers/pipeline.py:run_job()` is exactly `{stage, pct, state, metrics, records}` (verified by reading its `return` statement) — **not** `{stage, pct, per_market[], confidence, eta, rows}` as previously documented, and **not** `started_at`/`finished_at` either (those two column names exist only on the unused `jobs` SQL table in `docs/03-data-model.md` — the in-memory job dict `run_job` actually returns has no timestamps at all). `metrics` itself is `{rows, dropped_below_threshold, per_market, avg_confidence}`; `records` holds the delivered rows. `GET /orders/{id}/events` (SSE) **does not exist** — target only. ## Uploads (R2 presigned) ``` POST /uploads body:{ order_id, name, mime, bytes } -> { file_id, put_url } (client PUTs to R2) GET /orders/{id}/files -> [files] ``` +**Status: target only.** Neither route exists in `main.py`. ## Customer DB access (read-only grant) ``` @@ -65,18 +90,28 @@ POST /connections/db-grant -> { grant_id, status:'pending'|'verified' } DELETE /connections/db-grant/{id} -> revoke ``` +**Status: target only.** Neither route exists in `main.py`. The `db_grants` table exists in `sql/001_init.sql` (see `docs/03-data-model.md`) but nothing writes to it. ## Deliveries ``` GET /orders/{id}/delivery -> { formats, destinations, record_count, avg_confidence, quality_report, files[], provenance_export_url } POST /orders/{id}/delivery/destinations body:{ destinations[] } -> updates targets (CSV/Parquet/JSON/API/NocoDB/Metabase/FiveCRM/Sheets/webhook) ``` +**Status: target only.** Neither route exists in `main.py`. Note also that a "delivery" concept and a "quality_report" are described here as if built — per this doc corpus's broader honesty pass, the underlying quality/QA-check logic these would report on (`quality-check-service.ts`, `quality-report-service.ts` and friends) does not exist in this repo at all (see CLAUDE.md's pointer to README's "Known issues"). This section is aspirational on two levels: the route, and the quality data it would need to serve. ## Sources (the transparency moat) ``` -GET /sources -> [{source_id,name,market,access,licence,cadence,confidence}] -GET /sources/{market} -> sources + freshness for a market +GET /sources -> [{source_id,name,market,access,licence,cadence}] +GET /sources/{market} -> full sources.yaml entries (source_id,name,market,access,licence,cadence,url,entities,fields,base_url,robots_ok,notes) filtered to that market + 'global' ``` +**Status: implemented, with two shape corrections.** `GET /sources` does **not** return a `confidence` field — `main.py` explicitly projects only `source_id,name,market,access,licence,cadence` from `sources.yaml` (verified: `{k: s.get(k) for k in (...)}` with no `confidence` key). `GET /sources/{market}` returns the *entire* raw `sources.yaml` entry for each match (richer than `/sources` — includes `base_url`, `robots_ok`, `entities`, `fields`, `notes`), not a "sources + freshness" shape; there is no `freshness` field anywhere in `sources.yaml` or added by this endpoint. Both read from `services/api/sources/sources.yaml` (**8** entries as of this pass — verified by count: `gleif`, `brreg`, `companies_house`, `prh`, `cvr`, `bolagsverket`, `rdw`, `opensanctions`; no `traficom` entry exists in this file, unlike the edge-native `GET /api/sources` route below) — an entirely separate, static list from the per-tenant `Source` registry (`src/types/source.ts`) the edge-native `0pt.query_source`/`0pt.source_rate_policy_manage` capabilities act against; see `docs/03-data-model.md` and `docs/12-edge-capability-api.md`. **A third, separate static list exists too:** `app/api/sources/route.ts` is a hardcoded, `GET`-only Next.js route with its own 9-entry array (the same 8 markets plus `traficom`/FI, each with a `confidence` field sources.yaml does not have) — not part of this FastAPI contract, not wired to sources.yaml in any way, and not itself documented in detail here (out of this doc's FastAPI scope; see `docs/12-edge-capability-api.md`). + +## Vehicles & sanctions screening (implemented, previously undocumented) +``` +GET /vehicles/{plate} -> RDW (NL) lookup result, or { vehicle: null } if not found +GET /screen?q= -> { source: "opensanctions", matches: [{name, score, schema}] } +``` +These two endpoints exist in `main.py` today but were missing from every prior version of this doc. `GET /vehicles/{plate}` proxies `sources/connectors.py:rdw_vehicle()` (Dutch open vehicle data, technical/fleet fields only, no owner PII, per the connector's own comment). `GET /screen` proxies `sources/connectors.py:opensanctions_screen()` against `api.opensanctions.org` — the edge route `app/api/screen/route.ts` calls this same path via `engineFetch('/screen?q=...')` first and falls back to calling OpenSanctions directly if the engine isn't reachable, which is a working (not broken) reconciliation, unlike `/ai/parse` above. ## Subscriptions / feeds ``` @@ -84,14 +119,41 @@ POST /subscriptions body:{ order_id, cadence, webhook_url } -> { GET /subscriptions -> [subs] POST /subscriptions/{id}/pause | /resume ``` +**Status: target only.** No route exists in `main.py`. The `subscriptions` table exists in `sql/001_init.sql` but nothing writes to it from the engine. ## Webhooks out (to customers) - On delivery / feed update: POST to `subscriptions.webhook_url` with `{ order_id, delta_url, summary }`. +**Status: target only** — depends on Deliveries and Subscriptions above, neither of which exist yet. + ## Health ``` -GET /health -> { ok, db, workers, queue } +GET /health -> { ok, service, db, sources } ``` +**Status: implemented**, with a corrected shape: the real response is `{ok: true, service: "0pt-data-engine", db: bool(DATABASE_URL is set), sources: len(_SOURCES)}` — not `{ok, db, workers, queue}` as previously documented. There is no `workers` or `queue` field, and `db` is a presence check on an env var, not a live connectivity probe (see "Implementation status" above). ## Existing-code reconciliation -Map current endpoints (orders, estimates, ai-assistent, catalog, webhooks, health) onto the names above; rename for consistency, keep behavior. Add: `/orders/{id}/events` (SSE), `/payments/*` escrow split, `/connections/db-grant`, `/sources`, `/subscriptions`. Remove any endpoint that drives a bypass/stealth fetch path. + +| This doc's target | `main.py` reality | Status | +|---|---|---| +| `GET/POST /catalog*` | — | not implemented (edge-only, unverified) | +| `POST /estimates` | `POST /estimates` | implemented, different request/response fields (see above) | +| `POST /ai/parse` | — | not implemented; edge layer calls it anyway and silently falls back | +| `POST /orders`, `GET /orders`, `GET /orders/{id}` | same paths, exist | implemented, no auth/org filtering, in-memory only | +| `POST /orders/{id}/scope\|accept\|revise` | `PATCH /orders/{oid}`, `POST /orders/{oid}/run` | different shape: generic status PATCH + a synchronous run trigger, no escrow gate | +| `POST /payments/*`, `POST /webhooks/stripe` | — | not implemented in engine (may exist at the edge — unverified this pass) | +| `GET /orders/{id}/status` | `GET /orders/{oid}/status` | implemented, smaller/different shape | +| `GET /orders/{id}/events` (SSE) | — | not implemented | +| `POST /uploads`, `GET /orders/{id}/files` | — | not implemented | +| `POST/DELETE /connections/db-grant*` | — | not implemented | +| `GET/POST /orders/{id}/delivery*` | — | not implemented; underlying quality data doesn't exist either | +| `GET /sources`, `GET /sources/{market}` | same paths, exist | implemented, different projected fields (no `confidence`/`freshness`) | +| `POST/GET /subscriptions*` | — | not implemented | +| `GET /health` | `GET /health` | implemented, different shape (no `workers`/`queue`) | +| — | `GET /vehicles/{plate}`, `GET /screen` | implemented, previously undocumented | + +Remove any endpoint that drives a bypass/stealth fetch path (CLAUDE.md §1) — none of the 12 real endpoints in `main.py` do; the WAF-bypass concern in this repo's history is in `services/scrapers/*` on the Gitea remote, not here (see CLAUDE.md's note on the two diverged remotes — an open question for Simon, not resolved by this pass). + +## Related: the edge-native TypeScript capability surface + +`app/api/0pt/*`, `app/api/screen/route.ts`, and `app/api/ai/parse/route.ts` are **not** part of the FastAPI contract above — they're Next.js Route Handlers, run on Vercel, and implement several of the newer P0 capabilities registered in CL0UD's capability registry (`0pt.query_source`, `0pt.document_ocr`, `0pt.lakehouse_mapping_manage`, `0pt.lakehouse_write`, `0pt.source_rate_policy_manage`). Full route-by-route documentation, including several confirmed always-fails-over-HTTP bugs, lives in **`docs/12-edge-capability-api.md`**. diff --git a/docs/05-design-system.md b/docs/05-design-system.md index 5f6ff0b..25e29af 100644 --- a/docs/05-design-system.md +++ b/docs/05-design-system.md @@ -1,27 +1,42 @@ # 05 · Design system (mono) -Self-contained spec. Component-level v0 prompts live in `docs/design/0pt-v0-build-brief.md`; the full page/component inventory + visual rules in `docs/design/0pt-design-coverage-and-visuals.md`. This file is the source of truth for tokens + motion; build them once and reuse. +Self-contained spec. This file is the source of truth for tokens + motion; the +tokens and CSS below are **already implemented** in `app/globals.css` and +consumed across `components/` and `app/` — this is not aspirational, it's a +description of what ships today, kept here so new screens stay on-brand. -Derived from the **mono / EVASION** template (itself v0-generated). Light, editorial, monochrome, sharp, with a single red accent and animated grain + scroll-reveal motion. **No photography** — imagery is rendered data. +Derived from the **mono / EVASION** template (itself v0-generated). Light, editorial, monochrome, sharp, with a single red accent and animated grain + scroll-reveal motion. **No photography** — imagery is rendered data. (Verified: no `` or `next/image` usage anywhere in `app/` or `components/` except a single unused leftover component, see "Vestigial component, not on the 'no photography' path" further down.) + +> **Doc-index gap:** `CLAUDE.md`'s documentation index lists two files this doc used to point to — `docs/design/0pt-v0-build-brief.md` (per-component v0 prompts) and `docs/design/0pt-design-coverage-and-visuals.md` (full page/component inventory + visual rules). **Neither file exists anywhere in this repo** (confirmed via full-repo search). They are referenced but were never written. This doc no longer treats them as available references — anything they were supposed to cover (per-component prompts, the full visual-mapping table) is not documented anywhere right now. Whoever owns design docs next should either write them or remove the rows from `CLAUDE.md`'s index; not resolved in this pass. ## Tokens +Verified against `app/globals.css` (the file actually imported by `app/layout.tsx`): + ``` THEME (light, default): --background #ffffff --foreground #0a0a0a --muted-foreground #737373 --border #e5e5e5 --card #ffffff --input #f5f5f5 -THEME (dark sections — footer, one editorial block): +THEME (dark sections — footer, editorial/manifesto block, technology/engine block): --background #0a0a0a --foreground #fafafa --muted-foreground #a3a3a3 --border #262626 -ACCENT (single, sparing): red --accent #e40014 (one CTA / one number / one marker per screen) +ACCENT (single, sparing): red --accent-red #e40014 (one CTA / one number / one marker per screen) PROVENANCE squares (ONLY non-mono color besides the red CTA): level 0 #737373 (unavailable/licensed) · 1 #143aff (single source) · 2 #fcbb00 (partial) · 3 #00d294 (corroborated) -RADIUS: 0 everywhere (max 2px on tiny chips). No drop shadows; structure via 1px hairlines. -TYPE: Inter (everything; headlines weight 500–600, letter-spacing -0.02em, large) · JetBrains Mono (eyebrows, labels, numbers, codes, table headers — UPPERCASE, letter-spacing 0.12em, ~11px, #737373) +RADIUS: 0 everywhere (`--radius: 0rem` in app/globals.css; shadcn/ui components inherit --radius-sm/md/lg/xl derived from it, so they render square too). +TYPE: Inter (everything; headlines weight 500–600, letter-spacing -0.02em, large) · JetBrains Mono (eyebrows, labels, numbers, codes, table headers — UPPERCASE, letter-spacing mostly 0.08–0.22em but ranging down to 0.02em on some labels/table headers (e.g. `provenance-badge.tsx`, the source table headers in `app/kallor/page.tsx`), ~10–12px, #737373). Loaded via `next/font/google` in `app/layout.tsx` as CSS vars `--font-inter` / `--font-jetbrains-mono`. GRID: 8px base; graph-paper hairlines; generous whitespace; left-aligned headlines. -SECTION SPINE: real ordered indices 00–NN as mono eyebrows. +SECTION SPINE: real ordered indices 00–06 as mono eyebrows (hero=00, philosophy=01, catalog=02, [03 unused/reserved], collection/coverage=04, testimonials=05, pricing=06). The two dark sections (editorial "Manifesto", technology "The engine") intentionally break the numbered spine — they carry a text eyebrow instead of a number. ``` -## globals.css (add verbatim — guarantees identical motion + grain) +**Correction — a third section also breaks the spine, and it isn't dark:** verified render order in `app/page.tsx` is Hero(00) → Philosophy(01) → FeaturedProducts/Catalog(02) → **Technology (dark, no number)** → **GallerySection (no number)** → Collection(04) → **Editorial (dark, no number)** → Testimonials(05) → Pricing(06) → Footer(dark). `components/sections/gallery-section.tsx` sits exactly where a "03" would fall but carries a text eyebrow (`"— Visualizations"`) like the dark sections, plus its own unrelated local `01`–`05` indices on its five tiles (a per-tile counter, not a continuation of the page's section spine — don't confuse the two). So "03 unused/reserved" is accurate in that no section ever shows the numeral 03, but three sections break the numbered spine, not two — only two of those three happen to be dark-themed. + +**Correction vs. previous version of this doc:** the dark theme was previously described as appearing on "footer, one editorial block" (2 places). Verified in `components/sections/`: **three** sections actually use the dark theme — `footer-section.tsx`, `editorial-section.tsx` (the "Manifesto" block), and `technology-section.tsx` (the "#motorn" / "The engine" block, `id="motorn"`). All three set `className="dark ..." ` and a `bg-[#0a0a0a] text-[#fafafa]` pairing. + +**Implementation note (not yet reconciled in code):** `editorial-section.tsx` and `technology-section.tsx` use the documented dark border `#262626` consistently. `footer-section.tsx` instead hardcodes `#1f1f1f` for its borders — a different, undocumented shade. All three sections hardcode hex literals (`#0a0a0a`, `#fafafa`, `#737373`, `#a3a3a3`, plus the two different border shades) rather than referencing the `.dark` CSS-variable overrides that already exist in `app/globals.css`'s `--border`/`--foreground`/`--muted-foreground` tokens. This works today because the values happen to match (except the footer border), but it means the dark sections don't actually pick up `.dark`-scoped custom-property overrides — they're hand-pinned. Flagging as a small cleanup opportunity, not fixing here (out of scope for a docs pass). + +## globals.css (already live in `app/globals.css` — do not re-add elsewhere) + +The block below is the actual motion + grain CSS, condensed for reference. It ships today in `app/globals.css` (with the keyframes/classes spelled out at full length rather than this compact form — functionally identical): ```css @keyframes reveal-up{0%{opacity:0;transform:translateY(60px)}to{opacity:1;transform:translateY(0)}} @@ -44,44 +59,73 @@ SECTION SPINE: real ordered indices 00–NN as mono eyebrows. @media(prefers-reduced-motion:reduce){.reveal,.reveal-left,.reveal-right,.reveal-scale,.reveal-fade{opacity:1;animation:none}.animate-float,.grain-overlay::before{animation:none}} ``` -## useReveal hook +The provenance colors and `--accent-red` also live in `app/globals.css` as CSS custom properties (`--accent-red`, `--prov-0..3`), exposed to Tailwind via `@theme inline` as `--color-accent-red`, `--color-prov-0..3`. -```tsx -"use client"; -import { useEffect, useRef } from "react"; -export function useReveal() { - const ref = useRef(null); - useEffect(() => { - const el = ref.current; if (!el) return; - const io = new IntersectionObserver((es) => es.forEach((e) => { - if (e.isIntersecting) { - el.querySelectorAll(".reveal,.reveal-left,.reveal-right,.reveal-scale,.reveal-fade") - .forEach((n) => n.classList.add("in-view")); - io.unobserve(el); - } - }), { threshold: 0.15, rootMargin: "0px 0px -10% 0px" }); - io.observe(el); return () => io.disconnect(); - }, []); - return ref; -} -``` -Every section wraps in a tag using `useReveal()`; children get `reveal[/-left/-right/-scale/-fade]` + `delay-N`. +**Known leftover, unused file:** `styles/globals.css` also exists in this repo and is **not** imported anywhere (`app/layout.tsx` imports `./globals.css`, i.e. `app/globals.css`, only). `styles/globals.css` is a stray copy of the original shadcn/v0 scaffold — it uses `oklch()` colors, Geist/Geist Mono fonts, and `--radius: 0.625rem`, none of which match this design system. It is dead code. Don't edit it expecting it to affect the app, and consider deleting it in a future cleanup (not done in this pass — out of scope). + +## Reveal / motion system + +Three pieces work together; the doc previously only described one of them: + +1. **`useReveal()` hook** — `hooks/use-reveal.ts`. Wire a section's ref with it; descendants carrying `.reveal` (+ optional `.reveal-left/-right/-scale/-fade` and `.delay-N`) animate to `.in-view` via `IntersectionObserver` (`threshold: 0.15`, `rootMargin: "0px 0px -10% 0px"`) when the section scrolls into view. **Difference from a naive IntersectionObserver:** if the section's top edge is already inside the viewport on mount (e.g. direct navigation to a page whose hero is already visible), the hook applies `.in-view` immediately via `requestAnimationFrame` instead of waiting for a scroll event — this avoids a flash of invisible content on first paint. + ```tsx + "use client"; + import { useEffect, useRef } from "react"; + export function useReveal() { + const ref = useRef(null); + useEffect(() => { + const el = ref.current; + if (!el) return; + const applyReveal = () => { + el.querySelectorAll(".reveal, .reveal-left, .reveal-right, .reveal-scale, .reveal-fade") + .forEach((n) => n.classList.add("in-view")); + }; + const rect = el.getBoundingClientRect(); + if (rect.top < window.innerHeight * 0.9) { + requestAnimationFrame(applyReveal); + return; + } + const io = new IntersectionObserver((entries) => entries.forEach((e) => { + if (e.isIntersecting) { applyReveal(); io.unobserve(el); } + }), { threshold: 0.15, rootMargin: "0px 0px -10% 0px" }); + io.observe(el); + return () => io.disconnect(); + }, []); + return ref; + } + ``` +2. **``** — `components/route-reveal.tsx`, mounted once, globally, in `app/layout.tsx` inside the providers. On every client-side route change (via `usePathname()`), it force-applies `.in-view` to any `.reveal*` element whose top edge is already within 95% of the viewport height — covering standalone pages (e.g. `/bestall`, `/katalog/[id]`) that render fully visible on load, where per-section `IntersectionObserver`s might not fire before paint. Runs on an immediate pass, a `requestAnimationFrame` pass, and a 600ms-delayed pass (for async-loaded content). **This component was not documented in the previous version of this file** — it's load-bearing for anything not covered by `useReveal()`'s own "already visible" check (e.g. content that becomes visible slightly after mount). +3. **CSS keyframes/classes** in `app/globals.css` (above) define the actual animation. + +Every section wraps in a tag using `useReveal()`; children get `reveal[/-left/-right/-scale/-fade]` + `delay-N`. This pattern is used consistently across `components/sections/*.tsx`. ## Buttons & labels -- Primary: solid `#0a0a0a` bg, white text, square, mono uppercase 12px / spacing 0.1em, padding 14px 26px; hover → bg `#e40014`. -- Secondary: transparent, 1px `#0a0a0a` border; hover → bg `#0a0a0a`, white text. -- Eyebrow: mono uppercase 11px, spacing 0.14em, `#737373`, with a 2-digit index ("01 — KATALOG"). + +- Primary: solid `#0a0a0a` (`--foreground`) bg, `--background` text, square, mono uppercase ~12px, padding varies by placement (header CTA and hero CTAs use `px-5..7 py-2.5..4`, i.e. roughly 10–16px vertical / 20–28px horizontal — treat the doc's original "14px 26px" as an approximate baseline, not an exact contract); hover → bg `var(--accent-red)`. +- Secondary: transparent, 1px `border-foreground`; hover → bg `--foreground`, `--background` text. +- Eyebrow: mono uppercase 11px, spacing ~0.14–0.18em, `--muted-foreground`, with a 2-digit index where the section spine applies (e.g. `"01 — Philosophy"`). - Headlines: sentence case, never uppercase. +- **Implementation note:** there is no shared `Button`/`MonoButton` component implementing this pattern — `components/ui/button.tsx` is the generic, unmodified shadcn/ui button (rounded via `--radius-*` tokens, which happen to resolve to 0 here, but otherwise a stock `cva` variant set: default/destructive/outline/secondary/ghost/link). The mono square CTA look described above is hand-rolled per call site as inline Tailwind classes on raw ``/`