0pt/docs/04-api.md

04 · API contract (FastAPI)

Canonical REST contract for the Python data engine — services/api/orchestrator/main.py, run on a VPS/worker host, not on Vercel. JSON in/out, snake_case. All amounts in öre.

Scope note: this doc covers only the FastAPI engine. The newer edge-native TypeScript P0 capability surface (app/api/0pt/*, /api/screen, /api/ai/parse — Next.js Route Handlers that run on Vercel) is a separate, additive layer with its own contract: see docs/12-edge-capability-api.md. The two are not the same service and, as of this pass, are not fully wired together (see "Existing-code reconciliation" below for a concrete case where the edge layer expects an engine route that doesn't exist).

Implementation status (verified this pass — read before trusting any endpoint below)

The engine file is services/api/orchestrator/main.py (162 lines) — not src/orchestrator/main.py, which does not exist in this repo (older doc text, including CLAUDE.md's "Build on existing code" section, still uses that stale path). It implements exactly 12 endpoints, all listed below — not "25+" as CLAUDE.md and the previous version of this doc claimed. That number does not match anything in the current codebase; flagging it here since it's this doc's own error to fix (CLAUDE.md's copy of the same claim is outside this file's scope).

Endpoints that exist in main.py today: GET /health, GET /sources, GET /sources/{market}, GET /vehicles/{plate}, GET /screen, POST /estimates, POST /orders, GET /orders, GET /orders/{oid}, PATCH /orders/{oid}, POST /orders/{oid}/run, GET /orders/{oid}/status.

Everything else in this document — payments/escrow, uploads, customer DB-grant, deliveries, subscriptions, outbound webhooks, /orders/{id}/events (SSE), /orders/{id}/scope|accept|revise — is a target contract for the engine, not yet implemented in main.py. Some of it already exists as a separate Next.js edge route instead (payments, orders, catalog, estimates, vehicles, sources all have edge-layer counterparts under app/api/ — not read in detail for this pass, since they're outside this doc's FastAPI scope; noted only so you don't assume they don't exist anywhere).

Three structural gaps apply to every endpoint below that main.py does implement:

• No persistence. main.py has no asyncpg/psycopg/sqlalchemy import anywhere. _ORDERS and _JOBS are plain module-level Python dicts — state is lost on every restart, and never touches the orders/jobs tables docs/03-data-model.md documents. GET /health's db field only reports whether the DATABASE_URL env var is set (bool(os.getenv(...))) — it never opens a connection, so it does not indicate the database is actually reachable or in use.
• No authentication, no tenant scoping. main.py never reads any request header (no X-Internal-Token check, despite the edge layer sending one — see lib/engine.ts), and no request/response model anywhere carries org_id or user_id. Concretely: GET /orders returns every order held by the process, for every org, with no filtering at all. The "Auth: … customer identity passed as org_id/user_id resolved from the Auth.js session server-side" framing below is the target design, not current behavior — this is the same platform-wide "no auth/session system exists yet" gap noted across this doc corpus (see docs/09-build-plan.md Phase 3), not something specific to fix in this file.
• No idempotency, no pagination. Neither is implemented by any endpoint in main.py today, despite being listed as target conventions below.

Conventions (target contract — see gaps above for current reality)

• Errors: { "error": { "code", "message" } }, proper HTTP status. (Not implemented in main.py: it raises plain FastAPI HTTPException(status, "message"), which serializes as {"detail": "message"}, not the {error:{code,message}} shape below.) Interface-voice messages (Swedish) are produced in the frontend, not here.
• Idempotency: accept Idempotency-Key on POSTs that create orders/payments. (Target only — not implemented.)
• Pagination: ?limit=&cursor=. (Target only — not implemented; GET /orders returns the full unfiltered list.)
• Auth: Next.js → FastAPI over a shared internal token (X-Internal-Token, lib/engine.ts sends it via INTERNAL_TOKEN) — the edge side of this is real, but main.py never validates the header, so today it has no effect. Customer identity as org_id/user_id resolved from an Auth.js session is target design; no such session system exists in this repo yet.

Catalog

GET  /catalog                      -> [{id,title,markets,entity_type,count,updated,price_ore,confidence}]
GET  /catalog/{id}                 -> full dataset (overview, coverage, schema, quality summary)
GET  /catalog/{id}/sample?n=10     -> sample rows WITH per-field provenance

Status: target only. No /catalog route exists in main.py. A separate GET /api/catalog exists at the edge (app/api/catalog/route.ts, registered in CL0UD's capability registry as 0pt.catalog_list) — not read in detail for this pass; out of scope here.

Estimate (configurator → price/time/confidence)

POST /estimates
  body: { markets[], entity_type, fields[], enrichment[], volume, frequency, confidence_target }
  -> { total_ore, deposit_ore, balance_ore, est_days, est_confidence, recurring }

Status: implemented, but the shape differs from the previous version of this doc: the request body has no flow field, and the response has no breakdown[] field — main.py's EstimateIn model is exactly {markets, entity_type, fields, enrichment, volume, frequency, confidence_target} (defaults: entity_type="Bolag", volume=1000, frequency="Engångs", confidence_target=92), and compute_estimate() returns exactly {total_ore, deposit_ore, balance_ore, est_days, est_confidence, recurring} — recurring is a bool (frequency != "Engångs"), not a breakdown[] array. There is also no formats[] field on the request.

AI-assist (NL → config)

POST /ai/parse
  body: { text }                   -- "5000 svenska SaaS-bolag med e-post och omsättning"
  -> { config: <estimate body>, rationale }   -- prefilled configurator; uses pgvector match to catalog/fields

Status: not implemented — confirmed contract mismatch. main.py has no /ai/parse route at all (grep-verified: only the 12 endpoints listed in "Implementation status" exist). This matters beyond "target only": the edge route app/api/ai/parse/route.ts actively calls engineFetch("/ai/parse", {method:"POST", ...}) expecting the engine to have this route, checks if (eng && eng.ok), and — because the engine 404s — silently falls through to a local keyword-heuristic parser every time, even when ENGINE_URL is configured and the engine is reachable. See docs/12-edge-capability-api.md for the edge side of this route. There is also no pgvector usage anywhere in main.py (no embeddings table exists either — see docs/03-data-model.md).

Orders (lifecycle)

POST /orders                       body: { flow, config }            -> { order_id, status:'draft', deposit_ore, total_ore }
GET  /orders                       -> [orders]  -- NOT filtered by org; returns every order in the process (no auth)
GET  /orders/{id}                  -> { ...order, job }               -- job is null if none exists yet; no "delivery" field (deliveries aren't implemented)
PATCH /orders/{id}                 body: { status }                  -> updates order.status to any caller-supplied string, no validation against a lifecycle enum
POST /orders/{id}/run              -> runs the pipeline SYNCHRONOUSLY (awaits workers/pipeline.py:run_job), sets status "in_production" then "review", returns the job record
GET  /orders/{id}/status           -> the job record, or {stage:"queued", pct:0, state:"queued"} if none exists (never 404s)

Status: partially implemented, but shaped differently from the target below. What actually exists: POST /orders, GET /orders, GET /orders/{oid}, PATCH /orders/{oid}, POST /orders/{oid}/run, GET /orders/{oid}/status — six endpoints, listed above with their real behavior. What the original target contract described — POST /orders/{id}/scope (manual quote step), POST /orders/{id}/accept (triggers balance release), POST /orders/{id}/revise — do not exist in main.py at all. Order-status transitions today happen only through the generic PATCH /orders/{oid} (accepts any string, no state-machine enforcement) or implicitly through POST /orders/{oid}/run. POST /orders/{oid}/run itself is undocumented in any prior version of this doc despite being the only way to actually trigger the sourcing pipeline — it has no payment/escrow gate of any kind (see Payments below).

Payments (50/50 escrow)

POST /payments/{order_id}/deposit  -> { checkout_url }   (Stripe Checkout for 50%; on success → IN_PRODUCTION)
POST /payments/{order_id}/release  -> captures/charges the held balance (called on accept)
POST /webhooks/stripe              (Stripe → here) verify signature; update payments + order status

Status: not implemented in the engine at all. There is no Stripe reference anywhere in services/api/ (grep-verified). POST /orders/{oid}/run — the engine's only way to start production — has no payment dependency whatsoever; nothing in main.py checks that a deposit was paid before running a job. If Stripe/escrow exists anywhere in this repo, it's at the edge (app/api/payments/[order_id]/deposit, app/api/payments/[order_id]/release, app/api/webhooks/stripe — registered in CL0UD's capability registry as 0pt.order_payment, evidence comment notes both routes 501 when STRIPE_SECRET_KEY is unset) — not read in detail this pass, out of scope here. Open question: whether escrow/payment gating is meant to live in the FastAPI engine (as this doc's original framing implies) or stay purely at the edge is unresolved; this doc doesn't decide it for you.

Jobs / live status

GET  /orders/{id}/status           -> { stage, pct, per_market[], confidence, eta, rows }
GET  /orders/{id}/events           -> Server-Sent Events stream (preferred for the production tracker)

Status: GET /orders/{id}/status exists but returns a different, smaller shape — the actual dict returned by workers/pipeline.py:run_job() is exactly {stage, pct, state, metrics, records} (verified by reading its return statement) — not {stage, pct, per_market[], confidence, eta, rows} as previously documented, and not started_at/finished_at either (those two column names exist only on the unused jobs SQL table in docs/03-data-model.md — the in-memory job dict run_job actually returns has no timestamps at all). metrics itself is {rows, dropped_below_threshold, per_market, avg_confidence}; records holds the delivered rows. GET /orders/{id}/events (SSE) does not exist — target only.

Uploads (R2 presigned)

POST /uploads                      body:{ order_id, name, mime, bytes } -> { file_id, put_url }  (client PUTs to R2)
GET  /orders/{id}/files            -> [files]

Status: target only. Neither route exists in main.py.

Customer DB access (read-only grant)

POST /connections/db-grant
  body: { order_id, conn_string, scope, expires_at }   -- stored ENCRYPTED; validated read-only
  -> { grant_id, status:'pending'|'verified' }
DELETE /connections/db-grant/{id}  -> revoke

Status: target only. Neither route exists in main.py. The db_grants table exists in sql/001_init.sql (see docs/03-data-model.md) but nothing writes to it.

Deliveries

GET  /orders/{id}/delivery         -> { formats, destinations, record_count, avg_confidence, quality_report, files[], provenance_export_url }
POST /orders/{id}/delivery/destinations  body:{ destinations[] }    -> updates targets (CSV/Parquet/JSON/API/NocoDB/Metabase/FiveCRM/Sheets/webhook)

Status: target only. Neither route exists in main.py. Note also that a "delivery" concept and a "quality_report" are described here as if built — per this doc corpus's broader honesty pass, the underlying quality/QA-check logic these would report on (quality-check-service.ts, quality-report-service.ts and friends) does not exist in this repo at all (see CLAUDE.md's pointer to README's "Known issues"). This section is aspirational on two levels: the route, and the quality data it would need to serve.

Sources (the transparency moat)

GET  /sources                      -> [{source_id,name,market,access,licence,cadence}]
GET  /sources/{market}             -> full sources.yaml entries (source_id,name,market,access,licence,cadence,url,entities,fields,base_url,robots_ok,notes) filtered to that market + 'global'

Status: implemented, with two shape corrections. GET /sources does not return a confidence field — main.py explicitly projects only source_id,name,market,access,licence,cadence from sources.yaml (verified: {k: s.get(k) for k in (...)} with no confidence key). GET /sources/{market} returns the entire raw sources.yaml entry for each match (richer than /sources — includes base_url, robots_ok, entities, fields, notes), not a "sources + freshness" shape; there is no freshness field anywhere in sources.yaml or added by this endpoint. Both read from services/api/sources/sources.yaml (8 entries as of this pass — verified by count: gleif, brreg, companies_house, prh, cvr, bolagsverket, rdw, opensanctions; no traficom entry exists in this file, unlike the edge-native GET /api/sources route below) — an entirely separate, static list from the per-tenant Source registry (src/types/source.ts) the edge-native 0pt.query_source/0pt.source_rate_policy_manage capabilities act against; see docs/03-data-model.md and docs/12-edge-capability-api.md. A third, separate static list exists too: app/api/sources/route.ts is a hardcoded, GET-only Next.js route with its own 9-entry array (the same 8 markets plus traficom/FI, each with a confidence field sources.yaml does not have) — not part of this FastAPI contract, not wired to sources.yaml in any way, and not itself documented in detail here (out of this doc's FastAPI scope; see docs/12-edge-capability-api.md).

Vehicles & sanctions screening (implemented, previously undocumented)

GET  /vehicles/{plate}             -> RDW (NL) lookup result, or { vehicle: null } if not found
GET  /screen?q=                    -> { source: "opensanctions", matches: [{name, score, schema}] }

These two endpoints exist in main.py today but were missing from every prior version of this doc. GET /vehicles/{plate} proxies sources/connectors.py:rdw_vehicle() (Dutch open vehicle data, technical/fleet fields only, no owner PII, per the connector's own comment). GET /screen proxies sources/connectors.py:opensanctions_screen() against api.opensanctions.org — the edge route app/api/screen/route.ts calls this same path via engineFetch('/screen?q=...') first and falls back to calling OpenSanctions directly if the engine isn't reachable, which is a working (not broken) reconciliation, unlike /ai/parse above.

Subscriptions / feeds

POST /subscriptions                body:{ order_id, cadence, webhook_url } -> { sub_id }
GET  /subscriptions                -> [subs]
POST /subscriptions/{id}/pause | /resume

Status: target only. No route exists in main.py. The subscriptions table exists in sql/001_init.sql but nothing writes to it from the engine.

Webhooks out (to customers)

• On delivery / feed update: POST to subscriptions.webhook_url with { order_id, delta_url, summary }.

Status: target only — depends on Deliveries and Subscriptions above, neither of which exist yet.

Health

GET  /health                       -> { ok, service, db, sources }

Status: implemented, with a corrected shape: the real response is {ok: true, service: "0pt-data-engine", db: bool(DATABASE_URL is set), sources: len(_SOURCES)} — not {ok, db, workers, queue} as previously documented. There is no workers or queue field, and db is a presence check on an env var, not a live connectivity probe (see "Implementation status" above).

Existing-code reconciliation

This doc's target	main.py reality	Status
GET/POST /catalog*	—	not implemented (edge-only, unverified)
POST /estimates	POST /estimates	implemented, different request/response fields (see above)
POST /ai/parse	—	not implemented; edge layer calls it anyway and silently falls back
POST /orders, GET /orders, GET /orders/{id}	same paths, exist	implemented, no auth/org filtering, in-memory only
POST /orders/{id}/scope|accept|revise	PATCH /orders/{oid}, POST /orders/{oid}/run	different shape: generic status PATCH + a synchronous run trigger, no escrow gate
POST /payments/*, POST /webhooks/stripe	—	not implemented in engine (may exist at the edge — unverified this pass)
GET /orders/{id}/status	GET /orders/{oid}/status	implemented, smaller/different shape
GET /orders/{id}/events (SSE)	—	not implemented
POST /uploads, GET /orders/{id}/files	—	not implemented
POST/DELETE /connections/db-grant*	—	not implemented
GET/POST /orders/{id}/delivery*	—	not implemented; underlying quality data doesn't exist either
GET /sources, GET /sources/{market}	same paths, exist	implemented, different projected fields (no confidence/freshness)
POST/GET /subscriptions*	—	not implemented
GET /health	GET /health	implemented, different shape (no workers/queue)
—	GET /vehicles/{plate}, GET /screen	implemented, previously undocumented

Remove any endpoint that drives a bypass/stealth fetch path (CLAUDE.md §1) — none of the 12 real endpoints in main.py do; the WAF-bypass concern in this repo's history is in services/scrapers/* on the Gitea remote, not here (see CLAUDE.md's note on the two diverged remotes — an open question for Simon, not resolved by this pass).

Related: the edge-native TypeScript capability surface

app/api/0pt/*, app/api/screen/route.ts, and app/api/ai/parse/route.ts are not part of the FastAPI contract above — they're Next.js Route Handlers, run on Vercel, and implement several of the newer P0 capabilities registered in CL0UD's capability registry (0pt.query_source, 0pt.document_ocr, 0pt.lakehouse_mapping_manage, 0pt.lakehouse_write, 0pt.source_rate_policy_manage). Full route-by-route documentation, including several confirmed always-fails-over-HTTP bugs, lives in docs/12-edge-capability-api.md.