0pt/docs/04-api.md

# 04 · API contract (FastAPI)

Canonical REST contract for the **Python data engine** — `services/api/orchestrator/main.py`, run on a VPS/worker host, not on Vercel. JSON in/out, `snake_case`. All amounts in **öre**.

**Scope note:** this doc covers *only* the FastAPI engine. The newer edge-native TypeScript P0 capability surface (`app/api/0pt/*`, `/api/screen`, `/api/ai/parse` — Next.js Route Handlers that run on Vercel) is a separate, additive layer with its own contract: see `docs/12-edge-capability-api.md`. The two are not the same service and, as of this pass, are not fully wired together (see "Existing-code reconciliation" below for a concrete case where the edge layer expects an engine route that doesn't exist).

## Implementation status (verified this pass — read before trusting any endpoint below)

The engine file is `services/api/orchestrator/main.py` (162 lines) — not `src/orchestrator/main.py`, which does not exist in this repo (older doc text, including CLAUDE.md's "Build on existing code" section, still uses that stale path). It implements **exactly 12 endpoints**, all listed below — not "25+" as CLAUDE.md and the previous version of this doc claimed. That number does not match anything in the current codebase; flagging it here since it's this doc's own error to fix (CLAUDE.md's copy of the same claim is outside this file's scope).

Endpoints that exist in `main.py` today: `GET /health`, `GET /sources`, `GET /sources/{market}`, `GET /vehicles/{plate}`, `GET /screen`, `POST /estimates`, `POST /orders`, `GET /orders`, `GET /orders/{oid}`, `PATCH /orders/{oid}`, `POST /orders/{oid}/run`, `GET /orders/{oid}/status`.

Everything else in this document — payments/escrow, uploads, customer DB-grant, deliveries, subscriptions, outbound webhooks, `/orders/{id}/events` (SSE), `/orders/{id}/scope|accept|revise` — is a **target contract for the engine**, not yet implemented in `main.py`. Some of it already exists as a *separate* Next.js edge route instead (payments, orders, catalog, estimates, vehicles, sources all have edge-layer counterparts under `app/api/` — not read in detail for this pass, since they're outside this doc's FastAPI scope; noted only so you don't assume they don't exist anywhere).

Three structural gaps apply to **every** endpoint below that main.py does implement:

- **No persistence.** `main.py` has no `asyncpg`/`psycopg`/`sqlalchemy` import anywhere. `_ORDERS` and `_JOBS` are plain module-level Python `dict`s — state is lost on every restart, and never touches the `orders`/`jobs` tables `docs/03-data-model.md` documents. `GET /health`'s `db` field only reports whether the `DATABASE_URL` env var is *set* (`bool(os.getenv(...))`) — it never opens a connection, so it does not indicate the database is actually reachable or in use.
- **No authentication, no tenant scoping.** `main.py` never reads any request header (no `X-Internal-Token` check, despite the edge layer sending one — see `lib/engine.ts`), and no request/response model anywhere carries `org_id` or `user_id`. Concretely: **`GET /orders` returns every order held by the process, for every org, with no filtering at all.** The "Auth: … customer identity passed as `org_id`/`user_id` resolved from the Auth.js session server-side" framing below is the target design, not current behavior — this is the same platform-wide "no auth/session system exists yet" gap noted across this doc corpus (see `docs/09-build-plan.md` Phase 3), not something specific to fix in this file.
- **No idempotency, no pagination.** Neither is implemented by any endpoint in `main.py` today, despite being listed as target conventions below.

## Conventions (target contract — see gaps above for current reality)

- Errors: `{ "error": { "code", "message" } }`, proper HTTP status. *(Not implemented in `main.py`: it raises plain FastAPI `HTTPException(status, "message")`, which serializes as `{"detail": "message"}`, not the `{error:{code,message}}` shape below.)* Interface-voice messages (Swedish) are produced in the frontend, not here.
- Idempotency: accept `Idempotency-Key` on POSTs that create orders/payments. *(Target only — not implemented.)*
- Pagination: `?limit=&cursor=`. *(Target only — not implemented; `GET /orders` returns the full unfiltered list.)*
- Auth: Next.js → FastAPI over a shared internal token (`X-Internal-Token`, `lib/engine.ts` sends it via `INTERNAL_TOKEN`) — the edge side of this is real, but `main.py` never validates the header, so today it has no effect. Customer identity as `org_id`/`user_id` resolved from an Auth.js session is target design; no such session system exists in this repo yet.

## Catalog

```
GET  /catalog                      -> [{id,title,markets,entity_type,count,updated,price_ore,confidence}]
GET  /catalog/{id}                 -> full dataset (overview, coverage, schema, quality summary)
GET  /catalog/{id}/sample?n=10     -> sample rows WITH per-field provenance
```
**Status: target only.** No `/catalog` route exists in `main.py`. A separate `GET /api/catalog` exists at the edge (`app/api/catalog/route.ts`, registered in CL0UD's capability registry as `0pt.catalog_list`) — not read in detail for this pass; out of scope here.

## Estimate (configurator → price/time/confidence)
```
POST /estimates
  body: { markets[], entity_type, fields[], enrichment[], volume, frequency, confidence_target }
  -> { total_ore, deposit_ore, balance_ore, est_days, est_confidence, recurring }
```
**Status: implemented**, but the shape differs from the previous version of this doc: the request body has **no `flow` field**, and the response has **no `breakdown[]` field** — `main.py`'s `EstimateIn` model is exactly `{markets, entity_type, fields, enrichment, volume, frequency, confidence_target}` (defaults: `entity_type="Bolag"`, `volume=1000`, `frequency="Engångs"`, `confidence_target=92`), and `compute_estimate()` returns exactly `{total_ore, deposit_ore, balance_ore, est_days, est_confidence, recurring}` — `recurring` is a bool (`frequency != "Engångs"`), not a `breakdown[]` array. There is also no `formats[]` field on the request.

## AI-assist (NL → config)
```
POST /ai/parse
  body: { text }                   -- "5000 svenska SaaS-bolag med e-post och omsättning"
  -> { config: <estimate body>, rationale }   -- prefilled configurator; uses pgvector match to catalog/fields
```
**Status: not implemented — confirmed contract mismatch.** `main.py` has no `/ai/parse` route at all (grep-verified: only the 12 endpoints listed in "Implementation status" exist). This matters beyond "target only": the edge route `app/api/ai/parse/route.ts` **actively calls** `engineFetch("/ai/parse", {method:"POST", ...})` expecting the engine to have this route, checks `if (eng && eng.ok)`, and — because the engine 404s — silently falls through to a local keyword-heuristic parser every time, even when `ENGINE_URL` is configured and the engine is reachable. See `docs/12-edge-capability-api.md` for the edge side of this route. There is also no pgvector usage anywhere in `main.py` (no `embeddings` table exists either — see `docs/03-data-model.md`).

## Orders (lifecycle)
```
POST /orders                       body: { flow, config }            -> { order_id, status:'draft', deposit_ore, total_ore }
GET  /orders                       -> [orders]  -- NOT filtered by org; returns every order in the process (no auth)
GET  /orders/{id}                  -> { ...order, job }               -- job is null if none exists yet; no "delivery" field (deliveries aren't implemented)
PATCH /orders/{id}                 body: { status }                  -> updates order.status to any caller-supplied string, no validation against a lifecycle enum
POST /orders/{id}/run              -> runs the pipeline SYNCHRONOUSLY (awaits workers/pipeline.py:run_job), sets status "in_production" then "review", returns the job record
GET  /orders/{id}/status           -> the job record, or {stage:"queued", pct:0, state:"queued"} if none exists (never 404s)
```
**Status: partially implemented, but shaped differently from the target below.** What actually exists: `POST /orders`, `GET /orders`, `GET /orders/{oid}`, `PATCH /orders/{oid}`, `POST /orders/{oid}/run`, `GET /orders/{oid}/status` — six endpoints, listed above with their real behavior. What the original target contract described — `POST /orders/{id}/scope` (manual quote step), `POST /orders/{id}/accept` (triggers balance release), `POST /orders/{id}/revise` — **do not exist in `main.py` at all.** Order-status transitions today happen only through the generic `PATCH /orders/{oid}` (accepts any string, no state-machine enforcement) or implicitly through `POST /orders/{oid}/run`. `POST /orders/{oid}/run` itself is undocumented in any prior version of this doc despite being the only way to actually trigger the sourcing pipeline — it has no payment/escrow gate of any kind (see Payments below).

## Payments (50/50 escrow)
```
POST /payments/{order_id}/deposit  -> { checkout_url }   (Stripe Checkout for 50%; on success → IN_PRODUCTION)
POST /payments/{order_id}/release  -> captures/charges the held balance (called on accept)
POST /webhooks/stripe              (Stripe → here) verify signature; update payments + order status
```
**Status: not implemented in the engine at all.** There is no Stripe reference anywhere in `services/api/` (grep-verified). `POST /orders/{oid}/run` — the engine's only way to start production — has no payment dependency whatsoever; nothing in `main.py` checks that a deposit was paid before running a job. If Stripe/escrow exists anywhere in this repo, it's at the edge (`app/api/payments/[order_id]/deposit`, `app/api/payments/[order_id]/release`, `app/api/webhooks/stripe` — registered in CL0UD's capability registry as `0pt.order_payment`, evidence comment notes both routes 501 when `STRIPE_SECRET_KEY` is unset) — not read in detail this pass, out of scope here. **Open question:** whether escrow/payment gating is meant to live in the FastAPI engine (as this doc's original framing implies) or stay purely at the edge is unresolved; this doc doesn't decide it for you.

## Jobs / live status
```
GET  /orders/{id}/status           -> { stage, pct, per_market[], confidence, eta, rows }
GET  /orders/{id}/events           -> Server-Sent Events stream (preferred for the production tracker)
```
**Status: `GET /orders/{id}/status` exists but returns a different, smaller shape** — the actual dict returned by `workers/pipeline.py:run_job()` is exactly `{stage, pct, state, metrics, records}` (verified by reading its `return` statement) — **not** `{stage, pct, per_market[], confidence, eta, rows}` as previously documented, and **not** `started_at`/`finished_at` either (those two column names exist only on the unused `jobs` SQL table in `docs/03-data-model.md` — the in-memory job dict `run_job` actually returns has no timestamps at all). `metrics` itself is `{rows, dropped_below_threshold, per_market, avg_confidence}`; `records` holds the delivered rows. `GET /orders/{id}/events` (SSE) **does not exist** — target only.

## Uploads (R2 presigned)
```
POST /uploads                      body:{ order_id, name, mime, bytes } -> { file_id, put_url }  (client PUTs to R2)
GET  /orders/{id}/files            -> [files]
```
**Status: target only.** Neither route exists in `main.py`.

## Customer DB access (read-only grant)
```
POST /connections/db-grant
  body: { order_id, conn_string, scope, expires_at }   -- stored ENCRYPTED; validated read-only
  -> { grant_id, status:'pending'|'verified' }
DELETE /connections/db-grant/{id}  -> revoke
```
**Status: target only.** Neither route exists in `main.py`. The `db_grants` table exists in `sql/001_init.sql` (see `docs/03-data-model.md`) but nothing writes to it.

## Deliveries
```
GET  /orders/{id}/delivery         -> { formats, destinations, record_count, avg_confidence, quality_report, files[], provenance_export_url }
POST /orders/{id}/delivery/destinations  body:{ destinations[] }    -> updates targets (CSV/Parquet/JSON/API/NocoDB/Metabase/FiveCRM/Sheets/webhook)
```
**Status: target only.** Neither route exists in `main.py`. Note also that a "delivery" concept and a "quality_report" are described here as if built — per this doc corpus's broader honesty pass, the underlying quality/QA-check logic these would report on (`quality-check-service.ts`, `quality-report-service.ts` and friends) does not exist in this repo at all (see CLAUDE.md's pointer to README's "Known issues"). This section is aspirational on two levels: the route, and the quality data it would need to serve.

## Sources (the transparency moat)
```
GET  /sources                      -> [{source_id,name,market,access,licence,cadence}]
GET  /sources/{market}             -> full sources.yaml entries (source_id,name,market,access,licence,cadence,url,entities,fields,base_url,robots_ok,notes) filtered to that market + 'global'
```
**Status: implemented, with two shape corrections.** `GET /sources` does **not** return a `confidence` field — `main.py` explicitly projects only `source_id,name,market,access,licence,cadence` from `sources.yaml` (verified: `{k: s.get(k) for k in (...)}` with no `confidence` key). `GET /sources/{market}` returns the *entire* raw `sources.yaml` entry for each match (richer than `/sources` — includes `base_url`, `robots_ok`, `entities`, `fields`, `notes`), not a "sources + freshness" shape; there is no `freshness` field anywhere in `sources.yaml` or added by this endpoint. Both read from `services/api/sources/sources.yaml` (**8** entries as of this pass — verified by count: `gleif`, `brreg`, `companies_house`, `prh`, `cvr`, `bolagsverket`, `rdw`, `opensanctions`; no `traficom` entry exists in this file, unlike the edge-native `GET /api/sources` route below) — an entirely separate, static list from the per-tenant `Source` registry (`src/types/source.ts`) the edge-native `0pt.query_source`/`0pt.source_rate_policy_manage` capabilities act against; see `docs/03-data-model.md` and `docs/12-edge-capability-api.md`. **A third, separate static list exists too:** `app/api/sources/route.ts` is a hardcoded, `GET`-only Next.js route with its own 9-entry array (the same 8 markets plus `traficom`/FI, each with a `confidence` field sources.yaml does not have) — not part of this FastAPI contract, not wired to sources.yaml in any way, and not itself documented in detail here (out of this doc's FastAPI scope; see `docs/12-edge-capability-api.md`).

## Vehicles & sanctions screening (implemented, previously undocumented)
```
GET  /vehicles/{plate}             -> RDW (NL) lookup result, or { vehicle: null } if not found
GET  /screen?q=                    -> { source: "opensanctions", matches: [{name, score, schema}] }
```
These two endpoints exist in `main.py` today but were missing from every prior version of this doc. `GET /vehicles/{plate}` proxies `sources/connectors.py:rdw_vehicle()` (Dutch open vehicle data, technical/fleet fields only, no owner PII, per the connector's own comment). `GET /screen` proxies `sources/connectors.py:opensanctions_screen()` against `api.opensanctions.org` — the edge route `app/api/screen/route.ts` calls this same path via `engineFetch('/screen?q=...')` first and falls back to calling OpenSanctions directly if the engine isn't reachable, which is a working (not broken) reconciliation, unlike `/ai/parse` above.

## Subscriptions / feeds
```
POST /subscriptions                body:{ order_id, cadence, webhook_url } -> { sub_id }
GET  /subscriptions                -> [subs]
POST /subscriptions/{id}/pause | /resume
```
**Status: target only.** No route exists in `main.py`. The `subscriptions` table exists in `sql/001_init.sql` but nothing writes to it from the engine.

## Webhooks out (to customers)
- On delivery / feed update: POST to `subscriptions.webhook_url` with `{ order_id, delta_url, summary }`.

**Status: target only** — depends on Deliveries and Subscriptions above, neither of which exist yet.

## Health
```
GET  /health                       -> { ok, service, db, sources }
```
**Status: implemented**, with a corrected shape: the real response is `{ok: true, service: "0pt-data-engine", db: bool(DATABASE_URL is set), sources: len(_SOURCES)}` — not `{ok, db, workers, queue}` as previously documented. There is no `workers` or `queue` field, and `db` is a presence check on an env var, not a live connectivity probe (see "Implementation status" above).

## Existing-code reconciliation

| This doc's target | `main.py` reality | Status |
|---|---|---|
| `GET/POST /catalog*` | — | not implemented (edge-only, unverified) |
| `POST /estimates` | `POST /estimates` | implemented, different request/response fields (see above) |
| `POST /ai/parse` | — | not implemented; edge layer calls it anyway and silently falls back |
| `POST /orders`, `GET /orders`, `GET /orders/{id}` | same paths, exist | implemented, no auth/org filtering, in-memory only |
| `POST /orders/{id}/scope\|accept\|revise` | `PATCH /orders/{oid}`, `POST /orders/{oid}/run` | different shape: generic status PATCH + a synchronous run trigger, no escrow gate |
| `POST /payments/*`, `POST /webhooks/stripe` | — | not implemented in engine (may exist at the edge — unverified this pass) |
| `GET /orders/{id}/status` | `GET /orders/{oid}/status` | implemented, smaller/different shape |
| `GET /orders/{id}/events` (SSE) | — | not implemented |
| `POST /uploads`, `GET /orders/{id}/files` | — | not implemented |
| `POST/DELETE /connections/db-grant*` | — | not implemented |
| `GET/POST /orders/{id}/delivery*` | — | not implemented; underlying quality data doesn't exist either |
| `GET /sources`, `GET /sources/{market}` | same paths, exist | implemented, different projected fields (no `confidence`/`freshness`) |
| `POST/GET /subscriptions*` | — | not implemented |
| `GET /health` | `GET /health` | implemented, different shape (no `workers`/`queue`) |
| — | `GET /vehicles/{plate}`, `GET /screen` | implemented, previously undocumented |

Remove any endpoint that drives a bypass/stealth fetch path (CLAUDE.md §1) — none of the 12 real endpoints in `main.py` do; the WAF-bypass concern in this repo's history is in `services/scrapers/*` on the Gitea remote, not here (see CLAUDE.md's note on the two diverged remotes — an open question for Simon, not resolved by this pass).

## Related: the edge-native TypeScript capability surface

`app/api/0pt/*`, `app/api/screen/route.ts`, and `app/api/ai/parse/route.ts` are **not** part of the FastAPI contract above — they're Next.js Route Handlers, run on Vercel, and implement several of the newer P0 capabilities registered in CL0UD's capability registry (`0pt.query_source`, `0pt.document_ocr`, `0pt.lakehouse_mapping_manage`, `0pt.lakehouse_write`, `0pt.source_rate_policy_manage`). Full route-by-route documentation, including several confirmed always-fails-over-HTTP bugs, lives in **`docs/12-edge-capability-api.md`**.