0pt/docs/06-data-engine.md

# 06 · Data engine

The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.

> ### Known issue — two diverged git remotes for this repo (unresolved, Simon's call)
> This doc describes the compliant-only sourcing chain as *the law*. That law was briefly violated on one
> of this repo's two remotes, so anyone reading this doc needs to know both exist:
> - `github.com/sax3l/0pt` (this session's `origin`) — the remote this documentation pass was verified against.
> - `git.siax.io/sax3l/0pt` (Gitea) — a **separate, diverged** remote that shares origin history with GitHub for
>   at least part of the codebase (`services/api/workers/adaptive_router.py` is byte-for-byte identical between
>   GitHub's current `main` and Gitea's post-fix PR branch, so they were once the same commit lineage).
>
> Gitea's `main` independently accumulated three commits on 2026-07-13 that added a real WAF-bypass /
> proxy-rotation / fake-identity scraper stack (`services/scrapers/*`) — exactly the category of code
> §"Prohibited" below and CLAUDE.md §1 forbid. Those commits never touched GitHub's `main`. The stack was
> since removed on Gitea via PR #1, but:
> - **Which remote is authoritative / deployed to `0pt.app` is unknown** — not resolved this session.
> - **Whether the two remotes are supposed to be kept in sync is unknown** — not resolved this session.
>
> This pass does not resolve either question. Simon: decide which remote is canonical, confirm the other has
> no lingering bypass code, and either merge or retire the loser before treating "the 0pt repo" as singular
> in any future doc pass.

## Sourcing chain (strict priority — this is the law)

For each field × market, acquire in this order; stop at the first that works:

1. **Official API / bulk** — registries with programmatic/bulk access (highest confidence).
2. **Authorized public web** — clearly public pages, **respecting `robots.txt` and rate limits**, no auth walls.
3. **Licensed partner data** — contractually obtained.
4. **Otherwise → "not available / licensed only"** — delivered honestly as level 0. **Never guessed.**

**Current implementation covers tier 1 only.** All five working connectors in `services/api/sources/connectors.py`
(`gleif_by_lei`, `brreg_by_orgnr`, `companies_house_by_number`, `prh_by_businessid`, `cvr_by_number`) call
official JSON APIs over `httpx`, nothing else. There is no HTML-scraping/parsing library anywhere in
`services/api` (no `bs4`/`lxml`/`selectolax`, no `playwright`, no `scrapy`/`crawlee` — checked against
`requirements.txt` and the source tree), so tier 2 (authorized public web) has no code path yet. Tier 3
(licensed partner) also has no connector — `sources.yaml`'s `bolagsverket` entry is `access: licensed` with
no matching function in `connectors.py`; it correctly falls through to level 0 today. Both tiers remain
target design, not built.

### Prohibited (do not build, keep, or extend)
- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection.
- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
- **Reconciliation note (stale — kept for history):** this section used to say "if `browser_worker.py` has an
  `HTTP → FlareSolverr → Playwright` chain, drop the FlareSolverr stage." **`browser_worker.py` does not exist
  anywhere in this repository** (verified this pass — no file of that name on GitHub's `main`). There is
  nothing to reconcile in the current Python engine. The bypass code that *did* exist, briefly, lived in
  `services/scrapers/*` on the Gitea remote only (see the Known Issue callout above) and has since been
  removed there via PR #1. `adaptive_router.py` (below) has never contained a bypass tier on either remote —
  it chooses among **compliant** methods only (API > bulk > authorized fetch > licensed), by design.

## Pipeline stages (per job)

```
sourcing → extraction/OCR → structuring → enrichment/fusion → QA
```

The stage names below are the target architecture. `services/api/workers/pipeline.py::run_job` is the one
function that actually runs today, and it implements a much smaller slice of this than the stage list
implies — see the "built vs. not built" status called out under each stage below.

1. **Sourcing** — `adaptive_router.plan_for_market(market)` returns a hardcoded, ordered list of compliant
   `source_id`s per market (see table below); `pipeline.py` tries each in order for a given key and **stops at
   the first one that returns a result** (`break` after first hit). This is sequential fallback, not
   concurrent racing — the phrase "source-racing = query several authorized sources and take the
   fastest/freshest" describes a target behavior that isn't implemented; today only one source is ever queried
   per key unless the first fails. Writes raw payload fields directly into the record dict (no separate
   raw-payload store).
2. **Extraction / OCR — not built.** There is no `ocr_worker`, no `unstructured`, no `paddleocr`/`pytesseract`
   anywhere in this repo (checked `services/api/requirements.txt` and the full source tree — none present).
   "Extraction" today is just each connector in `connectors.py` parsing the JSON response of its own API call
   into a `CompanyResult`. Document/PDF/image ingestion and the OCR cascade described here are spec only.
3. **Structuring — partially built.** `services/api/workers/resolve.py::resolve()` does real, deterministic
   entity resolution: block by `(market, name[:1].upper())`, then `rapidfuzz.fuzz.token_sort_ratio(...) >= 92`
   to merge duplicates, bumping `confidence` by +0.1 per corroborating match (capped 0.99) and incrementing
   `agreed_sources`. That part is real and unit-testable. What's **not** built: no spaCy/transformers, no NER,
   no multilingual normalization (`sv/no/da/fi/de/fr` from the original text), and no explicit LEI-based
   cross-market entity linking — matching is name-string-similarity only, within the same market. `spacy` and
   `transformers` are not in `requirements.txt`.
4. **Enrichment / fusion — partially built, not fully wired into the job.** The confidence/agreement bump
   described above happens inside `resolve()`. Separately, `opensanctions_screen()` (sanctions/PEP) and
   `rdw_vehicle()` (NL vehicle data) are real, working connectors in `connectors.py` — but `pipeline.py` never
   imports or calls either one. They're only reachable as their own standalone endpoints
   (`GET /screen`, `GET /vehicles/{plate}` in `orchestrator/main.py`), not as an automatic enrichment step on
   an order's records. NACE/SNI classification and "website/tech" enrichment have no code anywhere.
5. **QA — not built beyond a single threshold filter.** The entire QA stage in `pipeline.py` is:
   `delivered = [r for r in deduped if r["confidence"] >= threshold]`. That's real and does drop
   below-threshold rows (deliver fewer, never fabricate — that principle holds). But there is no
   Great Expectations (not in `requirements.txt`, not imported anywhere), no separate validity/format,
   completeness, freshness, or post-dedup-uniqueness checks beyond the confidence number already computed in
   structuring, and no quality-report object emitted or stored anywhere — `deliveries` (the SQL table defined
   in `sql/001_init.sql`) is never written to by this service. This mirrors a confirmed, broader finding this
   session: 0pt's data-quality/QA layer was at one point documented elsewhere (TypeScript side,
   `src/services/quality-check-service.ts` etc.) as if built and tested, but those files **do not exist** in
   this repo's history at all (zero commits, any branch — see `docs/07-dependencies.md` note). `docs/ICEBERG_WRITE_BOUNDARY.md`,
   `docs/11-lakehouse-write-plan.md`, and `docs/P0_QUERY_SOURCE_FLOW.md` have all since been corrected — each
   now carries its own explicit "fabricated, not built" callout rather than asserting those files as real —
   and `docs/P0_DATASET_QUALITY_FLOW.md` documents this gap as target-spec-only. The Python engine
   independently confirms the same gap: **no real quality-check enforcement exists in this codebase today**,
   in either language.

**"Each stage updates `jobs` ... so the production tracker / SSE can render it live" is not what happens.**
`run_job` runs synchronously start-to-finish and returns one final dict (`stage: "qa", pct: 100, state: "done",
metrics, records`). `orchestrator/main.py`'s `POST /orders/{oid}/run` awaits the whole thing and stores that one
result in an in-memory `_JOBS` dict; `GET /orders/{oid}/status` just returns whatever's there, or a static
`{"stage": "queued", "pct": 0, "state": "queued"}` placeholder before it's run. There is no per-stage
incremental progress, no `pct` that climbs, and **no SSE endpoint exists anywhere in `orchestrator/main.py`**
(grepped — none). Live progress rendering is target design, not current behavior.

### `adaptive_router` — actual market coverage (verified against `services/api/workers/adaptive_router.py`)

`MARKET_SOURCES` currently has real entries for exactly five markets, each a short compliant list — not the
"elaborate multi-source lists" an earlier version of this router may have implied:

| Market | Ordered plan | Note |
|---|---|---|
| NO | `brreg`, `gleif` | |
| GB | `companies_house`, `gleif` | |
| FI | `prh`, `gleif` | |
| DK | `cvr`, `gleif` | |
| SE | `gleif` only | Bolagsverket bulk is licensed (see above); SE company core relies on GLEIF + a not-yet-built licensed/partner route |
| anything else | `["gleif"]` | fallback in `plan_for_market()` |

`ACCESS_RANK` (`api=0, bulk=1, authorized_web=2, licensed=3`) is defined in the same file but is **not
actually used** by `plan_for_market()` to sort anything — the lists above are hand-ordered constants already
in the desired priority, and the rank dict is currently dead code (documentation-as-code, not logic).

## Confidence & provenance scoring

- Per field, `connectors.py`'s `Provenance` dataclass carries: `field_name`, `source_id`, `confidence` (0–1),
  `level` (0–3), `freshness`. **`agreed_sources` is not a field on `Provenance`** — it's tracked separately, at
  the merged-record level, by `resolve()` (`match["agreed_sources"] = ... + 1`). If a doc or downstream
  consumer expects `agreed_sources` inside each `bridge_provenance` row, that's a mismatch to reconcile against
  the actual dataclass shape, not a fact to assume.
- Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
- `est_confidence` on the order = aggregate target (computed in `orchestrator/main.py::compute_estimate`);
  delivered `avg_confidence` = actual, computed in `pipeline.py` as the mean confidence of delivered rows.
- **None of this is persisted to Postgres by the current service.** `sql/001_init.sql` defines the
  `bridge_provenance` table (plus `dim_company`, `dim_source`, `fact_financials`, `deliveries`, etc.), and
  `asyncpg` is a listed dependency — but nothing in `services/api` actually calls `asyncpg` or issues a query.
  `DATABASE_URL` is read exactly once, in `orchestrator/main.py`'s `/health` handler, purely to report a
  boolean (`"db": bool(os.getenv("DATABASE_URL"))`). `orchestrator/main.py` stores orders and jobs in
  in-memory dicts (`_ORDERS`, `_JOBS`) that reset on every restart. The schema exists; the wiring from
  pipeline output → Postgres rows does not yet.

### Network-layer provenance: `1p` (not yet in use)

`bridge_provenance` answers *which source a field came from*. It does not answer *what
actually happened on the wire* — which policy authorized the fetch, whether `robots.txt` was
read and what it said, or whether the bytes we parsed were the page or a bot wall served with
HTTP 200. **[`1p`](https://git.siax.io/sax3l/1p)**, the SIAX egress measurement point, closes
that gap and extends the provenance chain all the way down to the network layer.

1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance,
allowed methods), a single egress choke point, block detection, and one provenance record per
fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD).

Why this matters for 0pt specifically — three facts that are hard to assert without it:

| Claim 0pt makes | What 1p adds |
|---|---|
| "acquired from an authorized public page, respecting `robots.txt`" | The record carries the robots decision **plus a sha256 of the `robots.txt` it was taken from**. A policy that says `respect` and cannot read robots denies the fetch — silence is not permission. |
| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. |
| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. |

**It is a measurement point, not a bypass.** This is consistent with, not an exception to,
the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove*
provenance, never to evade. Its purchased-residential egress pool is an interface with a stub
that throws — wiring it up would be a separate, explicit decision. Block detection also
alarms on a rising *trend*, never on single events, and deliberately excludes `error`
(their 5xx, our timeouts) from the blocking measure.

**Status: not in use.** 1p is Phase 0 — code, contracts, tests; not deployed, no credentials
set. No 0pt worker routes through it today, and `bridge_provenance` remains the only
provenance 0pt writes. Adopting it is a later, separate step.

## `sources.yaml` (machine-readable registry)

Single registry that drives the router, the `/sources` API, and the (still-to-be-built) public sourcing-map
page. `services/api/sources/sources.yaml` currently has **8 entries**: `gleif`, `brreg`, `companies_house`,
`prh`, `cvr`, `bolagsverket`, `rdw`, `opensanctions`. Schema per entry (as actually used, confirmed against the
file — no `entities`/`fields` mismatch found):

```yaml
- source_id: gleif
  name: GLEIF (LEI)
  market: global
  access: api            # api | bulk | authorized_web | licensed
  cadence: daily
  licence: "open (CC0) — VERIFY current terms"
  entities: [company]
  fields: [lei, legal_name, address, status]
  base_url: https://api.gleif.org
  robots_ok: true        # for authorized_web only; must be honored
  notes: "cross-border key"
```

`orchestrator/main.py` loads this file once at import time (`yaml.safe_load(...)`) and serves it through
`GET /sources` (id/name/market/access/licence/cadence only) and `GET /sources/{market}`. It is genuinely the
live registry behind those two endpoints — that part of the doc's original claim holds.

### Starter set (VERIFY current access/licence per source before relying — terms change)
- **GLEIF / LEI** — typically open; cross-border key. Connector: `gleif_by_lei` — real, working.
- **Brønnøysund (Brreg, NO)** — typically open API. Connector: `brreg_by_orgnr` — real, working.
- **Companies House (UK)** — free API (key required, `COMPANIES_HOUSE_API_KEY`). Connector:
  `companies_house_by_number` — real; degrades honestly to level-0 unavailable if the key is unset (does not
  fabricate).
- **PRH (FI)** — Connector: `prh_by_businessid` — real, working.
- **CVR / Virk (DK)** — connector actually calls `cvrapi.dk` (a third-party aggregator), **not** Virk's own API
  directly, despite the registry entry's `base_url: https://datacvr.virk.dk`. `sources.yaml`'s `base_url` for
  `cvr` does not match what `connectors.py::cvr_by_number` actually calls — flag for reconciliation (either
  update `base_url` to `cvrapi.dk` or switch the connector to call Virk directly; not resolved this pass).
- **Bolagsverket (SE)** — restricted for bulk; `sources.yaml` marks it `access: licensed`; **no connector
  exists** for it in `connectors.py`. Consistent with the doc's own "plan licensed/partner route" framing —
  not a false claim, just confirming it's still unbuilt.
- **RDW (NL)** vehicle data — open, Socrata-backed. Connector: `rdw_vehicle` — real, working, but **not called
  by the job pipeline** (only reachable via `GET /vehicles/{plate}`).
- **OpenSanctions** — global sanctions/PEP screening. Connector: `opensanctions_screen` — real, working, same
  caveat: reachable only via `GET /screen`, not part of `run_job`.
- Vehicle beyond RDW (**Traficom FI**, **Transportstyrelsen SE**, **DVLA UK**) — no connectors exist yet;
  *(verify each before building; vehicle/PII rules differ)*.

> Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots
> status and record it in `sources.yaml`. If unclear or disallowed for bulk, route to licensed/partner or mark
> fields level 0.

## Structured financials (XBRL) — real but unwired

`services/api/sources/financials.py::companies_house_filings()` is a real, working function: given a
Companies House number and `COMPANIES_HOUSE_API_KEY`, it fetches the accounts filing history and returns up to
10 filings. It is **not imported or called anywhere else in the codebase** (grepped `services/api` — only
`financials.py` itself references the function name). The file's closing comment describes a further XBRL
extraction step ("with `arelle` installed, download each accounts filing's iXBRL document and pull tagged
facts...") that has no code at all — `arelle` is not in `requirements.txt`. Treat the whole file as a real but
disconnected building block, not a wired pipeline stage.

## Libraries used here

Target list only — see `docs/07-dependencies.md` for what's actually installed today vs. still to be added,
with an explicit install-status callout per package. As of this pass, the *only* Python packages installed for
this engine are `fastapi`, `uvicorn`, `pydantic`, `httpx`, `asyncpg` (unused), `python-dotenv`, `pyyaml`,
`rapidfuzz`, `tenacity` — i.e. the API-only tier 1 sourcing + structuring-by-fuzzy-match slice described above,
nothing from the extraction/OCR/NER/quality layers this section used to list as if already in place.