# 06 · Data engine The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec. > ### Known issue — two diverged git remotes for this repo (unresolved, Simon's call) > This doc describes the compliant-only sourcing chain as *the law*. That law was briefly violated on one > of this repo's two remotes, so anyone reading this doc needs to know both exist: > - `github.com/sax3l/0pt` (this session's `origin`) — the remote this documentation pass was verified against. > - `git.siax.io/sax3l/0pt` (Gitea) — a **separate, diverged** remote that shares origin history with GitHub for > at least part of the codebase (`services/api/workers/adaptive_router.py` is byte-for-byte identical between > GitHub's current `main` and Gitea's post-fix PR branch, so they were once the same commit lineage). > > Gitea's `main` independently accumulated three commits on 2026-07-13 that added a real WAF-bypass / > proxy-rotation / fake-identity scraper stack (`services/scrapers/*`) — exactly the category of code > §"Prohibited" below and CLAUDE.md §1 forbid. Those commits never touched GitHub's `main`. The stack was > since removed on Gitea via PR #1, but: > - **Which remote is authoritative / deployed to `0pt.app` is unknown** — not resolved this session. > - **Whether the two remotes are supposed to be kept in sync is unknown** — not resolved this session. > > This pass does not resolve either question. Simon: decide which remote is canonical, confirm the other has > no lingering bypass code, and either merge or retire the loser before treating "the 0pt repo" as singular > in any future doc pass. ## Sourcing chain (strict priority — this is the law) For each field × market, acquire in this order; stop at the first that works: 1. **Official API / bulk** — registries with programmatic/bulk access (highest confidence). 2. **Authorized public web** — clearly public pages, **respecting `robots.txt` and rate limits**, no auth walls. 3. **Licensed partner data** — contractually obtained. 4. **Otherwise → "not available / licensed only"** — delivered honestly as level 0. **Never guessed.** **Current implementation covers tier 1 only.** All five working connectors in `services/api/sources/connectors.py` (`gleif_by_lei`, `brreg_by_orgnr`, `companies_house_by_number`, `prh_by_businessid`, `cvr_by_number`) call official JSON APIs over `httpx`, nothing else. There is no HTML-scraping/parsing library anywhere in `services/api` (no `bs4`/`lxml`/`selectolax`, no `playwright`, no `scrapy`/`crawlee` — checked against `requirements.txt` and the source tree), so tier 2 (authorized public web) has no code path yet. Tier 3 (licensed partner) also has no connector — `sources.yaml`'s `bolagsverket` entry is `access: licensed` with no matching function in `connectors.py`; it correctly falls through to level 0 today. Both tiers remain target design, not built. ### Prohibited (do not build, keep, or extend) - WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection. - Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.). - **Reconciliation note (stale — kept for history):** this section used to say "if `browser_worker.py` has an `HTTP → FlareSolverr → Playwright` chain, drop the FlareSolverr stage." **`browser_worker.py` does not exist anywhere in this repository** (verified this pass — no file of that name on GitHub's `main`). There is nothing to reconcile in the current Python engine. The bypass code that *did* exist, briefly, lived in `services/scrapers/*` on the Gitea remote only (see the Known Issue callout above) and has since been removed there via PR #1. `adaptive_router.py` (below) has never contained a bypass tier on either remote — it chooses among **compliant** methods only (API > bulk > authorized fetch > licensed), by design. ## Pipeline stages (per job) ``` sourcing → extraction/OCR → structuring → enrichment/fusion → QA ``` The stage names below are the target architecture. `services/api/workers/pipeline.py::run_job` is the one function that actually runs today, and it implements a much smaller slice of this than the stage list implies — see the "built vs. not built" status called out under each stage below. 1. **Sourcing** — `adaptive_router.plan_for_market(market)` returns a hardcoded, ordered list of compliant `source_id`s per market (see table below); `pipeline.py` tries each in order for a given key and **stops at the first one that returns a result** (`break` after first hit). This is sequential fallback, not concurrent racing — the phrase "source-racing = query several authorized sources and take the fastest/freshest" describes a target behavior that isn't implemented; today only one source is ever queried per key unless the first fails. Writes raw payload fields directly into the record dict (no separate raw-payload store). 2. **Extraction / OCR — not built.** There is no `ocr_worker`, no `unstructured`, no `paddleocr`/`pytesseract` anywhere in this repo (checked `services/api/requirements.txt` and the full source tree — none present). "Extraction" today is just each connector in `connectors.py` parsing the JSON response of its own API call into a `CompanyResult`. Document/PDF/image ingestion and the OCR cascade described here are spec only. 3. **Structuring — partially built.** `services/api/workers/resolve.py::resolve()` does real, deterministic entity resolution: block by `(market, name[:1].upper())`, then `rapidfuzz.fuzz.token_sort_ratio(...) >= 92` to merge duplicates, bumping `confidence` by +0.1 per corroborating match (capped 0.99) and incrementing `agreed_sources`. That part is real and unit-testable. What's **not** built: no spaCy/transformers, no NER, no multilingual normalization (`sv/no/da/fi/de/fr` from the original text), and no explicit LEI-based cross-market entity linking — matching is name-string-similarity only, within the same market. `spacy` and `transformers` are not in `requirements.txt`. 4. **Enrichment / fusion — partially built, not fully wired into the job.** The confidence/agreement bump described above happens inside `resolve()`. Separately, `opensanctions_screen()` (sanctions/PEP) and `rdw_vehicle()` (NL vehicle data) are real, working connectors in `connectors.py` — but `pipeline.py` never imports or calls either one. They're only reachable as their own standalone endpoints (`GET /screen`, `GET /vehicles/{plate}` in `orchestrator/main.py`), not as an automatic enrichment step on an order's records. NACE/SNI classification and "website/tech" enrichment have no code anywhere. 5. **QA — not built beyond a single threshold filter.** The entire QA stage in `pipeline.py` is: `delivered = [r for r in deduped if r["confidence"] >= threshold]`. That's real and does drop below-threshold rows (deliver fewer, never fabricate — that principle holds). But there is no Great Expectations (not in `requirements.txt`, not imported anywhere), no separate validity/format, completeness, freshness, or post-dedup-uniqueness checks beyond the confidence number already computed in structuring, and no quality-report object emitted or stored anywhere — `deliveries` (the SQL table defined in `sql/001_init.sql`) is never written to by this service. This mirrors a confirmed, broader finding this session: 0pt's data-quality/QA layer was at one point documented elsewhere (TypeScript side, `src/services/quality-check-service.ts` etc.) as if built and tested, but those files **do not exist** in this repo's history at all (zero commits, any branch — see `docs/07-dependencies.md` note). `docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, and `docs/P0_QUERY_SOURCE_FLOW.md` have all since been corrected — each now carries its own explicit "fabricated, not built" callout rather than asserting those files as real — and `docs/P0_DATASET_QUALITY_FLOW.md` documents this gap as target-spec-only. The Python engine independently confirms the same gap: **no real quality-check enforcement exists in this codebase today**, in either language. **"Each stage updates `jobs` ... so the production tracker / SSE can render it live" is not what happens.** `run_job` runs synchronously start-to-finish and returns one final dict (`stage: "qa", pct: 100, state: "done", metrics, records`). `orchestrator/main.py`'s `POST /orders/{oid}/run` awaits the whole thing and stores that one result in an in-memory `_JOBS` dict; `GET /orders/{oid}/status` just returns whatever's there, or a static `{"stage": "queued", "pct": 0, "state": "queued"}` placeholder before it's run. There is no per-stage incremental progress, no `pct` that climbs, and **no SSE endpoint exists anywhere in `orchestrator/main.py`** (grepped — none). Live progress rendering is target design, not current behavior. ### `adaptive_router` — actual market coverage (verified against `services/api/workers/adaptive_router.py`) `MARKET_SOURCES` currently has real entries for exactly five markets, each a short compliant list — not the "elaborate multi-source lists" an earlier version of this router may have implied: | Market | Ordered plan | Note | |---|---|---| | NO | `brreg`, `gleif` | | | GB | `companies_house`, `gleif` | | | FI | `prh`, `gleif` | | | DK | `cvr`, `gleif` | | | SE | `gleif` only | Bolagsverket bulk is licensed (see above); SE company core relies on GLEIF + a not-yet-built licensed/partner route | | anything else | `["gleif"]` | fallback in `plan_for_market()` | `ACCESS_RANK` (`api=0, bulk=1, authorized_web=2, licensed=3`) is defined in the same file but is **not actually used** by `plan_for_market()` to sort anything — the lists above are hand-ordered constants already in the desired priority, and the rank dict is currently dead code (documentation-as-code, not logic). ## Confidence & provenance scoring - Per field, `connectors.py`'s `Provenance` dataclass carries: `field_name`, `source_id`, `confidence` (0–1), `level` (0–3), `freshness`. **`agreed_sources` is not a field on `Provenance`** — it's tracked separately, at the merged-record level, by `resolve()` (`match["agreed_sources"] = ... + 1`). If a doc or downstream consumer expects `agreed_sources` inside each `bridge_provenance` row, that's a mismatch to reconcile against the actual dataclass shape, not a fact to assume. - Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree. - `est_confidence` on the order = aggregate target (computed in `orchestrator/main.py::compute_estimate`); delivered `avg_confidence` = actual, computed in `pipeline.py` as the mean confidence of delivered rows. - **None of this is persisted to Postgres by the current service.** `sql/001_init.sql` defines the `bridge_provenance` table (plus `dim_company`, `dim_source`, `fact_financials`, `deliveries`, etc.), and `asyncpg` is a listed dependency — but nothing in `services/api` actually calls `asyncpg` or issues a query. `DATABASE_URL` is read exactly once, in `orchestrator/main.py`'s `/health` handler, purely to report a boolean (`"db": bool(os.getenv("DATABASE_URL"))`). `orchestrator/main.py` stores orders and jobs in in-memory dicts (`_ORDERS`, `_JOBS`) that reset on every restart. The schema exists; the wiring from pipeline output → Postgres rows does not yet. ### Network-layer provenance: `1p` (not yet in use) `bridge_provenance` answers *which source a field came from*. It does not answer *what actually happened on the wire* — which policy authorized the fetch, whether `robots.txt` was read and what it said, or whether the bytes we parsed were the page or a bot wall served with HTTP 200. **[`1p`](https://git.siax.io/sax3l/1p)**, the SIAX egress measurement point, closes that gap and extends the provenance chain all the way down to the network layer. 1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance, allowed methods), a single egress choke point, block detection, and one provenance record per fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD). Why this matters for 0pt specifically — three facts that are hard to assert without it: | Claim 0pt makes | What 1p adds | |---|---| | "acquired from an authorized public page, respecting `robots.txt`" | The record carries the robots decision **plus a sha256 of the `robots.txt` it was taken from**. A policy that says `respect` and cannot read robots denies the fetch — silence is not permission. | | "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. | | "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. | **It is a measurement point, not a bypass.** This is consistent with, not an exception to, the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove* provenance, never to evade. Its purchased-residential egress pool is an interface with a stub that throws — wiring it up would be a separate, explicit decision. Block detection also alarms on a rising *trend*, never on single events, and deliberately excludes `error` (their 5xx, our timeouts) from the blocking measure. **Status: not in use.** 1p is Phase 0 — code, contracts, tests; not deployed, no credentials set. No 0pt worker routes through it today, and `bridge_provenance` remains the only provenance 0pt writes. Adopting it is a later, separate step. ## `sources.yaml` (machine-readable registry) Single registry that drives the router, the `/sources` API, and the (still-to-be-built) public sourcing-map page. `services/api/sources/sources.yaml` currently has **8 entries**: `gleif`, `brreg`, `companies_house`, `prh`, `cvr`, `bolagsverket`, `rdw`, `opensanctions`. Schema per entry (as actually used, confirmed against the file — no `entities`/`fields` mismatch found): ```yaml - source_id: gleif name: GLEIF (LEI) market: global access: api # api | bulk | authorized_web | licensed cadence: daily licence: "open (CC0) — VERIFY current terms" entities: [company] fields: [lei, legal_name, address, status] base_url: https://api.gleif.org robots_ok: true # for authorized_web only; must be honored notes: "cross-border key" ``` `orchestrator/main.py` loads this file once at import time (`yaml.safe_load(...)`) and serves it through `GET /sources` (id/name/market/access/licence/cadence only) and `GET /sources/{market}`. It is genuinely the live registry behind those two endpoints — that part of the doc's original claim holds. ### Starter set (VERIFY current access/licence per source before relying — terms change) - **GLEIF / LEI** — typically open; cross-border key. Connector: `gleif_by_lei` — real, working. - **Brønnøysund (Brreg, NO)** — typically open API. Connector: `brreg_by_orgnr` — real, working. - **Companies House (UK)** — free API (key required, `COMPANIES_HOUSE_API_KEY`). Connector: `companies_house_by_number` — real; degrades honestly to level-0 unavailable if the key is unset (does not fabricate). - **PRH (FI)** — Connector: `prh_by_businessid` — real, working. - **CVR / Virk (DK)** — connector actually calls `cvrapi.dk` (a third-party aggregator), **not** Virk's own API directly, despite the registry entry's `base_url: https://datacvr.virk.dk`. `sources.yaml`'s `base_url` for `cvr` does not match what `connectors.py::cvr_by_number` actually calls — flag for reconciliation (either update `base_url` to `cvrapi.dk` or switch the connector to call Virk directly; not resolved this pass). - **Bolagsverket (SE)** — restricted for bulk; `sources.yaml` marks it `access: licensed`; **no connector exists** for it in `connectors.py`. Consistent with the doc's own "plan licensed/partner route" framing — not a false claim, just confirming it's still unbuilt. - **RDW (NL)** vehicle data — open, Socrata-backed. Connector: `rdw_vehicle` — real, working, but **not called by the job pipeline** (only reachable via `GET /vehicles/{plate}`). - **OpenSanctions** — global sanctions/PEP screening. Connector: `opensanctions_screen` — real, working, same caveat: reachable only via `GET /screen`, not part of `run_job`. - Vehicle beyond RDW (**Traficom FI**, **Transportstyrelsen SE**, **DVLA UK**) — no connectors exist yet; *(verify each before building; vehicle/PII rules differ)*. > Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots > status and record it in `sources.yaml`. If unclear or disallowed for bulk, route to licensed/partner or mark > fields level 0. ## Structured financials (XBRL) — real but unwired `services/api/sources/financials.py::companies_house_filings()` is a real, working function: given a Companies House number and `COMPANIES_HOUSE_API_KEY`, it fetches the accounts filing history and returns up to 10 filings. It is **not imported or called anywhere else in the codebase** (grepped `services/api` — only `financials.py` itself references the function name). The file's closing comment describes a further XBRL extraction step ("with `arelle` installed, download each accounts filing's iXBRL document and pull tagged facts...") that has no code at all — `arelle` is not in `requirements.txt`. Treat the whole file as a real but disconnected building block, not a wired pipeline stage. ## Libraries used here Target list only — see `docs/07-dependencies.md` for what's actually installed today vs. still to be added, with an explicit install-status callout per package. As of this pass, the *only* Python packages installed for this engine are `fastapi`, `uvicorn`, `pydantic`, `httpx`, `asyncpg` (unused), `python-dotenv`, `pyyaml`, `rapidfuzz`, `tenacity` — i.e. the API-only tier 1 sourcing + structuring-by-fuzzy-match slice described above, nothing from the extraction/OCR/NER/quality layers this section used to list as if already in place.