diff --git a/docs/06-data-engine.md b/docs/06-data-engine.md index b16d090..ddfaf17 100644 --- a/docs/06-data-engine.md +++ b/docs/06-data-engine.md @@ -36,6 +36,37 @@ Each stage updates `jobs` (`stage`, `pct`, `metrics` incl. per-market progress, - Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree. - `est_confidence` on the order = aggregate target; delivered `avg_confidence` = actual. +### Network-layer provenance: `1p` (not yet in use) + +`bridge_provenance` answers *which source a field came from*. It does not answer *what +actually happened on the wire* — which policy authorized the fetch, whether `robots.txt` was +read and what it said, or whether the bytes we parsed were the page or a bot wall served with +HTTP 200. **[`1p`](https://git.siax.io/sax3l/1p)**, the SIAX egress measurement point, closes +that gap and extends the provenance chain all the way down to the network layer. + +1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance, +allowed methods), a single egress choke point, block detection, and one provenance record per +fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD). + +Why this matters for 0pt specifically — three facts that are hard to assert without it: + +| Claim 0pt makes | What 1p adds | +|---|---| +| "acquired from an authorized public page, respecting `robots.txt`" | The record carries the robots decision **plus a sha256 of the `robots.txt` it was taken from**. A policy that says `respect` and cannot read robots denies the fetch — silence is not permission. | +| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. | +| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. | + +**It is a measurement point, not a bypass.** This is consistent with, not an exception to, +the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove* +provenance, never to evade. Its purchased-residential egress pool is an interface with a stub +that throws — wiring it up would be a separate, explicit decision. Block detection also +alarms on a rising *trend*, never on single events, and deliberately excludes `error` +(their 5xx, our timeouts) from the blocking measure. + +**Status: not in use.** 1p is Phase 0 — code, contracts, tests; not deployed, no credentials +set. No 0pt worker routes through it today, and `bridge_provenance` remains the only +provenance 0pt writes. Adopting it is a later, separate step. + ## `sources.yaml` (machine-readable registry) Single registry that drives the router, the `/sources` API, and the public sourcing-map page. Schema per entry: