From a06f86095fb09ad1131ada2b0aefc49988253eb6 Mon Sep 17 00:00:00 2001 From: Simon Axelsson Date: Sun, 26 Jul 2026 13:57:38 +0200 Subject: [PATCH] docs(data-engine): 1p extends the provenance chain to the network layer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bridge_provenance answers which source a field came from; it does not answer what happened on the wire. 1p (the SIAX egress measurement point) adds the robots decision + sha256 of the robots.txt it was read from, body_sha256 per fetch, and blocked_soft detection — HTTP 200 whose body is a block page, which otherwise gets scored as corroborating data. Consistent with CLAUDE.md §1: a measurement point, not a bypass. Documentation only; not in use, no worker routes through it. Co-Authored-By: Claude Opus 5 --- docs/06-data-engine.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/06-data-engine.md b/docs/06-data-engine.md index b16d090..ddfaf17 100644 --- a/docs/06-data-engine.md +++ b/docs/06-data-engine.md @@ -36,6 +36,37 @@ Each stage updates `jobs` (`stage`, `pct`, `metrics` incl. per-market progress, - Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree. - `est_confidence` on the order = aggregate target; delivered `avg_confidence` = actual. +### Network-layer provenance: `1p` (not yet in use) + +`bridge_provenance` answers *which source a field came from*. It does not answer *what +actually happened on the wire* — which policy authorized the fetch, whether `robots.txt` was +read and what it said, or whether the bytes we parsed were the page or a bot wall served with +HTTP 200. **[`1p`](https://git.siax.io/sax3l/1p)**, the SIAX egress measurement point, closes +that gap and extends the provenance chain all the way down to the network layer. + +1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance, +allowed methods), a single egress choke point, block detection, and one provenance record per +fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD). + +Why this matters for 0pt specifically — three facts that are hard to assert without it: + +| Claim 0pt makes | What 1p adds | +|---|---| +| "acquired from an authorized public page, respecting `robots.txt`" | The record carries the robots decision **plus a sha256 of the `robots.txt` it was taken from**. A policy that says `respect` and cannot read robots denies the fetch — silence is not permission. | +| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. | +| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. | + +**It is a measurement point, not a bypass.** This is consistent with, not an exception to, +the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove* +provenance, never to evade. Its purchased-residential egress pool is an interface with a stub +that throws — wiring it up would be a separate, explicit decision. Block detection also +alarms on a rising *trend*, never on single events, and deliberately excludes `error` +(their 5xx, our timeouts) from the blocking measure. + +**Status: not in use.** 1p is Phase 0 — code, contracts, tests; not deployed, no credentials +set. No 0pt worker routes through it today, and `bridge_provenance` remains the only +provenance 0pt writes. Adopting it is a later, separate step. + ## `sources.yaml` (machine-readable registry) Single registry that drives the router, the `/sources` API, and the public sourcing-map page. Schema per entry: -- 2.54.0