docs(data-engine): 1p extends the provenance chain to the network layer #3

Merged
admin merged 2 commits from docs/1p-egress-measurement-point into main 2026-08-09 08:50:39 +00:00
+31
View File
@@ -36,6 +36,37 @@ Each stage updates `jobs` (`stage`, `pct`, `metrics` incl. per-market progress,
- Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
- `est_confidence` on the order = aggregate target; delivered `avg_confidence` = actual.
### Network-layer provenance: `1p` (not yet in use)
`bridge_provenance` answers *which source a field came from*. It does not answer *what
actually happened on the wire* — which policy authorized the fetch, whether `robots.txt` was
read and what it said, or whether the bytes we parsed were the page or a bot wall served with
HTTP 200. **[`1p`](https://git.siax.io/sax3l/1p)**, the SIAX egress measurement point, closes
that gap and extends the provenance chain all the way down to the network layer.
1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance,
allowed methods), a single egress choke point, block detection, and one provenance record per
fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD).
Why this matters for 0pt specifically — three facts that are hard to assert without it:
| Claim 0pt makes | What 1p adds |
|---|---|
| "acquired from an authorized public page, respecting `robots.txt`" | The record carries the robots decision **plus a sha256 of the `robots.txt` it was taken from**. A policy that says `respect` and cannot read robots denies the fetch — silence is not permission. |
| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. |
| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. |
**It is a measurement point, not a bypass.** This is consistent with, not an exception to,
the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove*
provenance, never to evade. Its purchased-residential egress pool is an interface with a stub
that throws — wiring it up would be a separate, explicit decision. Block detection also
alarms on a rising *trend*, never on single events, and deliberately excludes `error`
(their 5xx, our timeouts) from the blocking measure.
**Status: not in use.** 1p is Phase 0 — code, contracts, tests; not deployed, no credentials
set. No 0pt worker routes through it today, and `bridge_provenance` remains the only
provenance 0pt writes. Adopting it is a later, separate step.
## `sources.yaml` (machine-readable registry)
Single registry that drives the router, the `/sources` API, and the public sourcing-map page. Schema per entry: