docs(data-engine): 1p extends the provenance chain to the network layer #3
@@ -36,6 +36,37 @@ Each stage updates `jobs` (`stage`, `pct`, `metrics` incl. per-market progress,
|
||||
- Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
|
||||
- `est_confidence` on the order = aggregate target; delivered `avg_confidence` = actual.
|
||||
|
||||
### Network-layer provenance: `1p` (not yet in use)
|
||||
|
||||
`bridge_provenance` answers *which source a field came from*. It does not answer *what
|
||||
actually happened on the wire* — which policy authorized the fetch, whether `robots.txt` was
|
||||
read and what it said, or whether the bytes we parsed were the page or a bot wall served with
|
||||
HTTP 200. **[`1p`](https://git.siax.io/sax3l/1p)**, the SIAX egress measurement point, closes
|
||||
that gap and extends the provenance chain all the way down to the network layer.
|
||||
|
||||
1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance,
|
||||
allowed methods), a single egress choke point, block detection, and one provenance record per
|
||||
fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD).
|
||||
|
||||
Why this matters for 0pt specifically — three facts that are hard to assert without it:
|
||||
|
||||
| Claim 0pt makes | What 1p adds |
|
||||
|---|---|
|
||||
| "acquired from an authorized public page, respecting `robots.txt`" | The record carries the robots decision **plus a sha256 of the `robots.txt` it was taken from**. A policy that says `respect` and cannot read robots denies the fetch — silence is not permission. |
|
||||
| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. |
|
||||
| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. |
|
||||
|
||||
**It is a measurement point, not a bypass.** This is consistent with, not an exception to,
|
||||
the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove*
|
||||
provenance, never to evade. Its purchased-residential egress pool is an interface with a stub
|
||||
that throws — wiring it up would be a separate, explicit decision. Block detection also
|
||||
alarms on a rising *trend*, never on single events, and deliberately excludes `error`
|
||||
(their 5xx, our timeouts) from the blocking measure.
|
||||
|
||||
**Status: not in use.** 1p is Phase 0 — code, contracts, tests; not deployed, no credentials
|
||||
set. No 0pt worker routes through it today, and `bridge_provenance` remains the only
|
||||
provenance 0pt writes. Adopting it is a later, separate step.
|
||||
|
||||
## `sources.yaml` (machine-readable registry)
|
||||
|
||||
Single registry that drives the router, the `/sources` API, and the public sourcing-map page. Schema per entry:
|
||||
|
||||
Reference in New Issue
Block a user