chore(salvage): preserve uncommitted WIP from /opt/vehiq-work/0pt-p060-p068_0pt before machine cleanup (2026-09-27)
CI (SIAX Cloud) / quality (push) Successful in 49s
CI (SIAX Cloud) / security (push) Successful in 7m5s
Security Scan (SIAX Cloud) / security (push) Successful in 7m5s
CI (SIAX Cloud) / integration (push) Successful in 18m53s
CI (SIAX Cloud) / security (pull_request) Successful in 46s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 58s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m38s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m45s

This snapshot was found sitting uncommitted in a stale /opt/vehiq-work
lane clone dated ~2026-09-09..18, base branch: feat/p060-p068-ingestion-plane.
Not reviewed, not verified against current main. Pushed only so the
directory could be deleted without losing the diff.
This commit is contained in:
opt-vehiq-work salvage
2026-09-27 12:58:50 +02:00
parent 9267576b7c
commit 1d63f63dfd
18 changed files with 189 additions and 80 deletions
+16 -10
View File
@@ -1,9 +1,15 @@
# 06 · Data engine
The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.
> **Capability/egress decision 2026-09-16:** proxy, rotation, stealth browser
> and managed unblock are optional 1p capabilities. Older anti-bypass language
> in this document is superseded as product policy and remains only as history
> about the local implementation. See
> `docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md`.
The engine turns sources into delivered, provenance-tagged data. It is the governed data motor. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.
> ### Known issue — two diverged git remotes for this repo (unresolved, Simon's call)
> This doc describes the compliant-only sourcing chain as *the law*. That law was briefly violated on one
> This doc previously treated the standard sourcing chain as the only allowed path. One
> of this repo's two remotes, so anyone reading this doc needs to know both exist:
> - `github.com/sax3l/0pt` (this session's `origin`) — the remote this documentation pass was verified against.
> - `git.siax.io/sax3l/0pt` (Gitea) — a **separate, diverged** remote that shares origin history with GitHub for
@@ -11,8 +17,8 @@ The engine turns sources into delivered, provenance-tagged data. It is the legal
> GitHub's current `main` and Gitea's post-fix PR branch, so they were once the same commit lineage).
>
> Gitea's `main` independently accumulated three commits on 2026-07-13 that added a real WAF-bypass /
> proxy-rotation / fake-identity scraper stack (`services/scrapers/*`) — exactly the category of code
> §"Prohibited" below and CLAUDE.md §1 forbid. Those commits never touched GitHub's `main`. The stack was
> proxy-rotation / identity-profile scraper stack (`services/scrapers/*`). Under the current
> architecture those capabilities belong to b0ts + 1p rather than a local 0pt transport owner. Those commits never touched GitHub's `main`. The stack was
> since removed on Gitea via PR #1, but:
> - **Which remote is authoritative / deployed to `0pt.app` is unknown** — not resolved this session.
> - **Whether the two remotes are supposed to be kept in sync is unknown** — not resolved this session.
@@ -21,7 +27,7 @@ The engine turns sources into delivered, provenance-tagged data. It is the legal
> no lingering bypass code, and either merge or retire the loser before treating "the 0pt repo" as singular
> in any future doc pass.
## Sourcing chain (strict priority — this is the law)
## Standard sourcing chain
For each field × market, acquire in this order; stop at the first that works:
@@ -39,9 +45,9 @@ official JSON APIs over `httpx`, nothing else. There is no HTML-scraping/parsing
no matching function in `connectors.py`; it correctly falls through to level 0 today. Both tiers remain
target design, not built.
### Prohibited (do not build, keep, or extend)
- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection.
- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
### Optional high-risk capabilities (canonical owner: 1p)
- WAF/challenge handling, stealth/patched browsers and undetected-style browser profiles are modeled as explicit 1p execution profiles.
- Proxy/fingerprint rotation, authenticated-session collection and account automation require their own scope, entitlement, isolation, budget and audit; they are not implicit stages in the standard 0pt pipeline.
- **Reconciliation note (stale — kept for history):** this section used to say "if `browser_worker.py` has an
`HTTP → FlareSolverr → Playwright` chain, drop the FlareSolverr stage." **`browser_worker.py` does not exist
anywhere in this repository** (verified this pass — no file of that name on GitHub's `main`). There is
@@ -164,8 +170,8 @@ Why this matters for 0pt specifically — three facts that are hard to assert wi
| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. |
| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. |
**It is a measurement point, not a bypass.** This is consistent with, not an exception to,
the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove*
**The currently built 1p core is still the measurement point.** The decided target also
includes governed proxy, stealth and managed-unblock modes; 1p continues to *detect* blocking and *prove*
provenance, never to evade. Its purchased-residential egress pool is an interface with a stub
that throws — wiring it up would be a separate, explicit decision. Block detection also
alarms on a rising *trend*, never on single events, and deliberately excludes `error`