chore(salvage): preserve uncommitted WIP from /opt/vehiq-work/0pt-p060-p068_0pt before machine cleanup (2026-09-27)
CI (SIAX Cloud) / quality (push) Successful in 49s
CI (SIAX Cloud) / security (push) Successful in 7m5s
Security Scan (SIAX Cloud) / security (push) Successful in 7m5s
CI (SIAX Cloud) / integration (push) Successful in 18m53s
CI (SIAX Cloud) / security (pull_request) Successful in 46s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 58s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m38s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m45s
CI (SIAX Cloud) / quality (push) Successful in 49s
CI (SIAX Cloud) / security (push) Successful in 7m5s
Security Scan (SIAX Cloud) / security (push) Successful in 7m5s
CI (SIAX Cloud) / integration (push) Successful in 18m53s
CI (SIAX Cloud) / security (pull_request) Successful in 46s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 58s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m38s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m45s
This snapshot was found sitting uncommitted in a stale /opt/vehiq-work lane clone dated ~2026-09-09..18, base branch: feat/p060-p068-ingestion-plane. Not reviewed, not verified against current main. Pushed only so the directory could be deleted without losing the diff.
This commit is contained in:
+16
-10
@@ -1,9 +1,15 @@
|
||||
# 06 · Data engine
|
||||
|
||||
The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.
|
||||
> **Capability/egress decision 2026-09-16:** proxy, rotation, stealth browser
|
||||
> and managed unblock are optional 1p capabilities. Older anti-bypass language
|
||||
> in this document is superseded as product policy and remains only as history
|
||||
> about the local implementation. See
|
||||
> `docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md`.
|
||||
|
||||
The engine turns sources into delivered, provenance-tagged data. It is the governed data motor. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.
|
||||
|
||||
> ### Known issue — two diverged git remotes for this repo (unresolved, Simon's call)
|
||||
> This doc describes the compliant-only sourcing chain as *the law*. That law was briefly violated on one
|
||||
> This doc previously treated the standard sourcing chain as the only allowed path. One
|
||||
> of this repo's two remotes, so anyone reading this doc needs to know both exist:
|
||||
> - `github.com/sax3l/0pt` (this session's `origin`) — the remote this documentation pass was verified against.
|
||||
> - `git.siax.io/sax3l/0pt` (Gitea) — a **separate, diverged** remote that shares origin history with GitHub for
|
||||
@@ -11,8 +17,8 @@ The engine turns sources into delivered, provenance-tagged data. It is the legal
|
||||
> GitHub's current `main` and Gitea's post-fix PR branch, so they were once the same commit lineage).
|
||||
>
|
||||
> Gitea's `main` independently accumulated three commits on 2026-07-13 that added a real WAF-bypass /
|
||||
> proxy-rotation / fake-identity scraper stack (`services/scrapers/*`) — exactly the category of code
|
||||
> §"Prohibited" below and CLAUDE.md §1 forbid. Those commits never touched GitHub's `main`. The stack was
|
||||
> proxy-rotation / identity-profile scraper stack (`services/scrapers/*`). Under the current
|
||||
> architecture those capabilities belong to b0ts + 1p rather than a local 0pt transport owner. Those commits never touched GitHub's `main`. The stack was
|
||||
> since removed on Gitea via PR #1, but:
|
||||
> - **Which remote is authoritative / deployed to `0pt.app` is unknown** — not resolved this session.
|
||||
> - **Whether the two remotes are supposed to be kept in sync is unknown** — not resolved this session.
|
||||
@@ -21,7 +27,7 @@ The engine turns sources into delivered, provenance-tagged data. It is the legal
|
||||
> no lingering bypass code, and either merge or retire the loser before treating "the 0pt repo" as singular
|
||||
> in any future doc pass.
|
||||
|
||||
## Sourcing chain (strict priority — this is the law)
|
||||
## Standard sourcing chain
|
||||
|
||||
For each field × market, acquire in this order; stop at the first that works:
|
||||
|
||||
@@ -39,9 +45,9 @@ official JSON APIs over `httpx`, nothing else. There is no HTML-scraping/parsing
|
||||
no matching function in `connectors.py`; it correctly falls through to level 0 today. Both tiers remain
|
||||
target design, not built.
|
||||
|
||||
### Prohibited (do not build, keep, or extend)
|
||||
- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection.
|
||||
- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
|
||||
### Optional high-risk capabilities (canonical owner: 1p)
|
||||
- WAF/challenge handling, stealth/patched browsers and undetected-style browser profiles are modeled as explicit 1p execution profiles.
|
||||
- Proxy/fingerprint rotation, authenticated-session collection and account automation require their own scope, entitlement, isolation, budget and audit; they are not implicit stages in the standard 0pt pipeline.
|
||||
- **Reconciliation note (stale — kept for history):** this section used to say "if `browser_worker.py` has an
|
||||
`HTTP → FlareSolverr → Playwright` chain, drop the FlareSolverr stage." **`browser_worker.py` does not exist
|
||||
anywhere in this repository** (verified this pass — no file of that name on GitHub's `main`). There is
|
||||
@@ -164,8 +170,8 @@ Why this matters for 0pt specifically — three facts that are hard to assert wi
|
||||
| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. |
|
||||
| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. |
|
||||
|
||||
**It is a measurement point, not a bypass.** This is consistent with, not an exception to,
|
||||
the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove*
|
||||
**The currently built 1p core is still the measurement point.** The decided target also
|
||||
includes governed proxy, stealth and managed-unblock modes; 1p continues to *detect* blocking and *prove*
|
||||
provenance, never to evade. Its purchased-residential egress pool is an interface with a stub
|
||||
that throws — wiring it up would be a separate, explicit decision. Block detection also
|
||||
alarms on a rising *trend*, never on single events, and deliberately excludes `error`
|
||||
|
||||
Reference in New Issue
Block a user