chore(salvage): preserve uncommitted WIP from /opt/vehiq-work/0pt-p060-p068_0pt before machine cleanup (2026-09-27) #360

Closed
admin wants to merge 1 commits from opt-vehiq-work-salvage-20260927/0pt-p060-p068_0pt-uncommitted-wip into main
18 changed files with 189 additions and 80 deletions
+17 -11
View File
@@ -1,5 +1,14 @@
# AGENTS.md — 0pt
## Web Data SSOT
Läs `sax3l/siax-standard`-filerna
`standard/estate/SIAX_WEB_DATA_CAPABILITY_REGISTRY.json`,
`standard/SIAX-WEB-DATA-PLATFORM-MASTERPLAN-v1.0.md` och ADR-018. 0pt får
inte skapa en lokal kopia av den globala capabilitykatalogen.
AR-4 hindrar egna providerberoenden i 0pt men upphäver inte 1p:s daterade
upstream-overlay eller registrerade tekniska capabilities.
> Operativ brief för AI-agenter (Claude Code / opencode / liknande) som arbetar
> i detta repo. Läs `CLAUDE.md` + `docs/compliance/` innan du gör ändringar.
@@ -18,16 +27,13 @@ entity-resolution, provenance per fält, adaptive router, orderflöde med
## Viktiga förhållanden
- **Bypass-stacken** (`services/scrapers/*`, `waf_bypass.py`, data-stack-compose,
`adaptive_router.py`) är **behållen** — rör den inte. Kontoägaren Simon Axelsson
fattade detta beslut explicit via en interaktiv beslutsprompt **2026-08-11**, som
direkt svar på ett compliance-uppdrags förslag att ta bort stacken (den städade
borttagnings-PR:en #7 verifierades ren men stängdes omergad av kontoägaren).
Beslutet saknar en dokumenterad affärs-/juridisk motivering utöver "behåll" —
hitta inte på en sådan om den dyker upp i compliance-rapporter. Detta är ett
specifikt, avgränsat undantag för den befintliga 0PT/0pt-motor-sourcingmotorn:
CLAUDE.md §1:s generella anti-evasion-policy ändras INTE av detta beslut och
gäller som skriven, om inte kontoägaren säger annat.
- **Egressägarskap ändrat 2026-09-16.** Den äldre stacken i
`services/scrapers/*` är historiskt behållen, men nya produktflöden ska inte
bygga ut en konkurrerande proxy-/stealthrouter i 0pt. Providerpooler,
rotation, `stealth_browser` och `managed_unblock` ägs av 1p och används via
dess entitlement-, scope- och destinationspolicy-styrda kontrakt. Läs
`docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md`. Ingen teknisk
kapabilitet blanket-förbjuds eller raderas; den görs valbar och isolerad.
- **Auth:** `/api/0pt/*` fail-closed i kod (Zitadel via middleware), men live-instansen
svarade 2026-08-10 anonymt 200 (lucka S1 — rotorsaksanalys ej klar). Påstå aldrig
att produktionen är skyddad innan en anonym request fått 401.
@@ -58,6 +64,6 @@ pnpm run lint # advisory
## Vid osäkerhet
Fråga Simon (simon@siax.io) innan du: raderar något, ändrar bypass-stacken,
Fråga Simon (simon@siax.io) innan du: raderar något, aktiverar äldre lokal bypasskod,
ändrar auth-beteende, eller gör produktbeslut som är öppna frågor (§8 i
ARCHITECTURE.md).
+3 -3
View File
@@ -14,9 +14,9 @@ Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, Veh
## Non-negotiable constraints (do not violate)
1. **Legal sourcing only.** The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect `robots.txt` + rate limits) → licensed partner data → otherwise the field is delivered as **"not available / licensed only"**. We never invent values.
- **Do NOT build, keep, or extend**: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.).
- **Path corrected this pass:** no `src/workers/browser_worker.py` exists anywhere in this checkout (verified via full-repo search) — the engine's actual workers live under `services/api/workers/` (`adaptive_router.py`, `pipeline.py`, `queue.py`, `resolve.py`, `run.py`), none of which contain a FlareSolverr/bypass path today. The precedent for this concern is real and, **as of this GitHub<->Gitea unification merge, it is in this tree**: Gitea's `git.siax.io/sax3l/0pt` `main` independently accumulated a WAF-bypass/proxy-rotation/fake-identity scraper stack (`services/scrapers/*` — `curl_cffi`, `cloudscraper`, Tor circuit rotation) in three 2026-07-13 commits, and this branch inherits it because it branches from Gitea `main`. **The deletion PR (`feature/0pt-remove-noncompliant-scrapers`) is open, NOT merged** — an earlier version of this paragraph claimed it was "since removed", which was false. That removal is deliberately left to its own PR and was *not* folded into this merge; nothing in this merge resurrects, extends, or re-wires any of it. If new code (from either remote) reintroduces a bypass path anywhere, **remove or disable it** and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat.
1. **Sourcing and egress boundary.** The data engine prioritizes official APIs/bulk, configured public-web collection and partner data, and never invents values. Proxy routing is not implemented locally.
- **Capability-neutral rule:** no technical function category is blanket-prohibited or deleted. Proxy rotation, stealth browser, managed unblock and other high-risk modes are modeled as explicit, optional 1p capabilities with CL0UD entitlement, destination scope, isolation, budget and audit. 0pt consumes those contracts through b0ts/1p instead of building a second provider router or holding providercredentials.
- **Verified historical state:** no `src/workers/browser_worker.py` exists; the active engine workers live under `services/api/workers/`. This checkout also contains the older `services/scrapers/*` stack. Do not describe it as removed or as the new 1p implementation. Preserve it until a scoped migration, prevent new product paths from depending on it, and move active network ownership to b0ts + 1p.
2. **Postgres is the backbone.** One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the *same* DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth.
3. **Provenance per field.** Every fact carries source, freshness, confidence (`bridge_provenance`). This drives the UI signature too.
4. **One tool per job.** Do not add overlapping tools. The approved stack is in `docs/02-architecture.md` and `docs/07-dependencies.md`. Anything not listed needs a documented reason.
+7 -1
View File
@@ -1,5 +1,10 @@
# 0pt — your data
> **Web Data-programmet:** global capabilitykatalog, owner och status finns
> endast i `sax3l/siax-standard:standard/estate/SIAX_WEB_DATA_CAPABILITY_REGISTRY.json`.
> Masterplanen finns i `standard/SIAX-WEB-DATA-PLATFORM-MASTERPLAN-v1.0.md`;
> 0pt:s lokala ansvar finns i `docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md`.
Data services on demand for the Nordics + Europe. Compose a data job, pay 50 % to start,
watch production live, validate with full provenance, deliver anywhere. Owned by **SIAX
Technology AB**. Honest accuracy is the moat: every field carries source, freshness and
@@ -23,7 +28,8 @@ confidence; nothing is guessed.
admin portal, and the API route handlers. Defers to the data engine when `ENGINE_URL` is set.
- **Engine (`services/api`, runs on a VPS)** — FastAPI orchestrator + worker pipeline doing
the **compliant** sourcing (official API → bulk → authorized web → licensed → else level 0).
Never on Vercel (long jobs). Bypass/stealth/proxy-rotation/fake-account paths are excluded
Never on Vercel (long jobs). Proxy/rotation/stealth/unblock is consumed through 1p;
technical capabilities are optional and gated rather than blanket-prohibited
by design (CLAUDE.md §1).
## Routes
+7 -7
View File
@@ -73,7 +73,7 @@ rather than surfacing something new:
admin metrics, payments (including the escrow `release` that flips an order to
`delivered`), subscriptions, uploads and DB-grant endpoints are unprotected in
code (live-verified 200 anonymously on `/api/orders` and `/api/admin/metrics`).
- **The WAF-bypass scraper stack is kept deliberately.** `services/scrapers/*`
- **The legacy WAF-bypass scraper stack is kept for migration.** `services/scrapers/*`
(`waf_bypass.py`: FlareSolverr, Tor, cloudscraper, curl_cffi, proxy rotation,
fake identities) and `services/infra/docker-compose.data-stack.yml`
(flaresolverr/tor containers) contradict this repo's own compliance docs.
@@ -81,12 +81,12 @@ rather than surfacing something new:
tests) was closed unmerged by the account owner. **The account owner, Simon
Axelsson, was asked directly via an interactive decision prompt on 2026-08-11
whether to keep or remove this code, and explicitly chose to keep it.** No
business or legal justification beyond "keep it" was recorded — none is
invented here. This is a narrow, scoped exception for the existing 0PT/0pt-motor
sourcing engine; it does not change `CLAUDE.md` §1's general anti-evasion policy,
which remains in force as written unless/until the account owner says otherwise.
Out of scope for deletion — see `docs/compliance/GOLDEN-STANDARD-REPORT.md`
control C6.
additional rationale beyond "keep it" was recorded — none is invented here.
The 2026-09-16 capability-neutral decision supersedes the earlier blanket
anti-evasion rule: functions are preserved and made optional, scoped and
isolated. Active provider/route ownership moves to b0ts + 1p; this local stack
remains migration-era code until parity is verified. See
`docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md`.
- Beyond those items, other authentication gaps on internal API routes may still
exist and are tracked internally pending an intentional, reviewed fix; reporting
any you find is still welcome and helps confirm priority.
+9 -3
View File
@@ -1,5 +1,11 @@
# 02 · Architecture
> **Capability/egress decision 2026-09-16:**
> `docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md` supersedes older blanket
> prohibitions. Technical functions are optional and execution-gated; proxy,
> rotation, stealth and managed unblock are owned by 1p rather than rebuilt in
> 0pt. Current-code observations below remain valid historical facts.
> **Known issues affecting this doc, in brief** — full detail + evidence in [`README.md`](../README.md)'s
> "Known issues" section: (1) no auth/session system exists yet, so several newer routes trust
> client-supplied `tenant_id` (see "Known gaps" below); (2) the CL0UD/N0D/AUD0/ST0RE platform clients are
@@ -35,7 +41,7 @@ This principle describes — and is largely realized by — the **original edge
- **Next.js (Vercel)** — all customer/admin UI: marketing site, catalog, configurator, checkout, production tracker, validation, portal, admin. It also hosts **two distinct families** of API route handlers under `app/api/`: the original order/catalog/estimate surface (some of which defer to the FastAPI engine), and the newer P0 data-platform surface under `app/api/0pt/*` (does not defer to the engine at all — see below).
- **FastAPI orchestrator** (`services/api/orchestrator/main.py`) — REST contract in `docs/04-api.md`: estimates, orders, sources, vehicle lookup, sanctions screening, job run/status. **Not on Vercel** (serverless timeouts kill long jobs). *Correction: earlier drafts of this doc pointed at `src/orchestrator`; that was the pre-migration location. `docs/09-build-plan.md` Phase 0 called for moving it into `services/api`, and that move has happened — the code lives at `services/api/orchestrator` today.*
- **Workers** (`services/api/workers/`) — adaptive router (`adaptive_router.py`, verified: chooses only among compliant methods — official API → bulk → authorized web → licensed, no bypass/stealth/proxy-rotation tier exists in it, by design) + pipeline stages (`pipeline.py`, `queue.py`, `resolve.py`, `run.py`). See `docs/06-data-engine.md`.
- **Workers** (`services/api/workers/`) — adaptive router (`adaptive_router.py`, verified: currently chooses official API → bulk → authorized web → licensed). Optional bypass/stealth/proxy-rotation transport is delegated to b0ts + 1p rather than duplicated here. Pipeline stages: `pipeline.py`, `queue.py`, `resolve.py`, `run.py`. See `docs/06-data-engine.md`.
- **Postgres** — backbone (`docs/03-data-model.md`) incl. `pgvector` for matching/embeddings, for the original edge + engine plane. Wired via `lib/db.ts` (env-guarded: `DATABASE_URL`/`POSTGRES_URL`; falls back to an in-memory, non-durable store when unset, e.g. `lib/orders-store.ts`).
- **NocoDB / Metabase** — both are defined as services in `infra/docker-compose.yml`; embedding them into the portal as "Min data" / customer dashboards is the target, not yet confirmed wired into a portal page in this pass.
- **Cloudflare R2** — file storage (uploads + deliverables) via presigned URLs. Env-guarded (`app/api/uploads/route.ts` checks `R2_ACCESS_KEY_ID`/`R2_BUCKET`/`R2_ENDPOINT` and returns `501` until set); the presign call itself is still a stub (`put_url: null`) even when the env vars are present — not wired end-to-end yet.
@@ -147,7 +153,7 @@ Flagged so it is not repeated: `src/services/quality-check-service.ts`, `src/ser
### Two diverged git remotes
`origin` in local checkouts of this repo is `github.com/sax3l/0pt`. A separate Gitea remote, `git.siax.io/sax3l/0pt`, shares at least part of the same history (`services/api/workers/adaptive_router.py` is byte-for-byte identical between GitHub's current `main` and a Gitea post-fix PR branch) but independently accumulated commits GitHub's `main` never received — including, at one point, a WAF-bypass/proxy-rotation/fake-identity scraper stack that directly violated the CLAUDE.md §1 sourcing rule (since removed on Gitea via a separate PR). **Which remote is authoritative, which is actually deployed to `0pt.app`, and whether the two should be kept in sync is unresolved and is Simon's decision** — do not assume `origin`/GitHub is the sole copy of this codebase, here or in any other doc.
`origin` in local checkouts of this repo is `github.com/sax3l/0pt`. A separate Gitea remote, `git.siax.io/sax3l/0pt`, shares at least part of the same history (`services/api/workers/adaptive_router.py` is byte-for-byte identical between GitHub's current `main` and a Gitea post-fix PR branch) but independently accumulated commits GitHub's `main` never received — including, at one point, a local WAF-bypass/proxy-rotation/identity-profile scraper stack that now overlaps b0ts + 1p ownership (since removed on Gitea via a separate PR). **Which remote is authoritative, which is actually deployed to `0pt.app`, and whether the two should be kept in sync is unresolved and is Simon's decision** — do not assume `origin`/GitHub is the sole copy of this codebase, here or in any other doc.
## Stack decisions — IN
@@ -174,7 +180,7 @@ Flagged so it is not repeated: `src/services/quality-check-service.ts`, `src/ser
| Meilisearch (v1) | Postgres full-text suffices to a few thousand rows; add later |
| Delta Sharing (v1) | unnecessary; customer pull via scoped role + exports |
| Trigger.dev | the existing queue already schedules jobs; Resend covers mail |
| FlareSolverr / stealth browsers / undetected-* / proxy-fingerprint rotation / fake-account creation | **prohibited** — see CLAUDE.md §1. Replace with compliant fetching. (The Gitea-only scraper stack described under "Two diverged git remotes" above is exactly this — removed there, never present on GitHub `main`.) |
| Proxy rotation / stealth browser / managed unblock / account automation | Optional high-risk capabilities. Route-level ownership belongs to 1p; activate through explicit scope, entitlement, isolation, budget and audit rather than a blanket ban or local duplicate implementation. |
## Hosting (cheapest coherent) — **SUPERSEDED** (BACKLOG P2-5)
+1 -1
View File
@@ -152,7 +152,7 @@ GET /health -> { ok, service, db, sources }
| `GET /health` | `GET /health` | implemented, different shape (no `workers`/`queue`) |
| — | `GET /vehicles/{plate}`, `GET /screen` | implemented, previously undocumented |
Remove any endpoint that drives a bypass/stealth fetch path (CLAUDE.md §1) — none of the 12 real endpoints in `main.py` do; the WAF-bypass concern in this repo's history is in `services/scrapers/*` on the Gitea remote, not here (see CLAUDE.md's note on the two diverged remotes — an open question for Simon, not resolved by this pass).
Any endpoint that requests a bypass/stealth fetch path delegates transport to an explicit b0ts + 1p profile — none of the 12 real endpoints in `main.py` does that today. The earlier local implementation in `services/scrapers/*` on the Gitea remote is migration-era code (see CLAUDE.md's note on the two diverged remotes).
## Related: the edge-native TypeScript capability surface
+16 -10
View File
@@ -1,9 +1,15 @@
# 06 · Data engine
The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.
> **Capability/egress decision 2026-09-16:** proxy, rotation, stealth browser
> and managed unblock are optional 1p capabilities. Older anti-bypass language
> in this document is superseded as product policy and remains only as history
> about the local implementation. See
> `docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md`.
The engine turns sources into delivered, provenance-tagged data. It is the governed data motor. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.
> ### Known issue — two diverged git remotes for this repo (unresolved, Simon's call)
> This doc describes the compliant-only sourcing chain as *the law*. That law was briefly violated on one
> This doc previously treated the standard sourcing chain as the only allowed path. One
> of this repo's two remotes, so anyone reading this doc needs to know both exist:
> - `github.com/sax3l/0pt` (this session's `origin`) — the remote this documentation pass was verified against.
> - `git.siax.io/sax3l/0pt` (Gitea) — a **separate, diverged** remote that shares origin history with GitHub for
@@ -11,8 +17,8 @@ The engine turns sources into delivered, provenance-tagged data. It is the legal
> GitHub's current `main` and Gitea's post-fix PR branch, so they were once the same commit lineage).
>
> Gitea's `main` independently accumulated three commits on 2026-07-13 that added a real WAF-bypass /
> proxy-rotation / fake-identity scraper stack (`services/scrapers/*`) — exactly the category of code
> §"Prohibited" below and CLAUDE.md §1 forbid. Those commits never touched GitHub's `main`. The stack was
> proxy-rotation / identity-profile scraper stack (`services/scrapers/*`). Under the current
> architecture those capabilities belong to b0ts + 1p rather than a local 0pt transport owner. Those commits never touched GitHub's `main`. The stack was
> since removed on Gitea via PR #1, but:
> - **Which remote is authoritative / deployed to `0pt.app` is unknown** — not resolved this session.
> - **Whether the two remotes are supposed to be kept in sync is unknown** — not resolved this session.
@@ -21,7 +27,7 @@ The engine turns sources into delivered, provenance-tagged data. It is the legal
> no lingering bypass code, and either merge or retire the loser before treating "the 0pt repo" as singular
> in any future doc pass.
## Sourcing chain (strict priority — this is the law)
## Standard sourcing chain
For each field × market, acquire in this order; stop at the first that works:
@@ -39,9 +45,9 @@ official JSON APIs over `httpx`, nothing else. There is no HTML-scraping/parsing
no matching function in `connectors.py`; it correctly falls through to level 0 today. Both tiers remain
target design, not built.
### Prohibited (do not build, keep, or extend)
- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection.
- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
### Optional high-risk capabilities (canonical owner: 1p)
- WAF/challenge handling, stealth/patched browsers and undetected-style browser profiles are modeled as explicit 1p execution profiles.
- Proxy/fingerprint rotation, authenticated-session collection and account automation require their own scope, entitlement, isolation, budget and audit; they are not implicit stages in the standard 0pt pipeline.
- **Reconciliation note (stale — kept for history):** this section used to say "if `browser_worker.py` has an
`HTTP → FlareSolverr → Playwright` chain, drop the FlareSolverr stage." **`browser_worker.py` does not exist
anywhere in this repository** (verified this pass — no file of that name on GitHub's `main`). There is
@@ -164,8 +170,8 @@ Why this matters for 0pt specifically — three facts that are hard to assert wi
| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. |
| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. |
**It is a measurement point, not a bypass.** This is consistent with, not an exception to,
the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove*
**The currently built 1p core is still the measurement point.** The decided target also
includes governed proxy, stealth and managed-unblock modes; 1p continues to *detect* blocking and *prove*
provenance, never to evade. Its purchased-residential egress pool is an interface with a stub
that throws — wiring it up would be a separate, explicit decision. Block detection also
alarms on a rising *trend*, never on single events, and deliberately excludes `error`
+19 -6
View File
@@ -1,6 +1,10 @@
# 07 · Dependencies & repos to import
Curated. Each entry: what it is, where it slots in, how to install. **Do not import on reputation/star-count** — several items in earlier brainstorming were unverified or hallucinated (see Exclusions). Verify each repo exists and is maintained before adopting.
> **Capability rule 2026-09-16:** this document no longer blanket-prohibits
> technical categories. Provider/transport functions enter through 1p and
> siax-integrations as optional, classified and execution-gated capabilities.
Curated. Each entry: what it is, where it slots in, how to install. Reputation/star-count is not verification — several items in earlier brainstorming were unverified or hallucinated (see Integration routing). Verify each repo exists and is maintained before adopting.
> **This is a plan, not a status report.** Every table below lists what *should* be imported for the target
> architecture in `docs/06-data-engine.md`. A **Status** column has been added to each table (verified this
@@ -69,11 +73,20 @@ sourcing and rapidfuzz-based dedup are real today; extraction/OCR, NER, and qual
- **Resend** — email. Not installed (see frontend table).
- **Neon** *(optional)* — managed Postgres. `infra/docker-compose.yml` runs its own Postgres container instead; no evidence either way of a Neon instance actually in use.
## EXCLUSIONS — do not import
## Integration routing and ownership
- **Bypass/stealth/evasion**: FlareSolverr, "CloakBrowser", undetected-chromedriver/undetected-playwright (as bypass), proxy/fingerprint-rotation engines, any "WAF bypass" repo. Prohibited by CLAUDE.md §1 — they destroy the honest-accuracy positioning and create legal exposure. **This is not hypothetical**: a real WAF-bypass/proxy-rotation/fake-identity scraper stack (`services/scrapers/*`) briefly existed on this repo's Gitea remote (`git.siax.io/sax3l/0pt`, three commits dated 2026-07-13) and has since been removed there via PR #1. It never touched `origin` (`github.com/sax3l/0pt`), which this doc corpus was verified against. See `docs/06-data-engine.md`'s Known Issue callout for the full, unresolved detail — which remote is canonical is Simon's call, not decided this pass.
- **Fake-account automation** on Apollo/Cognism/etc.
- **Unverified / likely-hallucinated** from earlier brainstorming: "CloakBrowser/CloakHQ", "Rota proxy engine", "OpenClaw". Do not adopt unless you confirm a real, maintained project — and even then, not if it falls in the bypass category.
- **Overlapping tools** already decided against (Superset, DataHub/OpenMetadata, Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`.
- **Proxy/stealth/unblock engines:** 1p is the canonical product owner; 0pt
integrates verified components through 1p + siax-integrations and exposes
them as optional, scoped, isolated and audited profiles. The older
`services/scrapers/*` stack is migration-era code, not the future provider
boundary.
- **Account automation** is an optional high-risk profile routed through the
canonical 1p session/browser contract rather than embedded in 0pt.
- **Unverified components** from earlier brainstorming: verify existence,
maintenance and provider conformance before registering them. Unverified is
a status, not a blanket prohibition.
- **Overlapping tools** (Superset, DataHub/OpenMetadata, Baserow/Teable,
Trigger.dev, Delta Sharing) are catalogued as alternatives; adopting one is
an explicit architecture decision to avoid duplicate active owners.
> Note on real-but-overhyped tools (Firecrawl, browser-use, etc.): they exist, but ignore inflated star-counts and only adopt if they fit a *compliant* role and don't duplicate the stack.
+6 -7
View File
@@ -125,7 +125,7 @@ Phased, in dependency order. **Finish a phase to its acceptance criteria before
`ENGINE_URL` is set. This one holds up as described.
## Phase 5 — Real data engine (Nordic slice)
- Compliant connectors first: **GLEIF + Brreg + Companies House** (verify access terms; record in `sources.yaml`). Remove any FlareSolverr/bypass path (`docs/06`).
- Standard connectors first: **GLEIF + Brreg + Companies House** (record provider and access metadata in `sources.yaml`). Route any FlareSolverr/stealth/unblock profile through b0ts + 1p instead of implementing a second transport owner (`docs/06`).
- Pipeline stages produce real `dim_company` + `fact_financials` with `bridge_provenance`; QA threshold drops low-confidence rows.
- Production tracker now reflects **real** job metrics.
- **Accept:** a Nordic company dataset is produced for a real order, every field has provenance, quality report generated, level-0 fields shown honestly (not fabricated).
@@ -133,11 +133,10 @@ Phased, in dependency order. **Finish a phase to its acceptance criteria before
- `services/api/sources/sources.yaml` and `connectors.py` are real: GLEIF, Brreg, Companies House,
PRH, and CVR are all registered, each with an explicit `licence: "... — VERIFY"` tag rather than an
assumed-clean claim — matches the "verify access terms" instruction.
- `services/api/workers/adaptive_router.py`'s own comment confirms: "There are NO bypass/stealth/
proxy-rotation tiers here, by design (CLAUDE.md §1)" — the FlareSolverr/bypass removal is done, at
least on this repo's `main` (see `docs/08-infrastructure.md`'s "Known open question" about the
diverged Gitea remote, which independently added such a stack on its own `main` — unresolved which
remote is canonical).
- `services/api/workers/adaptive_router.py` currently has only the standard connector tiers. Optional
bypass/stealth/proxy-rotation transport is target-owned by b0ts + 1p rather than this local router
(see `docs/08-infrastructure.md`'s "Known open question" about the diverged Gitea remote, which
independently added a local stack on its own `main` — unresolved which remote is canonical).
- `services/api/workers/pipeline.py` has a real, working confidence-threshold QA drop: `delivered =
[r for r in deduped if r["confidence"] >= threshold]`, plus an `avg_confidence` figure in its output.
This is a genuine, narrow QA mechanism inside the Python engine.
@@ -236,7 +235,7 @@ Phased, in dependency order. **Finish a phase to its acceptance criteria before
- No secrets in code; update `.env.example` when adding env. (`.env.example` itself currently lives on
`feature/0pt-env-example` / PR #3 — merge that first, or this instruction has nothing to update yet.)
- Add/extend tests for engine logic (estimate math, fusion/confidence, dedup) and API contract.
- When a source's access/licence is unclear, **verify and record in `sources.yaml`** before relying; if disallowed for bulk, route to licensed/partner or mark level 0 — never bypass, never fabricate.
- When a source profile is unclear, **classify and record it in `sources.yaml`** before relying on it. Standard, licensed, stealth and managed-unblock modes remain explicit profile choices; values are never fabricated.
- Touch the design tokens/motion only via `docs/05`; keep red ≤ once per screen; no photography.
- If anything here conflicts with existing code, **flag it** in the PR description and propose the reconciliation rather than guessing.
- **Two git remotes claim to be this repo** (`github.com/sax3l/0pt` and `git.siax.io/sax3l/0pt`), and
+1 -1
View File
@@ -207,7 +207,7 @@ for what's still actually pending.)*
`IcebergWriteService` calls `N0dClient.createJob`, which — like the other platform clients — has
no live `N0D_URL` wired from this repo's environment either.
## Explicitly out of scope for 0pt (do not build here)
## Canonical ownership outside 0pt
- An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's
registry existing — that repeats the `n0d` mistake this plan is repairing.
@@ -0,0 +1,50 @@
# 0pt → 1p integrationsgräns
## Kanonisk dokumentordning
- Capability-ID, ägarskap, profiler och status:
`sax3l/siax-standard:standard/estate/SIAX_WEB_DATA_CAPABILITY_REGISTRY.json`
- Programordning:
`sax3l/siax-standard:standard/SIAX-WEB-DATA-PLATFORM-MASTERPLAN-v1.0.md`
- Beslut: `sax3l/siax-standard:docs/architecture/decisions/ADR-018-web-data-platform-canonical-ownership.md`
- Exekveringskort: `sax3l/siax-standard:prompts/007-web-data-platform-full-program.md`
Detta dokument äger endast 0pt:s lokala integrationsgräns och observerade
nuläge. Den globala katalogen kopieras inte hit.
**Beslut:** 2026-09-16
**Kanonisk produktdefinition:**
`https://git.siax.io/sax3l/1p/src/branch/main/docs/architecture/SIAX_PROXY_PLATFORM.md`
0pt äger dataprodukt, källmodell, kvalitet, provenance per fält och leverans.
0pt äger inte proxyprovider, IP-pool, session, rotation eller web access-route.
Dessa förmågor ägs av 1p och anropas via b0ts/1p-kontraktet.
## Regler
- Ingen teknisk kapabilitet blanket-förbjuds eller raderas. Funktioner görs
valbara, klassificerade, isolerade och styrda vid exekvering.
- Nya dataflöden bygger inte en konkurrerande provider-/proxyrouter i 0pt.
- Standardhämtning används om ingen explicit 1p-policy väljer annat.
- Stealth och managed unblock är 1p-lägen, avstängda som standard och styrda
av CL0UD-entitlement, scope, budget och destinationspolicy.
- Ingen direkt fallback runt 1p när ett 1p-styrt flöde nekas eller är nere.
- 0pt lagrar route-/artifact-/proveniensreferenser, aldrig providercredentials.
- High-risk-funktioner får egna profiler och scope i stället för blanket-förbud.
## Nuläge
`services/scrapers/*` innehåller äldre lokal transport-/bypasskod och README
säger samtidigt att scraperansvaret flyttats till b0ts. Koden är historiskt
bevarad men är inte den framtida ägargränsen. Den ska isoleras från nya
produktflöden och migreras till b0ts + 1p utan att resultat- eller datakvalitet
förloras. 1p:s nya providerförmågor är ännu target, inte driftsatt funktion.
Äldre compliance- och arkitekturrapporter är historisk evidens. Om de beskriver
en teknisk funktionskategori som förbjuden eller lokalt ägd i 0pt ersätts den
produktregeln av detta dokument; deras kod- och driftsobservationer gäller för
den tidpunkt de verifierades.
Extraction- och OCR-funktioner publiceras som fristående moduler med manifest,
versionerat kontrakt och fake-port-tester. Produktflöden komponerar dessa
moduler utan att importera intern implementation eller skapa en ny nätväg.
+12 -4
View File
@@ -1,5 +1,11 @@
# 0PT — Arkitektur
> **Produktregel uppdaterad 2026-09-16:** tekniska kapabiliteter blanket-
> förbjuds inte. `1P_PROXY_PLATFORM_INTEGRATION.md` är kanonisk för egress:
> 1p äger proxy/rotation/stealth/unblock och 0pt konsumerar dessa som valbara,
> scope-/entitlement-styrda funktioner. Äldre konfliktbeskrivningar nedan är
> historisk kodanalys, inte aktuell funktionspolicy.
**Kategori:** platform (K1) **Underkategori:** data-os **Status:** under utveckling (deployad frontend i drift, capability-lagret ej produktionsklart)
**Senast verifierad mot kod:** 2026-08-07 (commit `6531384`, main) · **Klassificeringsuppdatering:** 2026-08-10 (`siax.repo.json`) · **v2.0-standardmigrering (§5 tillagd, §6–§9 omnumrerade; §6 Deploy-fältet nedan uppdaterat mot `lib/health-checks.ts`):** 2026-08-11 — scoped till dessa avsnitt, ingen fullständig radrad-omverifiering av hela dokumentet gjord denna pass
@@ -54,16 +60,18 @@ Repot innehåller i praktiken **två separata system** i samma träd:
- **Ingen egen policymotor.** Beslut delegeras till CL0UD (`src/clients/cl0ud-client.ts`).
- **Ingen tung batch i webblagret.** Långa jobb ska gå till engine/N0D.
### Uttryckligt förbud som koden BRYTER MOT
### Historisk dokumentkonflikt och donatorkod
`CLAUDE.md` §1 och `README.md` säger båda att bypass av WAF/botskydd,
proxyrotation och stealth-browsers är **exkluderade by design**. Det stämmer inte:
Äldre versioner av `CLAUDE.md` och `README.md` beskrev bypass av WAF/botskydd,
proxyrotation och stealth-browsers som **exkluderade by design**. Det beslutet
är ersatt av ADR-018 och den capabilityneutrala målbilden. Kodfyndet kvarstår:
`services/scrapers/waf_bypass.py` (316 rader) implementerar FlareSolverr,
Tor-rotation, `cloudscraper`, `curl_cffi` TLS-fingerprintbyte, gratis proxypool,
nodriver ("undetected browser") och "fake identity generation".
`services/infra/docker-compose.data-stack.yml` startar `flaresolverr` och
`tor-proxy` som tjänster. Commit `4a164c5` (Simon Axelsson, 2026-07-13) heter
ordagrant *"refactor: remove compliance restrictions"*. Se §8 lucka **S9**.
ordagrant *"refactor: remove compliance restrictions"*. Koden är donator- och
migrationsevidens, inte en ny lokal ägargräns. Se §8 lucka **S9**.
---
+5 -1
View File
@@ -1,5 +1,9 @@
# BACKLOG — 0pt (FAS 5)
> **Beslut 2026-09-16:** `[BYPASS]` betyder hädanefter migrationsarbete mot
> b0ts + 1p, inte en förbjuden teknisk kategori. Förmågor bevaras och görs
> valbara; provider-/routeägarskap konsolideras i 1p.
> P0 = säkerhetskritiskt · P1 = hög · P2 = medel · P3 = låg · P4 = hygien · P5 = struktur · P6 = senare.
> Alla luckor hänvisar till docs/architecture/ARCHITECTURE.md §7 (S1–S12) och GOLDEN-STANDARD-REPORT.
> **[BYPASS]** = berört av det medvetna användarbeslutet (rör ej kod).
@@ -24,7 +28,7 @@
| P1-1 | **S6** — AUD0 no-op utan URL | Wire AUD0_URL från Infisical; verifiera event-emission live |
| P1-2 | **S7** — db-grant påstår kryptering | Implementera AES-GCM med GRANT_ENC_KEY, read-only-validering, lagring + audit — eller ta bort påståendet |
| P1-3 | **S12** — svag session | exp-kontroll, rotation, användar-binding |
| P1-4 | **S9** **[BYPASS]** | Produktbeslut: behåll/bygg-om bypass-stacken (GPL-fläckar + juridiskt läge). Teknisk radering är INTE planerad — beslut om framtida läge fattas av Simon |
| P1-4 | **S9** **[BYPASS]** | Migrera aktiv transport, rotation och high-risk-profiler till b0ts + 1p; behåll historisk funktionalitet isolerad tills paritet är verifierad. Ingen blanket-radering eller blanket-förbud. |
| P1-5 | **D4** — Coolify-app ej provisionerad | Provisionera app i cloud.siax.io, fyll COOLIFY_APP_ID + COOLIFY_TOKEN (Infisical) |
| P1-6 | **C8** — Infisical ej inkopplat | Infisical-projekt 0pt + env-mapping; Vercel-env avvecklas i takt med cutover |
+8 -3
View File
@@ -1,5 +1,10 @@
# BLOCKERS.md — 0pt (§1.4)
> **Omklassificering 2026-09-16:** B-1 är inte längre en fråga om huruvida en
> teknisk funktionskategori får finnas. Den är en ägarskaps- och migrationsfråga:
> aktiv nättransport ska konsolideras i b0ts + 1p, med valbara profiler och
> exekveringsgrindar. Historiken nedan behålls som evidens.
> Policy: enligt SIAX-F0KUSREPO-PROMPT §1.4 (Blockerarpolicy) dokumenteras varje
> blockerare som inte kan lösas själv eller plattformsbrett här — exakt vad som
> krävs för att lyfta den. En obesvarad blockerare som inte står i denna fil
@@ -19,10 +24,10 @@
| Fält | Innehåll |
|---|---|
| **Vad blockerar** | `services/scrapers/waf_bypass.py`, `engine.py` och `engine_v2.py` innehåller en FlareSolverr-/Tor-/proxyrotationsstack som strider mot `CLAUDE.md` §1:s uttryckliga förbud mot WAF/bot-detektionsbypass. En kirurgiskt ren borttagnings-PR (**#7**, `chore(compliance): ta bort den förbjudna WAF-bypass-/evasionstacken`) verifierades ren men stängdes **omergad** 2026-08-09. |
| **Vad krävs för att lyfta den** | Ett uttryckligt, dokumenterat kontoägarbeslut: behåll, bygg om, eller ta bort stacken. |
| **Vad blockerar** | `services/scrapers/waf_bypass.py`, `engine.py` och `engine_v2.py` innehåller en lokal FlareSolverr-/Tor-/proxyrotationsstack som överlappar b0ts + 1p:s kanoniska ägarskap. En äldre borttagnings-PR (**#7**) stängdes **omergad** 2026-08-09; ADR-018 ersätter dess teknikförbudsgrund med ägarskapsmigrering. |
| **Vad krävs för att lyfta den** | Migrera aktiv transport till b0ts + 1p och verifiera profilparitet innan den lokala dubletten avvecklas. |
| **Vem beslutar** | Simon Axelsson (kontoägare). |
| **Nuvarande status** | **Redan beslutad (behåll) — 2026-08-11.** Kontoägaren tillfrågades direkt via en interaktiv beslutsprompt, som svar på PR #7:s borttagningsförslag, och valde uttryckligen **behåll**. Beslutet är nu dokumenterat i `AGENTS.md` (rad ~21–30) och `SECURITY.md`, mergat till main via PR **#15** (docs-only, merge-commit `5c36dc3` — samma commit som är denna verifierings `main`-HEAD). PR #7 kvarstår **closed/unmerged** och rörs inte av detta dokument eller detta uppdrag. Ingen affärs- eller juridisk motivering utöver "behåll" är dokumenterad, och ingen ska hittas på om den saknas i framtida rapporter. `services/scrapers/{waf_bypass,engine,engine_v2}.py` är oförändrade av denna PR och ligger utanför detta dokuments och detta uppdrags mandat. |
| **Nuvarande status** | **Beslutad: bevara kapabiliteten och migrera ägarskapet.** Det tidigare beslutet **behåll** från 2026-08-11 kvarstår. Produktbeslutet 2026-09-16 placerar aktiv transport, provider, pool, session, stealth/unblock och route receipt i b0ts + 1p. PR #7 kvarstår **closed/unmerged**. |
| **Bevis** | `git log -- services/scrapers/waf_bypass.py` (`bb6be0f`, `afd301f`); Gitea API `repos/sax3l/0pt/pulls/7` (`state: closed`, `merged: false`, `closed_at: 2026-08-09T09:49:05Z`); Gitea API `repos/sax3l/0pt/pulls/15` (`merged: true`, `merged_at: 2026-08-11T09:36:17Z`); `AGENTS.md` rad 21–30 (denna checkout). |
---
+7 -8
View File
@@ -2,7 +2,8 @@
> Bedömning mot SIAX Cloud K1-golden-standard (plattformsrepo). Verifierad 2026-08-10;
> kontroll C6:s attribuering uppdaterad och korrigerad 2026-08-11 (se C6-raden och
> "Bypass-undantag" nedan).
> "Proxy-/profilbeslut" nedan). Produktpolicyn är uppdaterad 2026-09-16 av
> `docs/architecture/1P_PROXY_PLATFORM_INTEGRATION.md`.
> Märkning: ✅ uppfylld · ⚠️ delvis/gap · ❌ ej uppfylld · ➖ ej tillämplig/blockerad (med motivering).
> Bevis per kontroll (AR-5). Poäng: ✅=1, ⚠️=0.5, ❌=0, ➖ exkluderas ur nämnaren.
@@ -92,7 +93,7 @@
E6 (SLO + runbooks) ✅ → **✅ 21 · ⚠️ 8 · ❌ 10 · ➖ 2**
**Poäng efter: (21×1 + 8×0.5) / 39 = 25/39 ≈ 64 %** (verifierat i SLUTRAPPORT)
## Bypass-undantag (formell notering)
## Proxy-/profilbeslut (formell notering)
`services/scrapers/waf_bypass.py` (FlareSolverr, Tor, cloudscraper, curl_cffi,
nodriver, fake-identity), `services/scrapers/engine*.py`,
@@ -101,12 +102,10 @@ intakt och orörda. Kontoägaren Simon Axelsson fattade detta beslut explicit vi
en interaktiv beslutsprompt **2026-08-11**, som direkt svar på ett
compliance-uppdrags förslag att ta bort stacken (den städade
borttagnings-PR:en #7 verifierades ren, 128/128 test, men stängdes omergad av
kontoägaren). Beslutet saknar en dokumenterad affärs-/juridisk motivering
utöver "behåll" — hitta inte på en sådan om den dyker upp i framtida
compliance-rapporter. Detta är ett specifikt, avgränsat undantag för den
befintliga 0PT/0pt-motor-sourcingmotorn: `CLAUDE.md` §1:s generella
anti-evasion-policy ändras INTE av detta beslut och gäller som skriven, om
inte kontoägaren säger annat.
kontoägaren). Produktbeslutet 2026-09-16 ersätter undantagsmodellen:
kapabiliteten är tillåten och bevaras, medan aktiv transport konsolideras i
b0ts + 1p som valbara, explicita och auditerade profiler. Äldre
anti-evasion-formuleringar är historik och inte längre aktiv produktpolicy.
Kontroll **C6** markeras ➖ med denna attribuerade, daterade motivering —
**ersätter det tidigare oattribuerade påståendet "medvetet användarbeslut"**
+4
View File
@@ -1,5 +1,9 @@
# RELATIONS — 0pt relationskarta (FAS 2)
> **Ägarskap uppdaterat 2026-09-16:** den äldre lokala bypassstacken visas som
> migrations-/historikrelation. Framtida nätväg ägs av 1p; 0pt → b0ts → 1p.
> Ingen teknisk kategori är förbjuden, men den ska ha ett kanoniskt ägarrepo.
> Verifierad mot kod 2026-08-10. Alla relationer är grep-/läs-verifierade (AR-5).
> Legenden: **sources.yaml** = kompliant källregistret · **bypass-stacken** = medvetet behållen
> per användarbeslut (se GOLDEN-STANDARD-REPORT), visas grått.
+4 -3
View File
@@ -44,6 +44,7 @@
## Undantag
Bypass-stacken (`services/scrapers/*`) är medvetet behållen per kontoägarens
daterade beslut (Simon Axelsson, 2026-08-11 — se SECURITY.md) och omfattas
inte av ovanstående SLO; dess körtid är ej mätbar i 0pt-SLO:er.
Den äldre transport-/bypassstacken (`services/scrapers/*`) är bevarad som
migrationskod men är inte framtida ägare av nätvägen. När flöden migreras ska
1p:s separata SLO och route receipts gälla; tills dess är stackens körtid inte
mätbar i 0pt-SLO:er.
+13 -11
View File
@@ -1,24 +1,25 @@
# 0pt data engine (`services/api`)
The legal data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
**compliant** sources into delivered, provenance-tagged data. Runs on a VPS via
The governed data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
configured sources into delivered, provenance-tagged data. Runs on a VPS via
[`../../infra/docker-compose.yml`](../../infra/docker-compose.yml) — **not on Vercel** (long jobs).
## Sourcing boundary (CLAUDE.md §1 — non-negotiable)
## Sourcing and transport boundary
Per field × market, in strict order: **official API/bulk → authorized public web
(robots + rate limits) → licensed partner data → otherwise level 0 "ej tillgänglig"**.
Per field × market, the standard profile prefers **official API/bulk → public web
→ partner data → otherwise level 0 "ej tillgänglig"**.
We never invent values.
**Deliberately excluded (never add here):** WAF/bot-detection bypass, FlareSolverr,
stealth/undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind
login walls, automated fake-account creation. This is the honest-accuracy moat.
**Canonical ownership:** WAF/challenge handling, FlareSolverr-compatible adapters,
stealth browsers, proxy/fingerprint rotation, authenticated sessions and account
automation are valid optional profiles owned by b0ts + 1p. They are selected through
explicit contracts and are not duplicated inside the standard 0pt worker pipeline.
## Layout
- `sources/connectors.py` — compliant connectors (GLEIF, Brønnøysund, Companies House live; PRH/CVR/Bolagsverket gated on verified terms → level 0 until wired).
- `sources/sources.yaml` — source registry (access/licence/cadence; VERIFY terms).
- `orchestrator/main.py` — REST API (health, sources, estimates, orders, jobs) per docs/04.
- `workers/adaptive_router.py` — picks the cheapest **compliant** method per market (api > bulk > authorized_web > licensed). No bypass tiers.
- `workers/adaptive_router.py` — picks the cheapest standard method per market (api > bulk > authorized_web > licensed). High-risk transport profiles are delegated to b0ts + 1p.
- `workers/pipeline.py` — sourcing → extraction → structuring (dedup) → enrichment/fusion → QA (drops rows below the order's confidence target). Writes provenance per field.
## Run locally
@@ -30,5 +31,6 @@ python -m workers.run # run one demo job
## Reconciliation note
Supersedes the earlier `nordic-eu-data-platform` backend: the compliant connectors +
pipeline are migrated here; the FlareSolverr/stealth/proxy/fake-account paths and the
`test_bypass_chain`/`test_proxy_pool` examples are intentionally **not** carried over.
pipeline are migrated here; FlareSolverr/stealth/proxy/account-session paths and the
`test_bypass_chain`/`test_proxy_pool` examples belong to b0ts + 1p instead of being
duplicated in this package.