0pt/IMPLEMENTATION_REPORT.md

IMPLEMENTATION_REPORT.md — P010 Data-OS completion (sax3l/0pt)

Branch: feat/p010-data-os-completion (renamed mid-pass from feat/p010-source-registry-persistence once scope grew beyond persistence alone). Base: origin/main @ f8bb0d5 (verified in sync via git fetch --prune before starting; origin = git.siax.io/sax3l/0pt, the canonical remote per CLAUDE.md — the GitHub copy was not touched). Commits on top of base: 07b13bd (Source Registry Postgres persistence + HTTP route surface), 9740e20 (TS/Python quality bridge + provenance fields

• fault-injection tests). Report date: 2026-09-01. Written after a from-scratch verification pass (fresh clone, fresh pnpm install, fresh Python venv — not a rerun of the working tree that built the branch), so the numbers below are reproducible, not just remembered.

This report follows BEFORE_STATE.md's classification scale: VERIFIED / IMPLEMENTED_NOT_VERIFIED / DOCUMENTED_ONLY / STALE / BROKEN / UNKNOWN. Read BEFORE_STATE.md first — this report only describes the delta from it.

---

1. What actually changed, mapped to BEFORE_STATE.md's 5 gaps

§1 Source Registry persistence — VERIFIED closed

• New src/repositories/* layer: SourceRepository / SourceRegistryRepository interfaces, each with an in-memory and a postgres implementation (postgres-source-repository.ts, postgres-source-registry-repository.ts). src/repositories/db.ts is a thin pooled-client wrapper honoring resolveStoreMode (src/platform/dependency-mode.ts) — the same guard BEFORE_STATE.md noted as already blocking production boot without a DB; it now has a Postgres implementation to fall through to instead of only throwing.
• sql/002_source_registry.sql — new migration: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, every table keyed by source_id with a tenant_id column and index — this is exactly the schema shape BEFORE_STATE.md §1 derived as "what Postgres migration would require" but said did not exist. It now exists and is applied.
• src/services/source-service.ts and source-registry-service.ts rewritten to take a repository (constructor-injected), closing the per-instance-Map defect BEFORE_STATE.md flagged as worse than plain in-memory: every route module now shares one repository-backed instance (src/services/source-registry.ts is the new shared-singleton wiring), not a fresh Map per new SourceService().
• New route surface closing BEFORE_STATE.md's "effectively zero HTTP routes" finding: app/api/0pt/sources/route.ts (create/list — did not exist before), [sourceId]/route.ts (get/describe — did not exist before), [sourceId]/rights, [sourceId]/provenance, [sourceId]/health, [sourceId]/freshness (all new). [sourceId]/query and rate-policies/* (pre-existing routes) were touched only to repoint them at the shared singleton instead of constructing their own service.
• tests/p0/source-registry-postgres.realinfra.test.ts (340 lines, 7 tests) — a real-Postgres suite, not a mock. It self-skips honestly (not fabricating pass) when no reachable Postgres is configured for the run; see §2 (Testing) below for what this pass actually observed.

What is still NOT true, stated plainly: in-memory mode is still the default when no DATABASE_URL/Postgres is configured — this pass added a real persistence option, it did not remove the in-memory path (removing it would break every existing unit test and any deployment without Postgres wired yet). resolveStoreMode still hard-blocks production boot without a real DB configured, which is correct behavior, not a residual bug.

§2 TS/Python split-brain — VERIFIED bridged (additive, advisory)

• New route surface, app/api/0pt/quality/*: rules (list/create), rules/[ruleId] (get/update), runs (list/create), runs/[runId] (get), issues (list), and POST /api/0pt/quality/bridge-run — the entry point the Python side calls. All sit over one shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance across route modules" class of defect the Source Registry had.
• src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use, so the Python engine never has to pre-register rules through the CRUD surface — it just calls bridge-run and gets a score back against rules that already exist.
• Python side, services/api/workers/quality_bridge.py: an HTTP client to the route above.
  • Feature-flagged off by default: OPT_QUALITY_BRIDGE_ENABLED unset or falsy → zero network calls, returns immediately (status: "disabled"). Verified in code (_enabled(), services/api/workers/ quality_bridge.py:59-60) and in tests (test_disabled_by_default_makes_zero_network_calls).
  • Fails open, unconditionally. Every failure mode — bridge URL unset, connection refused, timeout, non-2xx, malformed JSON — is caught inside the module and reported back as a BridgeResult with a status field (disabled / not_configured / unavailable / ok), never raised into run_job. pipeline.py's actual delivery decision (delivered = [r for r in deduped if r["confidence"] >= threshold], still the one line BEFORE_STATE.md quoted) is unchanged by any bridge outcome — success or failure. Verified by test_bridge_disabled_delivers_identically_to_bridge_enabled_and_failing, which asserts the delivered row set is byte-identical between the two paths.
  • Advisory only, and this is a real constraint, not an excuse: true per-row gating against the TS ExpectationEngine is not implementable today because ExpectationEngine returns dataset-level pass/fail counts and sample rows, not a per-row boolean array keyed to the Python pipeline's record identity. Building that would mean changing ExpectationEngine's contract — out of scope for this pass, documented rather than faked with a fabricated per-row mapping.
• Live-verified end-to-end (2026-09-01, this pass): a real next dev server (AUTH_ENFORCE=false for the local run only) plus the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge-run route, and receiving a real QualityService score back — not a contract double on either side. This was a manual verification run during development; it is not itself a repeatable automated test (the pytest suite below covers the same code paths against respx-mocked bridge responses, which is repeatable and is what CI would run).

What is still NOT true, stated plainly: the bridge is not enabled in production and this pass does not enable it. OPT_QUALITY_BRIDGE_ENABLED stays unset/false as shipped. Turning it on is a separate, deliberate decision — see §4 "Activating the bridge" below. Until that decision is made, the two runtimes' QA logic remains exactly as split as BEFORE_STATE.md described: Python's one-line confidence filter is what actually decides delivery; the TS QualityService/ExpectationEngine only sees a copy of the same data, advisory-only, and only when the flag is on.

§3 Provenance fields — VERIFIED extended (Python side)

services/api/sources/connectors.py's Provenance dataclass gains, via a new make_provenance() factory used by every connector: retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref — closing the "missing entirely from this struct" list BEFORE_STATE.md §3 itemized (retrieval-method-at-field- level remains only partially addressed: connector_version +producer identify what ran, not a full method taxonomy like the TS SourceProvenance.method enum).

• resolve.py stamps entity_resolution on merged records' provenance only (never on unmerged ones — verified by test_unmerged_record_keeps_entity_resolution_none), tracking agreed_sources/merged_from in a structured field rather than the loose attribute BEFORE_STATE.md found.
• pipeline.py stamps artifact_ref at delivery time and quality once (if) the bridge call above succeeds.

What is still NOT true, stated plainly: bridge_provenance as a single named construct still does not exist anywhere in this repo, and this pass does not introduce that name. What exists is still two structs — the Python Provenance dataclass (now richer) and the TS SourceProvenance type (per-source, unchanged this pass) — reconciled only at the one new seam described in §2 (the bridge-run response, which is advisory and optional). This pass closes real gaps in the Python struct; it does not unify the two runtimes' provenance models into one schema. Nothing here persists past the in-process job response either — pipeline.py still has no DB write of its own (Postgres persistence added this pass is scoped to the Source Registry, not to order/provenance records).

§4 Failure-mode tests — VERIFIED extended, with an honest residual

New pytest + respx suite, services/api/tests/:

File	Covers
test_resolve_determinism.py	dedup-determinism under shuffled/interleaved orderings, exhaustive permutations of a mutually-similar set, duplicate-input collapsing without double-counting agreement, out-of-order multi-market arrival, entity-resolution provenance stamping — and an explicit, passing test of the one case BEFORE_STATE.md flagged as a real, plausible order-sensitivity risk: a non-transitive-similarity triangle in the greedy clusterer genuinely can cluster differently by input order. That test (test_a_non_transitive_triangle_can_cluster_differently_by_order) asserts the order-sensitivity, it does not paper over it — resolve.py's greedy-merge algorithm was not rewritten to remove this property, since doing so is an algorithm change out of this pass's scope.
test_pipeline_fault_injection.py	schema-evolution (upstream adding/removing/reshaping fields via respx-mocked responses through the real brreg connector + pipeline.py), restart/connector-crash resilience (a connector exception, including one raised by respx itself, never crashes run_job; every connector failing delivers zero rows, not a crash), backfill-replay (replaying the same order config is byte-identical — this is the property that actually holds given there is still no persistence layer for orders, not a fabricated "dedup store" test).
test_quality_bridge.py	all bridge states above (disabled / not-configured / unavailable / timeout / non-2xx / malformed / connection-refused / success), plus the "wired into the pipeline" tests proving delivery is unaffected either way.
test_connectors_provenance.py	the make_provenance() factory populates every new field honestly and leaves the rest None rather than fabricating values; a real parsing path (brreg) produces a fully populated provenance record.
test_known_gaps.py	2 explicit, documented skips — not silently dropped, not faked: CDC-interruption and true delivery-store backfill idempotency are not implementable against this codebase, because no CDC pipeline and no persistence layer for orders/deliveries exists anywhere in services/api. Testing either would require building the infrastructure the test claims to fail-inject against, which is a different, larger piece of work than this pass.

Net: of BEFORE_STATE.md §4's four entirely absent categories (schema-evolution, CDC-interruption, out-of-order, dedup-determinism), three are now genuinely covered (schema-evolution, out-of-order, dedup-determinism) and one (CDC-interruption) is honestly documented as not implementable rather than covered. Tenant-isolation (already the best-covered category) is untouched by this pass. Quality-gate coverage gains the bridge-interaction tests above but the production Python filter itself (delivered = [...]) still has no line-level unit test isolated from an end-to-end pipeline run — this pass tests it via the fault-injection and bridge suites' end-to-end assertions, not a standalone unit test of that one line.

§5 1P/ST0RE/N0D/AUD0/CL0UD integration — unchanged this pass

Not touched. BEFORE_STATE.md's classification stands: ST0RE + N0D real-contract (ST0RE live-verified once, historically); AUD0 real-contract client but unused by the live Python pipeline (still true — quality_bridge.py does not emit AUD0 events, and neither does the rest of services/api); CL0UD real-contract but not platform-wide-enforced; 1P documented-only.

---

2. Testing — from-scratch verification (this pass, not a rerun)

To avoid reporting numbers from the working tree that built the branch, this verification was run against a fresh git clone of the pushed branch (feat/p010-data-os-completion @ 9740e20) in an isolated directory, with a fresh pnpm install --frozen-lockfile and a fresh Python venv — no reuse of node_modules, .next cache, or an existing venv from development.

Check	Result	Note
pnpm install --frozen-lockfile	OK	lockfile v9, no resolution drift
pnpm run typecheck (tsc -p tsconfig.check.json, CI-blocking)	0 errors	Clean on the pristine clone. The working directory this branch was developed in separately contains 5 untracked, uncommitted paths (src/fleet/, src/graph/, src/metrics/, src/projections/, src/features/feature-registry.ts) that were already present and flagged, unaudited, in BEFORE_STATE.md before this pass started. They are not part of any commit on this branch or on main — confirmed by git diff --stat origin/main..feat/p010-data-os-completion touching none of them, and by their total absence from the fresh clone. Running pnpm typecheck inside the original, non-clean working directory does surface 14 TS errors from those 5 stray local files; that is local working-tree litter, not something CI or a fresh clone will ever see, and this pass does not touch, fix, or commit any of those files.
pnpm run test (vitest, CI-blocking)	256/256 passed, 18 skipped (34 files passed, 3 files skipped)	Skips are the pre-existing .realinfra/.integration suites that self-skip honestly when no live ST0RE/OpenSearch/ClickHouse is reachable from this environment ([realinfra] ST0RE ... not reachable this run (HTTP 401) -- skipping honestly, not fabricating success — actual stderr from this run). 249 of the 256 predate this branch; 7 are new (tests/p0/quality-bridge.test.ts).
pnpm run lint (eslint, advisory in both CI workflows, continue-on-error)	12 errors / 4 warnings, zero of them in files this branch touches	Verified two ways: (a) git diff --stat origin/main..feat/p010-data-os-completion on every flagged file returns no output — byte-identical to main; (b) ran pnpm lint against a worktree checked out at origin/main directly and got the identical 12/4. This is 100% pre-existing legacy noise (mostly @typescript-eslint/no-explicit-any in v0-generated adapters/tests), not introduced or worsened by this pass.
pnpm run build (next build, CI-blocking in .gitea/workflows/ci.yml; not run by .github/workflows/ci.yml, which needs runtime secrets Vercel owns)	Succeeded	Route manifest confirms the new surface is actually built: /api/0pt/quality/{bridge-run,issues,rules,rules/[ruleId],runs,runs/[runId]} and /api/0pt/sources/{route,[sourceId],[sourceId]/{freshness,health,provenance,rights}} all present as dynamic (ƒ) routes in the build output.
python -m pytest -v (services/api, fresh venv from requirements-dev.txt)	28 passed, 2 skipped	The 2 skips are test_known_gaps.py's documented CDC/backfill-idempotency gaps (§4 above) — pytest.skip(reason=...), not silent omission. Not currently wired into either .gitea/workflows/ci.yml or .github/workflows/ci.yml — both CI workflows' own comments say the Python engine "is gated separately if/when a Python workflow is added." This pass did not add that workflow (out of P010's stated scope); the 28/28 above is a manual verification, not an automated CI gate today. Flagging this explicitly as a residual gap, not glossing over it.

Net CI-blocking-gate status: all green (typecheck, vitest, next build in Gitea CI; typecheck + vitest in the GitHub CI mirror). Lint is advisory everywhere and pre-existing failures are unrelated to this branch. Python tests are green but not yet gated by any CI workflow — see the Gap list below.

---

3. Files changed (47 files, +3119/-201, base origin/main → 9740e20)

New route surface: app/api/0pt/quality/{bridge-run,issues,rules,rules/[ruleId],runs,runs/[runId]}/route.ts, app/api/0pt/sources/{route.ts,[sourceId]/route.ts,[sourceId]/{freshness,health,provenance,rights}/route.ts}. New Python: services/api/workers/quality_bridge.py, services/api/pytest.ini, services/api/requirements-dev.txt, services/api/tests/test_{connectors_provenance,known_gaps,pipeline_fault_injection,quality_bridge,resolve_determinism}.py. New TS repositories/migration: src/repositories/* (7 files), sql/002_source_registry.sql. New TS quality bridge: src/quality/quality-bridge.ts, src/services/quality.ts, src/services/source-registry.ts. Modified: services/api/sources/connectors.py, services/api/workers/{pipeline,resolve}.py, src/services/{source-registry-service,source-service,source-rate-policy-service}.ts, src/validation/0pt-schemas.ts, src/http/siax-error-response.ts, .gitignore (added .venv/, .pytest_cache/), sql/004_iceberg_write_boundary.sql (minor). Test files modified: tests/p0/source-registry.test.ts (repointed at shared singleton), new tests/p0/quality-bridge.test.ts, tests/p0/source-registry-postgres.realinfra.test.ts.

No changes to: services/scrapers/ (the still-unmerged WAF-bypass deletion PR — untouched, not this pass's scope), the app/-root Next.js UI pages (bestall/, katalog/, portal/, etc. — untouched), any of the 5 untracked stray paths in the working tree.

---

4. Activating the pipeline bridge — Simon's decision, not made here

OPT_QUALITY_BRIDGE_ENABLED ships unset (default false) and this pass does not flip it, anywhere, including in any deploy config, .env.example, or CI workflow. This affects real customer orders once on — enabling it in production is a separate decision, deliberately left to Simon.

To turn it on later, the following must all be true first (this pass built the mechanism, not the readiness):

1. OPT_QUALITY_BRIDGE_URL (or OPT_QUALITY_BRIDGE_EDGE_URL, per quality_bridge.py's _bridge_url()) must point at a reachable, authenticated 0pt.app deployment — the Python engine and the Next.js edge app are typically different deploy targets (VPS vs. Vercel/SIAX Deploy Platform per the siax-0pt skill's two-plane model), so this is a real cross-service network dependency, not a same-process flag flip.
2. Even fully enabled and reachable, the bridge remains advisory-only by design — it stamps a quality field on delivered provenance and surfaces a score; it does not and, per §2's ExpectationEngine contract gap, currently cannot change which rows Python's delivered = [...] line keeps or drops. Anyone enabling the flag expecting stricter row-level gating will not get it from this code as shipped — that would require the ExpectationEngine contract change noted above, which is future work.
3. Recommended rollout, not yet executed: enable against one non-production tenant/order first, confirm bridge-run round-trips correctly (the manual live-verification in §1/§2 above did this once, ad hoc, during development — it was not repeated against a staging environment as a formal rollout step), then decide with Simon whether "advisory-only, fails open" is the intended production posture or whether the per-row gating gap needs to close first.

This pass leaves the flag off. No default, config file, or workflow in this PR turns it on.

---

5. Residual gaps (honest, not exhaustive elsewhere in this doc)

• Python engine tests (28/28) are not wired into either CI workflow yet — a .gitea/workflows//.github/workflows/ Python job is future work, not part of this pass.
• bridge_provenance as a single unified construct still does not exist; this pass narrows but does not close the TS/Python provenance-schema gap (§3 above).
• Per-row quality gating from the TS engine into the Python pipeline is not implementable against the current ExpectationEngine contract (§2, §4 above) — advisory-only is the ceiling of what this pass could honestly build, not a corner cut.
• resolve.py's greedy merge remains genuinely order-sensitive for non-transitive-similarity triangles — tested and documented, not fixed (fixing it is an algorithm change, out of scope here).
• CDC-interruption and true delivery-store backfill-idempotency tests remain unimplementable because the underlying infrastructure (a CDC pipeline, a delivery persistence store) does not exist anywhere in services/api — documented via pytest.skip with reasons in test_known_gaps.py, not silently dropped.
• 5 untracked/unaudited local paths (src/fleet/, src/graph/, src/metrics/, src/projections/, src/features/feature-registry.ts) remain exactly as found — not committed, not deleted, not audited by this pass. They are not part of this PR's diff and a fresh clone never sees them.
• services/scrapers/ (WAF-bypass stack) remains on main, unmerged deletion PR still open — unrelated to and untouched by this pass, flagged again here only because BEFORE_STATE.md flagged it and it has not changed.

---

6. Process notes

• No PR was open before this pass; one was opened as part of this pass (see PR URL reported separately / in PRODUCTION_EVIDENCE.json).
• BEFORE_STATE.md and the 5 pre-existing untracked/unaudited paths were left untouched and uncommitted throughout, exactly as found at the start.
• .venv/ and .pytest_cache/ (both created fresh under services/api/ while running the Python suite) were added to .gitignore so they are never accidentally committed; no leftover Python processes.
• The next dev server and any background verification process used for the live end-to-end bridge check (§1/§2) were both confirmed stopped before this pass ended — no lingering listeners.