feat(data-os): P010 completion — Source Registry persistence + TS/Python quality bridge + provenance + fault-injection tests #28

Merged

admin merged 3 commits from feat/p010-data-os-completion into main 2026-09-01 19:10:26 +00:00

Conversation 0 Commits 3 Files Changed 49

+3609 -201

3 Commits

Author	SHA1	Message	Date
Simon AxelssonandClaude Sonnet 5	e7516697b3	docs(p010): implementation report + production evidence for Data-OS completion ... From-scratch verification (fresh clone, fresh installs) backing up the P010 completion pass: 256/256 vitest, 28/28 pytest, clean typecheck/build on the committed tree, pre-existing-only lint noise. Documents that OPT_QUALITY_BRIDGE_ENABLED ships off and stays off in this PR -- activating it in production is left as Simon's decision, with a readiness checklist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:08:20 +02:00
Simon AxelssonandClaude Sonnet 5	9740e20818	feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... Continues P010 (branch renamed from feat/p010-source-registry-persistence): closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain, §3 missing provenance fields, §4 untested failure modes), all additive to the production pipeline — the Python QA filter's delivery decision is unchanged by any of this. TS/Python bridge (§2): - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD + POST /api/0pt/quality/bridge-run) over the shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance" defect the Source Registry had. - src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use so the Python engine never pre-registers rules through the CRUD surface. - services/api/workers/quality_bridge.py: the Python-side HTTP client. Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open on any error (never raises into run_job, never changes `delivered`). Advisory only — real per-row gating against the TS engine isn't implementable yet (ExpectationEngine returns dataset-level pass/fail counts + samples, not a per-row boolean array); documented, not faked. - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge route, receiving a real QualityService score. Provenance (§3): sources/connectors.py's Provenance dataclass gains retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref (all previously absent from the real dataflow entirely) via a make_provenance() factory used by every connector. resolve.py stamps entity_resolution on merged records' provenance (never on unmerged ones). pipeline.py stamps artifact_ref at delivery and quality once the bridge succeeds. Failure tests (§4), real fault injection against real code (services/api, new pytest + respx suite, not vitest mocks): - dedup-determinism + duplicate-input + out-of-order: resolve.py under shuffled/interleaved/exhaustive-permutation input, plus an explicit, verified test of the one case that IS legitimately order-sensitive (non-transitive similarity in a greedy clusterer) instead of hiding it. - schema-evolution: respx-mocked upstream responses with added/removed/ alien-shaped fields through the real brreg connector + pipeline. - restart: connector exceptions mid-job (including one from respx itself) never crash run_job. - backfill: the real property that holds given no persistence layer exists (replay is compute-idempotent) — NOT a fabricated dedup-store test. - CDC-interruption and true backfill/delivery-store idempotency: explicitly documented as not implementable against this codebase (no CDC pipeline, no delivery store exist anywhere) via skip markers with reasons, in tests/test_known_gaps.py, rather than silently dropped or faked. Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service tests) and 28/28 pytest (2 documented skips) green locally. tsc -p tsconfig.check.json has zero new errors (14 pre-existing errors, all in the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections paths this pass does not touch, per BEFORE_STATE.md). No PR opened, per instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:02:14 +02:00
Simon AxelssonandClaude Sonnet 5	07b13bd833	feat(data-os): Source Registry Postgres persistence + full HTTP route surface ... Replaces the in-memory (per-instance, not even per-process-shared) Maps in SourceService/SourceRegistryService with real Postgres persistence, and adds the register/query/describe/health/rights/provenance/freshness HTTP routes that never existed (grep for SourceRegistryService in app/ previously returned nothing). - sql/002_source_registry.sql: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, opt_source_freshness. Numbered 002 (not 006) because 004's opt_source_rate_policies already forward-references opt_sources, which no sql/ file defined until now — fixes 004's pre-existing broken migration ordering (opt_datasets remains a separate, unrelated, still-open gap in 004, flagged not fixed). - src/repositories/*: SourceRepository / SourceRegistryRepository seams, each with an in-memory implementation (business logic extracted verbatim) and a Postgres implementation, chosen via resolveStoreMode — production still fails hard with no DATABASE_URL (Master-DoD §3), but a configured production now actually works instead of always being blocked. - SourceService / SourceRegistryService: storage moved behind the repository seam; getHealth/getFreshness are now async and tenant-scoped at the accessor itself (needed once they're reachable directly over HTTP, not only after an earlier caller's tenant check). - src/services/source-registry.ts: shared singleton instances for the sources/* route group, fixing the "each route module owned its own private registry" defect (source-rate-policies and .../query previously couldn't see a source registered via POST /api/0pt/sources). - New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id], GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health, GET .../freshness. - Business logic unchanged: fail-closed purpose gate, cross-tenant NOT_FOUND (no existence oracle, §50), bounded 100-row health history. Tested against a real Postgres container (not mocked) — tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249 pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the full HTTP route surface end-to-end against next dev + live Postgres. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 20:42:41 +02:00