feat(data-os): P010 completion — Source Registry persistence + TS/Python quality bridge + provenance + fault-injection tests #28

Merged

admin merged 3 commits from feat/p010-data-os-completion into main 2026-09-01 19:10:26 +00:00

Conversation 0 Commits 3 Files Changed 49

+3609 -201

admin commented 2026-09-01 19:07:31 +00:00

Owner

Copy Link

Copy Source

Summary

Closes the three remaining gaps from BEFORE_STATE.md (the pre-pass recon,
committed at the head of this branch):

• §1 Source Registry persistence — new src/repositories/* layer
  (in-memory + Postgres implementations), sql/002_source_registry.sql
  migration, shared-singleton wiring fixing the "new Map() per route module"
  defect, and a full HTTP route surface for rights/provenance/health/
  freshness/describe that previously didn't exist.
• §2 TS/Python split-brain — new app/api/0pt/quality/* route surface
  over a shared QualityService, src/quality/quality-bridge.ts
  auto-provisioning baseline rules, and services/api/workers/ quality_bridge.py on the Python side: feature-flagged OFF by default
  (OPT_QUALITY_BRIDGE_ENABLED), fails open on every error mode, advisory
  only. Live-verified once end-to-end (real next dev + real Python
  pipeline + real brreg.no + real bridge round-trip) during development.
• §3 Provenance fields — Provenance dataclass in
  services/api/sources/connectors.py gains retrieved_at, transform,
  connector_version, producer, quality, entity_resolution,
  artifact_ref via a make_provenance() factory used by every connector.
• §4 Failure-mode tests — new pytest+respx suite: dedup-determinism,
  duplicate-input, out-of-order, schema-evolution, restart/crash-resilience,
  backfill-replay, plus 2 explicitly-documented pytest.skips for
  CDC-interruption and true delivery-store backfill-idempotency, which
  aren't implementable against this codebase (no CDC pipeline or delivery
  persistence layer exists anywhere in it).

Full detail, including what is honestly still NOT true after this pass, is
in IMPLEMENTATION_REPORT.md (this PR) and PRODUCTION_EVIDENCE.json
(machine-readable test evidence from a from-scratch verification run).

Production impact

None by default. OPT_QUALITY_BRIDGE_ENABLED ships unset/false and
this PR does not change that anywhere (no deploy config, no .env.example,
no CI workflow). Enabling it in production is a separate decision for
Simon — see IMPLEMENTATION_REPORT.md §4 for the readiness checklist.
Everything else in this PR is additive (new routes, new tables, new tests);
the production Python pipeline's delivery decision
(delivered = [r for r in deduped if r["confidence"] >= threshold]) is
unchanged.

Testing (from a fresh clone + fresh installs, not the dev working tree)

• pnpm run typecheck (blocking): 0 errors
• pnpm run test (vitest, blocking): 256/256 passed, 18 skipped
  (pre-existing realinfra/integration self-skips)
• pnpm run build (blocking in Gitea CI): succeeded, new routes present
  in the manifest
• pnpm run lint (advisory in both CI workflows): 12 errors/4 warnings,
  all pre-existing on main, zero in files this branch touches
  (verified via git diff + a direct lint run against origin/main)
• python -m pytest (services/api, fresh venv): 28/28 passed, 2
  documented skips — not yet wired into either CI workflow (flagged as a
  residual gap, not glossed over)

Not in scope / unchanged

services/scrapers/ (WAF-bypass stack, separate open deletion PR),
services/api CI wiring, the 5 pre-existing untracked/unaudited local
paths (src/fleet/, src/graph/, src/metrics/, src/projections/,
src/features/feature-registry.ts — not part of this diff, confirmed via
git diff --stat origin/main..HEAD, and absent entirely from a fresh
clone).

## Summary Closes the three remaining gaps from `BEFORE_STATE.md` (the pre-pass recon, committed at the head of this branch): - **§1 Source Registry persistence** — new `src/repositories/*` layer (in-memory + Postgres implementations), `sql/002_source_registry.sql` migration, shared-singleton wiring fixing the "new Map() per route module" defect, and a full HTTP route surface for rights/provenance/health/ freshness/describe that previously didn't exist. - **§2 TS/Python split-brain** — new `app/api/0pt/quality/*` route surface over a shared `QualityService`, `src/quality/quality-bridge.ts` auto-provisioning baseline rules, and `services/api/workers/ quality_bridge.py` on the Python side: **feature-flagged OFF by default** (`OPT_QUALITY_BRIDGE_ENABLED`), fails open on every error mode, advisory only. Live-verified once end-to-end (real `next dev` + real Python pipeline + real brreg.no + real bridge round-trip) during development. - **§3 Provenance fields** — `Provenance` dataclass in `services/api/sources/connectors.py` gains `retrieved_at`, `transform`, `connector_version`, `producer`, `quality`, `entity_resolution`, `artifact_ref` via a `make_provenance()` factory used by every connector. - **§4 Failure-mode tests** — new pytest+respx suite: dedup-determinism, duplicate-input, out-of-order, schema-evolution, restart/crash-resilience, backfill-replay, plus 2 explicitly-documented `pytest.skip`s for CDC-interruption and true delivery-store backfill-idempotency, which aren't implementable against this codebase (no CDC pipeline or delivery persistence layer exists anywhere in it). Full detail, including what is honestly still NOT true after this pass, is in `IMPLEMENTATION_REPORT.md` (this PR) and `PRODUCTION_EVIDENCE.json` (machine-readable test evidence from a from-scratch verification run). ## Production impact **None by default.** `OPT_QUALITY_BRIDGE_ENABLED` ships unset/`false` and this PR does not change that anywhere (no deploy config, no `.env.example`, no CI workflow). Enabling it in production is a separate decision for Simon — see `IMPLEMENTATION_REPORT.md` §4 for the readiness checklist. Everything else in this PR is additive (new routes, new tables, new tests); the production Python pipeline's delivery decision (`delivered = [r for r in deduped if r["confidence"] >= threshold]`) is unchanged. ## Testing (from a fresh clone + fresh installs, not the dev working tree) - `pnpm run typecheck` (blocking): **0 errors** - `pnpm run test` (vitest, blocking): **256/256 passed, 18 skipped** (pre-existing realinfra/integration self-skips) - `pnpm run build` (blocking in Gitea CI): **succeeded**, new routes present in the manifest - `pnpm run lint` (advisory in both CI workflows): 12 errors/4 warnings, **all pre-existing on `main`**, zero in files this branch touches (verified via `git diff` + a direct lint run against `origin/main`) - `python -m pytest` (`services/api`, fresh venv): **28/28 passed, 2 documented skips** — not yet wired into either CI workflow (flagged as a residual gap, not glossed over) ## Not in scope / unchanged `services/scrapers/` (WAF-bypass stack, separate open deletion PR), `services/api` CI wiring, the 5 pre-existing untracked/unaudited local paths (`src/fleet/`, `src/graph/`, `src/metrics/`, `src/projections/`, `src/features/feature-registry.ts` — not part of this diff, confirmed via `git diff --stat origin/main..HEAD`, and absent entirely from a fresh clone).

admin added 2 commits 2026-09-01 19:07:32 +00:00

feat(data-os): Source Registry Postgres persistence + full HTTP route surface ... 07b13bd833

Replaces the in-memory (per-instance, not even per-process-shared) Maps in
SourceService/SourceRegistryService with real Postgres persistence, and adds
the register/query/describe/health/rights/provenance/freshness HTTP routes
that never existed (grep for SourceRegistryService in app/ previously
returned nothing).

- sql/002_source_registry.sql: opt_sources, opt_source_rights,
  opt_source_provenance, opt_source_health_checks, opt_source_freshness.
  Numbered 002 (not 006) because 004's opt_source_rate_policies already
  forward-references opt_sources, which no sql/ file defined until now —
  fixes 004's pre-existing broken migration ordering (opt_datasets remains
  a separate, unrelated, still-open gap in 004, flagged not fixed).
- src/repositories/*: SourceRepository / SourceRegistryRepository seams,
  each with an in-memory implementation (business logic extracted verbatim)
  and a Postgres implementation, chosen via resolveStoreMode — production
  still fails hard with no DATABASE_URL (Master-DoD §3), but a configured
  production now actually works instead of always being blocked.
- SourceService / SourceRegistryService: storage moved behind the
  repository seam; getHealth/getFreshness are now async and tenant-scoped
  at the accessor itself (needed once they're reachable directly over HTTP,
  not only after an earlier caller's tenant check).
- src/services/source-registry.ts: shared singleton instances for the
  sources/* route group, fixing the "each route module owned its own
  private registry" defect (source-rate-policies and .../query previously
  couldn't see a source registered via POST /api/0pt/sources).
- New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id],
  GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health,
  GET .../freshness.
- Business logic unchanged: fail-closed purpose gate, cross-tenant
  NOT_FOUND (no existence oracle, §50), bounded 100-row health history.

Tested against a real Postgres container (not mocked) —
tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on
SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249
pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the
full HTTP route surface end-to-end against next dev + live Postgres.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... 9740e20818

Continues P010 (branch renamed from feat/p010-source-registry-persistence):
closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain,
§3 missing provenance fields, §4 untested failure modes), all additive to
the production pipeline — the Python QA filter's delivery decision is
unchanged by any of this.

TS/Python bridge (§2):
  - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD +
    POST /api/0pt/quality/bridge-run) over the shared QualityService
    singleton (src/services/quality.ts), fixing the same "no shared
    instance" defect the Source Registry had.
  - src/quality/quality-bridge.ts auto-provisions a deterministic,
    tenant-scoped baseline rule set on first use so the Python engine never
    pre-registers rules through the CRUD surface.
  - services/api/workers/quality_bridge.py: the Python-side HTTP client.
    Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open
    on any error (never raises into run_job, never changes `delivered`).
    Advisory only — real per-row gating against the TS engine isn't
    implementable yet (ExpectationEngine returns dataset-level pass/fail
    counts + samples, not a per-row boolean array); documented, not faked.
  - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real
    Python pipeline hitting the real brreg.no API, round-tripping through
    the real bridge route, receiving a real QualityService score.

Provenance (§3): sources/connectors.py's Provenance dataclass gains
retrieved_at, transform, connector_version, producer, quality,
entity_resolution, artifact_ref (all previously absent from the real
dataflow entirely) via a make_provenance() factory used by every connector.
resolve.py stamps entity_resolution on merged records' provenance (never on
unmerged ones). pipeline.py stamps artifact_ref at delivery and quality
once the bridge succeeds.

Failure tests (§4), real fault injection against real code (services/api,
new pytest + respx suite, not vitest mocks):
  - dedup-determinism + duplicate-input + out-of-order: resolve.py under
    shuffled/interleaved/exhaustive-permutation input, plus an explicit,
    verified test of the one case that IS legitimately order-sensitive
    (non-transitive similarity in a greedy clusterer) instead of hiding it.
  - schema-evolution: respx-mocked upstream responses with added/removed/
    alien-shaped fields through the real brreg connector + pipeline.
  - restart: connector exceptions mid-job (including one from respx itself)
    never crash run_job.
  - backfill: the real property that holds given no persistence layer
    exists (replay is compute-idempotent) — NOT a fabricated dedup-store
    test.
  - CDC-interruption and true backfill/delivery-store idempotency:
    explicitly documented as not implementable against this codebase
    (no CDC pipeline, no delivery store exist anywhere) via skip markers
    with reasons, in tests/test_known_gaps.py, rather than silently
    dropped or faked.

Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service
tests) and 28/28 pytest (2 documented skips) green locally. tsc -p
tsconfig.check.json has zero new errors (14 pre-existing errors, all in
the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections
paths this pass does not touch, per BEFORE_STATE.md).

No PR opened, per instructions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

admin added 1 commit 2026-09-01 19:08:26 +00:00

docs(p010): implementation report + production evidence for Data-OS completion ... e7516697b3

From-scratch verification (fresh clone, fresh installs) backing up the P010
completion pass: 256/256 vitest, 28/28 pytest, clean typecheck/build on the
committed tree, pre-existing-only lint noise. Documents that
OPT_QUALITY_BRIDGE_ENABLED ships off and stays off in this PR -- activating
it in production is left as Simon's decision, with a readiness checklist.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

admin merged commit into main 2026-09-01 19:10:26 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

siax-bot siax-ci siax-healer (siax-healer (CI self-healing agent)) admin (Simon Axelsson)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: admin/0pt#28