9 Commits

Include renames

Author	SHA1	Message	Date
admin	5137bf87a1	fix: distinguish failed sources from empty results	2026-11-02 06:58:15 +01:00
admin	ff11216813	feat(engine): fail-closed X-Internal-Token-gate + commit-kedja (GIT_SHA) i services/api ... - X-Internal-Token skickades av edgen men verifierades ALDRIG — gate som raw-ASGI-middleware: INTERNAL_TOKEN satt => alla routes utom /health kräver exakt token, 401 (constant-time) annars. Ospecificerad token = inert (dev). - Dockerfile: ARG GIT_SHA + OCI-label + /health.reporting revision - compose var redan skriven för INTERNAL_TOKEN — motorn hade bara aldrig deployats med den - tester: tests/test_internal_token_gate.py (3: off/public/401-pass) — hela services/api-sviten 62 passed, 2 skipped (dokumenterade known-gaps), 0 fail	2026-09-15 03:38:43 +02:00
admin	213294bc55	fix: use services/api pytest import convention	2026-09-08 22:50:05 +02:00
admin	97373bae0d	test: cover review-gated entity resolution	2026-09-08 22:50:05 +02:00
admin	6befaa7be7	test(er): resolv-determinism — match-pairs behöver hard-rule decided_by ... _apply_matches ignorerar medvetet decided_by=score (defence in depth: score/fuzzy får aldrig skapa canonical identity-länkar) — testernas stub-bridge returnerade score-pairs och förväntade ändå merge → CI-fail var implementationen som hade rätt. Pairs är nu decided_by=stable-identifiers.v3. Verifierat: pytest services/api/tests 6+14 pass, full unit 945/945, typecheck EXIT 0, real-Postgres 5/5.	2026-09-07 19:50:02 +02:00
admin	ad213b4400	test: prove hard non-match blocks transitive ER merge	2026-09-07 17:58:43 +02:00
admin	257e6d3410	test(er): cover canonical bridge application and fail-closed behavior	2026-09-07 17:21:19 +02:00
Simon AxelssonandClaude Sonnet 5	dc6aaa0e10	fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill ... Tenant isolation (security-critical): every app/api/0pt/** route reading tenant_id from client-controlled query/body never cross-checked it against the token middleware.ts already verified (x-0pt-verified-tenant). Adds a shared assertTenantMatchesVerified guard and wires it into all 20+ affected routes. Also closes two full-leak-by-omission bugs found in this pass (documents/capture and lakehouse/mappings GETs returned every tenant's rows when tenant_id was simply omitted) and two "no tenant check at all" reads (rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone. Duplicate-delivery idempotency (correctness-critical): POST /api/orders now honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres + in-memory); the Python orchestrator's POST /orders/{id}/run — the actual delivery trigger — is now idempotent under an asyncio lock, closing a real concurrent-request race, not just the sequential-retry case. Engine-duplication resolution: adaptive_router.py's docstring wrongly invited a bypass ("see b0ts for scraping") contradicting the product's own compliance stance; corrected, and given real (additive, not escalating) block-signal tracking so a rate-limited/blocked source is excluded from the plan instead of hammered. Also fixes a previously-undiscovered production bug: the Python engine's only call into the TS quality bridge (X-Internal-Token) was unconditionally 401'd by middleware.ts, which never checked that credential — the advisory quality bridge was unreachable in any environment enforcing auth. Restore drill: executed for real against throwaway Docker Postgres containers (not simulated). Found and fixed two real blockers along the way: sql/004 referenced opt_datasets, which no migration ever created (chain was never actually runnable end to end); lib/db.ts hardcoded ssl:"require", making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev container) unreachable. After both fixes: full migration chain applies cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts + content checksums), and real application code (lib/orders-store.ts, SourceRegistryService's Postgres repository) reads the restored data back correctly, including cross-tenant isolation still holding. 82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS + 55 Python) still green; typecheck clean on every changed/added file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-03 12:00:43 +02:00
Simon AxelssonandClaude Sonnet 5	9740e20818	feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... Continues P010 (branch renamed from feat/p010-source-registry-persistence): closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain, §3 missing provenance fields, §4 untested failure modes), all additive to the production pipeline — the Python QA filter's delivery decision is unchanged by any of this. TS/Python bridge (§2): - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD + POST /api/0pt/quality/bridge-run) over the shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance" defect the Source Registry had. - src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use so the Python engine never pre-registers rules through the CRUD surface. - services/api/workers/quality_bridge.py: the Python-side HTTP client. Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open on any error (never raises into run_job, never changes `delivered`). Advisory only — real per-row gating against the TS engine isn't implementable yet (ExpectationEngine returns dataset-level pass/fail counts + samples, not a per-row boolean array); documented, not faked. - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge route, receiving a real QualityService score. Provenance (§3): sources/connectors.py's Provenance dataclass gains retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref (all previously absent from the real dataflow entirely) via a make_provenance() factory used by every connector. resolve.py stamps entity_resolution on merged records' provenance (never on unmerged ones). pipeline.py stamps artifact_ref at delivery and quality once the bridge succeeds. Failure tests (§4), real fault injection against real code (services/api, new pytest + respx suite, not vitest mocks): - dedup-determinism + duplicate-input + out-of-order: resolve.py under shuffled/interleaved/exhaustive-permutation input, plus an explicit, verified test of the one case that IS legitimately order-sensitive (non-transitive similarity in a greedy clusterer) instead of hiding it. - schema-evolution: respx-mocked upstream responses with added/removed/ alien-shaped fields through the real brreg connector + pipeline. - restart: connector exceptions mid-job (including one from respx itself) never crash run_job. - backfill: the real property that holds given no persistence layer exists (replay is compute-idempotent) — NOT a fabricated dedup-store test. - CDC-interruption and true backfill/delivery-store idempotency: explicitly documented as not implementable against this codebase (no CDC pipeline, no delivery store exist anywhere) via skip markers with reasons, in tests/test_known_gaps.py, rather than silently dropped or faked. Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service tests) and 28/28 pytest (2 documented skips) green locally. tsc -p tsconfig.check.json has zero new errors (14 pre-existing errors, all in the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections paths this pass does not touch, per BEFORE_STATE.md). No PR opened, per instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:02:14 +02:00