8 Commits

Include renames

Author	SHA1	Message	Date
admin	93c01d2264	fix(er): refuse score-only matches at Python merge boundary	2026-09-07 18:18:33 +02:00
admin	6f525c5c9e	fix: pass registration country into canonical company identity rules	2026-09-07 18:15:47 +02:00
admin	3826f115ec	fix: enforce canonical hard non-match constraints across ER clusters	2026-09-07 17:58:06 +02:00
admin	ebb9521801	refactor(er): remove duplicate Python scoring engine	2026-09-07 17:20:26 +02:00
Simon AxelssonandClaude Opus 5	9267576b7c	feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill ... BEFORE_STATE (verified, with evidence): * P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/ freshness to Postgres, but carried no authorization, no schedule, no checkpoint, no schema and no owner distinct from created_by — and the activation DoD was inverted: postgres-source-repository.ts wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered. No activation operation existed at all. * P061 MISSING. services/api/sources/connectors.py had per-source fetch_*() functions only — no lifecycle, no cursor, no resume, no health, no conformance suite. * P062 MISSING, already documented as such in services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere; workers/pipeline.py has no persistence at all). * P063 MISSING. schema_version existed as a free string; no comparison, no policy, no versioned source schema. * P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only opened an issue and the data flowed on. No quarantine anywhere. * P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution- service.ts (canonical) and services/api/workers/resolve.py (simplified, still called by pipeline.py:110). * P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a lineage question — no store, no query, no route. * P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store persistence layer exists to dedupe a backfill against"). BUILT (only what was PARTIAL/BROKEN/MISSING): * P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/ checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot produce an unclassified-but-active source. SourceRegistryService gains classifySource/activateSource/deactivateSource + authorization/schedule accessors; registration NEVER activates (asking for status:"active" goes through the gate, not around it); a service identity cannot authorize a source. Routes: activate (GET classification / POST activate), authorization, schedule. * P061 src/connectors/connector.ts — all nine operations on one interface. src/connectors/conformance.ts — a runnable suite (15 checks, no "skip" verdict). HttpJsonConnector (real HTTP, opaque cursor) and JsonlFileConnector (real file, line offsets, CDC deletes) both pass it. The suite caught a real defect while being written: an API that drops next_cursor at the end left the connector unable to resume, which is why HttpJsonConnectorConfig has cursor_field. * P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts. One envelope for batch/stream/cdc; durable per-(source,partition) checkpoints; resume proven by killing a run inside the sink. * P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two enforcement points, both upstream of any consumer write: a discovery gate before the first read, and per-record conformance before the page reaches the sink. A producer cannot widen its own policy retroactively. * P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine (not a second engine) and adds the missing consequence: block / quarantine / warn, with quarantine durable and recoverable. No policy is reported as "none_declared", never as a pass; an unexecutable expectation counts as a violation, never a silent pass. * P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage. AUD0 stays the system of record; this is 0pt's own queryable projection. * P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent on repeat, leaves the live checkpoint alone, still re-emits real changes. A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft combined the idempotency check and its recording into one atomic call BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the page in flight when the sink died was already marked delivered, so the resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by splitting peekDeliveries/commitDeliveries and committing both the ledger and the checkpoint only after the sink resolves. NOT DONE, stated rather than deferred quietly: * P065 is NOT met. resolve.py is still a second simplified production engine. Closing it means rewriting it as an adapter over the canonical service, which cannot be verified from this repo today: services/api is not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI workflow runs its pytest suite. The duplication is now stated at both sites in code and the removal plan is in STATUS.md. The new ingestion plane performs no ER, so it adds no third engine. * P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract read only. * Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock). * No CDC *source* exists; the contract is met, a CDC connector is not built. * 0pt does not schedule itself — the schedule row is a declaration ACT0 would read. VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was 311/18); pnpm build clean with all four new routes in the route table; lint unchanged at 12 pre-existing errors. Real Postgres 16 (tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real `docker restart` of that container with an identical read-back through the production code path — that is P060's "a restart preserves all sources", measured. DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404; api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify app id. None of the capability layer runs in production. Nothing here is LIVE_VERIFIED and STATUS.md says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 17:03:14 +02:00
Simon AxelssonandClaude Sonnet 5	9740e20818	feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... Continues P010 (branch renamed from feat/p010-source-registry-persistence): closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain, §3 missing provenance fields, §4 untested failure modes), all additive to the production pipeline — the Python QA filter's delivery decision is unchanged by any of this. TS/Python bridge (§2): - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD + POST /api/0pt/quality/bridge-run) over the shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance" defect the Source Registry had. - src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use so the Python engine never pre-registers rules through the CRUD surface. - services/api/workers/quality_bridge.py: the Python-side HTTP client. Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open on any error (never raises into run_job, never changes `delivered`). Advisory only — real per-row gating against the TS engine isn't implementable yet (ExpectationEngine returns dataset-level pass/fail counts + samples, not a per-row boolean array); documented, not faked. - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge route, receiving a real QualityService score. Provenance (§3): sources/connectors.py's Provenance dataclass gains retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref (all previously absent from the real dataflow entirely) via a make_provenance() factory used by every connector. resolve.py stamps entity_resolution on merged records' provenance (never on unmerged ones). pipeline.py stamps artifact_ref at delivery and quality once the bridge succeeds. Failure tests (§4), real fault injection against real code (services/api, new pytest + respx suite, not vitest mocks): - dedup-determinism + duplicate-input + out-of-order: resolve.py under shuffled/interleaved/exhaustive-permutation input, plus an explicit, verified test of the one case that IS legitimately order-sensitive (non-transitive similarity in a greedy clusterer) instead of hiding it. - schema-evolution: respx-mocked upstream responses with added/removed/ alien-shaped fields through the real brreg connector + pipeline. - restart: connector exceptions mid-job (including one from respx itself) never crash run_job. - backfill: the real property that holds given no persistence layer exists (replay is compute-idempotent) — NOT a fabricated dedup-store test. - CDC-interruption and true backfill/delivery-store idempotency: explicitly documented as not implementable against this codebase (no CDC pipeline, no delivery store exist anywhere) via skip markers with reasons, in tests/test_known_gaps.py, rather than silently dropped or faked. Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service tests) and 28/28 pytest (2 documented skips) green locally. tsc -p tsconfig.check.json has zero new errors (14 pre-existing errors, all in the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections paths this pass does not touch, per BEFORE_STATE.md). No PR opened, per instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:02:14 +02:00
adminandClaude Opus 4.8	e6d3101501	fix: 17 confirmed bugs from full-repo audit (edge + Python engine) (#5) ... * docs: correct stale quality-check-service.ts comment references Two source comments still cited src/services/quality-check-service.ts as an existing precedent. It doesn't exist anywhere in this repo's history (git log --all: zero commits, any branch, ever) -- confirmed during the earlier full-doc-corpus accuracy pass. Points each comment at the real precedent instead (document-extraction-service.ts / iceberg-write-service.ts). Applied from task_05ec36aa, spun off during that pass. * fix: 17 confirmed bugs from full-repo audit (edge + Python engine) Found by an 8-finder bug hunt across app/api, src/services, src/clients, frontend, the Python engine, the data layer, config/infra, and tests, then independently adversarially verified per finding (32 raw findings -> 27 confirmed real bugs; 5 refuted). This commit applies the 17 confirmed as safe, mechanical fixes -- no architecture/design decision required. The other 10 need a decision and are reported separately, not touched here. Most consequential, app/api/payments/[order_id]/deposit/route.ts: the client-supplied Origin request header was trusted unsanitized both as a server-side fetch target (SSRF) and as Stripe Checkout success/cancel redirect URLs (open redirect tied to a real payment flow), and a failed order lookup silently fell through to a hardcoded deposit amount instead of 404. Fixed by reading the order directly via lib/orders-store.ts (mirroring the sibling release/route.ts, which already did this correctly) instead of both trusting the Origin header and self-issuing an HTTP fetch. Also fixed: - 5 routes (orders, orders/[id], estimates, subscriptions, ai/parse) that 500'd instead of 400ing on a literal JSON `null` body, since req.json() resolves (not rejects) for that input and bypasses the try/catch guard. - app/api/0pt/lakehouse/mappings/{,[mappingId]/{,writes/}}route.ts each instantiated their own private service instances, so a mapping created via POST was invisible to the sibling GET and writes routes even before the separately-tracked dataset-registration gap. Extracted a shared singleton module, src/services/lakehouse-mapping-registry.ts. - app/api/0pt/documents/[documentId]/extraction/route.ts collapsed every error (not-configured, policy-denied, policy-approval-required, not-found) into a blanket 400; now status-mapped like the sibling sources/[sourceId]/query route. - app/api/vehicles/route.ts: plate query param wasn't URL-encoded before being concatenated into the outbound RDW request, allowing extra query parameters to be injected; added a strict allowlist + encodeURIComponent. - app/api/uploads/route.ts: POST returned a fake 200 success with put_url: null even when "configured", instead of continuing to report 501 until presign generation is actually implemented. - src/adapters/document-intelligence-adapter.ts: the result-polling loop never checked response.ok, so a non-2xx poll response (auth expiry, 429, 5xx) was silently retried until a generic timeout masked the real error. - app/order/[id]/page.tsx: finish() never cleared the production-simulation setInterval, so it kept firing (and re-rendering) forever after delivery. - app/admin/page.tsx: the metrics fetch had no response/shape validation (unlike the adjacent orders fetch), rendering literal "undefined" stat tiles on a malformed response instead of falling back to <ErrorState/>. - services/api/workers/pipeline.py: `config.get("seeds") or DEMO_SEEDS` used truthy-or, so an explicit empty seeds dict was silently replaced with hardcoded demo org numbers instead of being respected as empty. - services/api/workers/resolve.py: corroboration (confidence/agreed_sources bump) no longer fires for two hits from the same source under fuzzy match -- only a genuinely independent source now counts, per docs/06's own ">=2 independent sources agree" definition -- and a merged record's provenance array now actually includes the corroborating source's fields instead of only a bare source-name string. - services/api/orchestrator/main.py: create_order's int(...) on unvalidated client volume/confidence fields raised on any decimal-formatted input (int("92.5") -> ValueError); now coerced via float() first with a safe default. /vehicles/{plate} and /screen also now catch connector exceptions and degrade gracefully, matching pipeline.py's existing pattern for the same connectors. - lib/orders-store.ts: postgres.js (pinned v3.4.9) returns bigint/numeric columns as strings by default; getOrder/listOrders/setStatus were silently handing callers stringified money fields. Added a narrow per-row coercion instead of touching the shared client's global type map (which would affect unrelated tables' bigint/numeric columns too). - lib/subscriptions-store.ts: the in-memory fallback returned oldest-first while the Postgres branch explicitly orders newest-first; now consistent, mirroring the identical fix already in orders-store.ts. - app/api/auth/login/route.ts: DEV_LOGIN_KEY was compared with a plain !==, a non-constant-time comparison; replaced with crypto.timingSafeEqual. Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo, 0 failures (13 files, unchanged from before this commit -- no existing test's assertions needed to change). services/api/workers/resolve.py's source-novelty and provenance-merge logic additionally verified by direct execution against synthetic same-source/cross-source inputs (no Python test suite exists in this repo to add to). services/api/workers/pipeline.py and services/api/orchestrator/main.py verified with python -m py_compile (no httpx/fastapi installed in this environment to run them directly). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(engine): treat explicit seeds:null as missing to avoid 500 in run_job Four-eyes follow-up to this PR's own seeds change. Switching from `config.get('seeds') or DEMO_SEEDS` to `config['seeds'] if 'seeds' in config` correctly stopped an explicit empty {} from being backfilled with DEMO_SEEDS, but regressed on client-controlled `{"seeds": null}`: seeds became None and the downstream `seeds.get(market, [])` raised AttributeError -> 500. That path is reachable (POST /orders stores raw body.config; POST /orders/{id}/run passes it straight to run_job). Using `config.get('seeds') is not None` keeps the intended behaviour (explicit {} honoured, never silently seeded with demo data) while restoring the None-tolerance the old idiom had. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-22 18:52:19 +02:00
adminandClaude Opus 4.8	be6909bce2	feat(engine+edge): live vehicles + screening endpoints, entity-resolution + job-queue + financials modules ... Edge /api/vehicles (RDW NL, live now) + /api/screen (OpenSanctions, live now), both defer to engine when ENGINE_URL set. Engine: /vehicles/{plate} + /screen endpoints; workers/resolve (blocking+fuzzy entity resolution, now used by pipeline); workers/queue (in-process job queue); sources/financials (Companies House accounts + XBRL path). py_compile + tsc + next build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-25 01:45:38 +02:00