From 9267576b7c1a462eeaaa97ad86cd71f3a1f74998 Mon Sep 17 00:00:00 2001 From: Simon Axelsson Date: Fri, 4 Sep 2026 17:03:14 +0200 Subject: [PATCH] =?UTF-8?q?feat(data-os):=20P060-P068=20durable=20ingestio?= =?UTF-8?q?n=20plane=20=E2=80=94=20source=20classification=20gate,=20conne?= =?UTF-8?q?ctor=20contract,=20common=20envelope,=20schema=20evolution,=20q?= =?UTF-8?q?uality=20enforcement,=20queryable=20lineage,=20idempotent=20bac?= =?UTF-8?q?kfill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BEFORE_STATE (verified, with evidence): * P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/ freshness to Postgres, but carried no authorization, no schedule, no checkpoint, no schema and no owner distinct from created_by — and the activation DoD was inverted: postgres-source-repository.ts wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered. No activation operation existed at all. * P061 MISSING. services/api/sources/connectors.py had per-source fetch_*() functions only — no lifecycle, no cursor, no resume, no health, no conformance suite. * P062 MISSING, already documented as such in services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere; workers/pipeline.py has no persistence at all). * P063 MISSING. schema_version existed as a free string; no comparison, no policy, no versioned source schema. * P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only opened an issue and the data flowed on. No quarantine anywhere. * P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution- service.ts (canonical) and services/api/workers/resolve.py (simplified, still called by pipeline.py:110). * P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a lineage question — no store, no query, no route. * P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store persistence layer exists to dedupe a backfill against"). BUILT (only what was PARTIAL/BROKEN/MISSING): * P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/ checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot produce an unclassified-but-active source. SourceRegistryService gains classifySource/activateSource/deactivateSource + authorization/schedule accessors; registration NEVER activates (asking for status:"active" goes through the gate, not around it); a service identity cannot authorize a source. Routes: activate (GET classification / POST activate), authorization, schedule. * P061 src/connectors/connector.ts — all nine operations on one interface. src/connectors/conformance.ts — a runnable suite (15 checks, no "skip" verdict). HttpJsonConnector (real HTTP, opaque cursor) and JsonlFileConnector (real file, line offsets, CDC deletes) both pass it. The suite caught a real defect while being written: an API that drops next_cursor at the end left the connector unable to resume, which is why HttpJsonConnectorConfig has cursor_field. * P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts. One envelope for batch/stream/cdc; durable per-(source,partition) checkpoints; resume proven by killing a run inside the sink. * P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two enforcement points, both upstream of any consumer write: a discovery gate before the first read, and per-record conformance before the page reaches the sink. A producer cannot widen its own policy retroactively. * P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine (not a second engine) and adds the missing consequence: block / quarantine / warn, with quarantine durable and recoverable. No policy is reported as "none_declared", never as a pass; an unexecutable expectation counts as a violation, never a silent pass. * P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage. AUD0 stays the system of record; this is 0pt's own queryable projection. * P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent on repeat, leaves the live checkpoint alone, still re-emits real changes. A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft combined the idempotency check and its recording into one atomic call BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the page in flight when the sink died was already marked delivered, so the resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by splitting peekDeliveries/commitDeliveries and committing both the ledger and the checkpoint only after the sink resolves. NOT DONE, stated rather than deferred quietly: * P065 is NOT met. resolve.py is still a second simplified production engine. Closing it means rewriting it as an adapter over the canonical service, which cannot be verified from this repo today: services/api is not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI workflow runs its pytest suite. The duplication is now stated at both sites in code and the removal plan is in STATUS.md. The new ingestion plane performs no ER, so it adds no third engine. * P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract read only. * Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock). * No CDC *source* exists; the contract is met, a CDC connector is not built. * 0pt does not schedule itself — the schedule row is a declaration ACT0 would read. VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was 311/18); pnpm build clean with all four new routes in the route table; lint unchanged at 12 pre-existing errors. Real Postgres 16 (tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real `docker restart` of that container with an identical read-back through the production code path — that is P060's "a restart preserves all sources", measured. DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404; api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify app id. None of the capability layer runs in production. Nothing here is LIVE_VERIFIED and STATUS.md says so. Co-Authored-By: Claude Opus 5 --- STATUS.md | 104 ++++ app/api/0pt/lineage/route.ts | 47 ++ .../0pt/sources/[sourceId]/activate/route.ts | 47 ++ .../sources/[sourceId]/authorization/route.ts | 45 ++ .../0pt/sources/[sourceId]/schedule/route.ts | 48 ++ docs/P060_INGESTION_PLANE.md | 189 ++++++ services/api/workers/resolve.py | 38 ++ sql/007_ingestion_plane.sql | 162 +++++ src/connectors/conformance.ts | 276 +++++++++ src/connectors/connector.ts | 163 +++++ src/connectors/http-json-connector.ts | 274 +++++++++ src/connectors/jsonl-file-connector.ts | 224 +++++++ src/er/entity-resolution-service.ts | 10 + src/ingestion/envelope.ts | 87 +++ src/ingestion/ingestion-runner.ts | 557 ++++++++++++++++++ src/lineage/lineage-edge.ts | 42 ++ src/lineage/lineage-store.ts | 121 ++++ src/quality/quality-policy.ts | 129 ++++ .../in-memory-ingestion-repository.ts | 120 ++++ .../in-memory-source-registry-repository.ts | 26 +- .../in-memory-source-repository.ts | 17 +- src/repositories/ingestion-repository.ts | 91 +++ .../postgres-ingestion-repository.ts | 308 ++++++++++ .../postgres-source-registry-repository.ts | 100 ++++ .../postgres-source-repository.ts | 24 +- .../source-registry-repository.ts | 16 +- src/repositories/source-repository.ts | 6 + src/schema/compatibility.ts | 170 ++++++ src/schema/record-schema.ts | 65 ++ src/schema/schema-registry.ts | 144 +++++ src/services/source-registry-service.ts | 202 ++++++- src/services/source-registry.ts | 16 + src/services/source-service.ts | 26 + src/types/ingestion.ts | 111 ++++ src/types/source-registry.ts | 69 ++- src/types/source.ts | 18 +- src/validation/0pt-schemas.ts | 33 +- tests/api/0pt-p060-routes.test.ts | 175 ++++++ tests/p0/connector-conformance.test.ts | 165 ++++++ ...ingestion-plane-postgres.realinfra.test.ts | 310 ++++++++++ tests/p0/ingestion-runner.test.ts | 412 +++++++++++++ tests/p0/schema-evolution.test.ts | 149 +++++ tests/p0/source-activation-gate.test.ts | 190 ++++++ 43 files changed, 5506 insertions(+), 20 deletions(-) create mode 100644 app/api/0pt/lineage/route.ts create mode 100644 app/api/0pt/sources/[sourceId]/activate/route.ts create mode 100644 app/api/0pt/sources/[sourceId]/authorization/route.ts create mode 100644 app/api/0pt/sources/[sourceId]/schedule/route.ts create mode 100644 docs/P060_INGESTION_PLANE.md create mode 100644 sql/007_ingestion_plane.sql create mode 100644 src/connectors/conformance.ts create mode 100644 src/connectors/connector.ts create mode 100644 src/connectors/http-json-connector.ts create mode 100644 src/connectors/jsonl-file-connector.ts create mode 100644 src/ingestion/envelope.ts create mode 100644 src/ingestion/ingestion-runner.ts create mode 100644 src/lineage/lineage-edge.ts create mode 100644 src/lineage/lineage-store.ts create mode 100644 src/quality/quality-policy.ts create mode 100644 src/repositories/in-memory-ingestion-repository.ts create mode 100644 src/repositories/ingestion-repository.ts create mode 100644 src/repositories/postgres-ingestion-repository.ts create mode 100644 src/schema/compatibility.ts create mode 100644 src/schema/record-schema.ts create mode 100644 src/schema/schema-registry.ts create mode 100644 src/types/ingestion.ts create mode 100644 tests/api/0pt-p060-routes.test.ts create mode 100644 tests/p0/connector-conformance.test.ts create mode 100644 tests/p0/ingestion-plane-postgres.realinfra.test.ts create mode 100644 tests/p0/ingestion-runner.test.ts create mode 100644 tests/p0/schema-evolution.test.ts create mode 100644 tests/p0/source-activation-gate.test.ts diff --git a/STATUS.md b/STATUS.md index 658d85c..7df9a86 100644 --- a/STATUS.md +++ b/STATUS.md @@ -82,3 +82,107 @@ accurate. 5. 0pt-motor executors for the delegated jobs (`0pt.quality.ge_run`, `0pt.features.*`, `0pt.er.batch_resolution`). 6. Observability: no metrics/traces emitted by the new services (§36). + +--- + +## P060-P068 ingestion plane (added 2026-09-04) + +Classification scale unchanged. Every row below states what was actually +verified, and by which command. + +| Requirement | BEFORE this pass | Status now | Evidence | +|---|---|---|---| +| **P060** Source Registry: durable canonical registry; restart preserves sources; an unclassified source cannot be activated | **PARTIAL / BROKEN.** The registry existed with Postgres persistence (`sql/002_source_registry.sql`, `src/repositories/postgres-source-registry-repository.ts`) and carried source_id/tenant/project/type/status/license/purpose/rights. It had NO authorization, NO schedule, NO checkpoint, NO schema, and no `owner` distinct from `created_by`. The activation DoD was inverted: `postgres-source-repository.ts:70` wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered, and no activation operation existed at all | **INTEGRATION_TESTED** | `sql/007_ingestion_plane.sql` (authorization/schedule/schemas/checkpoints + `ALTER … SET DEFAULT 'pending'` + `owner_id`); `SourceRegistryService.classifySource`/`activateSource`; `tests/p0/source-activation-gate.test.ts` (11); `tests/api/0pt-p060-routes.test.ts` (9); `tests/p0/ingestion-plane-postgres.realinfra.test.ts` against a live Postgres 16 | +| **P061** Connector contract: discover/validate/connect/checkpoint/read/resume/backfill/health/disconnect; two heterogeneous connectors pass one conformance suite | **MISSING.** `services/api/sources/connectors.py` had per-source `fetch_*()` functions only — no lifecycle, no cursor, no resume, no health, no disconnect, no conformance suite | **INTEGRATION_TESTED** | `src/connectors/connector.ts` (all nine operations); `src/connectors/conformance.ts` (15 checks); `HttpJsonConnector` (real node:http server, opaque cursor) and `JsonlFileConnector` (real file, line-offset cursor, CDC deletes) both pass the identical suite — `tests/p0/connector-conformance.test.ts`, which also proves the suite FAILS a fail-open connector | +| **P062** CDC/batch/stream normalized to one envelope; restart resumes from a durable checkpoint without silent gaps | **MISSING**, and documented as such: `services/api/tests/test_known_gaps.py:10-19` skipped the CDC case because no CDC pipeline existed anywhere, and `workers/pipeline.py` had no persistence at all | **INTEGRATION_TESTED** | `src/types/ingestion.ts` (one envelope for all three modes); `src/ingestion/ingestion-runner.ts`; `tests/p0/ingestion-runner.test.ts` kills a run inside the sink and proves the resumed run covers all 10 records exactly once; same proof against real Postgres across fresh processes | +| **P063** Schema versioning + compatibility policy; a breaking schema is detected before consumer corruption | **MISSING.** `schema_version` existed as a free-form string on datasets/Iceberg writes; there was no comparison, no policy, no versioned source schema | **INTEGRATION_TESTED** | `src/schema/compatibility.ts`, `src/schema/schema-registry.ts`; `tests/p0/schema-evolution.test.ts` (12); the runner's schema gate runs before the first read — `tests/p0/ingestion-runner.test.ts` asserts `sinkCalls === 0` on a break | +| **P064** Quality rules behind the SIAX abstraction; failures block or quarantine per declared policy | **PARTIAL.** `src/quality/*` had rules, runs, results, issues, scores and a GE-compatible engine, but a failing rule only opened an ISSUE — the data flowed on regardless. `grep -ri quarantin src/` found one unrelated hit | **INTEGRATION_TESTED** | `src/quality/quality-policy.ts` composes the EXISTING `ExpectationEngine` (not a second engine) and adds the consequence; block / quarantine / warn each proved in `tests/p0/ingestion-runner.test.ts`; quarantine durability proved against real Postgres | +| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **DUPLICATE (confirmed).** `src/er/entity-resolution-service.ts` (canonical: hard rules, review band, Splink delegation) AND `services/api/workers/resolve.py` (simplified fuzzy threshold, auto-merge) — the latter still called by `workers/pipeline.py:110` | **NOT DONE — documented, not built.** The duplication is now stated at BOTH sites in code with the removal plan. The new ingestion plane deliberately performs no ER, so it adds no third engine. See "P065 — why it was not closed" below | +| **P066** Iceberg | out of scope — ST0RE is owned by another workflow | **NOT TOUCHED** | Contract read only (`src/services/iceberg-write-service.ts`, `src/clients/st0re-lakehouse-client.ts`) | +| **P067** Lineage: OpenLineage/AUD0 events; source → output lineage is queryable | **PARTIAL.** `src/lineage/lineage-emitter.ts` emitted OpenLineage to AUD0, but nothing in this repo could ANSWER a lineage question — no store, no query, no route | **INTEGRATION_TESTED** | `src/lineage/lineage-edge.ts`, `src/lineage/lineage-store.ts` (local queryable projection; AUD0 stays the system of record), `GET /api/0pt/lineage`; `tests/p0/ingestion-runner.test.ts` queries both directions and proves tenant scoping | +| **P068** Durable bounded backfill/replay; repeated backfill is idempotent | **MISSING**, documented as such: `services/api/tests/test_known_gaps.py:44-52` skipped it because "no delivery-store persistence layer exists to dedupe a backfill against" | **INTEGRATION_TESTED** | `opt_ingestion_ledger` + `IngestionRunner.backfill`; `tests/p0/ingestion-runner.test.ts` (repeat emits 0, live checkpoint untouched, a real content change still re-emits); same against real Postgres | + +### What was actually run (not "should pass") + +- `pnpm run typecheck` — clean. +- `pnpm run test` — **365 passed, 25 skipped** (was 311/18 before this pass). +- `pnpm run build` — clean; the four new routes appear in the route table. +- `pnpm run lint` — 12 errors / 5 warnings, **all pre-existing** (identical + count on the base commit; advisory in CI). This pass adds none. +- Real Postgres 16: `tests/p0/ingestion-plane-postgres.realinfra.test.ts` + with `INGESTION_TEST_DATABASE_URL` — 7 passed. +- **Real container restart**, run manually against that same database: + `docker restart` the Postgres container, then re-read through the real + repositories. Before and after were identical — 2 sources (one `active` + with its owner, one `pending`), 1 checkpoint (cursor 6, 6 committed), + 6 ledger rows, 1 lineage edge, 4 quarantined records, 1 schema version. + That is P060's "a restart preserves all sources", measured rather than + asserted. + +### A real bug this pass wrote and then caught + +The first draft of `IngestionRepository` combined the idempotency check and +its recording into one atomic call, made BEFORE the sink. The crash/resume +test then delivered **7 of 10 records**: the page in flight when the sink +died had already been marked delivered, so the resumed run skipped it as a +duplicate — permanent, silent data loss. The fix is the `peekDeliveries` / +`commitDeliveries` split, with both the ledger commit and the checkpoint +commit after the sink resolves. Recorded here because the failure mode is +the interesting artifact, not the fix. + +### Honest gaps in the new plane (not hidden) + +1. **Delivery is AT-LEAST-ONCE, not exactly-once.** Two concurrent runs on + the same (source, partition) can both peek "new" and both emit. + `envelope_id` is deterministic so a dedup-capable sink survives it. There + is no per-(source, partition) run lock. Do not describe this plane as + exactly-once. +2. **No CDC *source* exists.** The envelope, the operations and the runner + handle change events, and `JsonlFileConnector` emits real deletes from an + `_op` field — but this repo still has no Debezium/WAL/replication-slot + consumer. P062's contract is met; a CDC connector is still MISSING. +3. **0pt does not schedule itself.** `opt_source_schedule` is a declaration + ACT0 would read; there is no cron daemon here. +4. **The sink is caller-supplied.** The runner stops at the sink; the + lakehouse write boundary is ST0RE's (P066, another workflow's repo). +5. **Nothing here is deployed.** See the deployment note below. +6. **`QualityService`'s own rule/run/issue registry is still in-process + Maps** (`src/quality/quality-service.ts:47-50`). The new *enforcement* + path is durable; the older rule registry is not. Unchanged by this pass. +7. **Python tests still run nowhere.** `grep -rn "pytest" .gitea/workflows + .github/workflows` finds nothing — `services/api/tests/*` (9 files) has + never been executed by CI. Flagged, not fixed (it needs a workflow change + and a Python toolchain decision). + +### Deployment reality, measured 2026-09-04 + +`https://0pt.app/api/health` answers **200** from **Vercel** with +`{"db":"not_configured"}`, and `https://0pt.app/api/0pt/sources` answers +**404** — the deployed build predates the entire `app/api/0pt/*` capability +layer. `https://api.0pt.app/health` answers **404** (the FastAPI engine is +not reachable), and `.gitea/workflows/cd.yml` still guards on +`COOLIFY_APP_ID = TODO-no-coolify-app-provisioned-yet`. So: none of the +capability layer — neither this pass's work nor P010/P014's — is running in +production, and P060's durability guarantee cannot be exercised there, +because production has no database configured and does not serve the routes. +Verified with `curl`; not inferred. + +### P065 — why it was not closed + +Closing P065 means turning `services/api/workers/resolve.py` into an adapter +over the canonical `src/er/entity-resolution-service.ts`. That change cannot +be verified from this repo today: `services/api` is not built by this repo's +Dockerfile (it builds only the Next.js app), `api.0pt.app` answers 404, and +no CI workflow runs its pytest suite. A blind rewrite would ship untested +into a path nobody can exercise — the exact failure mode this programme +exists to stop. + +The removal plan, for whoever picks it up: +1. Expose the canonical ER capability as `POST /api/0pt/er/candidates` + (service already exists; only a route + Zod schema are missing). +2. Add a Python CI job that actually runs `services/api/tests`. +3. Replace `resolve.py`'s body with an HTTP adapter over that route, + **fail-closed** when it is unreachable — never falling back to the local + scorer, which would preserve the duplicate under a different name. +4. Port `test_resolve_determinism.py` to assert the adapter's contract, and + delete the fuzzy scorer. diff --git a/app/api/0pt/lineage/route.ts b/app/api/0pt/lineage/route.ts new file mode 100644 index 0000000..8733ad3 --- /dev/null +++ b/app/api/0pt/lineage/route.ts @@ -0,0 +1,47 @@ +/** + * app/api/0pt/lineage/route.ts + * ============================= + * P067 DoD: "source -> output lineage is QUERYABLE". This is the query. + * + * GET /api/0pt/lineage?tenant_id=…&output_name=… -> what fed this output + * GET /api/0pt/lineage?tenant_id=…&input_name=… -> what this input produced + * + * The answer comes from 0pt's local projection of the OpenLineage events it + * emits to AUD0 (src/lineage/lineage-store.ts). AUD0 remains the system of + * record; this endpoint exists because "emitted to AUD0" was never the same + * thing as "answerable from 0pt", and before this route nothing in this repo + * could answer either question. + */ +import { NextRequest, NextResponse } from "next/server"; +import { lineageStore } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest) { + try { + const { searchParams } = new URL(request.url); + const tenantId = searchParams.get("tenant_id"); + const outputName = searchParams.get("output_name") ?? undefined; + const inputName = searchParams.get("input_name") ?? undefined; + if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + if (!outputName && !inputName) { + return NextResponse.json( + { error: "one of output_name or input_name is required — an unfiltered lineage dump is not a query" }, + { status: 400 }, + ); + } + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + const limitRaw = searchParams.get("limit"); + const limit = limitRaw ? Math.min(Math.max(Number(limitRaw), 1), 500) : undefined; + const edges = await lineageStore.query({ + tenant_id: tenantId, + output_name: outputName, + input_name: inputName, + limit, + }); + return NextResponse.json({ edges, count: edges.length }); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/sources/[sourceId]/activate/route.ts b/app/api/0pt/sources/[sourceId]/activate/route.ts new file mode 100644 index 0000000..4a10ef6 --- /dev/null +++ b/app/api/0pt/sources/[sourceId]/activate/route.ts @@ -0,0 +1,47 @@ +/** + * app/api/0pt/sources/[sourceId]/activate/route.ts + * ================================================== + * P060 activation gate over HTTP. POST activates a source ONLY when it is + * classified (owner + unexpired authorization + rights + provenance); + * otherwise it returns 403 AUTHORIZATION_DENIED naming every missing fact. + * GET returns the classification without changing anything, so an operator + * can see what is missing before trying. + */ +import { NextRequest, NextResponse } from "next/server"; +import { activateSourceSchema, parseBody } from "@/src/validation/0pt-schemas"; +import { sourceRegistryService } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id"); + if (!tenantId) { + return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + } + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + return NextResponse.json(await sourceRegistryService.classifySource(sourceId, tenantId)); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function POST(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const parsed = parseBody(activateSourceSchema, await request.json()); + if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status }); + const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id); + if (guard) return guard; + const source = await sourceRegistryService.activateSource( + sourceId, + parsed.data.tenant_id, + parsed.data.actor_id, + ); + return NextResponse.json(source); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/sources/[sourceId]/authorization/route.ts b/app/api/0pt/sources/[sourceId]/authorization/route.ts new file mode 100644 index 0000000..69b1c1f --- /dev/null +++ b/app/api/0pt/sources/[sourceId]/authorization/route.ts @@ -0,0 +1,45 @@ +/** + * app/api/0pt/sources/[sourceId]/authorization/route.ts + * ====================================================== + * P060 "authorization": the recorded decision that 0pt may connect to this + * source at all — separate from rights (what the DATA may be used for). PUT + * is an upsert; a service identity as `granted_by` is rejected (400), because + * a source may not authorize itself. + */ +import { NextRequest, NextResponse } from "next/server"; +import { parseBody, setSourceAuthorizationSchema } from "@/src/validation/0pt-schemas"; +import { sourceRegistryService } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id"); + if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + const auth = await sourceRegistryService.getAuthorization(sourceId, tenantId); + if (!auth) { + return NextResponse.json({ error: `source ${sourceId} has no authorization record` }, { status: 404 }); + } + return NextResponse.json(auth); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function PUT(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const parsed = parseBody(setSourceAuthorizationSchema, await request.json()); + if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status }); + const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id); + if (guard) return guard; + return NextResponse.json( + await sourceRegistryService.setAuthorization({ ...parsed.data, source_id: sourceId }), + ); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/sources/[sourceId]/schedule/route.ts b/app/api/0pt/sources/[sourceId]/schedule/route.ts new file mode 100644 index 0000000..0d2de4f --- /dev/null +++ b/app/api/0pt/sources/[sourceId]/schedule/route.ts @@ -0,0 +1,48 @@ +/** + * app/api/0pt/sources/[sourceId]/schedule/route.ts + * ================================================= + * P060 "schedule": ingestion mode + cadence, recorded on the source. 0pt does + * NOT run cron itself — ACT0 owns scheduling; this row is the declaration a + * scheduler reads, and saying otherwise would be a claim this repo cannot back. + */ +import { NextRequest, NextResponse } from "next/server"; +import { parseBody, setSourceScheduleSchema } from "@/src/validation/0pt-schemas"; +import { sourceRegistryService } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id"); + if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + const schedule = await sourceRegistryService.getSchedule(sourceId, tenantId); + if (!schedule) { + return NextResponse.json({ error: `source ${sourceId} has no schedule record` }, { status: 404 }); + } + return NextResponse.json(schedule); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function PUT(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const parsed = parseBody(setSourceScheduleSchema, await request.json()); + if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status }); + const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id); + if (guard) return guard; + return NextResponse.json( + await sourceRegistryService.setSchedule({ + ...parsed.data, + source_id: sourceId, + updated_at: new Date().toISOString(), + }), + ); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/docs/P060_INGESTION_PLANE.md b/docs/P060_INGESTION_PLANE.md new file mode 100644 index 0000000..b55d52a --- /dev/null +++ b/docs/P060_INGESTION_PLANE.md @@ -0,0 +1,189 @@ +# 0pt ingestion plane (P060–P068) + +> Written 2026-09-04. Every claim here is backed by a test path or a +> recorded command; anything not verified says so. Read `STATUS.md`'s +> "P060-P068 ingestion plane" section for the classification table and the +> honest-gaps list — this document is the mechanism, that one is the status. + +## The shape + +``` + Source Registry Connector Ingestion Runner + (P060: who, what, (P061: nine (P062: one envelope, + under what rights, operations, one one durable checkpoint) + on what schedule) conformance suite) + │ │ │ + │ classification gate │ discover / validate │ read ─┐ + │ blocks activation │ connect / read │ │ + ▼ ▼ resume / backfill ▼ ▼ + opt_sources records ───────────────▶ IngestionEnvelope + opt_source_rights │ + opt_source_authorization ├─▶ P063 schema gate + opt_source_schedule ├─▶ P064 quality gate + opt_source_schemas ├─▶ P068 ledger (peek) + opt_ingestion_checkpoints ├─▶ SINK (caller-supplied) + ├─▶ P068 ledger (commit) + ├─▶ checkpoint (commit) + └─▶ P067 lineage edge +``` + +## P060 — the registry, and the gate + +A source now carries: `source_id`, `tenant_id`, `project_id`, `source_type`, +`owner_id`, an **authorization** record, a **license/purpose rights** record, +**provenance** (how the data is obtained), a **schedule**, a versioned +**schema**, a durable **checkpoint**, and a `status`. + +`status` starts at **`pending`**. It was `active` before — both repository +implementations wrote `input.status ?? "active"` — which meant an +unclassified source went live the moment it was registered. `activateSource` +is now the only path to `active`, and it refuses unless ALL of these are on +record: + +- an `owner_id` (distinct from `created_by`: who is accountable, not who typed it), +- an **unexpired** authorization granted by an accountable actor + (`svc_*`/`system`/`automation` are rejected — a source may not authorize itself), +- rights with a `license_id` **and** at least one `allowed_purpose`, +- provenance with a sourcing `method`. + +Schedule, schema and checkpoint describe HOW a source runs, not WHETHER it +may, so they are deliberately not part of the gate. + +The default is enforced in two places on purpose: in the repositories, and as +`ALTER TABLE opt_sources ALTER COLUMN status SET DEFAULT 'pending'`, so a raw +`INSERT` from psql cannot produce an unclassified-but-active source either +(proved in the realinfra suite). + +Routes: `GET/POST /api/0pt/sources/{id}/activate` (GET returns the +classification without changing anything), `GET/PUT …/authorization`, +`GET/PUT …/schedule`. + +## P061 — the connector contract + +Nine operations on one interface: `validate` (pure, no IO, fail-closed), +`connect` (idempotent), `discover`, `health` (reports, never throws), `read` +(honours `limit` exactly), `checkpoint`, `resume`, `backfill` (bounded — an +open-ended range is refused), `disconnect` (idempotent, safe after an error). + +`src/connectors/conformance.ts` is a plain async function, not a test file, +so the same suite runs from tests, from an operator script against a real +configured source, and (later) from a pre-activation check. There is no +"skip" verdict: a connector that cannot honestly serve a check FAILS it. + +Two heterogeneous connectors pass it: + +| | `HttpJsonConnector` | `JsonlFileConnector` | +|---|---|---| +| transport | network, real HTTP | local filesystem | +| cursor | opaque server token, with a `cursor_field` fallback | line offset it computes itself | +| exhaustion | server-reported | exact (the file is finite) | +| health | HTTP round trip | `stat` | +| backfill bounds | server-defined (sequence/timestamp) | line numbers | +| CDC deletes | no — it does not claim `cdc` | yes, via `_op` | + +The `cursor_field` fallback exists because the conformance suite caught a +real defect: an API that stops returning `next_cursor` at the end of a page +left the connector with nothing to resume from, so `resume()` silently +restarted at the beginning. + +## P062 — one envelope, one durable checkpoint + +`IngestionEnvelope` is the single shape everything downstream sees: +`mode` (batch/stream/cdc), `operation` (upsert/insert/update/delete), +`partition_key`, `record_key`, `schema_version`, a monotonic `sequence`, +`occurred_at`/`ingested_at`, `payload` (null only for a delete) and +`provenance` (connector id + version + run id + cursor). + +**Ordering is the contract.** Per page: + +``` +read/resume → normalize → schema conformance → quality gate + → ledger PEEK → SINK → ledger COMMIT → checkpoint COMMIT +``` + +Both commits happen after the sink resolves. The two tempting shortcuts are +both data-loss bugs, and one of them was written here first — see STATUS.md's +"A real bug this pass wrote and then caught". + +The resulting guarantee is **at-least-once**, not exactly-once. +`envelope_id` is a deterministic hash of +`(source, partition, record_key, content)` so a dedup-capable sink survives a +re-delivery. + +## P063 — schema evolution + +`RecordSchema` is a small structural shape (field → type + required) plus a +`CompatibilityPolicy` (`backward` / `forward` / `full` / `none`). +`checkCompatibility` answers what breaks, in which direction, and why. It is +deliberately not a second JSON-Schema dialect: the question is "is version +N+1 of an UPSTREAM shape safe for consumers of version N?", which needs a +comparable structure, not a validator. + +Two enforcement points, both upstream of any consumer write: + +1. the runner's **discovery gate** — a shape change that breaks the previous + version's declared policy fails the whole run before the first `read()`; +2. **per-record conformance** — a source that drifted without redeclaring its + schema is caught on the offending record, before the page reaches the sink. + +A producer cannot widen its own policy: the PREVIOUS version's policy governs +whether a change is allowed. + +## P064 — quality with consequences + +`QualityGate` composes the existing `ExpectationEngine` (the Great +Expectations core subset already in this repo) and adds the missing half: a +declared `on_violation` action. + +- `block` — the page fails, and the checkpoint does **not** advance, so the + bad page stays unconsumed rather than being stepped over. +- `quarantine` — failing envelopes are persisted whole to + `opt_quality_quarantine` (recoverable, not merely counted); clean ones + continue and the checkpoint advances. +- `warn` — everything continues, violations are counted and reported. + +There is no default. With no policy the run report says +`quality_policy: "none_declared"` — an undeclared policy must never read as a +passed one. An expectation the local engine cannot execute counts as a +VIOLATION, never a silent pass. + +## P067 — queryable lineage + +`LineageEmitter` keeps emitting OpenLineage to AUD0, and **AUD0 remains the +system of record**. `LineageStore` adds 0pt's own flattened projection — one +row per (run, input, output) with a deterministic `edge_id` — because +"emitted to AUD0" was never the same thing as "answerable from 0pt". + +``` +GET /api/0pt/lineage?tenant_id=…&output_name=0pt:dataset:ds_x # what fed it +GET /api/0pt/lineage?tenant_id=…&input_name=source/src_x # what it produced +``` + +An unfiltered dump is refused (400): that is an export, not a query. + +## P068 — bounded, idempotent backfill + +`opt_ingestion_ledger` holds one row per `(source_id, record_key)` with the +last accepted content hash. A backfill runs through the same envelope, +schema, quality and ledger path as a normal run, and: + +- **never touches the live checkpoint** — a backfill must not rewind or + advance streaming ingestion; +- **re-emits nothing** on a repeat — every record comes back `duplicate`; +- **still re-emits a genuine content change** — idempotency is not amnesia; +- **refuses an unbounded range** at the connector contract level. + +## Running the real-infrastructure proof + +```bash +docker run -d --name opt-ingestion-pgtest \ + -e POSTGRES_USER=opt -e POSTGRES_PASSWORD=opt_test_pw -e POSTGRES_DB=opt_test \ + -p 15434:5432 postgres:16-alpine + +INGESTION_TEST_DATABASE_URL=postgres://opt:opt_test_pw@localhost:15434/opt_test \ + pnpm vitest run tests/p0/ingestion-plane-postgres.realinfra.test.ts +``` + +The suite applies `sql/002_source_registry.sql` and +`sql/007_ingestion_plane.sql` itself (both are idempotent). It skips +honestly, never fabricating success, when the env var is unset. diff --git a/services/api/workers/resolve.py b/services/api/workers/resolve.py index d8131dc..ae50623 100644 --- a/services/api/workers/resolve.py +++ b/services/api/workers/resolve.py @@ -3,6 +3,44 @@ Entity resolution (docs/06): block, score, and link records across sources so th same company from GLEIF + Brreg + CVR collapses to one record with corroborated confidence. rapidfuzz only; deterministic; no external calls. +======================================================================= +P065 — THIS IS THE SECOND ER ENGINE. IT IS NOT THE CANONICAL ONE. +======================================================================= +0pt has TWO entity-resolution implementations, and this is the simplified +one: + + * CANONICAL: src/er/entity-resolution-service.ts — the declared generic ER + capability. Deterministic blocking + an explainable per-pair score, + product-supplied HARD RULES that override the score in both directions, + a configurable review band that hands ambiguous pairs to humans instead + of auto-merging, and delegation of heavy probabilistic linkage (Splink) + to 0pt-motor via the N0D job "0pt.er.batch_resolution". + + * THIS FILE: blocking on (market, first initial) + a single + rapidfuzz token_sort_ratio threshold, first-match-wins, auto-merge with + no review band, no hard-rule hook, no explanation beyond the score. + +P065's definition of done is that no second simplified production engine +remains. It has NOT been met: this file is still what +`workers/pipeline.py` calls (line ~110, `deduped = resolve(records)`). + +WHY IT WAS NOT REPLACED IN THE P060-P068 PASS, stated plainly rather than +quietly deferred: replacing it means turning this module into an ADAPTER +that calls the canonical capability, and that change cannot be verified +from this repo today — + * `services/api` is not deployed by this repo's Dockerfile (which builds + only the Next.js app), and api.0pt.app answers 404, so there is no + running engine to test against; + * no CI workflow runs services/api's pytest suite at all (verified: + `grep -rn "pytest" .gitea/workflows .github/workflows` finds nothing), + so a rewrite here would land with zero automated coverage. +Rewriting it blind and calling P065 done would be exactly the +"claimed complete, broken in operation" failure this programme exists to +stop. The removal plan is recorded in STATUS.md. + +Until then: do NOT add capability to this file. Any new ER behaviour +belongs in the canonical TypeScript service. + Deterministic by construction: for a fixed `records` input, blocking is a pure function of (market, name-first-letter) and matching is first-match-wins within a block, so results do not depend on which source answered first — see diff --git a/sql/007_ingestion_plane.sql b/sql/007_ingestion_plane.sql new file mode 100644 index 0000000..508ae86 --- /dev/null +++ b/sql/007_ingestion_plane.sql @@ -0,0 +1,162 @@ +-- 0pt ingestion plane (P060-P068). +-- +-- Adds the durable state the Source Registry and the ingestion runner need +-- and that sql/002_source_registry.sql did NOT carry: +-- +-- * opt_source_authorization -- P060 "authorization": the recorded decision +-- that this tenant may use this source at all. Separate from rights +-- (license/purpose) on purpose: rights say WHAT the data may be used for, +-- authorization says WHO signed off that 0pt may connect. Both are +-- required to activate. +-- * opt_source_schedule -- P060 "schedule": ingestion mode + cadence. +-- * opt_source_schemas -- P060 "schema" + P063 schema evolution: +-- one row per (source_id, schema_version), plus the compatibility policy +-- that governs the NEXT version. +-- * opt_ingestion_checkpoints -- P060 "checkpoint" + P062: the durable +-- resume point per (source_id, partition_key). This is the table that +-- makes "restart resumes without silent gaps" true rather than aspirational. +-- * opt_ingestion_ledger -- P068: idempotency ledger. One row per +-- (source_id, record_key, payload hash); a repeated backfill that +-- re-delivers the same record is recognised, not re-emitted. +-- * opt_quality_quarantine -- P064: where envelopes go when the declared +-- enforcement policy says "quarantine" instead of "block". +-- * opt_lineage_edges -- P067: the LOCAL, QUERYABLE projection of +-- the OpenLineage events 0pt already emits to AUD0. AUD0 remains the +-- system of record; this table exists because "emitted to AUD0" is not +-- the same thing as "queryable from 0pt", and the P067 DoD asks for the +-- latter. +-- +-- Every statement is IF NOT EXISTS / idempotent so it can be applied to an +-- already-migrated database, matching sql/002_source_registry.sql's contract +-- (tests/p0/*.realinfra.test.ts apply the files themselves). +-- +-- Ordering: requires sql/002_source_registry.sql (opt_sources) to have run. + +-- ── P060 owner + the pre-activation status ──────────────────────────────── +-- `created_by` (002) records who typed the registration; `owner_id` records +-- who is ACCOUNTABLE for the source afterwards. The activation gate requires +-- the latter, so they cannot be collapsed. +ALTER TABLE opt_sources ADD COLUMN IF NOT EXISTS owner_id TEXT; +CREATE INDEX IF NOT EXISTS opt_sources_owner_idx ON opt_sources (owner_id); + +-- New sources are created 'pending' (see src/repositories/*-source-repository.ts). +-- The column default in 002 is 'active'; it is changed here so a plain INSERT +-- that omits `status` -- from any client, including psql -- cannot produce an +-- unclassified-but-active source. Rows that are already 'active' are left +-- exactly as they are: this migration does not retroactively deactivate +-- anything an operator has already approved. +ALTER TABLE opt_sources ALTER COLUMN status SET DEFAULT 'pending'; + +-- ── P060 authorization ──────────────────────────────────────────────────── +-- One row per source. `basis` is the legal/contractual ground; `granted_by` +-- is the accountable actor id (never "system"); `evidence_ref` points at the +-- AUD0 evidence or 1p/contract artifact. `expires_at` is honoured by the +-- activation gate -- an expired authorization does not classify a source. +CREATE TABLE IF NOT EXISTS opt_source_authorization ( + source_id TEXT PRIMARY KEY REFERENCES opt_sources (source_id), + tenant_id TEXT NOT NULL, + basis TEXT NOT NULL, -- contract | public_register | consent | legitimate_interest | internal + granted_by TEXT NOT NULL, -- accountable actor id + granted_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ, + evidence_ref TEXT, -- aud0 evidence ref / contract artifact ref + notes TEXT, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS opt_source_authorization_tenant_idx ON opt_source_authorization (tenant_id); + +-- ── P060 schedule ───────────────────────────────────────────────────────── +CREATE TABLE IF NOT EXISTS opt_source_schedule ( + source_id TEXT PRIMARY KEY REFERENCES opt_sources (source_id), + tenant_id TEXT NOT NULL, + mode TEXT NOT NULL, -- batch | stream | cdc + interval_seconds INT, -- batch/cdc poll cadence; NULL for continuous streams + cron TEXT, -- optional cron expression (documented, not executed here) + enabled BOOLEAN NOT NULL DEFAULT false, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS opt_source_schedule_tenant_idx ON opt_source_schedule (tenant_id); + +-- ── P060 schema + P063 schema evolution ─────────────────────────────────── +-- `fields` is the record schema as JSON: {"": {"type": "...", +-- "required": true|false}}. `compatibility` governs what the NEXT registered +-- version is allowed to change (see src/schema/compatibility.ts). +CREATE TABLE IF NOT EXISTS opt_source_schemas ( + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + schema_version INT NOT NULL, + tenant_id TEXT NOT NULL, + fields JSONB NOT NULL, + compatibility TEXT NOT NULL, -- backward | forward | full | none + registered_at TIMESTAMPTZ NOT NULL DEFAULT now(), + registered_by TEXT, + PRIMARY KEY (source_id, schema_version) +); +CREATE INDEX IF NOT EXISTS opt_source_schemas_tenant_idx ON opt_source_schemas (tenant_id); + +-- ── P060 checkpoint + P062 durable resume ───────────────────────────────── +-- `cursor` is opaque to 0pt -- the connector defines it (an offset, an LSN, +-- a page token). `sequence` is 0pt's own monotonic envelope counter for the +-- partition, used to detect gaps on resume. +CREATE TABLE IF NOT EXISTS opt_ingestion_checkpoints ( + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + partition_key TEXT NOT NULL, + tenant_id TEXT NOT NULL, + cursor TEXT, + sequence BIGINT NOT NULL DEFAULT 0, + records_committed BIGINT NOT NULL DEFAULT 0, + committed_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (source_id, partition_key) +); +CREATE INDEX IF NOT EXISTS opt_ingestion_checkpoints_tenant_idx ON opt_ingestion_checkpoints (tenant_id); + +-- ── P068 backfill idempotency ledger ────────────────────────────────────── +-- A backfill/replay that re-delivers a record 0pt has already accepted must +-- be a no-op, not a duplicate. `payload_hash` distinguishes "same record, +-- same content" (skip) from "same record, new content" (an update, emitted). +CREATE TABLE IF NOT EXISTS opt_ingestion_ledger ( + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + record_key TEXT NOT NULL, + tenant_id TEXT NOT NULL, + payload_hash TEXT NOT NULL, + first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + delivery_count BIGINT NOT NULL DEFAULT 1, + PRIMARY KEY (source_id, record_key) +); +CREATE INDEX IF NOT EXISTS opt_ingestion_ledger_tenant_idx ON opt_ingestion_ledger (tenant_id); + +-- ── P064 quarantine ─────────────────────────────────────────────────────── +CREATE TABLE IF NOT EXISTS opt_quality_quarantine ( + quarantine_id TEXT PRIMARY KEY, + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + tenant_id TEXT NOT NULL, + run_id TEXT NOT NULL, + record_key TEXT NOT NULL, + violations JSONB NOT NULL, -- [{expectation_type, kwargs, detail}] + envelope JSONB NOT NULL, + quarantined_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS opt_quality_quarantine_source_idx ON opt_quality_quarantine (source_id, quarantined_at DESC); +CREATE INDEX IF NOT EXISTS opt_quality_quarantine_tenant_idx ON opt_quality_quarantine (tenant_id); + +-- ── P067 queryable lineage projection ───────────────────────────────────── +-- One row per (run, input, output) edge. `edge_id` is deterministic +-- (sha256 of the tuple) so a re-emitted idempotent run does not duplicate +-- edges -- the same dedup property LineageEmitter relies on in AUD0. +CREATE TABLE IF NOT EXISTS opt_lineage_edges ( + edge_id TEXT PRIMARY KEY, + tenant_id TEXT NOT NULL, + run_id TEXT NOT NULL, + capability_id TEXT NOT NULL, + input_namespace TEXT NOT NULL, + input_name TEXT NOT NULL, + input_ref TEXT, + output_namespace TEXT NOT NULL, + output_name TEXT NOT NULL, + output_ref TEXT, + event_time TIMESTAMPTZ NOT NULL, + correlation_id TEXT +); +CREATE INDEX IF NOT EXISTS opt_lineage_edges_output_idx ON opt_lineage_edges (tenant_id, output_name); +CREATE INDEX IF NOT EXISTS opt_lineage_edges_input_idx ON opt_lineage_edges (tenant_id, input_name); +CREATE INDEX IF NOT EXISTS opt_lineage_edges_run_idx ON opt_lineage_edges (tenant_id, run_id); diff --git a/src/connectors/conformance.ts b/src/connectors/conformance.ts new file mode 100644 index 0000000..199c863 --- /dev/null +++ b/src/connectors/conformance.ts @@ -0,0 +1,276 @@ +/** + * src/connectors/conformance.ts + * ============================== + * P061 DoD: "two heterogeneous connectors pass the same conformance suite." + * + * This module IS that suite. It is a plain async function returning a report + * — not a vitest file — so it can be run three ways without duplication: + * 1. from tests (tests/p0/connector-conformance.test.ts runs it per connector), + * 2. from an operator script against a real configured source, + * 3. from a future capability route as a pre-activation check. + * + * Each check is a named assertion about the CONTRACT, not about any one + * connector's data. A connector that cannot serve a check honestly (e.g. it + * has no partitions under the given config) FAILS it — there is no "skip" + * verdict, because a skip that reads as a pass is how a conformance suite + * becomes decoration. + */ +import type { + BackfillRequest, + Connector, + ConnectorConfig, + ConnectorContext, +} from "./connector"; + +export interface ConformanceCheck { + name: string; + passed: boolean; + detail: string; +} + +export interface ConformanceReport { + connector_id: string; + connector_version: string; + passed: boolean; + checks: ConformanceCheck[]; +} + +export interface ConformanceFixture { + /** A config the connector must accept. */ + validConfig: ConnectorConfig; + /** A config the connector must REJECT in validate() without doing IO. */ + invalidConfig: ConnectorConfig; + ctx: ConnectorContext; + /** + * A bounded backfill range that overlaps data the connector can serve. + * `from`/`to` are connector-defined (documented per connector). + */ + backfill: Omit & { partition_key?: string }; + /** Total number of records the fixture's first partition holds, so paging can be checked exactly. */ + expectedRecordCount: number; +} + +async function check( + checks: ConformanceCheck[], + name: string, + fn: () => Promise | string, +): Promise { + try { + const detail = await fn(); + checks.push({ name, passed: true, detail }); + } catch (err) { + checks.push({ + name, + passed: false, + detail: err instanceof Error ? err.message : String(err), + }); + } +} + +function assert(condition: boolean, message: string): void { + if (!condition) throw new Error(message); +} + +/** + * Run the full conformance suite against one connector instance. + * The connector is left disconnected on return, pass or fail. + */ +export async function runConnectorConformance( + connector: Connector, + fixture: ConformanceFixture, +): Promise { + const checks: ConformanceCheck[] = []; + + await check(checks, "identity: non-empty id and version", () => { + assert(connector.connector_id.length > 0, "connector_id is empty"); + assert(connector.connector_version.length > 0, "connector_version is empty"); + return `${connector.connector_id}@${connector.connector_version}`; + }); + + await check(checks, "validate: accepts a valid config without IO", () => { + const res = connector.validate(fixture.validConfig); + assert(res.valid, `valid config rejected: ${res.errors.join("; ")}`); + assert(res.errors.length === 0, "valid config produced errors"); + return "valid config accepted"; + }); + + await check(checks, "validate: rejects an invalid config and says why", () => { + const res = connector.validate(fixture.invalidConfig); + assert(!res.valid, "invalid config was accepted (fail-open)"); + assert(res.errors.length > 0, "rejection carried no reason"); + return `rejected with ${res.errors.length} reason(s)`; + }); + + await check(checks, "operations before connect() fail closed", async () => { + let threw = false; + try { + await connector.read({ partition_key: "anything", limit: 1 }); + } catch { + threw = true; + } + assert(threw, "read() succeeded without a session"); + return "read() refused without a session"; + }); + + await check(checks, "connect: idempotent", async () => { + await connector.connect(fixture.validConfig, fixture.ctx); + await connector.connect(fixture.validConfig, fixture.ctx); + return "connect() called twice without error"; + }); + + let partitionKey = fixture.backfill.partition_key ?? ""; + await check(checks, "discover: reports at least one partition with a schema", async () => { + const discovery = await connector.discover(); + assert(discovery.partitions.length > 0, "no partitions discovered"); + assert(discovery.modes.length > 0, "no ingestion modes declared"); + const first = discovery.partitions[0]; + assert(first.partition_key.length > 0, "partition_key is empty"); + assert( + Object.keys(first.schema).length > 0, + `partition "${first.partition_key}" reported an empty schema`, + ); + if (!partitionKey) partitionKey = first.partition_key; + return `${discovery.partitions.length} partition(s), modes: ${discovery.modes.join(",")}`; + }); + + await check(checks, "health: answers with a status instead of throwing", async () => { + const health = await connector.health(); + assert( + ["up", "degraded", "down"].includes(health.status), + `unexpected health status "${health.status}"`, + ); + return `status=${health.status}`; + }); + + await check(checks, "read: honours the limit and advances the cursor", async () => { + const first = await connector.read({ partition_key: partitionKey, limit: 1 }); + assert(first.records.length <= 1, `limit 1 returned ${first.records.length} records`); + assert(first.records.length === 1, "limit 1 returned nothing — fixture has no data"); + assert(first.next_cursor !== undefined, "no cursor returned after a non-empty read"); + const second = await connector.read({ + partition_key: partitionKey, + cursor: first.next_cursor, + limit: 1, + }); + if (second.records.length > 0) { + assert( + second.records[0].record_key !== first.records[0].record_key, + "reading from the returned cursor re-delivered the same record (cursor is not exclusive)", + ); + } + return `cursor advanced past ${first.records[0].record_key}`; + }); + + await check(checks, "read: paging covers every record exactly once", async () => { + const seen: string[] = []; + let cursor: string | undefined; + for (let page = 0; page < 100; page++) { + const res = await connector.read({ partition_key: partitionKey, cursor, limit: 2 }); + for (const r of res.records) seen.push(r.record_key); + cursor = res.next_cursor; + if (res.exhausted || res.records.length === 0) break; + } + assert( + seen.length === fixture.expectedRecordCount, + `paged ${seen.length} records, fixture declares ${fixture.expectedRecordCount}`, + ); + assert(new Set(seen).size === seen.length, "paging delivered a duplicate record_key"); + return `${seen.length} records, no duplicates`; + }); + + await check(checks, "read: an exhausted partition reports exhausted", async () => { + const everything = await connector.read({ partition_key: partitionKey, limit: 1000 }); + const past = await connector.read({ + partition_key: partitionKey, + cursor: everything.next_cursor, + limit: 1000, + }); + assert(past.records.length === 0, "reading past the end returned records"); + assert(past.exhausted, "reading past the end did not report exhausted"); + return "end of partition reported honestly"; + }); + + await check(checks, "checkpoint: reflects the session position", async () => { + const cp = await connector.checkpoint(partitionKey); + assert(cp.partition_key === partitionKey, "checkpoint returned a different partition"); + return `cursor=${cp.cursor ?? ""}`; + }); + + await check(checks, "resume: continues from a committed checkpoint without re-delivery", async () => { + const firstPage = await connector.read({ partition_key: partitionKey, limit: 2 }); + assert(firstPage.records.length > 0, "fixture partition is empty"); + const committed = { + source_id: fixture.ctx.source_id, + tenant_id: fixture.ctx.tenant_id, + partition_key: partitionKey, + cursor: firstPage.next_cursor, + sequence: firstPage.records.length, + records_committed: firstPage.records.length, + committed_at: new Date().toISOString(), + }; + const resumed = await connector.resume(committed, 1000); + const firstKeys = new Set(firstPage.records.map((r) => r.record_key)); + for (const r of resumed.records) { + assert( + !firstKeys.has(r.record_key), + `resume re-delivered already-committed record "${r.record_key}"`, + ); + } + assert( + firstPage.records.length + resumed.records.length === fixture.expectedRecordCount, + `resume covered ${firstPage.records.length + resumed.records.length} of ${fixture.expectedRecordCount} records — a gap`, + ); + return `resumed ${resumed.records.length} remaining records with no gap and no overlap`; + }); + + await check(checks, "backfill: serves a bounded range", async () => { + const res = await connector.backfill({ + partition_key: partitionKey, + from: fixture.backfill.from, + to: fixture.backfill.to, + limit: 1000, + }); + assert(res.records.length > 0, "bounded backfill range returned nothing"); + return `${res.records.length} record(s) in range`; + }); + + await check(checks, "backfill: refuses an unbounded range", async () => { + let threw = false; + try { + await connector.backfill({ + partition_key: partitionKey, + from: fixture.backfill.from, + to: "", + limit: 1000, + }); + } catch { + threw = true; + } + assert(threw, "an unbounded backfill was accepted"); + return "unbounded backfill refused"; + }); + + await check(checks, "disconnect: idempotent and safe", async () => { + await connector.disconnect(); + await connector.disconnect(); + return "disconnect() called twice without error"; + }); + + await check(checks, "operations after disconnect() fail closed", async () => { + let threw = false; + try { + await connector.read({ partition_key: partitionKey, limit: 1 }); + } catch { + threw = true; + } + assert(threw, "read() succeeded after disconnect()"); + return "read() refused after disconnect()"; + }); + + return { + connector_id: connector.connector_id, + connector_version: connector.connector_version, + passed: checks.every((c) => c.passed), + checks, + }; +} diff --git a/src/connectors/connector.ts b/src/connectors/connector.ts new file mode 100644 index 0000000..3734598 --- /dev/null +++ b/src/connectors/connector.ts @@ -0,0 +1,163 @@ +/** + * src/connectors/connector.ts + * ============================ + * P061: THE connector contract. Nine operations, one interface, every + * connector implements all nine: + * + * validate — is this config usable? Pure, no IO, no network. Fail-closed. + * connect — open whatever session the source needs. Must be idempotent. + * discover — what partitions exist, and what shape do their records have? + * health — is the source answering right now? + * read — produce records from a cursor forward, bounded by `limit`. + * checkpoint — the cursor meaning "everything before this is produced". + * resume — continue from a previously committed checkpoint. + * backfill — re-read a bounded historical range. Bounded, never "all". + * disconnect — release the session. Must be idempotent and safe after an error. + * + * WHY AN INTERFACE AND NOT A BASE CLASS: connectors are heterogeneous by + * nature (an HTTP API paginates by opaque token, a file store seeks by byte + * offset, a change feed advances an LSN). The contract fixes the SHAPE of + * the interaction and the meaning of the cursor; it does not pretend the + * mechanics are the same. `src/connectors/conformance.ts` is what proves two + * unrelated implementations really do honour the same shape. + * + * WHAT THIS REPLACES, HONESTLY: services/api/sources/connectors.py has + * per-source `async def fetch_x()` functions and nothing else — no lifecycle, + * no cursor, no resume, no health, no disconnect. Those functions remain the + * Python engine's fetch layer; they are NOT retrofitted here (that engine is + * not deployed — api.0pt.app answers 404 — and rewriting it blind would be + * the exact "build it again" failure this programme forbids). This contract + * is the seam a Python connector is wrapped into when that engine is revived. + */ +import type { IngestionCheckpoint, IngestionMode } from "../types/ingestion"; +import type { RecordSchemaFields } from "../schema/record-schema"; + +/** Config is connector-specific; the contract only requires it be serializable. */ +export type ConnectorConfig = Record; + +export interface ConnectorContext { + tenant_id: string; + project_id: string; + source_id: string; + /** Correlates every operation of one ingestion run. */ + run_id: string; +} + +/** Result of `validate` — never throws, always answers. */ +export interface ValidationResult { + valid: boolean; + /** One entry per problem. Empty iff valid. */ + errors: string[]; +} + +/** One readable partition (a table, a topic, a file, a page family). */ +export interface PartitionDescriptor { + partition_key: string; + /** Human label; may equal partition_key. */ + name: string; + /** Record shape as the connector currently sees it — fed straight into the schema registry (P063). */ + schema: RecordSchemaFields; +} + +export interface DiscoveryResult { + partitions: PartitionDescriptor[]; + /** Modes this connector can actually serve for this config. */ + modes: IngestionMode[]; +} + +export interface ConnectorHealth { + status: "up" | "degraded" | "down"; + latency_ms?: number; + /** Short operator-facing text — never a raw upstream body. */ + detail?: string; +} + +export interface ReadRequest { + partition_key: string; + /** Start AFTER this cursor. Undefined = from the beginning of the partition. */ + cursor?: string; + /** Hard upper bound on records returned. Connectors MUST honour it. */ + limit: number; +} + +/** A record as the connector sees it, before the runner wraps it in an envelope. */ +export interface ConnectorRecord { + /** Stable business key within the partition. */ + record_key: string; + /** Null only for a delete in a change feed. */ + payload: Record | null; + operation?: "upsert" | "insert" | "update" | "delete"; + occurred_at?: string; + /** Cursor value AT this record — the runner records it per envelope. */ + cursor?: string; +} + +export interface ReadResult { + records: ConnectorRecord[]; + /** + * Cursor meaning "everything up to and including `records` has been + * produced". The runner commits this only after the records are accepted. + */ + next_cursor?: string; + /** False when the connector knows more records are immediately available. */ + exhausted: boolean; +} + +/** A bounded historical range. Unbounded backfills are rejected by the contract. */ +export interface BackfillRequest { + partition_key: string; + /** Inclusive lower bound (cursor or ISO timestamp — connector-defined, documented per connector). */ + from: string; + /** Exclusive upper bound. Required: a backfill with no end is a full reload, not a backfill. */ + to: string; + limit: number; +} + +export interface Connector { + readonly connector_id: string; + readonly connector_version: string; + + /** Pure config check. MUST NOT perform IO. MUST NOT throw. */ + validate(config: ConnectorConfig): ValidationResult; + + /** Open a session. Idempotent: calling it twice is a no-op, not an error. */ + connect(config: ConnectorConfig, ctx: ConnectorContext): Promise; + + /** Requires a live session. */ + discover(): Promise; + + /** Requires a live session. Never throws for an unhealthy source — reports it. */ + health(): Promise; + + /** Requires a live session. Honours `limit` exactly. */ + read(req: ReadRequest): Promise; + + /** + * The connector's own view of where it stands for a partition. The DURABLE + * checkpoint is owned by the runner (CheckpointStore); this is the + * in-session position, used to cross-check a resume. + */ + checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }>; + + /** + * Continue from a previously committed checkpoint. Equivalent to + * `read({cursor: checkpoint.cursor})` for cursor-based connectors, but + * explicit so a connector that must re-establish server-side state (a + * replication slot, a scroll context) has a place to do it. + */ + resume(checkpoint: IngestionCheckpoint, limit: number): Promise; + + /** Bounded historical re-read. MUST reject an unbounded range. */ + backfill(req: BackfillRequest): Promise; + + /** Idempotent; safe to call without a session and after an error. */ + disconnect(): Promise; +} + +/** Thrown by connectors when an operation needs a session that was never opened. */ +export class ConnectorNotConnectedError extends Error { + constructor(connectorId: string, operation: string) { + super(`connector ${connectorId}: ${operation} requires connect() first`); + this.name = "ConnectorNotConnectedError"; + } +} diff --git a/src/connectors/http-json-connector.ts b/src/connectors/http-json-connector.ts new file mode 100644 index 0000000..cb11627 --- /dev/null +++ b/src/connectors/http-json-connector.ts @@ -0,0 +1,274 @@ +/** + * src/connectors/http-json-connector.ts + * ====================================== + * P061 conformance connector #1: a real HTTP JSON API source. + * + * Heterogeneity (the point of having two): this connector talks to a REMOTE + * service over the network, pages by an OPAQUE token the server hands back, + * cannot know how many records exist until the server says it is done, and + * has a genuine health check (an HTTP round trip). Its sibling + * (jsonl-file-connector.ts) does none of those things. + * + * COMPLIANCE (CLAUDE.md §1): this connector is a plain, self-identifying + * JSON client. It sends a fixed User-Agent, honours the caller's timeout, and + * treats 401/403/429/451 as a STOP signal for that source — the same posture + * services/api/sources/connectors.py's BLOCKED_STATUSES already takes. There + * is no retry-harder path, no proxy rotation, no header spoofing, and adding + * any would be a compliance violation, not a feature. + * + * `fetchImpl` is injectable so tests drive a real request/response cycle + * without a network. That is a seam for TESTS, not a mock fallback: with no + * override it uses global fetch and talks to the configured URL for real. + */ +import { + ConnectorNotConnectedError, + type BackfillRequest, + type Connector, + type ConnectorConfig, + type ConnectorContext, + type ConnectorHealth, + type ConnectorRecord, + type DiscoveryResult, + type ReadRequest, + type ReadResult, + type ValidationResult, +} from "./connector"; +import type { IngestionCheckpoint } from "../types/ingestion"; +import type { RecordSchemaFields } from "../schema/record-schema"; + +export const HTTP_JSON_CONNECTOR_VERSION = "http-json-connector@1"; + +/** Statuses that mean "stop asking this way" — never "retry harder". */ +const BLOCKED_STATUSES = new Set([401, 403, 429, 451]); + +export interface HttpJsonConnectorConfig extends ConnectorConfig { + /** Absolute https:// (or http:// for a local fixture) base URL. */ + base_url: string; + /** Path serving records, e.g. "/v1/companies". */ + records_path: string; + /** Partition key this endpoint serves. */ + partition_key: string; + /** Declared record shape for the partition — the connector does not infer schemas from data. */ + schema: RecordSchemaFields; + /** JSON field holding the business key of each record. */ + record_key_field: string; + /** + * JSON field holding each record's own cursor value (an id, a sequence, an + * `updated_at`). Optional, but REQUIRED IN PRACTICE for any API that stops + * returning `next_cursor` once a page is exhausted: without it the + * connector has nothing to resume from after reading to the end, and + * resume() would silently restart at the beginning. The conformance suite + * catches exactly that ("read: an exhausted partition reports exhausted"), + * which is how this field came to exist. + */ + cursor_field?: string; + /** Request timeout (ms). Default 15000. */ + timeout_ms?: number; +} + +type FetchImpl = (url: string, init: RequestInit) => Promise; + +/** Wire shape this connector expects from the upstream endpoint. */ +interface UpstreamPage { + records?: Array>; + next_cursor?: string | null; +} + +function isConfig(config: ConnectorConfig): config is HttpJsonConnectorConfig { + return typeof config.base_url === "string" && typeof config.records_path === "string"; +} + +export class HttpJsonConnector implements Connector { + readonly connector_id = "http_json"; + readonly connector_version = HTTP_JSON_CONNECTOR_VERSION; + + private config?: HttpJsonConnectorConfig; + private ctx?: ConnectorContext; + private lastCursor = new Map(); + + constructor(private readonly fetchImpl: FetchImpl = (url, init) => fetch(url, init)) {} + + validate(config: ConnectorConfig): ValidationResult { + const errors: string[] = []; + if (typeof config.base_url !== "string" || config.base_url.length === 0) { + errors.push("base_url is required"); + } else if (!/^https?:\/\//.test(config.base_url)) { + errors.push("base_url must be an absolute http(s) URL"); + } + if (typeof config.records_path !== "string" || !config.records_path.startsWith("/")) { + errors.push("records_path is required and must start with /"); + } + if (typeof config.partition_key !== "string" || config.partition_key.length === 0) { + errors.push("partition_key is required"); + } + if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) { + errors.push("record_key_field is required"); + } + const schema = config.schema; + if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) { + errors.push("schema is required and must declare at least one field"); + } + return { valid: errors.length === 0, errors }; + } + + async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise { + const verdict = this.validate(config); + if (!verdict.valid) { + throw new Error(`http_json connector config invalid: ${verdict.errors.join("; ")}`); + } + if (!isConfig(config)) throw new Error("http_json connector config invalid"); + this.config = config; + this.ctx = ctx; + } + + async discover(): Promise { + const config = this.requireSession("discover"); + return { + partitions: [ + { + partition_key: config.partition_key, + name: `${config.base_url}${config.records_path}`, + schema: config.schema, + }, + ], + // A polled JSON endpoint serves batch and (by repeated polling) stream. + // It is NOT a change feed: it carries no delete events, so "cdc" is not + // declared. Declaring it would be a lie the runner would act on. + modes: ["batch", "stream"], + }; + } + + async health(): Promise { + const config = this.requireSession("health"); + const started = Date.now(); + try { + const res = await this.request(this.url(config.records_path, { limit: "1" }), config); + const latency = Date.now() - started; + if (BLOCKED_STATUSES.has(res.status)) { + return { status: "down", latency_ms: latency, detail: `source signalled a block (HTTP ${res.status})` }; + } + if (!res.ok) return { status: "degraded", latency_ms: latency, detail: `HTTP ${res.status}` }; + return { status: "up", latency_ms: latency }; + } catch (err) { + return { + status: "down", + latency_ms: Date.now() - started, + detail: err instanceof Error ? err.message : "unreachable", + }; + } + } + + async read(req: ReadRequest): Promise { + const config = this.requireSession("read"); + const query: Record = { limit: String(req.limit) }; + if (req.cursor !== undefined) query.cursor = req.cursor; + const res = await this.request(this.url(config.records_path, query), config); + if (BLOCKED_STATUSES.has(res.status)) { + throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`); + } + if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`); + const page = (await res.json()) as UpstreamPage; + const rawRecords = page.records ?? []; + // The contract says limit is a HARD bound; a server that over-delivers is + // truncated here rather than silently widening the caller's page. + const records = rawRecords.slice(0, req.limit).map((row) => this.toRecord(row, config)); + const serverCursor = page.next_cursor ?? undefined; + // When the server stops issuing a cursor at the end of a page, fall back + // to the last record's own cursor value so a later resume() starts AFTER + // the data already produced instead of at the beginning. + const trailing = records.length > 0 ? records[records.length - 1].cursor : undefined; + const nextCursor = serverCursor ?? trailing ?? req.cursor; + if (records.length > 0) this.lastCursor.set(req.partition_key, nextCursor); + return { + records, + next_cursor: nextCursor, + // The SERVER's cursor is what says "there is more"; a fallback cursor + // only preserves position, it never claims more data exists. + exhausted: serverCursor === undefined || records.length === 0, + }; + } + + async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> { + this.requireSession("checkpoint"); + return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) }; + } + + async resume(checkpoint: IngestionCheckpoint, limit: number): Promise { + this.requireSession("resume"); + return this.read({ + partition_key: checkpoint.partition_key, + cursor: checkpoint.cursor, + limit, + }); + } + + async backfill(req: BackfillRequest): Promise { + const config = this.requireSession("backfill"); + if (!req.from || !req.to) { + throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill"); + } + const res = await this.request( + this.url(config.records_path, { from: req.from, to: req.to, limit: String(req.limit) }), + config, + ); + if (BLOCKED_STATUSES.has(res.status)) { + throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`); + } + if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`); + const page = (await res.json()) as UpstreamPage; + const records = (page.records ?? []).slice(0, req.limit).map((row) => this.toRecord(row, config)); + return { records, next_cursor: page.next_cursor ?? undefined, exhausted: true }; + } + + async disconnect(): Promise { + this.config = undefined; + this.ctx = undefined; + this.lastCursor.clear(); + } + + // --- internals ----------------------------------------------------------- + + private toRecord(row: Record, config: HttpJsonConnectorConfig): ConnectorRecord { + const key = row[config.record_key_field]; + if (typeof key !== "string" && typeof key !== "number") { + throw new Error(`record is missing string/number key field "${config.record_key_field}"`); + } + let cursor: string | undefined; + if (config.cursor_field) { + const raw = row[config.cursor_field]; + if (typeof raw === "string" || typeof raw === "number") cursor = String(raw); + } + return { record_key: String(key), payload: row, operation: "upsert", cursor }; + } + + private requireSession(op: string): HttpJsonConnectorConfig { + if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op); + return this.config; + } + + private url(path: string, query: Record): string { + const config = this.config; + if (!config) throw new ConnectorNotConnectedError(this.connector_id, "url"); + const url = new URL(path, config.base_url); + for (const [k, v] of Object.entries(query)) url.searchParams.set(k, v); + return url.toString(); + } + + private async request(url: string, config: HttpJsonConnectorConfig): Promise { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), config.timeout_ms ?? 15_000); + try { + return await this.fetchImpl(url, { + method: "GET", + signal: controller.signal, + headers: { + Accept: "application/json", + "User-Agent": "0pt-connector/1 (+https://0pt.app; SIAX Technology AB)", + ...(this.ctx ? { "X-Correlation-Id": this.ctx.run_id } : {}), + }, + }); + } finally { + clearTimeout(timer); + } + } +} diff --git a/src/connectors/jsonl-file-connector.ts b/src/connectors/jsonl-file-connector.ts new file mode 100644 index 0000000..4386a3a --- /dev/null +++ b/src/connectors/jsonl-file-connector.ts @@ -0,0 +1,224 @@ +/** + * src/connectors/jsonl-file-connector.ts + * ======================================== + * P061 conformance connector #2: a JSONL file drop on local disk. + * + * Heterogeneity vs http-json-connector.ts — these two share nothing but the + * contract: + * - no network, no server, no opaque token; the cursor is a LINE OFFSET the + * connector computes itself, + * - it knows the full record count up front (the file is finite and local), + * so `exhausted` is exact rather than server-reported, + * - health is a filesystem stat, not an HTTP round trip, + * - it serves `cdc`-shaped records: a JSONL line may carry an `_op` field + * ("insert"/"update"/"delete"), so this connector emits real delete + * envelopes with a null payload. The HTTP connector cannot, + * - backfill bounds are line numbers, not timestamps. + * + * That last point is why the conformance suite takes `from`/`to` from a + * fixture: the contract fixes that a backfill is BOUNDED, not what the + * bounds mean. Each connector documents its own bound vocabulary. + */ +import { promises as fs } from "fs"; +import path from "path"; +import { + ConnectorNotConnectedError, + type BackfillRequest, + type Connector, + type ConnectorConfig, + type ConnectorContext, + type ConnectorHealth, + type ConnectorRecord, + type DiscoveryResult, + type ReadRequest, + type ReadResult, + type ValidationResult, +} from "./connector"; +import type { IngestionCheckpoint } from "../types/ingestion"; +import type { RecordSchemaFields } from "../schema/record-schema"; + +export const JSONL_FILE_CONNECTOR_VERSION = "jsonl-file-connector@1"; + +export interface JsonlFileConnectorConfig extends ConnectorConfig { + /** Absolute path to the .jsonl file. */ + file_path: string; + partition_key: string; + schema: RecordSchemaFields; + record_key_field: string; +} + +function isConfig(config: ConnectorConfig): config is JsonlFileConnectorConfig { + return typeof config.file_path === "string" && typeof config.partition_key === "string"; +} + +/** Cursor vocabulary: the decimal count of lines already produced. */ +function parseCursor(cursor: string | undefined): number { + if (cursor === undefined) return 0; + const n = Number.parseInt(cursor, 10); + if (!Number.isInteger(n) || n < 0) throw new Error(`invalid cursor "${cursor}" — expected a line offset`); + return n; +} + +export class JsonlFileConnector implements Connector { + readonly connector_id = "jsonl_file"; + readonly connector_version = JSONL_FILE_CONNECTOR_VERSION; + + private config?: JsonlFileConnectorConfig; + private lastCursor = new Map(); + + validate(config: ConnectorConfig): ValidationResult { + const errors: string[] = []; + if (typeof config.file_path !== "string" || config.file_path.length === 0) { + errors.push("file_path is required"); + } else if (!path.isAbsolute(config.file_path)) { + errors.push("file_path must be absolute"); + } else if (!config.file_path.endsWith(".jsonl")) { + errors.push("file_path must name a .jsonl file"); + } + if (typeof config.partition_key !== "string" || config.partition_key.length === 0) { + errors.push("partition_key is required"); + } + if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) { + errors.push("record_key_field is required"); + } + const schema = config.schema; + if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) { + errors.push("schema is required and must declare at least one field"); + } + return { valid: errors.length === 0, errors }; + } + + async connect(config: ConnectorConfig, _ctx: ConnectorContext): Promise { + const verdict = this.validate(config); + if (!verdict.valid) { + throw new Error(`jsonl_file connector config invalid: ${verdict.errors.join("; ")}`); + } + if (!isConfig(config)) throw new Error("jsonl_file connector config invalid"); + // Fail at connect time, not at first read: a file drop that is not there + // is a connection failure, and the runner must see it as one. + await fs.access(config.file_path); + this.config = config; + } + + async discover(): Promise { + const config = this.requireSession("discover"); + return { + partitions: [ + { partition_key: config.partition_key, name: path.basename(config.file_path), schema: config.schema }, + ], + // A file drop is a batch source; because a line may carry `_op`, it can + // also carry a change log, so cdc is honestly available here. + modes: ["batch", "cdc"], + }; + } + + async health(): Promise { + const config = this.requireSession("health"); + const started = Date.now(); + try { + const stat = await fs.stat(config.file_path); + if (stat.size === 0) { + return { status: "degraded", latency_ms: Date.now() - started, detail: "file is empty" }; + } + return { status: "up", latency_ms: Date.now() - started }; + } catch (err) { + return { + status: "down", + latency_ms: Date.now() - started, + detail: err instanceof Error ? err.message : "stat failed", + }; + } + } + + async read(req: ReadRequest): Promise { + const lines = await this.lines(); + const start = parseCursor(req.cursor); + const slice = lines.slice(start, start + req.limit); + const records = slice.map((line, i) => this.toRecord(line, start + i)); + const nextOffset = start + slice.length; + const cursor = String(nextOffset); + this.lastCursor.set(req.partition_key, cursor); + return { + records, + next_cursor: cursor, + exhausted: nextOffset >= lines.length, + }; + } + + async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> { + this.requireSession("checkpoint"); + return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) }; + } + + async resume(checkpoint: IngestionCheckpoint, limit: number): Promise { + this.requireSession("resume"); + return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit }); + } + + /** Bounds are LINE NUMBERS: `from` inclusive, `to` exclusive. */ + async backfill(req: BackfillRequest): Promise { + this.requireSession("backfill"); + if (!req.from || !req.to) { + throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill"); + } + const from = parseCursor(req.from); + const to = parseCursor(req.to); + if (to <= from) throw new Error(`backfill range is empty or inverted (from=${from}, to=${to})`); + const lines = await this.lines(); + const slice = lines.slice(from, Math.min(to, from + req.limit)); + return { + records: slice.map((line, i) => this.toRecord(line, from + i)), + next_cursor: String(from + slice.length), + exhausted: true, + }; + } + + async disconnect(): Promise { + this.config = undefined; + this.lastCursor.clear(); + } + + // --- internals ----------------------------------------------------------- + + private requireSession(op: string): JsonlFileConnectorConfig { + if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op); + return this.config; + } + + private async lines(): Promise { + const config = this.requireSession("read"); + const raw = await fs.readFile(config.file_path, "utf8"); + return raw.split("\n").filter((l) => l.trim().length > 0); + } + + private toRecord(line: string, lineNumber: number): ConnectorRecord { + const config = this.requireSession("read"); + let parsed: Record; + try { + parsed = JSON.parse(line) as Record; + } catch { + throw new Error(`line ${lineNumber + 1} is not valid JSON`); + } + const key = parsed[config.record_key_field]; + if (typeof key !== "string" && typeof key !== "number") { + throw new Error(`line ${lineNumber + 1} is missing key field "${config.record_key_field}"`); + } + const op = parsed._op; + const operation = + op === "insert" || op === "update" || op === "delete" ? op : ("upsert" as const); + const occurredAt = typeof parsed._occurred_at === "string" ? parsed._occurred_at : undefined; + // `_op`/`_occurred_at` are envelope metadata, not payload fields — leaving + // them in the payload would make every record fail schema conformance for + // an "unregistered field". + const payload: Record = { ...parsed }; + delete payload._op; + delete payload._occurred_at; + return { + record_key: String(key), + payload: operation === "delete" ? null : payload, + operation, + occurred_at: occurredAt, + cursor: String(lineNumber + 1), + }; + } +} diff --git a/src/er/entity-resolution-service.ts b/src/er/entity-resolution-service.ts index 2992a10..913d81f 100644 --- a/src/er/entity-resolution-service.ts +++ b/src/er/entity-resolution-service.ts @@ -23,6 +23,16 @@ * GENERIC means GENERIC: no person-matching logic, no Ratsit/PII pipelines — * records are opaque field maps supplied by the calling product under its * own legal basis; this module implements comparison mechanics only. + * + * P065 STATUS — THIS IS THE CANONICAL ENGINE, AND IT IS NOT YET THE ONLY ONE. + * `services/api/workers/resolve.py` is a second, simplified ER engine (fuzzy + * threshold, auto-merge, no hard rules, no review band) that + * `services/api/workers/pipeline.py` still calls. P065's DoD ("no second + * simplified duplicated production engine remains") is therefore NOT met. + * See that file's header for why it was not rewritten blind in the + * P060-P068 pass and STATUS.md for the removal plan. The ingestion plane + * added in that pass (src/ingestion/*) deliberately does no ER of its own — + * it stops at the sink, so it adds no THIRD engine. */ import crypto from "crypto"; import { SiaxError } from "../platform/errors"; diff --git a/src/ingestion/envelope.ts b/src/ingestion/envelope.ts new file mode 100644 index 0000000..f7038ee --- /dev/null +++ b/src/ingestion/envelope.ts @@ -0,0 +1,87 @@ +/** + * src/ingestion/envelope.ts + * ========================== + * P062: the one place a ConnectorRecord becomes an IngestionEnvelope. Every + * connector, every mode, one function — that is what "normalize all ingestion + * to a common envelope" means operationally. + * + * `hashPayload` is the stable content hash the idempotency ledger (P068) and + * the deterministic `envelope_id` are both built on. Key order must not + * change the hash — two connectors reading the same record can serialize the + * same object with different key order — so keys are sorted recursively. + * Undefined values are dropped (JSON has no undefined); null is kept, because + * "field explicitly null" is different data from "field absent". + */ +import crypto from "crypto"; +import type { ConnectorRecord } from "../connectors/connector"; +import type { EnvelopeOperation, IngestionEnvelope, IngestionMode } from "../types/ingestion"; + +/** Recursively key-sorted JSON — stable across key insertion order. */ +export function stableStringify(value: unknown): string { + if (value === null) return "null"; + if (Array.isArray(value)) return `[${value.map(stableStringify).join(",")}]`; + if (typeof value === "object") { + const entries = Object.entries(value as Record) + .filter(([, v]) => v !== undefined) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); + return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(",")}}`; + } + return JSON.stringify(value) ?? "null"; +} + +/** Content hash of an envelope payload. A delete (null payload) hashes as "deleted". */ +export function hashPayload(payload: Record | null): string { + if (payload === null) return "sha256:deleted"; + return `sha256:${crypto.createHash("sha256").update(stableStringify(payload)).digest("hex")}`; +} + +export interface BuildEnvelopeInput { + record: ConnectorRecord; + source_id: string; + tenant_id: string; + project_id: string; + mode: IngestionMode; + partition_key: string; + schema_version: number; + sequence: number; + connector_id: string; + connector_version: string; + run_id: string; + /** Cursor covering this record, when the connector reports one. */ + cursor?: string; + now?: string; +} + +export function buildEnvelope(input: BuildEnvelopeInput): IngestionEnvelope { + const operation: EnvelopeOperation = input.record.operation ?? "upsert"; + const payload = operation === "delete" ? null : input.record.payload; + const contentHash = hashPayload(payload); + return { + // Deterministic: the same record with the same content re-read after a + // crash produces the same envelope_id, so a downstream sink that dedups + // by id sees a retry, not a new record. + envelope_id: `env_${crypto + .createHash("sha256") + .update(`${input.source_id}|${input.partition_key}|${input.record.record_key}|${contentHash}`) + .digest("hex") + .slice(0, 32)}`, + source_id: input.source_id, + tenant_id: input.tenant_id, + project_id: input.project_id, + mode: input.mode, + operation, + partition_key: input.partition_key, + record_key: input.record.record_key, + schema_version: input.schema_version, + sequence: input.sequence, + occurred_at: input.record.occurred_at, + ingested_at: input.now ?? new Date().toISOString(), + payload, + provenance: { + connector_id: input.connector_id, + connector_version: input.connector_version, + run_id: input.run_id, + cursor: input.cursor ?? input.record.cursor, + }, + }; +} diff --git a/src/ingestion/ingestion-runner.ts b/src/ingestion/ingestion-runner.ts new file mode 100644 index 0000000..73b19a9 --- /dev/null +++ b/src/ingestion/ingestion-runner.ts @@ -0,0 +1,557 @@ +/** + * src/ingestion/ingestion-runner.ts + * ================================== + * The single ingestion highway (P062, P063 gate, P064 gate, P067 emission, + * P068 ledger). One loop, one envelope shape, one durable checkpoint. + * + * ORDERING IS THE WHOLE CONTRACT. Per page: + * + * read/resume -> normalize to envelopes -> schema conformance -> quality gate + * -> ledger PEEK (filter replays) -> SINK -> ledger COMMIT -> checkpoint COMMIT + * + * Both commits happen AFTER the sink resolves. That is what makes "a restart + * resumes without silent gaps" true: a process that dies anywhere before the + * commits leaves the page unconsumed, the next run re-reads it, and nothing is + * lost. The two tempting shortcuts are both data-loss bugs, and one of them + * was actually written here first: + * - committing the CHECKPOINT before the sink skips the page entirely; + * - committing the LEDGER before the sink marks records delivered that the + * sink never received, so the resumed run drops them as duplicates. This + * one shipped in the first draft of this file and was caught by + * tests/p0/ingestion-runner.test.ts's crash/resume case (7 of 10 records + * delivered). The peek/commit split in IngestionRepository exists because + * of it. + * + * The resulting delivery guarantee is AT-LEAST-ONCE, not exactly-once: two + * concurrent runs on the same partition can both peek "new". `envelope_id` is + * deterministic so a dedup-capable sink survives that. A per-(source, + * partition) run lock does not exist yet — see STATUS.md. + * + * A BLOCKED quality page also stops WITHOUT committing: a blocked page must + * stay unconsumed, not be skipped over. + * + * WHAT THIS RUNNER DOES NOT DO, stated plainly: + * - it does not schedule itself (ACT0 owns scheduling; the source's + * `schedule` row is a declaration, not a cron daemon in this repo), + * - it does not write to a lakehouse — the `sink` is supplied by the + * caller, because the Iceberg write boundary is ST0RE's contract + * (src/services/iceberg-write-service.ts) and is owned by another repo + * in this programme (P066), + * - it does not do entity resolution. The canonical ER engine is + * src/er/entity-resolution-service.ts and runs downstream of the sink. + */ +import crypto from "crypto"; +import { SiaxError } from "../platform/errors"; +import { conformanceErrors } from "../schema/compatibility"; +import type { SchemaRegistry } from "../schema/schema-registry"; +import { QualityGate, type QualityEnforcementPolicy } from "../quality/quality-policy"; +import type { IngestionRepository } from "../repositories/ingestion-repository"; +import type { LineageStore } from "../lineage/lineage-store"; +import type { + BackfillRequest, + Connector, + ConnectorConfig, + ReadResult, +} from "../connectors/connector"; +import type { + IngestionCheckpoint, + IngestionEnvelope, + IngestionMode, + QuarantineRecord, +} from "../types/ingestion"; +import { buildEnvelope, hashPayload } from "./envelope"; + +/** Where accepted envelopes go. Supplied by the caller — see the module note. */ +export type IngestionSink = (envelopes: IngestionEnvelope[]) => Promise; + +export interface IngestionRunInput { + source_id: string; + tenant_id: string; + project_id: string; + mode: IngestionMode; + connector: Connector; + config: ConnectorConfig; + sink: IngestionSink; + /** Deterministic run id. Defaults to a random one. */ + run_id?: string; + /** Records per connector read. Default 100. */ + page_size?: number; + /** Safety stop. Default 1000 pages. */ + max_pages?: number; + /** Only this partition; default is every partition discover() reports. */ + partition_key?: string; + /** P064. Omitted = no gate runs, and the report says so explicitly. */ + quality_policy?: QualityEnforcementPolicy; + /** Canonical ref of what the sink produces, for lineage (e.g. "0pt:dataset:ds_x"). */ + output_ref?: string; + correlation_id?: string; +} + +export interface PartitionRunReport { + partition_key: string; + schema_version: number; + pages_read: number; + records_read: number; + envelopes_emitted: number; + duplicates_skipped: number; + changed: number; + quarantined: number; + quality_violations: number; + checkpoint: IngestionCheckpoint; + /** Set when the partition stopped early. */ + stopped_because?: "blocked_by_quality" | "max_pages" | "sink_failed"; +} + +export interface IngestionRunReport { + run_id: string; + source_id: string; + tenant_id: string; + mode: IngestionMode; + status: "succeeded" | "failed"; + /** Honest marker: "none_declared" when the caller supplied no policy at all. */ + quality_policy: "block" | "quarantine" | "warn" | "none_declared"; + partitions: PartitionRunReport[]; + lineage_edges: number; + /** Present when status is "failed". */ + failure?: { code: string; detail: string }; +} + +export class IngestionRunner { + private readonly gate: QualityGate; + + constructor( + private readonly repo: IngestionRepository, + private readonly schemas: SchemaRegistry, + private readonly lineage?: LineageStore, + gate?: QualityGate, + ) { + this.gate = gate ?? new QualityGate(); + } + + async run(input: IngestionRunInput): Promise { + const runId = input.run_id ?? `run_${crypto.randomUUID()}`; + const pageSize = input.page_size ?? 100; + const maxPages = input.max_pages ?? 1000; + const report: IngestionRunReport = { + run_id: runId, + source_id: input.source_id, + tenant_id: input.tenant_id, + mode: input.mode, + status: "succeeded", + quality_policy: input.quality_policy?.on_violation ?? "none_declared", + partitions: [], + lineage_edges: 0, + }; + + const verdict = input.connector.validate(input.config); + if (!verdict.valid) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.ingestion", + `connector config rejected: ${verdict.errors.join("; ")}`, + ); + } + + await input.connector.connect(input.config, { + tenant_id: input.tenant_id, + project_id: input.project_id, + source_id: input.source_id, + run_id: runId, + }); + + try { + const discovery = await input.connector.discover(); + if (!discovery.modes.includes(input.mode)) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.ingestion", + `connector ${input.connector.connector_id} does not serve mode "${input.mode}" for this config (serves: ${discovery.modes.join(", ")})`, + ); + } + const partitions = input.partition_key + ? discovery.partitions.filter((p) => p.partition_key === input.partition_key) + : discovery.partitions; + if (partitions.length === 0) { + throw new SiaxError( + "NOT_FOUND", + "0pt.ingestion", + `partition "${input.partition_key}" was not discovered on source ${input.source_id}`, + ); + } + + // ---- P063: schema gate, BEFORE any record is read ------------------ + // Every partition's shape is checked and registered first. A break + // fails the whole run here, so no envelope — and therefore no consumer + // write — can happen under a broken schema. + const schemaVersions = new Map(); + for (const partition of partitions) { + const evaluated = await this.schemas.evaluate(input.source_id, partition.schema); + if (evaluated.verdict && !evaluated.verdict.compatible) { + throw new SiaxError( + "CONFLICT", + "0pt.ingestion", + `source ${input.source_id} partition "${partition.partition_key}" changed shape in a way that breaks its declared ` + + `"${evaluated.verdict.policy}" compatibility policy — refusing to ingest: ` + + evaluated.verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "), + ); + } + const registered = await this.schemas.register({ + source_id: input.source_id, + tenant_id: input.tenant_id, + fields: partition.schema, + compatibility: evaluated.previous?.compatibility ?? "backward", + registered_by: `run:${runId}`, + }); + schemaVersions.set(partition.partition_key, registered.schema.schema_version); + } + + for (const partition of partitions) { + report.partitions.push( + await this.runPartition(input, { + runId, + pageSize, + maxPages, + partitionKey: partition.partition_key, + schemaVersion: schemaVersions.get(partition.partition_key) ?? 1, + }), + ); + } + + // ---- P067: one lineage event per run, source -> output ------------- + if (this.lineage) { + report.lineage_edges = await this.lineage.record( + { + eventType: "COMPLETE", + eventTime: new Date().toISOString(), + producer: "https://git.siax.io/sax3l/0pt", + run: { runId }, + job: { namespace: "0pt", name: "0pt.ingestion.run.v1" }, + inputs: [ + { + namespace: "0pt", + name: `source/${input.source_id}`, + facets: { siaxResourceRef: { ref: `0pt:source:${input.source_id}` } }, + }, + ], + outputs: [ + { + namespace: "0pt", + name: input.output_ref ?? `ingestion/${input.source_id}`, + ...(input.output_ref ? { facets: { siaxResourceRef: { ref: input.output_ref } } } : {}), + }, + ], + }, + input.tenant_id, + input.correlation_id, + ); + } + + return report; + } finally { + await input.connector.disconnect(); + } + } + + /** + * P068: a bounded historical re-read. Runs through the SAME envelope, + * quality and ledger path as `run`, and deliberately does NOT touch the + * streaming checkpoint — a backfill must not rewind or advance live + * ingestion. Repeating an identical backfill emits nothing: every record + * comes back "duplicate" from the ledger. + */ + async backfill( + input: IngestionRunInput & { range: Omit }, + ): Promise { + const runId = input.run_id ?? `bkf_${crypto.randomUUID()}`; + const pageSize = input.page_size ?? 100; + const verdict = input.connector.validate(input.config); + if (!verdict.valid) { + throw new SiaxError("INVALID_REQUEST", "0pt.ingestion", `connector config rejected: ${verdict.errors.join("; ")}`); + } + await input.connector.connect(input.config, { + tenant_id: input.tenant_id, + project_id: input.project_id, + source_id: input.source_id, + run_id: runId, + }); + try { + const latest = await this.schemas.getLatest(input.source_id); + const schemaVersion = latest?.schema_version ?? 1; + const result = await input.connector.backfill({ ...input.range, limit: pageSize }); + const existing = + (await this.repo.getCheckpoint(input.source_id, input.range.partition_key)) ?? + this.emptyCheckpoint(input, input.range.partition_key); + const outcome = await this.consumePage(input, { + runId, + result, + partitionKey: input.range.partition_key, + schemaVersion, + startSequence: existing.sequence, + schemaFields: latest?.fields, + }); + return { + partition_key: input.range.partition_key, + schema_version: schemaVersion, + pages_read: 1, + records_read: result.records.length, + envelopes_emitted: outcome.emitted, + duplicates_skipped: outcome.duplicates, + changed: outcome.changed, + quarantined: outcome.quarantined, + quality_violations: outcome.violations, + // Unchanged on purpose — a backfill leaves the live checkpoint alone. + checkpoint: existing, + stopped_because: outcome.blocked ? "blocked_by_quality" : undefined, + }; + } finally { + await input.connector.disconnect(); + } + } + + // ------------------------------------------------------------------------- + + private emptyCheckpoint( + input: { source_id: string; tenant_id: string }, + partitionKey: string, + ): IngestionCheckpoint { + return { + source_id: input.source_id, + tenant_id: input.tenant_id, + partition_key: partitionKey, + cursor: undefined, + sequence: 0, + records_committed: 0, + committed_at: new Date(0).toISOString(), + }; + } + + private async runPartition( + input: IngestionRunInput, + opts: { + runId: string; + pageSize: number; + maxPages: number; + partitionKey: string; + schemaVersion: number; + }, + ): Promise { + const schema = await this.schemas.getLatest(input.source_id); + let checkpoint = + (await this.repo.getCheckpoint(input.source_id, opts.partitionKey)) ?? + this.emptyCheckpoint(input, opts.partitionKey); + + const stats = { + pages: 0, + records: 0, + emitted: 0, + duplicates: 0, + changed: 0, + quarantined: 0, + violations: 0, + }; + let stoppedBecause: PartitionRunReport["stopped_because"]; + + for (let page = 0; page < opts.maxPages; page++) { + // First page of a partition that has a durable checkpoint goes through + // resume(), so a connector that must re-establish server-side state has + // its chance; later pages are ordinary reads from the running cursor. + const result: ReadResult = + page === 0 && checkpoint.cursor !== undefined + ? await input.connector.resume(checkpoint, opts.pageSize) + : await input.connector.read({ + partition_key: opts.partitionKey, + cursor: checkpoint.cursor, + limit: opts.pageSize, + }); + + stats.pages++; + stats.records += result.records.length; + if (result.records.length === 0) break; + + const outcome = await this.consumePage(input, { + runId: opts.runId, + result, + partitionKey: opts.partitionKey, + schemaVersion: opts.schemaVersion, + startSequence: checkpoint.sequence, + schemaFields: schema?.fields, + }); + stats.emitted += outcome.emitted; + stats.duplicates += outcome.duplicates; + stats.changed += outcome.changed; + stats.quarantined += outcome.quarantined; + stats.violations += outcome.violations; + + if (outcome.blocked) { + // The checkpoint is NOT advanced: the offending page stays unconsumed + // so the next run sees it again rather than stepping over it. + stoppedBecause = "blocked_by_quality"; + break; + } + + checkpoint = { + source_id: input.source_id, + tenant_id: input.tenant_id, + partition_key: opts.partitionKey, + cursor: result.next_cursor, + sequence: checkpoint.sequence + result.records.length, + records_committed: checkpoint.records_committed + outcome.emitted, + committed_at: new Date().toISOString(), + }; + await this.repo.commitCheckpoint(checkpoint); + + if (result.exhausted) break; + if (page === opts.maxPages - 1) stoppedBecause = "max_pages"; + } + + return { + partition_key: opts.partitionKey, + schema_version: opts.schemaVersion, + pages_read: stats.pages, + records_read: stats.records, + envelopes_emitted: stats.emitted, + duplicates_skipped: stats.duplicates, + changed: stats.changed, + quarantined: stats.quarantined, + quality_violations: stats.violations, + checkpoint, + stopped_because: stoppedBecause, + }; + } + + /** + * Normalize -> schema conformance -> quality gate -> ledger -> sink. + * Throws if the sink throws; the caller must NOT have committed a + * checkpoint by then. + */ + private async consumePage( + input: IngestionRunInput, + opts: { + runId: string; + result: ReadResult; + partitionKey: string; + schemaVersion: number; + startSequence: number; + schemaFields?: Record; + }, + ): Promise<{ + emitted: number; + duplicates: number; + changed: number; + quarantined: number; + violations: number; + blocked: boolean; + }> { + const envelopes: IngestionEnvelope[] = opts.result.records.map((record, i) => + buildEnvelope({ + record, + source_id: input.source_id, + tenant_id: input.tenant_id, + project_id: input.project_id, + mode: input.mode, + partition_key: opts.partitionKey, + schema_version: opts.schemaVersion, + sequence: opts.startSequence + i + 1, + connector_id: input.connector.connector_id, + connector_version: input.connector.connector_version, + run_id: opts.runId, + cursor: opts.result.next_cursor, + }), + ); + + // ---- P063 per-record conformance ------------------------------------ + // A source that silently changed shape without a new discovery is caught + // here, on the record, before the sink sees anything from this page. + if (opts.schemaFields) { + for (const envelope of envelopes) { + if (envelope.payload === null) continue; + const errors = conformanceErrors( + opts.schemaFields as never, + envelope.payload, + ); + if (errors.length > 0) { + throw new SiaxError( + "CONFLICT", + "0pt.ingestion", + `record "${envelope.record_key}" does not conform to registered schema v${opts.schemaVersion} ` + + `for source ${input.source_id}: ${errors.join("; ")}`, + ); + } + } + } + + // ---- P064 quality gate ---------------------------------------------- + let accepted = envelopes; + let quarantinedCount = 0; + let violationCount = 0; + let blocked = false; + if (input.quality_policy) { + const gated = this.gate.evaluate(envelopes, input.quality_policy); + accepted = gated.accepted; + blocked = gated.blocked; + violationCount = gated.violations.length; + for (const item of gated.quarantined) { + const record: QuarantineRecord = { + quarantine_id: `qtn_${crypto + .createHash("sha256") + .update(`${opts.runId}|${item.envelope.envelope_id}`) + .digest("hex") + .slice(0, 32)}`, + source_id: input.source_id, + tenant_id: input.tenant_id, + run_id: opts.runId, + record_key: item.envelope.record_key, + violations: item.violations, + envelope: item.envelope, + quarantined_at: new Date().toISOString(), + }; + await this.repo.quarantine(record); + quarantinedCount++; + } + if (blocked) { + return { + emitted: 0, + duplicates: 0, + changed: 0, + quarantined: quarantinedCount, + violations: violationCount, + blocked: true, + }; + } + } + + // ---- P068 idempotency ledger, PEEK ONLY ------------------------------ + // Read the verdicts, filter replays, and commit NOTHING yet: a record is + // only "delivered" once the sink has actually taken it. Committing here + // would mark records delivered that a crashing sink never received, and + // the resumed run would then skip them as duplicates — silent data loss. + const entries = accepted.map((e) => ({ + record_key: e.record_key, + payload_hash: hashPayload(e.payload), + })); + const decisions = await this.repo.peekDeliveries(input.source_id, entries); + const verdictByKey = new Map(decisions.map((d) => [d.record_key, d.verdict])); + const toEmit = accepted.filter((e) => verdictByKey.get(e.record_key) !== "duplicate"); + const duplicates = accepted.length - toEmit.length; + const changed = decisions.filter((d) => d.verdict === "changed").length; + + // ---- sink (the consumer write) -------------------------------------- + if (toEmit.length > 0) await input.sink(toEmit); + + // ---- P068 ledger COMMIT, after the sink resolved --------------------- + const emitted = new Set(toEmit.map((e) => e.record_key)); + const committed = entries.filter((e) => emitted.has(e.record_key)); + if (committed.length > 0) { + await this.repo.commitDeliveries(input.source_id, input.tenant_id, committed); + } + + return { + emitted: toEmit.length, + duplicates, + changed, + quarantined: quarantinedCount, + violations: violationCount, + blocked: false, + }; + } +} diff --git a/src/lineage/lineage-edge.ts b/src/lineage/lineage-edge.ts new file mode 100644 index 0000000..cf6b3f7 --- /dev/null +++ b/src/lineage/lineage-edge.ts @@ -0,0 +1,42 @@ +/** + * src/lineage/lineage-edge.ts + * ============================ + * P067: the flattened, QUERYABLE form of an OpenLineage run event. + * + * LineageEmitter (src/lineage/lineage-emitter.ts) already emits spec-shaped + * run events to AUD0, and AUD0 remains the SYSTEM OF RECORD for them. But + * "emitted to AUD0" is not "answerable from 0pt": nothing in this repo could + * answer "which sources fed dataset ds_x?" without asking another product. + * These edges are 0pt's own projection — one row per (run, input, output) + * pair — written in the same call that emits, so the question is answerable + * locally and joins back to AUD0 by `run_id`. + * + * The projection is deliberately LOSSY (facets are not stored, only the + * canonical SIAX ref) — it is an index, not a second source of truth. + */ + +export interface LineageEdge { + /** Deterministic sha256 of (run_id, input, output) — a re-emitted idempotent run does not duplicate edges. */ + edge_id: string; + tenant_id: string; + run_id: string; + capability_id: string; + input_namespace: string; + input_name: string; + input_ref?: string; + output_namespace: string; + output_name: string; + output_ref?: string; + event_time: string; + correlation_id?: string; +} + +/** Query one direction of the graph. Exactly one of the two selectors is required. */ +export interface LineageQuery { + tenant_id: string; + /** Everything that fed this output (upstream). */ + output_name?: string; + /** Everything this input produced (downstream). */ + input_name?: string; + limit?: number; +} diff --git a/src/lineage/lineage-store.ts b/src/lineage/lineage-store.ts new file mode 100644 index 0000000..33526f2 --- /dev/null +++ b/src/lineage/lineage-store.ts @@ -0,0 +1,121 @@ +/** + * src/lineage/lineage-store.ts + * ============================= + * P067 DoD: "source -> output lineage is QUERYABLE." + * + * LineageEmitter already emits OpenLineage run events to AUD0 and that does + * not change — AUD0 stays the system of record. This class adds the missing + * half: it flattens the same run event into (input, output) edges, persists + * them, and answers both directions of the question. + * + * `RecordingLineageEmitter` is the composition point: it wraps the existing + * emitter so every existing caller (QualityService, DatasetVersionService, + * the ingestion runner) gets the local projection for free, without any of + * them learning about a second store. Emission to AUD0 keeps its existing + * never-throw contract; a projection write failure is likewise reported, not + * thrown, because lineage is evidence of work that already happened. + */ +import crypto from "crypto"; +import type { IngestionRepository } from "../repositories/ingestion-repository"; +import type { LineageEdge, LineageQuery } from "./lineage-edge"; +import type { LineageEmitInput, LineageEmitResult, LineageEmitter } from "./lineage-emitter"; +import type { OpenLineageRunEvent } from "./openlineage"; + +/** Deterministic edge id: the same run re-emitted produces the same edges. */ +export function lineageEdgeId(runId: string, input: string, output: string): string { + return `led_${crypto.createHash("sha256").update(`${runId}|${input}|${output}`).digest("hex").slice(0, 32)}`; +} + +function refOf(facets: Record | undefined): string | undefined { + const facet = facets?.siaxResourceRef as { ref?: string } | undefined; + return typeof facet?.ref === "string" ? facet.ref : undefined; +} + +/** + * Flatten one OpenLineage run event into edges. A run with N inputs and M + * outputs yields N*M edges; a run with no inputs or no outputs yields none + * (there is no edge to draw), which is why the runner always emits both. + */ +export function edgesFromRunEvent( + event: OpenLineageRunEvent, + tenantId: string, + correlationId?: string, +): LineageEdge[] { + const edges: LineageEdge[] = []; + for (const input of event.inputs ?? []) { + for (const output of event.outputs ?? []) { + edges.push({ + edge_id: lineageEdgeId(event.run.runId, `${input.namespace}/${input.name}`, `${output.namespace}/${output.name}`), + tenant_id: tenantId, + run_id: event.run.runId, + capability_id: event.job.name, + input_namespace: input.namespace, + input_name: input.name, + input_ref: refOf(input.facets), + output_namespace: output.namespace, + output_name: output.name, + output_ref: refOf(output.facets), + event_time: event.eventTime, + correlation_id: correlationId, + }); + } + } + return edges; +} + +export class LineageStore { + constructor(private readonly repo: IngestionRepository) {} + + async record(event: OpenLineageRunEvent, tenantId: string, correlationId?: string): Promise { + const edges = edgesFromRunEvent(event, tenantId, correlationId); + if (edges.length > 0) await this.repo.appendLineageEdges(edges); + return edges.length; + } + + /** Everything that fed `outputName`, newest first. */ + async upstreamOf(tenantId: string, outputName: string, limit?: number): Promise { + return this.repo.queryLineage({ tenant_id: tenantId, output_name: outputName, limit }); + } + + /** Everything `inputName` produced, newest first. */ + async downstreamOf(tenantId: string, inputName: string, limit?: number): Promise { + return this.repo.queryLineage({ tenant_id: tenantId, input_name: inputName, limit }); + } + + async query(query: LineageQuery): Promise { + return this.repo.queryLineage(query); + } +} + +/** + * A LineageEmitter that also writes the local projection. Drop-in for the + * existing emitter: same `emit` signature, same never-throw contract. + */ +export class RecordingLineageEmitter { + constructor( + private readonly inner: LineageEmitter, + private readonly store: LineageStore, + ) {} + + buildRunEvent(input: LineageEmitInput): OpenLineageRunEvent { + return this.inner.buildRunEvent(input); + } + + async emit(input: LineageEmitInput): Promise { + const event = this.inner.buildRunEvent(input); + const result = await this.inner.emit(input); + let projected = 0; + try { + projected = await this.store.record(event, input.tenant_id, input.correlation_id); + } catch (err) { + return { + ...result, + projected: 0, + warning: [result.warning, `lineage projection write failed: ${err instanceof Error ? err.message : String(err)}`] + .filter(Boolean) + .join("; "), + }; + } + return { ...result, projected }; + } +} diff --git a/src/quality/quality-policy.ts b/src/quality/quality-policy.ts new file mode 100644 index 0000000..712d56c --- /dev/null +++ b/src/quality/quality-policy.ts @@ -0,0 +1,129 @@ +/** + * src/quality/quality-policy.ts + * ============================== + * P064: the DECLARED enforcement policy — what happens to a record that fails + * its quality rules — and the gate that applies it on the ingestion path. + * + * WHAT EXISTED BEFORE (and is reused, not rebuilt): QualityService already + * owns the rule registry, runs, results, issues and scores, and + * ExpectationEngine already executes the Great Expectations core subset. What + * neither of them had was a CONSEQUENCE: a failing rule produced an ISSUE for + * a human, and the data flowed on regardless. That is the gap this closes. + * This module deliberately does NOT re-implement rule storage or execution — + * it composes ExpectationEngine and adds the decision. + * + * Three declared actions, and nothing implicit: + * - `block` — the whole page fails; the checkpoint is NOT advanced, so + * the run stops on the bad data instead of stepping over it. + * - `quarantine` — the failing envelopes are persisted to durable quarantine + * and excluded from the output; clean envelopes continue and + * the checkpoint advances past all of them. + * - `warn` — everything continues; violations are counted and reported. + * + * There is no default. A source with no policy gets `block` from the runner + * only if a policy was supplied; with no policy at all the runner skips the + * gate and SAYS SO in the run report (`quality_policy: "none_declared"`) — + * an undeclared policy must never look like a passed one. + */ +import { ExpectationEngine } from "./expectation-engine"; +import type { QualityExpectation } from "./quality-types"; +import type { IngestionEnvelope } from "../types/ingestion"; + +export type QualityEnforcementAction = "block" | "quarantine" | "warn"; + +export interface QualityEnforcementPolicy { + source_id: string; + tenant_id: string; + /** Rules applied per record. Expectations are evaluated against a one-row table. */ + expectations: QualityExpectation[]; + on_violation: QualityEnforcementAction; +} + +export interface EnvelopeViolation { + expectation_type: string; + detail: string; +} + +export interface QualityGateResult { + /** Envelopes that may proceed downstream. */ + accepted: IngestionEnvelope[]; + /** Envelopes routed to quarantine (only under `quarantine`). */ + quarantined: Array<{ envelope: IngestionEnvelope; violations: EnvelopeViolation[] }>; + /** Every violation seen, including under `warn` where nothing was withheld. */ + violations: Array<{ record_key: string; violations: EnvelopeViolation[] }>; + /** True when `block` fired: the caller must NOT advance the checkpoint. */ + blocked: boolean; + action: QualityEnforcementAction; +} + +export class QualityGate { + constructor(private readonly engine: ExpectationEngine = new ExpectationEngine()) {} + + /** + * Evaluate one page of envelopes against the policy. + * + * Delete envelopes (payload null) are passed through untouched: a CDC delete + * carries no columns to check, and failing it against a column expectation + * would quarantine the very deletions a source is asking us to honour. + */ + evaluate(envelopes: IngestionEnvelope[], policy: QualityEnforcementPolicy): QualityGateResult { + const accepted: IngestionEnvelope[] = []; + const quarantined: QualityGateResult["quarantined"] = []; + const allViolations: QualityGateResult["violations"] = []; + + for (const envelope of envelopes) { + if (envelope.payload === null) { + accepted.push(envelope); + continue; + } + const violations = this.violationsFor(envelope.payload, policy.expectations); + if (violations.length === 0) { + accepted.push(envelope); + continue; + } + allViolations.push({ record_key: envelope.record_key, violations }); + if (policy.on_violation === "quarantine") { + quarantined.push({ envelope, violations }); + } else if (policy.on_violation === "warn") { + accepted.push(envelope); + } + // `block`: neither accepted nor quarantined — the caller stops. + } + + const blocked = policy.on_violation === "block" && allViolations.length > 0; + return { + accepted: blocked ? [] : accepted, + quarantined, + violations: allViolations, + blocked, + action: policy.on_violation, + }; + } + + private violationsFor( + payload: Record, + expectations: QualityExpectation[], + ): EnvelopeViolation[] { + const violations: EnvelopeViolation[] = []; + for (const expectation of expectations) { + if (!this.engine.supports(expectation.expectation_type)) { + // An expectation this engine cannot execute is a VIOLATION of the + // policy's executability, not a silent pass. The record is treated as + // failing so an unrunnable rule can never look like a satisfied one. + violations.push({ + expectation_type: expectation.expectation_type, + detail: `expectation is not executable by the local engine; delegate it to the GE backend before declaring it in an enforcement policy`, + }); + continue; + } + const outcome = this.engine.execute(expectation, [payload]); + if (!outcome.success) { + violations.push({ + expectation_type: expectation.expectation_type, + detail: `failed on column "${String(expectation.kwargs.column ?? "")}" (${outcome.unexpected_count}/${outcome.evaluated_count} unexpected)`, + }); + } + } + return violations; + } +} diff --git a/src/repositories/in-memory-ingestion-repository.ts b/src/repositories/in-memory-ingestion-repository.ts new file mode 100644 index 0000000..31202b1 --- /dev/null +++ b/src/repositories/in-memory-ingestion-repository.ts @@ -0,0 +1,120 @@ +/** + * src/repositories/in-memory-ingestion-repository.ts + * ==================================================== + * Non-production implementation of IngestionRepository (hermetic tests, local + * dev). Semantics are kept IDENTICAL to the Postgres implementation on every + * point the runner depends on: + * - checkpoints are upserted per (source, partition), + * - ledger verdicts are new/changed/duplicate against the LAST accepted + * payload hash for the key, + * - quarantine is append-only, newest-first on read, + * - schemas are keyed by (source, version) and "latest" means highest version, + * - lineage edges are deduplicated by edge_id. + * + * Any divergence between the two implementations is a bug in one of them; + * tests/p0/ingestion-repository-parity.test.ts is what keeps them honest for + * the in-memory side, and tests/p0/ingestion-plane-postgres.realinfra.test.ts + * runs the same expectations against real Postgres. + */ +import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge"; +import type { RecordSchema } from "../schema/record-schema"; +import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion"; +import type { IngestionRepository, LedgerEntry } from "./ingestion-repository"; + +function checkpointKey(sourceId: string, partitionKey: string): string { + return `${sourceId}|${partitionKey}`; +} + +export class InMemoryIngestionRepository implements IngestionRepository { + private readonly checkpoints = new Map(); + /** source_id -> record_key -> last accepted payload hash. */ + private readonly ledger = new Map>(); + private readonly quarantined = new Map(); + private readonly schemas = new Map(); + private readonly lineage = new Map(); + + async getCheckpoint(sourceId: string, partitionKey: string): Promise { + return this.checkpoints.get(checkpointKey(sourceId, partitionKey)); + } + + async commitCheckpoint(checkpoint: IngestionCheckpoint): Promise { + this.checkpoints.set(checkpointKey(checkpoint.source_id, checkpoint.partition_key), { ...checkpoint }); + } + + async listCheckpoints(sourceId: string): Promise { + return [...this.checkpoints.values()] + .filter((c) => c.source_id === sourceId) + .sort((a, b) => a.partition_key.localeCompare(b.partition_key)); + } + + async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise { + const keys = this.ledger.get(sourceId); + return entries.map((entry) => { + const prior = keys?.get(entry.record_key); + return { + record_key: entry.record_key, + verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed", + } as LedgerDecision; + }); + } + + async commitDeliveries(sourceId: string, _tenantId: string, entries: LedgerEntry[]): Promise { + let keys = this.ledger.get(sourceId); + if (!keys) { + keys = new Map(); + this.ledger.set(sourceId, keys); + } + for (const entry of entries) keys.set(entry.record_key, entry.payload_hash); + } + + async countLedger(sourceId: string): Promise { + return this.ledger.get(sourceId)?.size ?? 0; + } + + async quarantine(record: QuarantineRecord): Promise { + const list = this.quarantined.get(record.source_id) ?? []; + list.push(record); + this.quarantined.set(record.source_id, list); + } + + async listQuarantine(sourceId: string, limit = 100): Promise { + const list = this.quarantined.get(sourceId) ?? []; + return [...list].reverse().slice(0, limit); + } + + async putSchema(schema: RecordSchema): Promise { + const list = this.schemas.get(schema.source_id) ?? []; + const existing = list.findIndex((s) => s.schema_version === schema.schema_version); + if (existing >= 0) list[existing] = { ...schema }; + else list.push({ ...schema }); + list.sort((a, b) => a.schema_version - b.schema_version); + this.schemas.set(schema.source_id, list); + } + + async getSchema(sourceId: string, version: number): Promise { + return this.schemas.get(sourceId)?.find((s) => s.schema_version === version); + } + + async getLatestSchema(sourceId: string): Promise { + const list = this.schemas.get(sourceId); + return list && list.length > 0 ? list[list.length - 1] : undefined; + } + + async listSchemas(sourceId: string): Promise { + return [...(this.schemas.get(sourceId) ?? [])]; + } + + async appendLineageEdges(edges: LineageEdge[]): Promise { + for (const edge of edges) this.lineage.set(edge.edge_id, { ...edge }); + } + + async queryLineage(query: LineageQuery): Promise { + const limit = query.limit ?? 200; + return [...this.lineage.values()] + .filter((e) => e.tenant_id === query.tenant_id) + .filter((e) => (query.output_name ? e.output_name === query.output_name : true)) + .filter((e) => (query.input_name ? e.input_name === query.input_name : true)) + .sort((a, b) => b.event_time.localeCompare(a.event_time)) + .slice(0, limit); + } +} diff --git a/src/repositories/in-memory-source-registry-repository.ts b/src/repositories/in-memory-source-registry-repository.ts index a0cf046..c5d8deb 100644 --- a/src/repositories/in-memory-source-registry-repository.ts +++ b/src/repositories/in-memory-source-registry-repository.ts @@ -10,7 +10,13 @@ * - 1]` for "most recent" and scanned backward for consecutive failures -- * callers now get the same data pre-ordered instead). */ -import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry"; +import type { + SourceAuthorization, + SourceHealthCheck, + SourceProvenance, + SourceRights, + SourceSchedule, +} from "../types/source-registry"; import type { SourceRegistryRepository } from "./source-registry-repository"; export class InMemorySourceRegistryRepository implements SourceRegistryRepository { @@ -18,6 +24,8 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor private readonly provenance = new Map(); private readonly healthChecks = new Map(); private readonly lastSuccess = new Map(); + private readonly authorizations = new Map(); + private readonly schedules = new Map(); async setRights(rights: SourceRights): Promise { this.rights.set(rights.source_id, rights); @@ -43,6 +51,22 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor return this.lastSuccess.get(sourceId); } + async setAuthorization(auth: SourceAuthorization): Promise { + this.authorizations.set(auth.source_id, auth); + } + + async getAuthorization(sourceId: string): Promise { + return this.authorizations.get(sourceId); + } + + async setSchedule(schedule: SourceSchedule): Promise { + this.schedules.set(schedule.source_id, schedule); + } + + async getSchedule(sourceId: string): Promise { + return this.schedules.get(sourceId); + } + async appendHealthCheck(check: SourceHealthCheck): Promise { const list = this.healthChecks.get(check.source_id) ?? []; list.push(check); diff --git a/src/repositories/in-memory-source-repository.ts b/src/repositories/in-memory-source-repository.ts index 53babf9..b7bfd07 100644 --- a/src/repositories/in-memory-source-repository.ts +++ b/src/repositories/in-memory-source-repository.ts @@ -27,6 +27,7 @@ interface SourceRecord { created_at: string; updated_at: string | null; created_by: string | null; + owner_id: string | null; } export class InMemorySourceRepository implements SourceRepository { @@ -50,11 +51,16 @@ export class InMemorySourceRepository implements SourceRepository { adapter_id: input.adapter_id ?? null, access: input.access ?? null, description: input.description ?? null, - status: input.status ?? "active", + // P060: "pending", not "active". A source with no rights, no provenance + // and no authorization used to be live the instant it was registered; + // reaching "active" now goes through + // SourceRegistryService.activateSource's classification gate. + status: input.status ?? "pending", correlation_id: input.correlation_id ?? null, created_at: now, updated_at: null, created_by: input.created_by ?? null, + owner_id: input.owner_id ?? null, }; this.sources.set(record.source_id, record); return toSource(record); @@ -66,6 +72,14 @@ export class InMemorySourceRepository implements SourceRepository { return toSource(record); } + async updateStatus(sourceId: string, status: SourceStatus): Promise { + const record = this.sources.get(sourceId); + if (!record) throw new Error(`Source ${sourceId} not found`); + record.status = status; + record.updated_at = new Date().toISOString(); + return toSource(record); + } + async list(filter?: SourceListFilter): Promise { let results = Array.from(this.sources.values()); if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id); @@ -91,5 +105,6 @@ function toSource(r: SourceRecord): Source { created_at: r.created_at, updated_at: r.updated_at ?? undefined, created_by: r.created_by ?? undefined, + owner_id: r.owner_id ?? undefined, }; } diff --git a/src/repositories/ingestion-repository.ts b/src/repositories/ingestion-repository.ts new file mode 100644 index 0000000..f7bdfc9 --- /dev/null +++ b/src/repositories/ingestion-repository.ts @@ -0,0 +1,91 @@ +/** + * src/repositories/ingestion-repository.ts + * ========================================== + * The storage seam for the P060-P068 ingestion plane: checkpoints (P060/P062), + * the idempotency ledger (P068), quarantine (P064), the schema registry + * (P060/P063) and the queryable lineage projection (P067). + * + * WHY ONE INTERFACE AND NOT FIVE. These five tables are written by ONE + * caller — IngestionRunner — inside one run, and their correctness is + * relative to each other (a checkpoint that advanced past records the ledger + * never accepted is a silent gap; a quarantined record whose checkpoint moved + * on is a silently dropped record). Splitting them into five factories would + * let a future caller wire four of them to Postgres and one to memory and + * still typecheck, which is exactly the failure this plane must not have. + * They are chosen once, together, by one `resolveStoreMode` call. + * + * Same production posture as every other 0pt store: `resolveStoreMode` fails + * hard in production with no DATABASE_URL rather than silently serving Maps + * (src/platform/dependency-mode.ts, Master-DoD §3/§10). + */ +import { resolveStoreMode } from "../platform/dependency-mode"; +import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge"; +import type { RecordSchema } from "../schema/record-schema"; +import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion"; +import { getRepositorySql, repositoryDatabaseUrl } from "./db"; +import { InMemoryIngestionRepository } from "./in-memory-ingestion-repository"; +import { PostgresIngestionRepository } from "./postgres-ingestion-repository"; + +/** One record offered to the ledger. */ +export interface LedgerEntry { + record_key: string; + /** Stable hash of the payload — see hashPayload() in src/ingestion/ingestion-ledger.ts. */ + payload_hash: string; +} + +export interface IngestionRepository { + // --- checkpoints (P060 registry field, P062 durable resume) -------------- + getCheckpoint(sourceId: string, partitionKey: string): Promise; + commitCheckpoint(checkpoint: IngestionCheckpoint): Promise; + listCheckpoints(sourceId: string): Promise; + + // --- idempotency ledger (P068) ------------------------------------------ + /** + * READ-ONLY verdicts, in input order: + * "new" — this record_key has never been accepted for this source, + * "changed" — it has, but with different content (a real update), + * "duplicate" — it has, with byte-identical content (a replay). + * + * Deliberately separate from {@link commitDeliveries}. An earlier draft of + * this interface combined them into one atomic check-and-record, which was + * WRONG in the way that matters: the runner has to consult the ledger + * BEFORE the sink (to filter replays) and commit AFTER it (so a crash + * between the two re-delivers rather than loses). A combined call marked a + * record "delivered" that the sink never received, and the resumed run then + * skipped it as a duplicate — permanent, silent data loss. That is not a + * hypothetical: tests/p0/ingestion-runner.test.ts caught it, and the + * "resumes with NO gap" case is what pins the fix. + */ + peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise; + + /** + * Record that these records were successfully delivered. Called only after + * the sink resolved. Idempotent: re-committing the same entry bumps the + * delivery count and leaves the verdict unchanged. + */ + commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise; + + countLedger(sourceId: string): Promise; + + // --- quarantine (P064) -------------------------------------------------- + quarantine(record: QuarantineRecord): Promise; + listQuarantine(sourceId: string, limit?: number): Promise; + + // --- schema registry (P060 registry field, P063 evolution) -------------- + putSchema(schema: RecordSchema): Promise; + getSchema(sourceId: string, version: number): Promise; + getLatestSchema(sourceId: string): Promise; + listSchemas(sourceId: string): Promise; + + // --- lineage projection (P067) ------------------------------------------ + appendLineageEdges(edges: LineageEdge[]): Promise; + queryLineage(query: LineageQuery): Promise; +} + +export function createIngestionRepository(databaseUrl?: string): IngestionRepository { + const mode = resolveStoreMode("0pt.ingestion", databaseUrl ?? repositoryDatabaseUrl()); + if (mode.mode === "postgres") { + return new PostgresIngestionRepository(getRepositorySql(mode.url)); + } + return new InMemoryIngestionRepository(); +} diff --git a/src/repositories/postgres-ingestion-repository.ts b/src/repositories/postgres-ingestion-repository.ts new file mode 100644 index 0000000..133096b --- /dev/null +++ b/src/repositories/postgres-ingestion-repository.ts @@ -0,0 +1,308 @@ +/** + * src/repositories/postgres-ingestion-repository.ts + * ================================================== + * Real Postgres backing for IngestionRepository (sql/007_ingestion_plane.sql). + * + * THE LEDGER IS THE INTERESTING ONE (P068), and it is deliberately SPLIT: + * `peekDeliveries` reads verdicts, `commitDeliveries` records them. The + * runner peeks before the sink and commits after it, so a crash in between + * re-delivers on resume instead of losing the record. An earlier version of + * this file combined the two into one atomic check-and-record statement; that + * was measurably wrong (see the interface's doc comment) and the crash/resume + * test in tests/p0/ingestion-runner.test.ts is what proved it. + * + * KNOWN LIMIT, stated rather than papered over: the split means two runs on + * the SAME (source_id, partition) can both peek "new" and both emit. The + * delivery guarantee is therefore AT-LEAST-ONCE, which is what the + * deterministic `envelope_id` (src/ingestion/envelope.ts) exists to make + * survivable downstream. Exactly-once across concurrent runs needs a run lock + * per (source, partition); there is none yet. That is a real open gap, not a + * solved one — do not describe this plane as exactly-once. + */ +import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge"; +import type { + CompatibilityPolicy, + RecordSchema, + RecordSchemaFields, +} from "../schema/record-schema"; +import type { + IngestionCheckpoint, + IngestionEnvelope, + LedgerDecision, + QuarantineRecord, +} from "../types/ingestion"; +import type { RepositorySql } from "./db"; +import type { IngestionRepository, LedgerEntry } from "./ingestion-repository"; + +/** postgres.js's `sql.json()` parameter type, named locally so the two JSONB columns can be passed without importing postgres' internals. */ +type JsonValue = Parameters[0]; + +function toIso(v: string | Date): string { + return v instanceof Date ? v.toISOString() : v; +} + +interface CheckpointRow { + source_id: string; + tenant_id: string; + partition_key: string; + cursor: string | null; + sequence: string | number; + records_committed: string | number; + committed_at: string | Date; +} + +function toCheckpoint(r: CheckpointRow): IngestionCheckpoint { + return { + source_id: r.source_id, + tenant_id: r.tenant_id, + partition_key: r.partition_key, + cursor: r.cursor ?? undefined, + sequence: Number(r.sequence), + records_committed: Number(r.records_committed), + committed_at: toIso(r.committed_at), + }; +} + +export class PostgresIngestionRepository implements IngestionRepository { + constructor(private readonly sql: RepositorySql) {} + + // --- checkpoints --------------------------------------------------------- + + async getCheckpoint(sourceId: string, partitionKey: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at + FROM opt_ingestion_checkpoints + WHERE source_id = ${sourceId} AND partition_key = ${partitionKey} + LIMIT 1`; + return rows[0] ? toCheckpoint(rows[0]) : undefined; + } + + async commitCheckpoint(cp: IngestionCheckpoint): Promise { + await this.sql` + INSERT INTO opt_ingestion_checkpoints + (source_id, partition_key, tenant_id, cursor, sequence, records_committed, committed_at) + VALUES + (${cp.source_id}, ${cp.partition_key}, ${cp.tenant_id}, ${cp.cursor ?? null}, + ${cp.sequence}, ${cp.records_committed}, ${cp.committed_at}) + ON CONFLICT (source_id, partition_key) DO UPDATE + SET cursor = EXCLUDED.cursor, + sequence = EXCLUDED.sequence, + records_committed = EXCLUDED.records_committed, + committed_at = EXCLUDED.committed_at, + tenant_id = EXCLUDED.tenant_id`; + } + + async listCheckpoints(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at + FROM opt_ingestion_checkpoints + WHERE source_id = ${sourceId} + ORDER BY partition_key ASC`; + return rows.map(toCheckpoint); + } + + // --- ledger -------------------------------------------------------------- + + async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise { + if (entries.length === 0) return []; + const keys = entries.map((e) => e.record_key); + const rows = await this.sql<{ record_key: string; payload_hash: string }[]>` + SELECT record_key, payload_hash FROM opt_ingestion_ledger + WHERE source_id = ${sourceId} AND record_key = ANY(${this.sql.array(keys)})`; + const priorByKey = new Map(rows.map((r) => [r.record_key, r.payload_hash])); + return entries.map((entry) => { + const prior = priorByKey.get(entry.record_key); + return { + record_key: entry.record_key, + verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed", + } as LedgerDecision; + }); + } + + async commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise { + for (const entry of entries) { + await this.sql` + INSERT INTO opt_ingestion_ledger (source_id, record_key, tenant_id, payload_hash) + VALUES (${sourceId}, ${entry.record_key}, ${tenantId}, ${entry.payload_hash}) + ON CONFLICT (source_id, record_key) DO UPDATE + SET last_seen_at = now(), + delivery_count = opt_ingestion_ledger.delivery_count + 1, + payload_hash = EXCLUDED.payload_hash`; + } + } + + async countLedger(sourceId: string): Promise { + const rows = await this.sql<{ n: string }[]>` + SELECT count(*)::text AS n FROM opt_ingestion_ledger WHERE source_id = ${sourceId}`; + return Number(rows[0]?.n ?? 0); + } + + // --- quarantine ---------------------------------------------------------- + + async quarantine(record: QuarantineRecord): Promise { + await this.sql` + INSERT INTO opt_quality_quarantine + (quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at) + VALUES + (${record.quarantine_id}, ${record.source_id}, ${record.tenant_id}, ${record.run_id}, + ${record.record_key}, ${this.sql.json(record.violations as unknown as JsonValue)}, + ${this.sql.json(record.envelope as unknown as JsonValue)}, + ${record.quarantined_at}) + ON CONFLICT (quarantine_id) DO NOTHING`; + } + + async listQuarantine(sourceId: string, limit = 100): Promise { + const rows = await this.sql< + { + quarantine_id: string; + source_id: string; + tenant_id: string; + run_id: string; + record_key: string; + violations: QuarantineRecord["violations"]; + envelope: IngestionEnvelope; + quarantined_at: string | Date; + }[] + >` + SELECT quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at + FROM opt_quality_quarantine + WHERE source_id = ${sourceId} + ORDER BY quarantined_at DESC, quarantine_id DESC + LIMIT ${limit}`; + return rows.map((r) => ({ + quarantine_id: r.quarantine_id, + source_id: r.source_id, + tenant_id: r.tenant_id, + run_id: r.run_id, + record_key: r.record_key, + violations: r.violations, + envelope: r.envelope, + quarantined_at: toIso(r.quarantined_at), + })); + } + + // --- schema registry ----------------------------------------------------- + + async putSchema(schema: RecordSchema): Promise { + await this.sql` + INSERT INTO opt_source_schemas + (source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by) + VALUES + (${schema.source_id}, ${schema.schema_version}, ${schema.tenant_id}, + ${this.sql.json(schema.fields as unknown as JsonValue)}, ${schema.compatibility}, + ${schema.registered_at}, ${schema.registered_by ?? null}) + ON CONFLICT (source_id, schema_version) DO UPDATE + SET fields = EXCLUDED.fields, + compatibility = EXCLUDED.compatibility, + registered_at = EXCLUDED.registered_at, + registered_by = EXCLUDED.registered_by`; + } + + async getSchema(sourceId: string, version: number): Promise { + const rows = await this.sql` + SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by + FROM opt_source_schemas + WHERE source_id = ${sourceId} AND schema_version = ${version} + LIMIT 1`; + return rows[0] ? toSchema(rows[0]) : undefined; + } + + async getLatestSchema(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by + FROM opt_source_schemas + WHERE source_id = ${sourceId} + ORDER BY schema_version DESC + LIMIT 1`; + return rows[0] ? toSchema(rows[0]) : undefined; + } + + async listSchemas(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by + FROM opt_source_schemas + WHERE source_id = ${sourceId} + ORDER BY schema_version ASC`; + return rows.map(toSchema); + } + + // --- lineage ------------------------------------------------------------- + + async appendLineageEdges(edges: LineageEdge[]): Promise { + for (const e of edges) { + await this.sql` + INSERT INTO opt_lineage_edges + (edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref, + output_namespace, output_name, output_ref, event_time, correlation_id) + VALUES + (${e.edge_id}, ${e.tenant_id}, ${e.run_id}, ${e.capability_id}, ${e.input_namespace}, + ${e.input_name}, ${e.input_ref ?? null}, ${e.output_namespace}, ${e.output_name}, + ${e.output_ref ?? null}, ${e.event_time}, ${e.correlation_id ?? null}) + ON CONFLICT (edge_id) DO NOTHING`; + } + } + + async queryLineage(query: LineageQuery): Promise { + const limit = query.limit ?? 200; + const rows = await this.sql` + SELECT edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref, + output_namespace, output_name, output_ref, event_time, correlation_id + FROM opt_lineage_edges + WHERE tenant_id = ${query.tenant_id} + ${query.output_name ? this.sql`AND output_name = ${query.output_name}` : this.sql``} + ${query.input_name ? this.sql`AND input_name = ${query.input_name}` : this.sql``} + ORDER BY event_time DESC, edge_id DESC + LIMIT ${limit}`; + return rows.map((r) => ({ + edge_id: r.edge_id, + tenant_id: r.tenant_id, + run_id: r.run_id, + capability_id: r.capability_id, + input_namespace: r.input_namespace, + input_name: r.input_name, + input_ref: r.input_ref ?? undefined, + output_namespace: r.output_namespace, + output_name: r.output_name, + output_ref: r.output_ref ?? undefined, + event_time: toIso(r.event_time), + correlation_id: r.correlation_id ?? undefined, + })); + } +} + +interface SchemaRow { + source_id: string; + schema_version: number; + tenant_id: string; + fields: RecordSchemaFields; + compatibility: string; + registered_at: string | Date; + registered_by: string | null; +} + +function toSchema(r: SchemaRow): RecordSchema { + return { + source_id: r.source_id, + schema_version: Number(r.schema_version), + tenant_id: r.tenant_id, + fields: r.fields, + compatibility: r.compatibility as CompatibilityPolicy, + registered_at: toIso(r.registered_at), + registered_by: r.registered_by ?? undefined, + }; +} + +interface LineageRow { + edge_id: string; + tenant_id: string; + run_id: string; + capability_id: string; + input_namespace: string; + input_name: string; + input_ref: string | null; + output_namespace: string; + output_name: string; + output_ref: string | null; + event_time: string | Date; + correlation_id: string | null; +} diff --git a/src/repositories/postgres-source-registry-repository.ts b/src/repositories/postgres-source-registry-repository.ts index e30abe3..27269d9 100644 --- a/src/repositories/postgres-source-registry-repository.ts +++ b/src/repositories/postgres-source-registry-repository.ts @@ -10,10 +10,13 @@ * is a single upserted "last success" row per source. */ import type { + AuthorizationBasis, + SourceAuthorization, SourceHealthCheck, SourceHealthStatus, SourceProvenance, SourceRights, + SourceSchedule, } from "../types/source-registry"; import type { RepositorySql } from "./db"; import type { SourceRegistryRepository } from "./source-registry-repository"; @@ -167,6 +170,57 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor return rows[0] ? toIsoString(rows[0].last_success_at) : undefined; } + // --- P060 authorization / schedule --------------------------------------- + + async setAuthorization(auth: SourceAuthorization): Promise { + await this.sql` + INSERT INTO opt_source_authorization + (source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes) + VALUES + (${auth.source_id}, ${auth.tenant_id}, ${auth.basis}, ${auth.granted_by}, ${auth.granted_at}, + ${auth.expires_at ?? null}, ${auth.evidence_ref ?? null}, ${auth.notes ?? null}) + ON CONFLICT (source_id) DO UPDATE + SET tenant_id = EXCLUDED.tenant_id, + basis = EXCLUDED.basis, + granted_by = EXCLUDED.granted_by, + granted_at = EXCLUDED.granted_at, + expires_at = EXCLUDED.expires_at, + evidence_ref = EXCLUDED.evidence_ref, + notes = EXCLUDED.notes, + updated_at = now()`; + } + + async getAuthorization(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes + FROM opt_source_authorization WHERE source_id = ${sourceId} LIMIT 1`; + return rows[0] ? toAuthorization(rows[0]) : undefined; + } + + async setSchedule(schedule: SourceSchedule): Promise { + await this.sql` + INSERT INTO opt_source_schedule + (source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at) + VALUES + (${schedule.source_id}, ${schedule.tenant_id}, ${schedule.mode}, + ${schedule.interval_seconds ?? null}, ${schedule.cron ?? null}, ${schedule.enabled}, + ${schedule.updated_at}) + ON CONFLICT (source_id) DO UPDATE + SET tenant_id = EXCLUDED.tenant_id, + mode = EXCLUDED.mode, + interval_seconds = EXCLUDED.interval_seconds, + cron = EXCLUDED.cron, + enabled = EXCLUDED.enabled, + updated_at = EXCLUDED.updated_at`; + } + + async getSchedule(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at + FROM opt_source_schedule WHERE source_id = ${sourceId} LIMIT 1`; + return rows[0] ? toSchedule(rows[0]) : undefined; + } + async appendHealthCheck(check: SourceHealthCheck): Promise { await this.sql` INSERT INTO opt_source_health_checks (source_id, tenant_id, status, checked_at, latency_ms, detail) @@ -195,3 +249,49 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor return rows.map(toHealthCheck); } } + +interface AuthorizationRow { + source_id: string; + tenant_id: string; + basis: string; + granted_by: string; + granted_at: string | Date; + expires_at: string | Date | null; + evidence_ref: string | null; + notes: string | null; +} + +function toAuthorization(r: AuthorizationRow): SourceAuthorization { + return { + source_id: r.source_id, + tenant_id: r.tenant_id, + basis: r.basis as AuthorizationBasis, + granted_by: r.granted_by, + granted_at: toIsoString(r.granted_at), + expires_at: r.expires_at ? toIsoString(r.expires_at) : undefined, + evidence_ref: r.evidence_ref ?? undefined, + notes: r.notes ?? undefined, + }; +} + +interface ScheduleRow { + source_id: string; + tenant_id: string; + mode: string; + interval_seconds: number | string | null; + cron: string | null; + enabled: boolean; + updated_at: string | Date; +} + +function toSchedule(r: ScheduleRow): SourceSchedule { + return { + source_id: r.source_id, + tenant_id: r.tenant_id, + mode: r.mode as SourceSchedule["mode"], + interval_seconds: r.interval_seconds === null ? undefined : Number(r.interval_seconds), + cron: r.cron ?? undefined, + enabled: r.enabled, + updated_at: toIsoString(r.updated_at), + }; +} diff --git a/src/repositories/postgres-source-repository.ts b/src/repositories/postgres-source-repository.ts index 970e911..fda7f90 100644 --- a/src/repositories/postgres-source-repository.ts +++ b/src/repositories/postgres-source-repository.ts @@ -25,6 +25,7 @@ interface SourceRow { created_at: string | Date; updated_at: string | Date | null; created_by: string | null; + owner_id: string | null; } function toIsoString(v: string | Date): string { @@ -46,6 +47,7 @@ function toSource(r: SourceRow): Source { created_at: toIsoString(r.created_at), updated_at: r.updated_at ? toIsoString(r.updated_at) : undefined, created_by: r.created_by ?? undefined, + owner_id: r.owner_id ?? undefined, }; } @@ -63,29 +65,41 @@ export class PostgresSourceRepository implements SourceRepository { const rows = await this.sql` INSERT INTO opt_sources - (source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by) + (source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by, owner_id) VALUES (${input.source_id}, ${input.tenant_id}, ${input.project_id}, ${input.source_type}, ${input.name}, ${input.adapter_id ?? null}, ${input.access ?? null}, ${input.description ?? null}, - ${input.status ?? "active"}, ${input.correlation_id ?? null}, ${input.created_by ?? null}) + ${input.status ?? "pending"}, ${input.correlation_id ?? null}, ${input.created_by ?? null}, + ${input.owner_id ?? null}) RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, - status, correlation_id, created_at, updated_at, created_by`; + status, correlation_id, created_at, updated_at, created_by, owner_id`; return toSource(rows[0]); } async get(sourceId: string): Promise { const rows = await this.sql` SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, - status, correlation_id, created_at, updated_at, created_by + status, correlation_id, created_at, updated_at, created_by, owner_id FROM opt_sources WHERE source_id = ${sourceId} LIMIT 1`; if (!rows[0]) throw new Error(`Source ${sourceId} not found`); return toSource(rows[0]); } + async updateStatus(sourceId: string, status: SourceStatus): Promise { + const rows = await this.sql` + UPDATE opt_sources + SET status = ${status}, updated_at = now() + WHERE source_id = ${sourceId} + RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, + status, correlation_id, created_at, updated_at, created_by, owner_id`; + if (!rows[0]) throw new Error(`Source ${sourceId} not found`); + return toSource(rows[0]); + } + async list(filter?: SourceListFilter): Promise { const rows = await this.sql` SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, - status, correlation_id, created_at, updated_at, created_by + status, correlation_id, created_at, updated_at, created_by, owner_id FROM opt_sources WHERE 1=1 ${filter?.tenant_id ? this.sql`AND tenant_id = ${filter.tenant_id}` : this.sql``} diff --git a/src/repositories/source-registry-repository.ts b/src/repositories/source-registry-repository.ts index 4560e26..941945b 100644 --- a/src/repositories/source-registry-repository.ts +++ b/src/repositories/source-registry-repository.ts @@ -16,7 +16,13 @@ * layer trusts the caller, same as the Maps did. */ import { resolveStoreMode } from "../platform/dependency-mode"; -import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry"; +import type { + SourceAuthorization, + SourceHealthCheck, + SourceProvenance, + SourceRights, + SourceSchedule, +} from "../types/source-registry"; import { getRepositorySql, repositoryDatabaseUrl } from "./db"; import { InMemorySourceRegistryRepository } from "./in-memory-source-registry-repository"; import { PostgresSourceRegistryRepository } from "./postgres-source-registry-repository"; @@ -31,6 +37,14 @@ export interface SourceRegistryRepository { recordSuccess(sourceId: string, tenantId: string, atIso: string): Promise; getLastSuccess(sourceId: string): Promise; + /** P060 authorization — the recorded decision that 0pt may connect at all. */ + setAuthorization(auth: SourceAuthorization): Promise; + getAuthorization(sourceId: string): Promise; + + /** P060 schedule — ingestion mode + cadence. */ + setSchedule(schedule: SourceSchedule): Promise; + getSchedule(sourceId: string): Promise; + appendHealthCheck(check: SourceHealthCheck): Promise; /** Newest-first, bounded to the latest `limit` observations (default 100) -- health is an operational signal, not an audit log; AUD0 owns history. */ listHealthChecks(sourceId: string, limit?: number): Promise; diff --git a/src/repositories/source-repository.ts b/src/repositories/source-repository.ts index f00e1cf..59b077c 100644 --- a/src/repositories/source-repository.ts +++ b/src/repositories/source-repository.ts @@ -33,6 +33,12 @@ export interface SourceRepository { /** Throws (message contains "not found") when no such source exists -- matches the pre-migration Map behavior every caller already handles. */ get(sourceId: string): Promise; list(filter?: SourceListFilter): Promise; + /** + * P060: the only way a source's lifecycle status changes. Deliberately NOT + * a general `update` -- status is the field the activation gate protects, + * and a generic patch method would be the obvious way around it. + */ + updateStatus(sourceId: string, status: SourceStatus): Promise; } export function createSourceRepository(databaseUrl?: string): SourceRepository { diff --git a/src/schema/compatibility.ts b/src/schema/compatibility.ts new file mode 100644 index 0000000..97a0f8d --- /dev/null +++ b/src/schema/compatibility.ts @@ -0,0 +1,170 @@ +/** + * src/schema/compatibility.ts + * ============================ + * P063: the pure comparison. `checkCompatibility(previous, next, policy)` + * returns every way `next` breaks readers/writers of `previous` under the + * declared policy. No IO, no throwing — the caller decides what a break + * means (SchemaRegistry refuses the registration; the ingestion runner + * refuses the RUN, before a single envelope reaches a consumer). + */ +import type { + CompatibilityPolicy, + CompatibilityVerdict, + RecordSchemaFields, + SchemaBreak, +} from "./record-schema"; + +/** + * BACKWARD breaks: a consumer written against `next` reading data produced + * under `previous` fails when + * - `next` adds a REQUIRED field `previous` never produced, or + * - `next` changes a field's type (old data has the old type), or + * - `next` drops a field that was REQUIRED in `previous` and is therefore + * part of the contract consumers rely on. + */ +function backwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] { + const breaks: SchemaBreak[] = []; + for (const [field, spec] of Object.entries(next)) { + const before = previous[field]; + if (!before) { + if (spec.required) { + breaks.push({ + field, + breaks: "backward", + detail: `new required field "${field}" is absent from every record written under the previous version`, + }); + } + continue; + } + if (before.type !== spec.type) { + breaks.push({ + field, + breaks: "backward", + detail: `field "${field}" changed type ${before.type} -> ${spec.type}; existing data still carries ${before.type}`, + }); + } + } + for (const [field, before] of Object.entries(previous)) { + if (!next[field] && before.required) { + breaks.push({ + field, + breaks: "backward", + detail: `required field "${field}" was removed; consumers of the previous version still depend on it`, + }); + } + } + return breaks; +} + +/** + * FORWARD breaks: a consumer written against `previous` reading data produced + * under `next` fails when + * - `next` removes a field `previous` marked REQUIRED, or + * - `next` changes a field's type. + * Adding a required field is forward-SAFE (old readers ignore it). + */ +function forwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] { + const breaks: SchemaBreak[] = []; + for (const [field, before] of Object.entries(previous)) { + const after = next[field]; + if (!after) { + if (before.required) { + breaks.push({ + field, + breaks: "forward", + detail: `required field "${field}" was removed; readers pinned to the previous version will not find it`, + }); + } + continue; + } + if (before.type !== after.type) { + breaks.push({ + field, + breaks: "forward", + detail: `field "${field}" changed type ${before.type} -> ${after.type}; readers pinned to the previous version expect ${before.type}`, + }); + } + } + return breaks; +} + +/** Deduplicate identical (field, direction, detail) triples produced by both directions under `full`. */ +function dedupe(breaks: SchemaBreak[]): SchemaBreak[] { + const seen = new Set(); + const out: SchemaBreak[] = []; + for (const b of breaks) { + const key = `${b.field}|${b.breaks}|${b.detail}`; + if (seen.has(key)) continue; + seen.add(key); + out.push(b); + } + return out; +} + +export function checkCompatibility( + previous: RecordSchemaFields, + next: RecordSchemaFields, + policy: CompatibilityPolicy, +): CompatibilityVerdict { + if (policy === "none") { + return { compatible: true, policy, breaks: [] }; + } + const breaks: SchemaBreak[] = []; + if (policy === "backward" || policy === "full") breaks.push(...backwardBreaks(previous, next)); + if (policy === "forward" || policy === "full") breaks.push(...forwardBreaks(previous, next)); + const deduped = dedupe(breaks); + return { compatible: deduped.length === 0, policy, breaks: deduped }; +} + +/** + * Does a concrete record conform to a schema? Used by the ingestion runner to + * detect that a source has SILENTLY changed shape — i.e. it is emitting data + * the registered schema does not describe — which is the case P063 exists to + * catch before consumers see it. Returns the problems; empty = conforming. + */ +export function conformanceErrors( + fields: RecordSchemaFields, + payload: Record, +): string[] { + const errors: string[] = []; + for (const [field, spec] of Object.entries(fields)) { + const value = payload[field]; + if (value === undefined || value === null) { + if (spec.required) errors.push(`required field "${field}" is missing`); + continue; + } + if (!matchesType(value, spec.type)) { + errors.push(`field "${field}" is ${describe(value)}, expected ${spec.type}`); + } + } + for (const field of Object.keys(payload)) { + if (!fields[field]) { + errors.push(`unregistered field "${field}" appeared in the payload`); + } + } + return errors; +} + +function matchesType(value: unknown, type: string): boolean { + switch (type) { + case "string": + return typeof value === "string"; + case "number": + return typeof value === "number" && Number.isFinite(value); + case "boolean": + return typeof value === "boolean"; + case "timestamp": + return typeof value === "string" && !Number.isNaN(Date.parse(value)); + case "array": + return Array.isArray(value); + case "object": + return typeof value === "object" && value !== null && !Array.isArray(value); + default: + return false; + } +} + +function describe(value: unknown): string { + if (Array.isArray(value)) return "array"; + return typeof value; +} diff --git a/src/schema/record-schema.ts b/src/schema/record-schema.ts new file mode 100644 index 0000000..e865162 --- /dev/null +++ b/src/schema/record-schema.ts @@ -0,0 +1,65 @@ +/** + * src/schema/record-schema.ts + * ============================ + * P063: the schema 0pt versions per source, and the compatibility policy that + * governs what the next version may change. + * + * Deliberately a SMALL structural schema (field name -> type + required), not + * a second JSON-Schema dialect: 0pt already ships JSON Schema in + * packages/schemas/ for its own contracts, and Zod in src/validation for HTTP + * bodies. Neither of those answers the question P063 actually asks — "is + * version N+1 of an UPSTREAM source's record shape safe for the consumers + * reading version N?" — which needs a comparable, field-level structure, not + * a validator. Anything richer belongs in a real schema registry + * (Confluent/Apicurio) and would be an integration, not a rewrite of this. + */ + +export type FieldType = "string" | "number" | "boolean" | "timestamp" | "object" | "array"; + +export interface FieldSpec { + type: FieldType; + required: boolean; +} + +/** A record shape: field name -> spec. */ +export type RecordSchemaFields = Record; + +/** + * Compatibility policy, using the standard schema-registry vocabulary: + * + * - `backward` — new readers can read data written under the OLD schema. + * Safe: adding an optional field, removing an optional field, relaxing a + * required field to optional. Breaking: adding a REQUIRED field, removing + * a field an old writer still emits as required, changing a type. + * - `forward` — old readers can read data written under the NEW schema. + * Safe: adding a required field, removing an optional field. Breaking: + * removing a REQUIRED field, changing a type. + * - `full` — both of the above. + * - `none` — no check. Recorded, never silently assumed: a source must + * declare `none` explicitly, and the runner logs it as an accepted risk. + */ +export type CompatibilityPolicy = "backward" | "forward" | "full" | "none"; + +export interface RecordSchema { + source_id: string; + tenant_id: string; + schema_version: number; + fields: RecordSchemaFields; + compatibility: CompatibilityPolicy; + registered_at: string; + registered_by?: string; +} + +/** One incompatibility between two schema versions. */ +export interface SchemaBreak { + field: string; + /** Which direction the change breaks. */ + breaks: "backward" | "forward"; + detail: string; +} + +export interface CompatibilityVerdict { + compatible: boolean; + policy: CompatibilityPolicy; + breaks: SchemaBreak[]; +} diff --git a/src/schema/schema-registry.ts b/src/schema/schema-registry.ts new file mode 100644 index 0000000..559f008 --- /dev/null +++ b/src/schema/schema-registry.ts @@ -0,0 +1,144 @@ +/** + * src/schema/schema-registry.ts + * ============================== + * P063: versioning + compatibility policy for a source's record shape. + * + * The DoD is "a breaking schema is detected BEFORE consumer corruption", so + * this registry is placed on the ingestion path in two places, both of them + * upstream of any consumer write: + * + * 1. `registerFromDiscovery` runs when the runner calls connector.discover(), + * i.e. before the first read. A shape change that breaks the declared + * policy makes the RUN fail with the field-level reasons — no envelope is + * produced at all. + * 2. `conformanceErrors` (src/schema/compatibility.ts) runs per envelope, so + * a source that changes shape mid-stream without changing its declared + * schema is caught on the offending record rather than on the next + * discovery, days later. + * + * A source whose policy is "none" opts out of (1) — explicitly, recorded in + * the registry row, never by default. + */ +import { SiaxError } from "../platform/errors"; +import type { IngestionRepository } from "../repositories/ingestion-repository"; +import { checkCompatibility } from "./compatibility"; +import type { + CompatibilityPolicy, + CompatibilityVerdict, + RecordSchema, + RecordSchemaFields, +} from "./record-schema"; + +export interface RegisterSchemaInput { + source_id: string; + tenant_id: string; + fields: RecordSchemaFields; + /** Policy governing THIS and future versions. Required — there is no default. */ + compatibility: CompatibilityPolicy; + registered_by?: string; +} + +export interface RegisterSchemaResult { + schema: RecordSchema; + /** True when the shape was identical to the current version and no new version was created. */ + unchanged: boolean; + /** Present when a comparison ran (i.e. a previous version existed). */ + verdict?: CompatibilityVerdict; +} + +/** Stable comparison of two field maps, order-independent. */ +function sameFields(a: RecordSchemaFields, b: RecordSchemaFields): boolean { + const ka = Object.keys(a).sort(); + const kb = Object.keys(b).sort(); + if (ka.length !== kb.length) return false; + for (let i = 0; i < ka.length; i++) { + if (ka[i] !== kb[i]) return false; + const fa = a[ka[i]]; + const fb = b[kb[i]]; + if (fa.type !== fb.type || fa.required !== fb.required) return false; + } + return true; +} + +export class SchemaRegistry { + constructor(private readonly repo: IngestionRepository) {} + + async getLatest(sourceId: string): Promise { + return this.repo.getLatestSchema(sourceId); + } + + async list(sourceId: string): Promise { + return this.repo.listSchemas(sourceId); + } + + /** + * Register a shape. Throws `SiaxError("CONFLICT")` — naming every break — + * when the new shape violates the previous version's declared policy. + * Nothing is persisted when it throws. + */ + async register(input: RegisterSchemaInput): Promise { + if (Object.keys(input.fields).length === 0) { + throw new SiaxError("INVALID_REQUEST", "0pt.schema_registry", "a schema must declare at least one field"); + } + const previous = await this.repo.getLatestSchema(input.source_id); + if (!previous) { + const schema: RecordSchema = { + source_id: input.source_id, + tenant_id: input.tenant_id, + schema_version: 1, + fields: input.fields, + compatibility: input.compatibility, + registered_at: new Date().toISOString(), + registered_by: input.registered_by, + }; + await this.repo.putSchema(schema); + return { schema, unchanged: false }; + } + + if (sameFields(previous.fields, input.fields) && previous.compatibility === input.compatibility) { + return { schema: previous, unchanged: true }; + } + + // The PREVIOUS version's policy governs whether this change is allowed — + // a producer must not be able to widen its own policy retroactively by + // declaring "none" on the breaking version. + const verdict = checkCompatibility(previous.fields, input.fields, previous.compatibility); + if (!verdict.compatible) { + throw new SiaxError( + "CONFLICT", + "0pt.schema_registry", + `schema v${previous.schema_version + 1} for source ${input.source_id} breaks the declared ` + + `"${previous.compatibility}" compatibility policy: ` + + verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "), + ); + } + + const schema: RecordSchema = { + source_id: input.source_id, + tenant_id: input.tenant_id, + schema_version: previous.schema_version + 1, + fields: input.fields, + compatibility: input.compatibility, + registered_at: new Date().toISOString(), + registered_by: input.registered_by, + }; + await this.repo.putSchema(schema); + return { schema, unchanged: false, verdict }; + } + + /** + * Non-throwing form used by the runner's pre-flight: answers whether the + * shape a connector just discovered would be accepted, without registering + * it. Used to report a break as a RUN failure with a full reason list + * rather than an exception message. + */ + async evaluate( + sourceId: string, + fields: RecordSchemaFields, + ): Promise<{ previous?: RecordSchema; verdict?: CompatibilityVerdict; changed: boolean }> { + const previous = await this.repo.getLatestSchema(sourceId); + if (!previous) return { changed: true }; + if (sameFields(previous.fields, fields)) return { previous, changed: false }; + return { previous, verdict: checkCompatibility(previous.fields, fields, previous.compatibility), changed: true }; + } +} diff --git a/src/services/source-registry-service.ts b/src/services/source-registry-service.ts index e9ec51d..e373ee7 100644 --- a/src/services/source-registry-service.ts +++ b/src/services/source-registry-service.ts @@ -44,6 +44,8 @@ import { SourceService, type SourceServiceOptions } from "./source-service"; import { SourceRatePolicyService } from "./source-rate-policy-service"; import type { CreateSourceInput, Source } from "../types/source"; import type { + SourceAuthorization, + SourceClassification, SourceDescriptor, SourceFreshness, SourceHealth, @@ -51,13 +53,24 @@ import type { SourceHealthStatus, SourceProvenance, SourceRights, + SourceSchedule, } from "../types/source-registry"; export interface RegisterSourceInput extends CreateSourceInput { rights?: Omit; provenance?: Omit; + authorization?: Omit; + schedule?: Omit; } +/** + * Actor ids that may never appear as an authorization's `granted_by`. + * P060 requires an accountable HUMAN decision behind a source going live; + * a service identity granting itself access is exactly the rubber stamp the + * gate exists to prevent. + */ +const NON_ACCOUNTABLE_ACTORS = /^(system|svc_|service:|robot|automation|anonymous)/i; + export class SourceRegistryService { private readonly repo: SourceRegistryRepository; private readonly aud0: Aud0Client; @@ -89,8 +102,16 @@ export class SourceRegistryService { // source.register // ------------------------------------------------------------------------- async registerSource(input: RegisterSourceInput): Promise { - const { rights, provenance, ...base } = input; - const source = await this.sources.createSource(base); + const { rights, provenance, authorization, schedule, ...base } = input; + // P060: registration NEVER activates. A caller asking for status "active" + // gets the source created "pending" and must go through activateSource, + // which checks the classification. Silently honouring `status: "active"` + // here would leave the gate wide open to its most obvious bypass. + const requestedActive = base.status === "active"; + const source = await this.sources.createSource({ + ...base, + status: requestedActive ? "pending" : base.status, + }); if (rights) { await this.setRights({ ...rights, source_id: source.source_id, tenant_id: source.tenant_id }); } @@ -102,9 +123,165 @@ export class SourceRegistryService { updated_at: new Date().toISOString(), }); } + if (authorization) { + await this.setAuthorization({ + ...authorization, + source_id: source.source_id, + tenant_id: source.tenant_id, + }); + } + if (schedule) { + await this.setSchedule({ + ...schedule, + source_id: source.source_id, + tenant_id: source.tenant_id, + updated_at: new Date().toISOString(), + }); + } + // A registration that arrived fully classified AND asked to be active is + // activated here — through the same gate, not around it. + if (requestedActive) { + await this.activateSource(source.source_id, source.tenant_id, base.created_by); + } return this.describeSource(source.source_id, source.tenant_id); } + // ------------------------------------------------------------------------- + // P060 authorization / schedule + // ------------------------------------------------------------------------- + async setAuthorization(auth: SourceAuthorization): Promise { + await this.assertSourceTenant(auth.source_id, auth.tenant_id); + if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.source_registry", + `authorization.granted_by must name an accountable actor, not "${auth.granted_by}"`, + ); + } + await this.repo.setAuthorization(auth); + await this.emit("0pt.source_authorization_set", auth.tenant_id, auth.source_id, { + basis: auth.basis, + granted_by: auth.granted_by, + expires_at: auth.expires_at, + evidence_ref: auth.evidence_ref, + }); + return auth; + } + + async getAuthorization(sourceId: string, tenantId: string): Promise { + await this.assertSourceTenant(sourceId, tenantId); + return this.repo.getAuthorization(sourceId); + } + + async setSchedule(schedule: SourceSchedule): Promise { + await this.assertSourceTenant(schedule.source_id, schedule.tenant_id); + if (schedule.interval_seconds !== undefined && schedule.interval_seconds <= 0) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.source_registry", + "schedule.interval_seconds must be positive", + ); + } + await this.repo.setSchedule(schedule); + await this.emit("0pt.source_schedule_set", schedule.tenant_id, schedule.source_id, { + mode: schedule.mode, + interval_seconds: schedule.interval_seconds, + enabled: schedule.enabled, + }); + return schedule; + } + + async getSchedule(sourceId: string, tenantId: string): Promise { + await this.assertSourceTenant(sourceId, tenantId); + return this.repo.getSchedule(sourceId); + } + + // ------------------------------------------------------------------------- + // P060 classification + activation gate + // ------------------------------------------------------------------------- + /** + * Is this source classified? A source is classified when ALL of the + * following are on record: + * - an accountable owner, + * - an unexpired authorization from an accountable actor, + * - rights: a license id AND at least one allowed purpose, + * - provenance: how the data is obtained. + * + * Everything else in the registry (schedule, schema, checkpoint) describes + * HOW a source runs, not WHETHER it may — those are deliberately not part + * of the gate, and that choice is documented here rather than left implicit. + */ + async classifySource( + sourceId: string, + tenantId: string, + now: Date = new Date(), + ): Promise { + const source = await this.assertSourceTenant(sourceId, tenantId); + const missing: string[] = []; + + if (!source.owner_id) missing.push("owner_id is not set"); + + const auth = await this.repo.getAuthorization(sourceId); + if (!auth) { + missing.push("no authorization on record"); + } else { + if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) { + missing.push(`authorization.granted_by "${auth.granted_by}" is not an accountable actor`); + } + if (auth.expires_at && new Date(auth.expires_at).getTime() <= now.getTime()) { + missing.push(`authorization expired at ${auth.expires_at}`); + } + } + + const rights = await this.repo.getRights(sourceId); + if (!rights) { + missing.push("no rights record (license/purpose) on record"); + } else { + if (!rights.license_id) missing.push("rights.license_id is empty"); + if (rights.allowed_purposes.length === 0) missing.push("rights.allowed_purposes is empty"); + } + + const provenance = await this.repo.getProvenance(sourceId); + if (!provenance) missing.push("no provenance record (sourcing method) on record"); + + return { source_id: sourceId, classified: missing.length === 0, missing }; + } + + /** + * P060 DoD: an UNCLASSIFIED source cannot be activated. Fail-closed — + * every missing fact is named at once, and nothing is written when the + * check fails. + */ + async activateSource(sourceId: string, tenantId: string, actorId?: string): Promise { + const classification = await this.classifySource(sourceId, tenantId); + if (!classification.classified) { + await this.emit("0pt.source_activation_denied", tenantId, sourceId, { + missing: classification.missing, + }); + throw new SiaxError( + "AUTHORIZATION_DENIED", + "0pt.source_registry", + `source ${sourceId} is not classified and cannot be activated: ${classification.missing.join("; ")}`, + ); + } + const source = await this.sources.updateStatus(sourceId, "active", actorId); + await this.emit("0pt.source_activated", tenantId, sourceId, { activated_by: actorId ?? null }); + return source; + } + + /** Take a source out of service. Always allowed — deactivating is never gated. */ + async deactivateSource( + sourceId: string, + tenantId: string, + status: "paused" | "archived" | "deprecated" = "paused", + actorId?: string, + ): Promise { + await this.assertSourceTenant(sourceId, tenantId); + const source = await this.sources.updateStatus(sourceId, status, actorId); + await this.emit("0pt.source_deactivated", tenantId, sourceId, { status }); + return source; + } + // ------------------------------------------------------------------------- // rights / license / purpose metadata // ------------------------------------------------------------------------- @@ -262,13 +439,17 @@ export class SourceRegistryService { async describeSource(sourceId: string, tenantId: string): Promise { const source = await this.assertSourceTenant(sourceId, tenantId); - const [policies, rights, provenance, health, freshness] = await Promise.all([ - this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }), - this.repo.getRights(sourceId), - this.repo.getProvenance(sourceId), - this.getHealth(sourceId, tenantId), - this.getFreshness(sourceId, tenantId), - ]); + const [policies, rights, provenance, health, freshness, authorization, schedule, classification] = + await Promise.all([ + this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }), + this.repo.getRights(sourceId), + this.repo.getProvenance(sourceId), + this.getHealth(sourceId, tenantId), + this.getFreshness(sourceId, tenantId), + this.repo.getAuthorization(sourceId), + this.repo.getSchedule(sourceId), + this.classifySource(sourceId, tenantId), + ]); return { source, rights, @@ -276,6 +457,9 @@ export class SourceRegistryService { health, freshness, rate_policy: policies[0], + authorization, + schedule, + classification, }; } diff --git a/src/services/source-registry.ts b/src/services/source-registry.ts index 50121ff..e68d719 100644 --- a/src/services/source-registry.ts +++ b/src/services/source-registry.ts @@ -28,3 +28,19 @@ export const sourceRegistryService = new SourceRegistryService({ sourceService, ratePolicyService: sourceRatePolicyService, }); + +/** + * P060-P068 ingestion plane singletons. Same reasoning as above: the schema + * registry, checkpoint store, ledger, quarantine and lineage projection all + * share ONE repository so the in-memory dev/test mode is coherent within a + * process, and one `resolveStoreMode` decision in production. + */ +import { createIngestionRepository } from "../repositories/ingestion-repository"; +import { SchemaRegistry } from "../schema/schema-registry"; +import { LineageStore } from "../lineage/lineage-store"; +import { IngestionRunner } from "../ingestion/ingestion-runner"; + +export const ingestionRepository = createIngestionRepository(); +export const schemaRegistry = new SchemaRegistry(ingestionRepository); +export const lineageStore = new LineageStore(ingestionRepository); +export const ingestionRunner = new IngestionRunner(ingestionRepository, schemaRegistry, lineageStore); diff --git a/src/services/source-service.ts b/src/services/source-service.ts index 5eade69..b114f10 100644 --- a/src/services/source-service.ts +++ b/src/services/source-service.ts @@ -77,6 +77,32 @@ export class SourceService { return this.repo.get(sourceId); } + /** + * P060: the ONLY status transition path. Callers do not get a generic + * update method, because status is what the activation gate protects — + * SourceRegistryService.activateSource is the only caller that moves a + * source to "active", and it runs the classification check first. + */ + async updateStatus(sourceId: string, status: SourceStatus, actorId?: string): Promise { + const source = await this.repo.updateStatus(sourceId, status); + await this.aud0.ingestEvent({ + event_id: `evt_0pt_source_status_${source.source_id}_${Date.now()}`, + event_type: "0pt.source_status_changed", + tenant_id: source.tenant_id, + project_id: source.project_id, + actor_id: actorId ?? source.owner_id ?? "system", + app_id: "0pt", + capability_id: "0pt.update_source_status", + resource_type: "source", + resource_id: source.source_id, + risk_level: "L2", + payload: { status: source.status, name: source.name }, + created_at: source.updated_at ?? new Date().toISOString(), + correlation_id: source.correlation_id, + }); + return source; + } + async listSources(filter?: { tenant_id?: string; project_id?: string; diff --git a/src/types/ingestion.ts b/src/types/ingestion.ts new file mode 100644 index 0000000..1f265a8 --- /dev/null +++ b/src/types/ingestion.ts @@ -0,0 +1,111 @@ +/** + * src/types/ingestion.ts + * ======================= + * P062: THE common ingestion envelope. Every record that enters 0pt through + * a connector — batch page, stream message, or change event — is normalized + * into exactly this shape before anything downstream (quality gate, entity + * resolution, dataset write, lineage) sees it. Nothing downstream is allowed + * to special-case "this came from a file" vs "this came from a change feed"; + * that difference is expressed by `mode` + `operation` and nothing else. + * + * HONEST SCOPE NOTE. `mode: "cdc"` and the delete/update operations are part + * of the CONTRACT, and the runner handles them (a delete envelope carries no + * payload and is never quality-gated as if it had one). This repo does NOT + * ship a CDC *source* — there is no Debezium consumer, no replication-slot + * reader, no WAL tailer here, exactly as services/api/tests/test_known_gaps.py + * has stated since P010. What changed with P062 is that a CDC connector can + * now be added without redesigning the pipeline, and that the checkpoint it + * would resume from is durable. A CDC connector itself is still MISSING. + * + * `checkpoint` on a ReadResult is what makes resume real: the connector hands + * back the cursor that means "everything up to and including these envelopes + * has been produced", the runner commits it durably AFTER the envelopes are + * accepted, and a restart passes it straight back into read(). Committing it + * before acceptance would create silent gaps — that ordering is the whole + * point and is asserted in tests/p0/ingestion-runner.test.ts. + */ + +/** How data arrives from the source. */ +export type IngestionMode = "batch" | "stream" | "cdc"; + +/** + * What the record means. Batch/stream sources normally emit "upsert" + * (they carry current state); a change feed distinguishes the three CDC + * operations. "delete" envelopes carry `payload: null`. + */ +export type EnvelopeOperation = "upsert" | "insert" | "update" | "delete"; + +/** Per-envelope provenance — which code, which version, which run produced it. */ +export interface EnvelopeProvenance { + connector_id: string; + connector_version: string; + /** The ingestion run that produced this envelope. Joins to lineage + checkpoints. */ + run_id: string; + /** Opaque connector cursor at which this record was produced. */ + cursor?: string; +} + +/** One normalized record. `payload` is null only for `operation: "delete"`. */ +export interface IngestionEnvelope { + envelope_id: string; + source_id: string; + tenant_id: string; + project_id: string; + mode: IngestionMode; + operation: EnvelopeOperation; + /** Partition/stream within the source (a table, a topic, a file). */ + partition_key: string; + /** Stable business key within the partition — the unit of idempotency (P068). */ + record_key: string; + /** Schema version the payload conforms to (P063). */ + schema_version: number; + /** Monotonic per-(source, partition) sequence assigned by the runner. */ + sequence: number; + /** When the change happened upstream, when known. */ + occurred_at?: string; + /** When 0pt normalized it. */ + ingested_at: string; + payload: Record | null; + provenance: EnvelopeProvenance; +} + +/** + * The durable resume point for one (source, partition). Committed only after + * the envelopes it covers have been accepted downstream. + */ +export interface IngestionCheckpoint { + source_id: string; + tenant_id: string; + partition_key: string; + /** Opaque connector cursor. `undefined` means "never read". */ + cursor?: string; + /** Highest envelope sequence committed for this partition. */ + sequence: number; + records_committed: number; + committed_at: string; +} + +/** + * P064: an envelope the declared quality policy sent to quarantine instead of + * downstream. Quarantine is DURABLE and inspectable — a dropped record that + * only exists in a log line is a silently lost record. + */ +export interface QuarantineRecord { + quarantine_id: string; + source_id: string; + tenant_id: string; + run_id: string; + record_key: string; + /** One entry per failed expectation. */ + violations: Array<{ expectation_type: string; detail: string }>; + envelope: IngestionEnvelope; + quarantined_at: string; +} + +/** P068: what the idempotency ledger decided about one delivered record. */ +export type LedgerVerdict = "new" | "changed" | "duplicate"; + +export interface LedgerDecision { + record_key: string; + verdict: LedgerVerdict; +} diff --git a/src/types/source-registry.ts b/src/types/source-registry.ts index 8fb17c0..65e65d9 100644 --- a/src/types/source-registry.ts +++ b/src/types/source-registry.ts @@ -87,7 +87,11 @@ export interface SourceFreshness { age_hours?: number; } -/** The full `source.describe` document (§10): base + rights + provenance + health + freshness + rate policy. */ +/** + * The full `source.describe` document (§10): base + rights + provenance + + * health + freshness + rate policy, plus the P060 additions (authorization, + * schedule, and the classification verdict the activation gate applies). + */ export interface SourceDescriptor { source: import("./source").Source; rights?: SourceRights; @@ -95,4 +99,67 @@ export interface SourceDescriptor { health: SourceHealth; freshness: SourceFreshness; rate_policy?: import("./source-rate-policy").SourceRatePolicy; + authorization?: SourceAuthorization; + schedule?: SourceSchedule; + /** Why this source may (or may not) be activated. */ + classification?: SourceClassification; +} + +// --------------------------------------------------------------------------- +// P060: the registry facts that were missing from the original §10 pass — +// authorization, schedule, schema reference, checkpoint — plus the +// classification verdict the activation gate produces. +// --------------------------------------------------------------------------- + +/** Legal/contractual ground on which this tenant may connect to the source at all. */ +export type AuthorizationBasis = + | "contract" + | "public_register" + | "consent" + | "legitimate_interest" + | "internal"; + +/** + * The recorded decision that 0pt MAY connect to this source (P060 + * "authorization"). Deliberately separate from {@link SourceRights}: rights + * say what the DATA may be used for, authorization says who signed off that + * the CONNECTION may exist. A source needs both to be classified. + */ +export interface SourceAuthorization { + source_id: string; + tenant_id: string; + basis: AuthorizationBasis; + /** Accountable actor id — never "system"/"svc_*" (the gate rejects those). */ + granted_by: string; + granted_at: string; + /** ISO timestamp; an authorization past this instant does not classify the source. */ + expires_at?: string; + /** AUD0 evidence ref or contract artifact ref backing the decision. */ + evidence_ref?: string; + notes?: string; +} + +/** How often 0pt pulls from the source, and in which ingestion mode (P060 "schedule"). */ +export interface SourceSchedule { + source_id: string; + tenant_id: string; + mode: "batch" | "stream" | "cdc"; + /** Poll cadence in seconds. Omitted for a continuously-open stream. */ + interval_seconds?: number; + /** Optional cron expression. Recorded here; 0pt does not execute cron itself (ACT0 owns scheduling). */ + cron?: string; + enabled: boolean; + updated_at: string; +} + +/** + * Why a source is (not) classified. `activateSource` refuses on + * `classified === false` and names every missing fact, so an operator gets + * the whole list at once instead of one error per round trip. + */ +export interface SourceClassification { + source_id: string; + classified: boolean; + /** Human-readable reasons, one per missing/expired fact. Empty when classified. */ + missing: string[]; } diff --git a/src/types/source.ts b/src/types/source.ts index 2489b40..62f356e 100644 --- a/src/types/source.ts +++ b/src/types/source.ts @@ -34,8 +34,16 @@ export type SourceType = "database" | "file" | "web" | "api" | "stream" | "manua * Union of both status vocabularies. "archived" and "deprecated" both survive * the unification merge -- they were introduced independently and no code * treats either as canonical. + * + * "pending" (P060) is the state every newly registered source starts in. It + * exists because "active" used to be the CREATE default + * (postgres-source-repository.ts / in-memory-source-repository.ts both did + * `input.status ?? "active"`), which meant a source with no rights, no + * provenance and no authorization was live the instant it was registered. + * Reaching "active" now requires passing SourceRegistryService.activateSource's + * classification gate. */ -export type SourceStatus = "active" | "paused" | "archived" | "deprecated"; +export type SourceStatus = "pending" | "active" | "paused" | "archived" | "deprecated"; /** Legal-sourcing vocabulary, mirroring dim_source.access. */ export type SourceAccess = "api" | "bulk" | "authorized_web" | "licensed"; @@ -60,7 +68,14 @@ export interface Source { correlation_id?: string; created_at: string; updated_at?: string; + /** Who registered the source. */ created_by?: string; + /** + * Who is ACCOUNTABLE for the source going forward (P060 "owner"). Distinct + * from `created_by`, which is only who typed the registration. Required by + * the activation gate. + */ + owner_id?: string; } export interface CreateSourceInput { @@ -75,4 +90,5 @@ export interface CreateSourceInput { status?: SourceStatus; correlation_id?: string; created_by?: string; + owner_id?: string; } diff --git a/src/validation/0pt-schemas.ts b/src/validation/0pt-schemas.ts index eb42f9e..a94989e 100644 --- a/src/validation/0pt-schemas.ts +++ b/src/validation/0pt-schemas.ts @@ -127,7 +127,8 @@ export const registerSourceSchema = z.object({ adapter_id: z.string().min(1).optional(), access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(), description: z.string().min(1).optional(), - status: z.enum(['active', 'paused', 'archived', 'deprecated']).optional(), + status: z.enum(['pending', 'active', 'paused', 'archived', 'deprecated']).optional(), + owner_id: z.string().min(1).optional(), correlation_id: z.string().min(1).optional(), created_by: z.string().min(1).optional(), rights: sourceRightsInputSchema.optional(), @@ -232,3 +233,33 @@ export const qualityBridgeRunSchema = z.object({ rows: z.array(jsonObject), correlation_id: z.string().min(1).optional(), }); + +/** Nested `authorization` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/authorization (P060). */ +const sourceAuthorizationInputSchema = z.object({ + basis: z.enum(['contract', 'public_register', 'consent', 'legitimate_interest', 'internal']), + granted_by: z.string().min(1), + granted_at: z.string().min(1), + expires_at: z.string().min(1).optional(), + evidence_ref: z.string().min(1).optional(), + notes: z.string().min(1).optional(), +}); + +/** PUT /api/0pt/sources/[sourceId]/authorization -- source.set_authorization (P060). */ +export const setSourceAuthorizationSchema = sourceAuthorizationInputSchema.extend({ + tenant_id: z.string().min(1), +}); + +/** PUT /api/0pt/sources/[sourceId]/schedule -- source.set_schedule (P060). */ +export const setSourceScheduleSchema = z.object({ + tenant_id: z.string().min(1), + mode: z.enum(['batch', 'stream', 'cdc']), + interval_seconds: z.number().int().positive().optional(), + cron: z.string().min(1).optional(), + enabled: z.boolean(), +}); + +/** POST /api/0pt/sources/[sourceId]/activate -- source.activate (P060 classification gate). */ +export const activateSourceSchema = z.object({ + tenant_id: z.string().min(1), + actor_id: z.string().min(1).optional(), +}); diff --git a/tests/api/0pt-p060-routes.test.ts b/tests/api/0pt-p060-routes.test.ts new file mode 100644 index 0000000..9259bda --- /dev/null +++ b/tests/api/0pt-p060-routes.test.ts @@ -0,0 +1,175 @@ +/** + * HTTP-level tests for the P060/P067 routes added in this pass, in the same + * shape as tests/api/0pt-cross-tenant-routes.test.ts: call the handler + * directly with a hand-built NextRequest carrying `x-0pt-verified-tenant`, + * which is exactly what middleware.ts forwards once a real Bearer token has + * been verified. + * + * Two things are proved here that the service-level tests cannot: + * - the new routes carry the SAME cross-tenant guard as the existing ones + * (a new route with no guard would be the P014 hole reopened), and + * - the activation gate is enforced at the HTTP boundary too — an + * unclassified source gets 403, not a 200 with a quiet no-op. + */ +import { describe, expect, it } from "vitest"; +import { NextRequest } from "next/server"; +import { POST as sourcesPOST } from "@/app/api/0pt/sources/route"; +import { GET as classifyGET, POST as activatePOST } from "@/app/api/0pt/sources/[sourceId]/activate/route"; +import { PUT as authorizationPUT } from "@/app/api/0pt/sources/[sourceId]/authorization/route"; +import { PUT as schedulePUT } from "@/app/api/0pt/sources/[sourceId]/schedule/route"; +import { GET as lineageGET } from "@/app/api/0pt/lineage/route"; + +const VERIFIED = "x-0pt-verified-tenant"; + +function getReq(url: string, verifiedTenant = "t_a"): NextRequest { + return new NextRequest(new URL(url, "http://localhost"), { headers: { [VERIFIED]: verifiedTenant } }); +} +function bodyReq(url: string, method: "POST" | "PUT", body: unknown, verifiedTenant = "t_a"): NextRequest { + return new NextRequest(new URL(url, "http://localhost"), { + method, + headers: { "content-type": "application/json", [VERIFIED]: verifiedTenant }, + body: JSON.stringify(body), + }); +} +const params = (sourceId: string) => ({ params: Promise.resolve({ sourceId }) }); + +describe("P060 routes — cross-tenant guard", () => { + it("POST /activate rejects a foreign tenant_id (403)", async () => { + const res = await activatePOST( + bodyReq("/api/0pt/sources/src_x/activate", "POST", { tenant_id: "t_victim" }), + params("src_x"), + ); + expect(res.status).toBe(403); + expect((await res.json()).error).toBe("OPT_ACCESS_DENIED"); + }); + + it("GET /activate (classification) rejects a foreign tenant_id (403)", async () => { + const res = await classifyGET( + getReq("/api/0pt/sources/src_x/activate?tenant_id=t_victim"), + params("src_x"), + ); + expect(res.status).toBe(403); + }); + + it("PUT /authorization rejects a foreign tenant_id (403)", async () => { + const res = await authorizationPUT( + bodyReq("/api/0pt/sources/src_x/authorization", "PUT", { + tenant_id: "t_victim", + basis: "contract", + granted_by: "usr_attacker", + granted_at: "2026-09-04T00:00:00.000Z", + }), + params("src_x"), + ); + expect(res.status).toBe(403); + }); + + it("PUT /schedule rejects a foreign tenant_id (403)", async () => { + const res = await schedulePUT( + bodyReq("/api/0pt/sources/src_x/schedule", "PUT", { tenant_id: "t_victim", mode: "batch", enabled: true }), + params("src_x"), + ); + expect(res.status).toBe(403); + }); + + it("GET /lineage rejects a foreign tenant_id (403)", async () => { + const res = await lineageGET(getReq("/api/0pt/lineage?tenant_id=t_victim&output_name=0pt:dataset:ds_x")); + expect(res.status).toBe(403); + }); + + it("GET /lineage refuses an unfiltered dump (400)", async () => { + const res = await lineageGET(getReq("/api/0pt/lineage?tenant_id=t_a")); + expect(res.status).toBe(400); + }); +}); + +describe("P060 routes — the activation gate over HTTP", () => { + it("an unclassified source cannot be activated through the API", async () => { + const created = await sourcesPOST( + bodyReq("/api/0pt/sources", "POST", { + source_id: "src_http_gate_1", + tenant_id: "t_a", + project_id: "p_a", + source_type: "api", + name: "Unclassified source", + }), + ); + expect(created.status).toBeLessThan(300); + const body = await created.json(); + expect(body.source.status).toBe("pending"); + + const classification = await classifyGET( + getReq("/api/0pt/sources/src_http_gate_1/activate?tenant_id=t_a"), + params("src_http_gate_1"), + ); + expect(classification.status).toBe(200); + expect((await classification.json()).classified).toBe(false); + + const activated = await activatePOST( + bodyReq("/api/0pt/sources/src_http_gate_1/activate", "POST", { tenant_id: "t_a" }), + params("src_http_gate_1"), + ); + expect(activated.status).toBe(403); + }); + + it("a classified source activates through the API", async () => { + await sourcesPOST( + bodyReq("/api/0pt/sources", "POST", { + source_id: "src_http_gate_2", + tenant_id: "t_a", + project_id: "p_a", + source_type: "api", + name: "Classified source", + owner_id: "usr_simon", + rights: { + license_id: "CC-BY-4.0", + allowed_purposes: ["catalog"], + pii_allowed: false, + redistribution: "internal", + verified_at: "2026-09-04", + verified_by: "usr_simon", + }, + provenance: { method: "official_api" }, + }), + ); + const auth = await authorizationPUT( + bodyReq("/api/0pt/sources/src_http_gate_2/authorization", "PUT", { + tenant_id: "t_a", + basis: "public_register", + granted_by: "usr_simon", + granted_at: "2026-09-01T00:00:00.000Z", + }), + params("src_http_gate_2"), + ); + expect(auth.status).toBe(200); + + const activated = await activatePOST( + bodyReq("/api/0pt/sources/src_http_gate_2/activate", "POST", { tenant_id: "t_a", actor_id: "usr_simon" }), + params("src_http_gate_2"), + ); + expect(activated.status).toBe(200); + expect((await activated.json()).status).toBe("active"); + }); + + it("a service identity cannot be recorded as the authorizing actor (400)", async () => { + await sourcesPOST( + bodyReq("/api/0pt/sources", "POST", { + source_id: "src_http_gate_3", + tenant_id: "t_a", + project_id: "p_a", + source_type: "api", + name: "Self-authorizing source", + }), + ); + const res = await authorizationPUT( + bodyReq("/api/0pt/sources/src_http_gate_3/authorization", "PUT", { + tenant_id: "t_a", + basis: "internal", + granted_by: "svc_0pt", + granted_at: "2026-09-01T00:00:00.000Z", + }), + params("src_http_gate_3"), + ); + expect(res.status).toBe(400); + }); +}); diff --git a/tests/p0/connector-conformance.test.ts b/tests/p0/connector-conformance.test.ts new file mode 100644 index 0000000..ed63b42 --- /dev/null +++ b/tests/p0/connector-conformance.test.ts @@ -0,0 +1,165 @@ +/** + * P061 DoD: "two heterogeneous connectors pass the same conformance suite." + * + * This file is that proof. It runs the SAME function + * (src/connectors/conformance.ts::runConnectorConformance) against two + * connectors that share nothing but the contract: + * + * HttpJsonConnector — a real node:http server on a real socket, opaque + * cursor paging, network health check, timestamp + * backfill bounds. + * JsonlFileConnector — a real file on disk, line-offset cursor, filesystem + * health check, line-number backfill bounds, and CDC + * delete records the HTTP connector cannot produce. + * + * Neither connector is mocked. The HTTP fixture is a genuine server that + * implements paging server-side; the file fixture is a genuine file written + * to a temp dir. A connector that only "passed" against a stub would prove + * nothing about the contract. + */ +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import http from "node:http"; +import type { AddressInfo } from "node:net"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { runConnectorConformance, type ConformanceFixture } from "@/src/connectors/conformance"; +import { HttpJsonConnector } from "@/src/connectors/http-json-connector"; +import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; + +const SCHEMA: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; + +/** 5 records, shared by both fixtures so the suite's counts are comparable. */ +const ROWS = [ + { org_number: "556001-0001", name: "Alfa AB", employees: 10, seq: 1 }, + { org_number: "556002-0002", name: "Beta AB", employees: 20, seq: 2 }, + { org_number: "556003-0003", name: "Gamma AB", employees: 30, seq: 3 }, + { org_number: "556004-0004", name: "Delta AB", employees: 40, seq: 4 }, + { org_number: "556005-0005", name: "Epsilon AB", employees: 50, seq: 5 }, +]; + +// --- fixture 1: a real paged HTTP API --------------------------------------- + +let server: http.Server; +let baseUrl = ""; + +beforeAll(async () => { + server = http.createServer((req, res) => { + const url = new URL(req.url ?? "/", "http://localhost"); + if (url.pathname !== "/v1/records") { + res.writeHead(404).end("{}"); + return; + } + const limit = Number(url.searchParams.get("limit") ?? "100"); + const from = url.searchParams.get("from"); + const to = url.searchParams.get("to"); + if (from !== null && to !== null) { + // Backfill: bounds are `seq` values, from inclusive, to exclusive. + const slice = ROWS.filter((r) => r.seq >= Number(from) && r.seq < Number(to)).slice(0, limit); + res.writeHead(200, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ records: slice, next_cursor: null })); + return; + } + // Cursor is the `seq` of the last record already delivered. + const after = Number(url.searchParams.get("cursor") ?? "0"); + const slice = ROWS.filter((r) => r.seq > after).slice(0, limit); + const next = slice.length > 0 ? String(slice[slice.length - 1].seq) : null; + const more = slice.length > 0 && slice[slice.length - 1].seq < ROWS[ROWS.length - 1].seq; + res.writeHead(200, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ records: slice, next_cursor: more ? next : null })); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + baseUrl = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; +}); + +afterAll(async () => { + await new Promise((resolve) => server.close(() => resolve())); + await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined); +}); + +// --- fixture 2: a real JSONL file ------------------------------------------- + +const tmpDir = path.join(os.tmpdir(), `opt-conformance-${process.pid}`); +const jsonlPath = path.join(tmpDir, "companies.jsonl"); + +beforeAll(async () => { + await fs.mkdir(tmpDir, { recursive: true }); + await fs.writeFile( + jsonlPath, + ROWS.map((r) => JSON.stringify({ org_number: r.org_number, name: r.name, employees: r.employees })).join("\n") + "\n", + "utf8", + ); +}); + +const ctx = { tenant_id: "t_test", project_id: "p_test", source_id: "src_conf", run_id: "run_conf" }; + +describe("P061 connector contract — the same conformance suite, two heterogeneous connectors", () => { + it("HttpJsonConnector passes every check", async () => { + const fixture: ConformanceFixture = { + validConfig: { + base_url: baseUrl, + records_path: "/v1/records", + partition_key: "companies", + schema: SCHEMA, + record_key_field: "org_number", + cursor_field: "seq", + }, + // Not an absolute URL and no schema — must be rejected without any IO. + invalidConfig: { base_url: "not-a-url", records_path: "v1/records", partition_key: "", record_key_field: "" }, + ctx, + backfill: { from: "2", to: "4" }, + expectedRecordCount: ROWS.length, + }; + const report = await runConnectorConformance(new HttpJsonConnector(), fixture); + const failed = report.checks.filter((c) => !c.passed); + expect(failed.map((c) => `${c.name}: ${c.detail}`)).toEqual([]); + expect(report.passed).toBe(true); + expect(report.checks.length).toBeGreaterThanOrEqual(14); + }); + + it("JsonlFileConnector passes every check — the identical suite", async () => { + const fixture: ConformanceFixture = { + validConfig: { + file_path: jsonlPath, + partition_key: "companies", + schema: SCHEMA, + record_key_field: "org_number", + }, + // Relative path, wrong extension, no schema. + invalidConfig: { file_path: "companies.csv", partition_key: "", record_key_field: "" }, + ctx, + // File-connector bounds are LINE NUMBERS — deliberately a different + // vocabulary from the HTTP fixture's timestamps/sequence values, which + // is exactly the heterogeneity the shared contract has to survive. + backfill: { from: "1", to: "3" }, + expectedRecordCount: ROWS.length, + }; + const report = await runConnectorConformance(new JsonlFileConnector(), fixture); + const failed = report.checks.filter((c) => !c.passed); + expect(failed.map((c) => `${c.name}: ${c.detail}`)).toEqual([]); + expect(report.passed).toBe(true); + }); + + it("the suite actually FAILS a connector that breaks the contract (it is not decoration)", async () => { + // A connector that accepts any config at all — the classic fail-open bug. + class FailOpenConnector extends JsonlFileConnector { + override validate() { + return { valid: true, errors: [] }; + } + } + const report = await runConnectorConformance(new FailOpenConnector(), { + validConfig: { file_path: jsonlPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + invalidConfig: { file_path: "nope.csv", partition_key: "", record_key_field: "" }, + ctx, + backfill: { from: "1", to: "3" }, + expectedRecordCount: ROWS.length, + }); + expect(report.passed).toBe(false); + expect(report.checks.find((c) => c.name.startsWith("validate: rejects"))?.passed).toBe(false); + }); +}); diff --git a/tests/p0/ingestion-plane-postgres.realinfra.test.ts b/tests/p0/ingestion-plane-postgres.realinfra.test.ts new file mode 100644 index 0000000..3a947a6 --- /dev/null +++ b/tests/p0/ingestion-plane-postgres.realinfra.test.ts @@ -0,0 +1,310 @@ +/** + * REAL Postgres proof for P060 DoD #1 ("a restart preserves every source") + * and for the durability the rest of the plane depends on (P062 checkpoints, + * P063 schema versions, P064 quarantine, P067 lineage, P068 ledger). + * + * Same honest-degrade convention as tests/p0/source-registry-postgres.realinfra.test.ts: + * gated on an env var, SKIPS HONESTLY when unset, never fabricates success. + * + * docker run -d --name opt-ingestion-pgtest \ + * -e POSTGRES_USER=opt -e POSTGRES_PASSWORD=opt_test_pw -e POSTGRES_DB=opt_test \ + * -p 15434:5432 postgres:16-alpine + * INGESTION_TEST_DATABASE_URL=postgres://opt:opt_test_pw@localhost:15434/opt_test \ + * pnpm vitest run tests/p0/ingestion-plane-postgres.realinfra.test.ts + * + * "RESTART" here is modelled the only way a test can model it without killing + * the runner: every assertion after a write is made through a FRESHLY + * CONSTRUCTED service/repository over a NEW connection, so nothing can be + * answered out of an in-process Map. The stronger proof — stopping and + * starting the Postgres CONTAINER itself and re-reading — was run manually + * during this pass and is recorded in STATUS.md; it is not automated here + * because a test that restarts its own database would need docker access + * from inside CI, which these runners do not grant. + */ +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import fs from "node:fs"; +import path from "node:path"; +import postgres from "postgres"; +import { SourceRegistryService } from "@/src/services/source-registry-service"; +import { SourceService } from "@/src/services/source-service"; +import { PostgresSourceRepository } from "@/src/repositories/postgres-source-repository"; +import { PostgresSourceRegistryRepository } from "@/src/repositories/postgres-source-registry-repository"; +import { PostgresIngestionRepository } from "@/src/repositories/postgres-ingestion-repository"; +import { SchemaRegistry } from "@/src/schema/schema-registry"; +import { LineageStore } from "@/src/lineage/lineage-store"; +import { IngestionRunner } from "@/src/ingestion/ingestion-runner"; +import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector"; +import { SiaxError } from "@/src/platform/errors"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; +import os from "node:os"; +import { promises as fsp } from "node:fs"; + +const TEST_DATABASE_URL = process.env.INGESTION_TEST_DATABASE_URL; +const run = TEST_DATABASE_URL ? describe : describe.skip; + +let adminSql: ReturnType | undefined; +let reachable = false; +const tmpDir = path.join(os.tmpdir(), `opt-realinfra-${process.pid}`); +const filePath = path.join(tmpDir, "companies.jsonl"); + +const SCHEMA: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; +const ROWS = Array.from({ length: 6 }, (_, i) => ({ + org_number: `55611${i}-111${i}`, + name: `Realinfra ${i} AB`, + employees: (i + 1) * 10, +})); + +const TENANT = "t_realinfra"; +const SOURCE = "src_realinfra_1"; + +beforeAll(async () => { + if (!TEST_DATABASE_URL) { + console.warn( + "[realinfra] INGESTION_TEST_DATABASE_URL not set this run -- skipping honestly, not fabricating success.", + ); + return; + } + adminSql = postgres(TEST_DATABASE_URL, { prepare: false, ssl: false, max: 2 }); + await Promise.race([ + adminSql`SELECT 1`, + new Promise((_, reject) => setTimeout(() => reject(new Error("connect timeout")), 5000)), + ]); + reachable = true; + // Only the two migrations this plane needs. sql/001_init.sql is skipped on + // purpose: it requires the `vector` extension (pgvector), which a plain + // postgres:16-alpine does not ship, and none of the tables under test + // depend on it. Same choice tests/p0/source-registry-postgres.realinfra.test.ts + // already makes. + for (const file of ["002_source_registry.sql", "007_ingestion_plane.sql"]) { + const p = path.join(process.cwd(), "sql", file); + if (!fs.existsSync(p)) continue; + await adminSql.unsafe(fs.readFileSync(p, "utf8")); + } + // Clean slate for this suite's ids only — never a blanket TRUNCATE. + await adminSql`DELETE FROM opt_lineage_edges WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_quality_quarantine WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_ingestion_ledger WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_ingestion_checkpoints WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_schemas WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_authorization WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_schedule WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_rights WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_provenance WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_freshness WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_health_checks WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_sources WHERE tenant_id = ${TENANT}`; + + await fsp.mkdir(tmpDir, { recursive: true }); + await fsp.writeFile(filePath, ROWS.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8"); +}); + +afterAll(async () => { + await adminSql?.end({ timeout: 5 }); + await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined); +}); + +/** A brand-new service over a brand-new connection — the "restarted process". */ +function freshRegistry(): SourceRegistryService { + const sql = postgres(TEST_DATABASE_URL as string, { prepare: false, ssl: false, max: 2 }); + return new SourceRegistryService({ + repository: new PostgresSourceRegistryRepository(sql), + sourceService: new SourceService({ repository: new PostgresSourceRepository(sql) }), + }); +} + +function freshIngestion(): { + repo: PostgresIngestionRepository; + runner: IngestionRunner; + lineage: LineageStore; +} { + const sql = postgres(TEST_DATABASE_URL as string, { prepare: false, ssl: false, max: 2 }); + const repo = new PostgresIngestionRepository(sql); + const lineage = new LineageStore(repo); + return { repo, runner: new IngestionRunner(repo, new SchemaRegistry(repo), lineage), lineage }; +} + +run("P060 real Postgres — a restart preserves every source and every registry fact", () => { + it("is reachable (otherwise the rest of this suite proves nothing)", () => { + expect(reachable).toBe(true); + }); + + it("registers pending, refuses activation unclassified, then activates — all surviving a fresh process", async () => { + const writer = freshRegistry(); + await writer.registerSource({ + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + source_type: "file", + name: "Realinfra file drop", + owner_id: "usr_simon", + schedule: { mode: "batch", interval_seconds: 900, enabled: true }, + }); + + // A DIFFERENT service instance over a DIFFERENT connection. + const reader = freshRegistry(); + expect((await reader.sources.getSource(SOURCE)).status).toBe("pending"); + await expect(reader.activateSource(SOURCE, TENANT)).rejects.toBeInstanceOf(SiaxError); + + const classifier = freshRegistry(); + await classifier.setRights({ + source_id: SOURCE, + tenant_id: TENANT, + license_id: "proprietary:realinfra", + allowed_purposes: ["catalog"], + pii_allowed: false, + redistribution: "internal", + verified_at: "2026-09-04", + verified_by: "usr_simon", + }); + await classifier.setProvenance({ + source_id: SOURCE, + tenant_id: TENANT, + method: "bulk_download", + updated_at: new Date().toISOString(), + }); + await classifier.setAuthorization({ + source_id: SOURCE, + tenant_id: TENANT, + basis: "contract", + granted_by: "usr_simon", + granted_at: new Date().toISOString(), + }); + + const activator = freshRegistry(); + expect((await activator.classifySource(SOURCE, TENANT)).classified).toBe(true); + expect((await activator.activateSource(SOURCE, TENANT, "usr_simon")).status).toBe("active"); + + // Read back through yet another fresh process AND directly in SQL, so the + // assertion cannot be satisfied by any in-process bookkeeping. + const verifier = freshRegistry(); + const descriptor = await verifier.describeSource(SOURCE, TENANT); + expect(descriptor.source.status).toBe("active"); + expect(descriptor.source.owner_id).toBe("usr_simon"); + expect(descriptor.authorization?.basis).toBe("contract"); + expect(descriptor.schedule?.interval_seconds).toBe(900); + expect(descriptor.classification?.classified).toBe(true); + + const rows = await (adminSql as ReturnType)< + { status: string; owner_id: string | null }[] + >`SELECT status, owner_id FROM opt_sources WHERE source_id = ${SOURCE}`; + expect(rows[0]).toEqual({ status: "active", owner_id: "usr_simon" }); + }); + + it("a raw INSERT that omits status lands PENDING — the gate is in the schema too", async () => { + const sql = adminSql as ReturnType; + await sql` + INSERT INTO opt_sources (source_id, tenant_id, project_id, source_type, name) + VALUES ('src_realinfra_raw', ${TENANT}, 'p_realinfra', 'api', 'Raw insert') + ON CONFLICT (source_id) DO NOTHING`; + const rows = await sql<{ status: string }[]>` + SELECT status FROM opt_sources WHERE source_id = 'src_realinfra_raw'`; + expect(rows[0].status).toBe("pending"); + }); +}); + +run("P062/P063/P064/P067/P068 real Postgres — the durable ingestion plane", () => { + it("runs, crashes, and resumes across fresh processes with no gap and no duplicate", async () => { + const first = freshIngestion(); + const delivered: string[] = []; + let pages = 0; + const input = { + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + mode: "batch" as const, + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + page_size: 2, + output_ref: "0pt:dataset:ds_realinfra", + }; + await expect( + first.runner.run({ + ...input, + sink: async (batch) => { + pages++; + if (pages === 2) throw new Error("simulated process death"); + for (const e of batch) delivered.push(e.record_key); + }, + }), + ).rejects.toThrow(/simulated process death/); + expect(delivered).toHaveLength(2); + + // Fresh everything — only Postgres carries the position forward. + const second = freshIngestion(); + const report = await second.runner.run({ + ...input, + connector: new JsonlFileConnector(), + sink: async (batch) => { + for (const e of batch) delivered.push(e.record_key); + }, + }); + expect(report.status).toBe("succeeded"); + expect(delivered).toHaveLength(ROWS.length); + expect(new Set(delivered).size).toBe(ROWS.length); + + const cp = await second.repo.getCheckpoint(SOURCE, "companies"); + expect(cp?.records_committed).toBe(ROWS.length); + }); + + it("the schema version, ledger, and lineage are all readable from a fresh process", async () => { + const fresh = freshIngestion(); + expect((await new SchemaRegistry(fresh.repo).getLatest(SOURCE))?.schema_version).toBe(1); + expect(await fresh.repo.countLedger(SOURCE)).toBe(ROWS.length); + const upstream = await fresh.lineage.upstreamOf(TENANT, "0pt:dataset:ds_realinfra"); + expect(upstream.length).toBeGreaterThanOrEqual(1); + expect(upstream[0].input_ref).toBe(`0pt:source:${SOURCE}`); + }); + + it("a repeated backfill is idempotent against the durable ledger", async () => { + const fresh = freshIngestion(); + const emitted: string[] = []; + const report = await fresh.runner.backfill({ + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + mode: "batch", + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + sink: async (batch) => { + for (const e of batch) emitted.push(e.record_key); + }, + page_size: 100, + range: { partition_key: "companies", from: "0", to: String(ROWS.length) }, + }); + expect(report.envelopes_emitted).toBe(0); + expect(report.duplicates_skipped).toBe(ROWS.length); + expect(emitted).toHaveLength(0); + }); + + it("quarantine survives to a fresh process with the full envelope", async () => { + const fresh = freshIngestion(); + await fresh.runner.backfill({ + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + mode: "batch", + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + sink: async () => undefined, + page_size: 100, + quality_policy: { + source_id: SOURCE, + tenant_id: TENANT, + expectations: [ + { expectation_type: "expect_column_values_to_be_between", kwargs: { column: "employees", min_value: 0, max_value: 20 } }, + ], + on_violation: "quarantine", + }, + range: { partition_key: "companies", from: "0", to: String(ROWS.length) }, + }); + const reader = freshIngestion(); + const quarantined = await reader.repo.listQuarantine(SOURCE); + expect(quarantined.length).toBe(ROWS.length - 2); + expect(quarantined[0].envelope.record_key).toBeTruthy(); + expect(quarantined[0].violations[0].expectation_type).toBe("expect_column_values_to_be_between"); + }); +}); diff --git a/tests/p0/ingestion-runner.test.ts b/tests/p0/ingestion-runner.test.ts new file mode 100644 index 0000000..14085e8 --- /dev/null +++ b/tests/p0/ingestion-runner.test.ts @@ -0,0 +1,412 @@ +/** + * The ingestion plane end to end: + * + * P062 — every connector's output becomes the same envelope, and a run that + * DIES mid-page resumes from the durable checkpoint with no gap and + * no double-delivery. + * P063 — a breaking schema change stops the run BEFORE the sink is called. + * P064 — quality failures block or quarantine according to the DECLARED + * policy, and a blocked page does not advance the checkpoint. + * P067 — a run's source -> output lineage is queryable afterwards. + * P068 — a repeated backfill is idempotent. + * + * The connectors here are the real ones (JsonlFileConnector against a real + * file on disk); the "crash" is a sink that throws, which is the closest + * faithful analogue of a process dying between the consumer write and the + * checkpoint commit. + */ +import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { IngestionRunner, type IngestionRunInput } from "@/src/ingestion/ingestion-runner"; +import { InMemoryIngestionRepository } from "@/src/repositories/in-memory-ingestion-repository"; +import { SchemaRegistry } from "@/src/schema/schema-registry"; +import { LineageStore } from "@/src/lineage/lineage-store"; +import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector"; +import { hashPayload } from "@/src/ingestion/envelope"; +import { SiaxError } from "@/src/platform/errors"; +import type { IngestionEnvelope } from "@/src/types/ingestion"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; + +const SCHEMA: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; + +const tmpDir = path.join(os.tmpdir(), `opt-runner-${process.pid}`); +const filePath = path.join(tmpDir, "companies.jsonl"); + +const ROWS = Array.from({ length: 10 }, (_, i) => ({ + org_number: `55600${i}-000${i}`, + name: `Bolag ${i} AB`, + employees: (i + 1) * 10, +})); + +beforeAll(async () => { + await fs.mkdir(tmpDir, { recursive: true }); + await fs.writeFile(filePath, ROWS.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8"); +}); +afterAll(async () => { + await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined); +}); + +let repo: InMemoryIngestionRepository; +let runner: IngestionRunner; +let lineage: LineageStore; + +beforeEach(() => { + repo = new InMemoryIngestionRepository(); + lineage = new LineageStore(repo); + runner = new IngestionRunner(repo, new SchemaRegistry(repo), lineage); +}); + +function baseInput(sink: IngestionRunInput["sink"], overrides: Partial = {}): IngestionRunInput { + return { + source_id: "src_ing", + tenant_id: "t_test", + project_id: "p_test", + mode: "batch", + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + sink, + page_size: 3, + output_ref: "0pt:dataset:ds_companies", + ...overrides, + }; +} + +describe("P062 common envelope", () => { + it("normalizes every record into one envelope shape with provenance and a monotonic sequence", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run(baseInput(async (batch) => void seen.push(...batch))); + + expect(report.status).toBe("succeeded"); + expect(seen).toHaveLength(ROWS.length); + expect(seen.map((e) => e.sequence)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]); + for (const envelope of seen) { + expect(envelope.source_id).toBe("src_ing"); + expect(envelope.tenant_id).toBe("t_test"); + expect(envelope.mode).toBe("batch"); + expect(envelope.operation).toBe("upsert"); + expect(envelope.partition_key).toBe("companies"); + expect(envelope.schema_version).toBe(1); + expect(envelope.provenance.connector_id).toBe("jsonl_file"); + expect(envelope.provenance.connector_version).toBe("jsonl-file-connector@1"); + expect(envelope.provenance.run_id).toBe(report.run_id); + expect(envelope.ingested_at).toMatch(/^\d{4}-\d{2}-\d{2}T/); + } + }); + + it("the durable checkpoint advances to the end of the partition", async () => { + await runner.run(baseInput(async () => undefined)); + const checkpoint = await repo.getCheckpoint("src_ing", "companies"); + expect(checkpoint?.cursor).toBe(String(ROWS.length)); + expect(checkpoint?.sequence).toBe(ROWS.length); + expect(checkpoint?.records_committed).toBe(ROWS.length); + }); + + it("a run that dies inside the sink resumes with NO gap and NO double delivery", async () => { + const delivered: string[] = []; + let pages = 0; + + // First run: the sink dies on the third page (records 7-9), AFTER two + // pages have been committed. This is the crash window the ordering exists + // to protect: the checkpoint for page 3 was never committed. + const crashing = baseInput(async (batch) => { + pages++; + if (pages === 3) throw new Error("process died mid-write"); + for (const e of batch) delivered.push(e.record_key); + }); + await expect(runner.run(crashing)).rejects.toThrow(/process died mid-write/); + + expect(delivered).toHaveLength(6); + const afterCrash = await repo.getCheckpoint("src_ing", "companies"); + expect(afterCrash?.cursor).toBe("6"); + + // Second run: a FRESH runner and a FRESH connector, exactly as a restarted + // process would be — only the durable checkpoint carries over. + const resumed = new IngestionRunner(repo, new SchemaRegistry(repo), lineage); + const report = await resumed.run( + baseInput(async (batch) => { + for (const e of batch) delivered.push(e.record_key); + }), + ); + + expect(report.status).toBe("succeeded"); + // Every record exactly once across both runs: no gap, no duplicate. + expect(delivered).toHaveLength(ROWS.length); + expect(new Set(delivered).size).toBe(ROWS.length); + expect([...delivered].sort()).toEqual(ROWS.map((r) => r.org_number).sort()); + }); + + it("refuses a mode the connector does not serve", async () => { + // JsonlFileConnector declares batch + cdc; it never claims "stream". + await expect(runner.run(baseInput(async () => undefined, { mode: "stream" }))).rejects.toThrow( + /does not serve mode "stream"/, + ); + }); +}); + +describe("P063 breaking schema is caught before the consumer sees anything", () => { + it("fails the run BEFORE the sink is called when the shape breaks the declared policy", async () => { + // v1 registered by a clean run. + await runner.run(baseInput(async () => undefined)); + expect((await new SchemaRegistry(repo).getLatest("src_ing"))?.schema_version).toBe(1); + + // The source now claims a new REQUIRED field — backward-breaking. + let sinkCalls = 0; + const breaking = baseInput(async () => void sinkCalls++, { + source_id: "src_ing", + config: { + file_path: filePath, + partition_key: "companies", + schema: { ...SCHEMA, sni_code: { type: "string", required: true } }, + record_key_field: "org_number", + }, + }); + const err = await runner.run(breaking).catch((e: unknown) => e); + expect(err).toBeInstanceOf(SiaxError); + expect((err as SiaxError).code).toBe("CONFLICT"); + expect((err as SiaxError).message).toContain("refusing to ingest"); + // The whole point: no consumer write happened. + expect(sinkCalls).toBe(0); + }); + + it("catches a record that stopped conforming mid-stream, before emitting the page", async () => { + // A file whose DECLARED schema is unchanged but whose DATA drifted: + // `employees` arrives as a string. A distinct source_id so this run starts + // from a clean checkpoint rather than resuming past the file. + const driftPath = path.join(tmpDir, "drift.jsonl"); + await fs.writeFile( + driftPath, + JSON.stringify({ org_number: "556999-9999", name: "Drift AB", employees: "many" }) + "\n", + "utf8", + ); + let sinkCalls = 0; + const drifting = baseInput(async () => void sinkCalls++, { + source_id: "src_drift", + config: { file_path: driftPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + }); + await expect(runner.run(drifting)).rejects.toThrow(/does not conform to registered schema v1/); + expect(sinkCalls).toBe(0); + }); +}); + +describe("P064 quality enforcement follows the DECLARED policy", () => { + const blockingExpectation = { + expectation_type: "expect_column_values_to_be_between", + kwargs: { column: "employees", min_value: 0, max_value: 50 }, + }; + + it("'block' stops the run and does NOT advance the checkpoint past the bad page", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run( + baseInput(async (b) => void seen.push(...b), { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [blockingExpectation], + on_violation: "block", + }, + }), + ); + expect(report.quality_policy).toBe("block"); + const partition = report.partitions[0]; + expect(partition.stopped_because).toBe("blocked_by_quality"); + // Records 0-4 have employees <= 50; the block fires on the page containing + // employees 60, i.e. page 3 (records 6,7,8 -> 70,80,90 ... employees 60 is + // record index 5 on page 2). Whatever the exact page, nothing from the + // blocked page was emitted and the checkpoint stayed behind it. + expect(partition.envelopes_emitted).toBeLessThan(ROWS.length); + const checkpoint = await repo.getCheckpoint("src_ing", "companies"); + expect(Number(checkpoint?.cursor ?? "0")).toBeLessThan(ROWS.length); + expect(seen.every((e) => Number(e.payload?.employees) <= 50)).toBe(true); + }); + + it("'quarantine' persists the failing records, lets the clean ones through, and advances", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run( + baseInput(async (b) => void seen.push(...b), { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [blockingExpectation], + on_violation: "quarantine", + }, + }), + ); + expect(report.quality_policy).toBe("quarantine"); + const partition = report.partitions[0]; + expect(partition.stopped_because).toBeUndefined(); + expect(partition.quarantined).toBe(5); // employees 60..100 + expect(partition.envelopes_emitted).toBe(5); // employees 10..50 + expect(seen.every((e) => Number(e.payload?.employees) <= 50)).toBe(true); + + const quarantined = await repo.listQuarantine("src_ing"); + expect(quarantined).toHaveLength(5); + expect(quarantined[0].violations[0].expectation_type).toBe("expect_column_values_to_be_between"); + // The full envelope is preserved — a quarantined record is recoverable, + // not just counted. + expect(quarantined[0].envelope.payload).toBeTruthy(); + + const checkpoint = await repo.getCheckpoint("src_ing", "companies"); + expect(checkpoint?.cursor).toBe(String(ROWS.length)); + }); + + it("'warn' lets everything through but still reports the violations", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run( + baseInput(async (b) => void seen.push(...b), { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [blockingExpectation], + on_violation: "warn", + }, + }), + ); + expect(seen).toHaveLength(ROWS.length); + expect(report.partitions[0].quality_violations).toBe(5); + expect(report.partitions[0].quarantined).toBe(0); + }); + + it("no declared policy is reported as 'none_declared', never as a pass", async () => { + const report = await runner.run(baseInput(async () => undefined)); + expect(report.quality_policy).toBe("none_declared"); + }); + + it("an expectation the local engine cannot execute is a violation, not a silent pass", async () => { + const report = await runner.run( + baseInput(async () => undefined, { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [{ expectation_type: "expect_column_kl_divergence_to_be_less_than", kwargs: { column: "name" } }], + on_violation: "quarantine", + }, + }), + ); + expect(report.partitions[0].quarantined).toBe(ROWS.length); + const quarantined = await repo.listQuarantine("src_ing"); + expect(quarantined[0].violations[0].detail).toContain("not executable by the local engine"); + }); +}); + +describe("P067 lineage is queryable after the run", () => { + it("answers 'which source fed this output' and 'what did this source produce'", async () => { + const report = await runner.run(baseInput(async () => undefined, { correlation_id: "corr-lineage-1" })); + expect(report.lineage_edges).toBe(1); + + const upstream = await lineage.upstreamOf("t_test", "0pt:dataset:ds_companies"); + expect(upstream).toHaveLength(1); + expect(upstream[0].input_name).toBe("source/src_ing"); + expect(upstream[0].input_ref).toBe("0pt:source:src_ing"); + expect(upstream[0].capability_id).toBe("0pt.ingestion.run.v1"); + expect(upstream[0].run_id).toBe(report.run_id); + expect(upstream[0].correlation_id).toBe("corr-lineage-1"); + + const downstream = await lineage.downstreamOf("t_test", "source/src_ing"); + expect(downstream.map((e) => e.output_name)).toEqual(["0pt:dataset:ds_companies"]); + + // Tenant-scoped: another tenant sees nothing. + expect(await lineage.upstreamOf("t_other", "0pt:dataset:ds_companies")).toEqual([]); + }); + + it("re-emitting the same run id does not duplicate edges (deterministic edge ids)", async () => { + await runner.run(baseInput(async () => undefined, { run_id: "run_fixed" })); + // A second identical run of the same id — e.g. an idempotent retry. + const repo2Runner = new IngestionRunner(repo, new SchemaRegistry(repo), lineage); + await repo2Runner.run(baseInput(async () => undefined, { run_id: "run_fixed" })); + const edges = await lineage.upstreamOf("t_test", "0pt:dataset:ds_companies"); + expect(edges).toHaveLength(1); + }); +}); + +describe("P068 backfill is bounded and idempotent", () => { + it("a repeated identical backfill emits nothing the second time", async () => { + const first: IngestionEnvelope[] = []; + const firstReport = await runner.backfill({ + ...baseInput(async (b) => void first.push(...b)), + range: { partition_key: "companies", from: "0", to: "5" }, + page_size: 100, + }); + expect(firstReport.envelopes_emitted).toBe(5); + expect(firstReport.duplicates_skipped).toBe(0); + expect(first).toHaveLength(5); + + const second: IngestionEnvelope[] = []; + const secondReport = await runner.backfill({ + ...baseInput(async (b) => void second.push(...b)), + range: { partition_key: "companies", from: "0", to: "5" }, + page_size: 100, + }); + expect(secondReport.envelopes_emitted).toBe(0); + expect(secondReport.duplicates_skipped).toBe(5); + expect(second).toHaveLength(0); + }); + + it("a backfill does NOT move the live checkpoint", async () => { + await runner.run(baseInput(async () => undefined)); + const before = await repo.getCheckpoint("src_ing", "companies"); + await runner.backfill({ + ...baseInput(async () => undefined), + range: { partition_key: "companies", from: "0", to: "3" }, + }); + const after = await repo.getCheckpoint("src_ing", "companies"); + expect(after).toEqual(before); + }); + + it("a backfill overlapping a completed run re-delivers nothing (the ledger already has it)", async () => { + await runner.run(baseInput(async () => undefined)); + const emitted: IngestionEnvelope[] = []; + const report = await runner.backfill({ + ...baseInput(async (b) => void emitted.push(...b)), + range: { partition_key: "companies", from: "0", to: "10" }, + page_size: 100, + }); + expect(report.duplicates_skipped).toBe(10); + expect(emitted).toHaveLength(0); + }); + + it("a record whose CONTENT changed is re-emitted — idempotency is not amnesia", async () => { + await runner.run(baseInput(async () => undefined)); + const changedPath = path.join(tmpDir, "changed.jsonl"); + await fs.writeFile( + changedPath, + JSON.stringify({ ...ROWS[0], name: "Bolag 0 AB (namnändrad)" }) + "\n", + "utf8", + ); + const emitted: IngestionEnvelope[] = []; + const report = await runner.backfill({ + ...baseInput(async (b) => void emitted.push(...b), { + config: { file_path: changedPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + }), + range: { partition_key: "companies", from: "0", to: "1" }, + page_size: 100, + }); + expect(report.changed).toBe(1); + expect(report.envelopes_emitted).toBe(1); + expect(emitted[0].payload?.name).toBe("Bolag 0 AB (namnändrad)"); + }); + + it("an unbounded backfill is refused by the connector contract", async () => { + await expect( + runner.backfill({ + ...baseInput(async () => undefined), + range: { partition_key: "companies", from: "0", to: "" }, + }), + ).rejects.toThrow(/unbounded backfill|requires both/); + }); +}); + +describe("envelope hashing", () => { + it("is stable across key order and distinguishes a real content change", () => { + expect(hashPayload({ a: 1, b: 2 })).toBe(hashPayload({ b: 2, a: 1 })); + expect(hashPayload({ a: 1, b: 2 })).not.toBe(hashPayload({ a: 1, b: 3 })); + // null and absent are different data, and must hash differently. + expect(hashPayload({ a: 1, b: null })).not.toBe(hashPayload({ a: 1 })); + expect(hashPayload(null)).toBe("sha256:deleted"); + }); +}); diff --git a/tests/p0/schema-evolution.test.ts b/tests/p0/schema-evolution.test.ts new file mode 100644 index 0000000..e8230cc --- /dev/null +++ b/tests/p0/schema-evolution.test.ts @@ -0,0 +1,149 @@ +/** + * P063 DoD: "a breaking schema is detected BEFORE consumer corruption." + * + * Two levels, both tested here: + * - the pure comparison (checkCompatibility) per policy direction, and + * - the registry refusing to record a version that breaks the PREVIOUS + * version's declared policy — including the bypass where a producer tries + * to widen its own policy to "none" on the breaking version. + * + * The end-to-end "no consumer write happened" proof lives in + * tests/p0/ingestion-runner.test.ts, which asserts the sink was never called. + */ +import { describe, expect, it } from "vitest"; +import { checkCompatibility, conformanceErrors } from "@/src/schema/compatibility"; +import { SchemaRegistry } from "@/src/schema/schema-registry"; +import { InMemoryIngestionRepository } from "@/src/repositories/in-memory-ingestion-repository"; +import { SiaxError } from "@/src/platform/errors"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; + +const V1: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; + +describe("P063 compatibility rules", () => { + it("adding an OPTIONAL field is backward-compatible", () => { + const next = { ...V1, sni_code: { type: "string" as const, required: false } }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(true); + }); + + it("adding a REQUIRED field breaks backward compatibility", () => { + const next = { ...V1, sni_code: { type: "string" as const, required: true } }; + const verdict = checkCompatibility(V1, next, "backward"); + expect(verdict.compatible).toBe(false); + expect(verdict.breaks[0].field).toBe("sni_code"); + expect(verdict.breaks[0].detail).toContain("absent from every record written under the previous version"); + }); + + it("adding a required field is FORWARD-safe — direction matters", () => { + const next = { ...V1, sni_code: { type: "string" as const, required: true } }; + expect(checkCompatibility(V1, next, "forward").compatible).toBe(true); + expect(checkCompatibility(V1, next, "full").compatible).toBe(false); + }); + + it("removing a required field breaks BOTH directions", () => { + const next: RecordSchemaFields = { org_number: V1.org_number, employees: V1.employees }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(false); + expect(checkCompatibility(V1, next, "forward").compatible).toBe(false); + }); + + it("removing an OPTIONAL field is backward-safe but not forward-safe", () => { + const next: RecordSchemaFields = { org_number: V1.org_number, name: V1.name }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(true); + expect(checkCompatibility(V1, next, "forward").compatible).toBe(true); + }); + + it("a type change breaks every policy except an explicit 'none'", () => { + const next: RecordSchemaFields = { ...V1, employees: { type: "string", required: false } }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(false); + expect(checkCompatibility(V1, next, "forward").compatible).toBe(false); + expect(checkCompatibility(V1, next, "full").compatible).toBe(false); + expect(checkCompatibility(V1, next, "none").compatible).toBe(true); + }); + + it("conformanceErrors catches a record that silently stopped matching its schema", () => { + expect(conformanceErrors(V1, { org_number: "556001-0001", name: "Alfa AB" })).toEqual([]); + expect(conformanceErrors(V1, { org_number: "556001-0001" })).toContain('required field "name" is missing'); + expect(conformanceErrors(V1, { org_number: "556001-0001", name: "Alfa", employees: "ten" })).toContain( + 'field "employees" is string, expected number', + ); + expect(conformanceErrors(V1, { org_number: "1", name: "A", surprise: true })).toContain( + 'unregistered field "surprise" appeared in the payload', + ); + }); +}); + +describe("P063 schema registry — versioning and refusal", () => { + function registry() { + return new SchemaRegistry(new InMemoryIngestionRepository()); + } + + it("the first registration is v1; an identical re-registration does not create v2", async () => { + const reg = registry(); + const first = await reg.register({ source_id: "src_a", tenant_id: "t", fields: V1, compatibility: "backward" }); + expect(first.schema.schema_version).toBe(1); + const again = await reg.register({ source_id: "src_a", tenant_id: "t", fields: V1, compatibility: "backward" }); + expect(again.unchanged).toBe(true); + expect((await reg.list("src_a")).length).toBe(1); + }); + + it("a compatible change becomes v2", async () => { + const reg = registry(); + await reg.register({ source_id: "src_b", tenant_id: "t", fields: V1, compatibility: "backward" }); + const v2 = await reg.register({ + source_id: "src_b", + tenant_id: "t", + fields: { ...V1, sni_code: { type: "string", required: false } }, + compatibility: "backward", + }); + expect(v2.schema.schema_version).toBe(2); + expect((await reg.getLatest("src_b"))?.schema_version).toBe(2); + }); + + it("a BREAKING change is refused and nothing is persisted", async () => { + const reg = registry(); + await reg.register({ source_id: "src_c", tenant_id: "t", fields: V1, compatibility: "backward" }); + const err = await reg + .register({ + source_id: "src_c", + tenant_id: "t", + fields: { ...V1, sni_code: { type: "string", required: true } }, + compatibility: "backward", + }) + .catch((e: unknown) => e); + expect(err).toBeInstanceOf(SiaxError); + expect((err as SiaxError).code).toBe("CONFLICT"); + expect((err as SiaxError).message).toContain("breaks the declared"); + // Still only v1 on record — a refused registration writes nothing. + expect((await reg.list("src_c")).map((s) => s.schema_version)).toEqual([1]); + }); + + it("a producer cannot widen its OWN policy to 'none' on the breaking version", async () => { + const reg = registry(); + await reg.register({ source_id: "src_d", tenant_id: "t", fields: V1, compatibility: "full" }); + // The PREVIOUS version's policy governs — declaring "none" here must not help. + await expect( + reg.register({ + source_id: "src_d", + tenant_id: "t", + fields: { ...V1, employees: { type: "string", required: false } }, + compatibility: "none", + }), + ).rejects.toThrow(/breaks the declared "full" compatibility policy/); + }); + + it("a source that explicitly declared 'none' opts out — recorded, never assumed", async () => { + const reg = registry(); + await reg.register({ source_id: "src_e", tenant_id: "t", fields: V1, compatibility: "none" }); + const v2 = await reg.register({ + source_id: "src_e", + tenant_id: "t", + fields: { ...V1, employees: { type: "string", required: false } }, + compatibility: "none", + }); + expect(v2.schema.schema_version).toBe(2); + expect(v2.schema.compatibility).toBe("none"); + }); +}); diff --git a/tests/p0/source-activation-gate.test.ts b/tests/p0/source-activation-gate.test.ts new file mode 100644 index 0000000..f6f9b88 --- /dev/null +++ b/tests/p0/source-activation-gate.test.ts @@ -0,0 +1,190 @@ +/** + * P060 DoD #2: "an unclassified source can NOT be activated." + * + * Before this pass the opposite was true, and provably so: + * src/repositories/postgres-source-repository.ts wrote + * `${input.status ?? "active"}` and there was no activation operation at all + * — registering a source with no rights, no provenance and no authorization + * made it `active` on the spot. These tests pin the closed gate AND its + * obvious bypasses (asking for status "active" at registration; a service + * identity authorizing itself; an expired authorization). + */ +import { describe, expect, it } from "vitest"; +import { SourceRegistryService } from "@/src/services/source-registry-service"; +import { SourceService } from "@/src/services/source-service"; +import { InMemorySourceRepository } from "@/src/repositories/in-memory-source-repository"; +import { InMemorySourceRegistryRepository } from "@/src/repositories/in-memory-source-registry-repository"; +import { SiaxError } from "@/src/platform/errors"; + +function newService(): SourceRegistryService { + return new SourceRegistryService({ + repository: new InMemorySourceRegistryRepository(), + sourceService: new SourceService({ repository: new InMemorySourceRepository() }), + }); +} + +const TENANT = "t_test"; +const base = { + source_id: "src_gate_1", + tenant_id: TENANT, + project_id: "p_test", + source_type: "api" as const, + name: "Bolagsverket öppna data", + owner_id: "usr_simon", +}; + +const RIGHTS = { + license_id: "CC-BY-4.0", + allowed_purposes: ["catalog", "enrichment"], + pii_allowed: false, + redistribution: "customer_deliverable" as const, + verified_at: "2026-09-04", + verified_by: "usr_simon", +}; + +const PROVENANCE = { method: "official_api" as const, authority: "Bolagsverket" }; + +const AUTHORIZATION = { + basis: "public_register" as const, + granted_by: "usr_simon", + granted_at: "2026-09-01T08:00:00.000Z", +}; + +describe("P060 source registry — the activation gate", () => { + it("a newly registered source is PENDING, never active", async () => { + const svc = newService(); + const descriptor = await svc.registerSource(base); + expect(descriptor.source.status).toBe("pending"); + }); + + it("asking for status:'active' at registration does NOT bypass the gate", async () => { + const svc = newService(); + // The most obvious bypass: just ask for it. + await expect(svc.registerSource({ ...base, status: "active" })).rejects.toBeInstanceOf(SiaxError); + // ...and the source is still on record, still pending — not half-created. + const source = await svc.sources.getSource(base.source_id); + expect(source.status).toBe("pending"); + }); + + it("activating an unclassified source is denied and names every missing fact", async () => { + const svc = newService(); + await svc.registerSource({ ...base, owner_id: undefined }); + const err = await svc.activateSource(base.source_id, TENANT).catch((e: unknown) => e); + expect(err).toBeInstanceOf(SiaxError); + const siax = err as SiaxError; + expect(siax.code).toBe("AUTHORIZATION_DENIED"); + expect(siax.message).toContain("owner_id is not set"); + expect(siax.message).toContain("no authorization on record"); + expect(siax.message).toContain("no rights record"); + expect(siax.message).toContain("no provenance record"); + expect((await svc.sources.getSource(base.source_id)).status).toBe("pending"); + }); + + it("a fully classified source activates, and describe() reports the classification", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + rights: RIGHTS, + provenance: PROVENANCE, + authorization: AUTHORIZATION, + }); + const classification = await svc.classifySource(base.source_id, TENANT); + expect(classification).toEqual({ source_id: base.source_id, classified: true, missing: [] }); + + const activated = await svc.activateSource(base.source_id, TENANT, "usr_simon"); + expect(activated.status).toBe("active"); + + const descriptor = await svc.describeSource(base.source_id, TENANT); + expect(descriptor.classification?.classified).toBe(true); + expect(descriptor.authorization?.basis).toBe("public_register"); + }); + + it("registering fully classified WITH status:'active' activates through the gate, not around it", async () => { + const svc = newService(); + const descriptor = await svc.registerSource({ + ...base, + status: "active", + rights: RIGHTS, + provenance: PROVENANCE, + authorization: AUTHORIZATION, + }); + expect(descriptor.source.status).toBe("active"); + }); + + it("a service identity cannot authorize a source (no self-granted rubber stamp)", async () => { + const svc = newService(); + await svc.registerSource(base); + await expect( + svc.setAuthorization({ + source_id: base.source_id, + tenant_id: TENANT, + basis: "internal", + granted_by: "svc_0pt", + granted_at: "2026-09-01T08:00:00.000Z", + }), + ).rejects.toThrow(/accountable actor/); + }); + + it("an EXPIRED authorization does not classify the source", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + rights: RIGHTS, + provenance: PROVENANCE, + authorization: { ...AUTHORIZATION, expires_at: "2026-01-01T00:00:00.000Z" }, + }); + const classification = await svc.classifySource(base.source_id, TENANT); + expect(classification.classified).toBe(false); + expect(classification.missing.join(" ")).toContain("expired"); + await expect(svc.activateSource(base.source_id, TENANT)).rejects.toThrow(/not classified/); + }); + + it("rights with an empty purpose list do not classify (purpose limitation is not optional)", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + rights: { ...RIGHTS, allowed_purposes: [] }, + provenance: PROVENANCE, + authorization: AUTHORIZATION, + }); + const classification = await svc.classifySource(base.source_id, TENANT); + expect(classification.classified).toBe(false); + expect(classification.missing.join(" ")).toContain("allowed_purposes is empty"); + }); + + it("deactivating is never gated — a source can always be taken out of service", async () => { + const svc = newService(); + await svc.registerSource({ ...base, rights: RIGHTS, provenance: PROVENANCE, authorization: AUTHORIZATION }); + await svc.activateSource(base.source_id, TENANT); + const paused = await svc.deactivateSource(base.source_id, TENANT, "paused"); + expect(paused.status).toBe("paused"); + }); + + it("the gate is tenant-scoped: another tenant cannot activate this source", async () => { + const svc = newService(); + await svc.registerSource({ ...base, rights: RIGHTS, provenance: PROVENANCE, authorization: AUTHORIZATION }); + await expect(svc.activateSource(base.source_id, "t_attacker")).rejects.toThrow(/not found/); + expect((await svc.sources.getSource(base.source_id)).status).toBe("pending"); + }); + + it("schedule is recorded as a registry field and validated", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + schedule: { mode: "batch", interval_seconds: 3600, enabled: true }, + }); + const schedule = await svc.getSchedule(base.source_id, TENANT); + expect(schedule?.mode).toBe("batch"); + expect(schedule?.interval_seconds).toBe(3600); + await expect( + svc.setSchedule({ + source_id: base.source_id, + tenant_id: TENANT, + mode: "stream", + interval_seconds: 0, + enabled: true, + updated_at: new Date().toISOString(), + }), + ).rejects.toThrow(/interval_seconds must be positive/); + }); +}); -- 2.54.0