diff --git a/STATUS.md b/STATUS.md index 658d85c..7df9a86 100644 --- a/STATUS.md +++ b/STATUS.md @@ -82,3 +82,107 @@ accurate. 5. 0pt-motor executors for the delegated jobs (`0pt.quality.ge_run`, `0pt.features.*`, `0pt.er.batch_resolution`). 6. Observability: no metrics/traces emitted by the new services (§36). + +--- + +## P060-P068 ingestion plane (added 2026-09-04) + +Classification scale unchanged. Every row below states what was actually +verified, and by which command. + +| Requirement | BEFORE this pass | Status now | Evidence | +|---|---|---|---| +| **P060** Source Registry: durable canonical registry; restart preserves sources; an unclassified source cannot be activated | **PARTIAL / BROKEN.** The registry existed with Postgres persistence (`sql/002_source_registry.sql`, `src/repositories/postgres-source-registry-repository.ts`) and carried source_id/tenant/project/type/status/license/purpose/rights. It had NO authorization, NO schedule, NO checkpoint, NO schema, and no `owner` distinct from `created_by`. The activation DoD was inverted: `postgres-source-repository.ts:70` wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered, and no activation operation existed at all | **INTEGRATION_TESTED** | `sql/007_ingestion_plane.sql` (authorization/schedule/schemas/checkpoints + `ALTER … SET DEFAULT 'pending'` + `owner_id`); `SourceRegistryService.classifySource`/`activateSource`; `tests/p0/source-activation-gate.test.ts` (11); `tests/api/0pt-p060-routes.test.ts` (9); `tests/p0/ingestion-plane-postgres.realinfra.test.ts` against a live Postgres 16 | +| **P061** Connector contract: discover/validate/connect/checkpoint/read/resume/backfill/health/disconnect; two heterogeneous connectors pass one conformance suite | **MISSING.** `services/api/sources/connectors.py` had per-source `fetch_*()` functions only — no lifecycle, no cursor, no resume, no health, no disconnect, no conformance suite | **INTEGRATION_TESTED** | `src/connectors/connector.ts` (all nine operations); `src/connectors/conformance.ts` (15 checks); `HttpJsonConnector` (real node:http server, opaque cursor) and `JsonlFileConnector` (real file, line-offset cursor, CDC deletes) both pass the identical suite — `tests/p0/connector-conformance.test.ts`, which also proves the suite FAILS a fail-open connector | +| **P062** CDC/batch/stream normalized to one envelope; restart resumes from a durable checkpoint without silent gaps | **MISSING**, and documented as such: `services/api/tests/test_known_gaps.py:10-19` skipped the CDC case because no CDC pipeline existed anywhere, and `workers/pipeline.py` had no persistence at all | **INTEGRATION_TESTED** | `src/types/ingestion.ts` (one envelope for all three modes); `src/ingestion/ingestion-runner.ts`; `tests/p0/ingestion-runner.test.ts` kills a run inside the sink and proves the resumed run covers all 10 records exactly once; same proof against real Postgres across fresh processes | +| **P063** Schema versioning + compatibility policy; a breaking schema is detected before consumer corruption | **MISSING.** `schema_version` existed as a free-form string on datasets/Iceberg writes; there was no comparison, no policy, no versioned source schema | **INTEGRATION_TESTED** | `src/schema/compatibility.ts`, `src/schema/schema-registry.ts`; `tests/p0/schema-evolution.test.ts` (12); the runner's schema gate runs before the first read — `tests/p0/ingestion-runner.test.ts` asserts `sinkCalls === 0` on a break | +| **P064** Quality rules behind the SIAX abstraction; failures block or quarantine per declared policy | **PARTIAL.** `src/quality/*` had rules, runs, results, issues, scores and a GE-compatible engine, but a failing rule only opened an ISSUE — the data flowed on regardless. `grep -ri quarantin src/` found one unrelated hit | **INTEGRATION_TESTED** | `src/quality/quality-policy.ts` composes the EXISTING `ExpectationEngine` (not a second engine) and adds the consequence; block / quarantine / warn each proved in `tests/p0/ingestion-runner.test.ts`; quarantine durability proved against real Postgres | +| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **DUPLICATE (confirmed).** `src/er/entity-resolution-service.ts` (canonical: hard rules, review band, Splink delegation) AND `services/api/workers/resolve.py` (simplified fuzzy threshold, auto-merge) — the latter still called by `workers/pipeline.py:110` | **NOT DONE — documented, not built.** The duplication is now stated at BOTH sites in code with the removal plan. The new ingestion plane deliberately performs no ER, so it adds no third engine. See "P065 — why it was not closed" below | +| **P066** Iceberg | out of scope — ST0RE is owned by another workflow | **NOT TOUCHED** | Contract read only (`src/services/iceberg-write-service.ts`, `src/clients/st0re-lakehouse-client.ts`) | +| **P067** Lineage: OpenLineage/AUD0 events; source → output lineage is queryable | **PARTIAL.** `src/lineage/lineage-emitter.ts` emitted OpenLineage to AUD0, but nothing in this repo could ANSWER a lineage question — no store, no query, no route | **INTEGRATION_TESTED** | `src/lineage/lineage-edge.ts`, `src/lineage/lineage-store.ts` (local queryable projection; AUD0 stays the system of record), `GET /api/0pt/lineage`; `tests/p0/ingestion-runner.test.ts` queries both directions and proves tenant scoping | +| **P068** Durable bounded backfill/replay; repeated backfill is idempotent | **MISSING**, documented as such: `services/api/tests/test_known_gaps.py:44-52` skipped it because "no delivery-store persistence layer exists to dedupe a backfill against" | **INTEGRATION_TESTED** | `opt_ingestion_ledger` + `IngestionRunner.backfill`; `tests/p0/ingestion-runner.test.ts` (repeat emits 0, live checkpoint untouched, a real content change still re-emits); same against real Postgres | + +### What was actually run (not "should pass") + +- `pnpm run typecheck` — clean. +- `pnpm run test` — **365 passed, 25 skipped** (was 311/18 before this pass). +- `pnpm run build` — clean; the four new routes appear in the route table. +- `pnpm run lint` — 12 errors / 5 warnings, **all pre-existing** (identical + count on the base commit; advisory in CI). This pass adds none. +- Real Postgres 16: `tests/p0/ingestion-plane-postgres.realinfra.test.ts` + with `INGESTION_TEST_DATABASE_URL` — 7 passed. +- **Real container restart**, run manually against that same database: + `docker restart` the Postgres container, then re-read through the real + repositories. Before and after were identical — 2 sources (one `active` + with its owner, one `pending`), 1 checkpoint (cursor 6, 6 committed), + 6 ledger rows, 1 lineage edge, 4 quarantined records, 1 schema version. + That is P060's "a restart preserves all sources", measured rather than + asserted. + +### A real bug this pass wrote and then caught + +The first draft of `IngestionRepository` combined the idempotency check and +its recording into one atomic call, made BEFORE the sink. The crash/resume +test then delivered **7 of 10 records**: the page in flight when the sink +died had already been marked delivered, so the resumed run skipped it as a +duplicate — permanent, silent data loss. The fix is the `peekDeliveries` / +`commitDeliveries` split, with both the ledger commit and the checkpoint +commit after the sink resolves. Recorded here because the failure mode is +the interesting artifact, not the fix. + +### Honest gaps in the new plane (not hidden) + +1. **Delivery is AT-LEAST-ONCE, not exactly-once.** Two concurrent runs on + the same (source, partition) can both peek "new" and both emit. + `envelope_id` is deterministic so a dedup-capable sink survives it. There + is no per-(source, partition) run lock. Do not describe this plane as + exactly-once. +2. **No CDC *source* exists.** The envelope, the operations and the runner + handle change events, and `JsonlFileConnector` emits real deletes from an + `_op` field — but this repo still has no Debezium/WAL/replication-slot + consumer. P062's contract is met; a CDC connector is still MISSING. +3. **0pt does not schedule itself.** `opt_source_schedule` is a declaration + ACT0 would read; there is no cron daemon here. +4. **The sink is caller-supplied.** The runner stops at the sink; the + lakehouse write boundary is ST0RE's (P066, another workflow's repo). +5. **Nothing here is deployed.** See the deployment note below. +6. **`QualityService`'s own rule/run/issue registry is still in-process + Maps** (`src/quality/quality-service.ts:47-50`). The new *enforcement* + path is durable; the older rule registry is not. Unchanged by this pass. +7. **Python tests still run nowhere.** `grep -rn "pytest" .gitea/workflows + .github/workflows` finds nothing — `services/api/tests/*` (9 files) has + never been executed by CI. Flagged, not fixed (it needs a workflow change + and a Python toolchain decision). + +### Deployment reality, measured 2026-09-04 + +`https://0pt.app/api/health` answers **200** from **Vercel** with +`{"db":"not_configured"}`, and `https://0pt.app/api/0pt/sources` answers +**404** — the deployed build predates the entire `app/api/0pt/*` capability +layer. `https://api.0pt.app/health` answers **404** (the FastAPI engine is +not reachable), and `.gitea/workflows/cd.yml` still guards on +`COOLIFY_APP_ID = TODO-no-coolify-app-provisioned-yet`. So: none of the +capability layer — neither this pass's work nor P010/P014's — is running in +production, and P060's durability guarantee cannot be exercised there, +because production has no database configured and does not serve the routes. +Verified with `curl`; not inferred. + +### P065 — why it was not closed + +Closing P065 means turning `services/api/workers/resolve.py` into an adapter +over the canonical `src/er/entity-resolution-service.ts`. That change cannot +be verified from this repo today: `services/api` is not built by this repo's +Dockerfile (it builds only the Next.js app), `api.0pt.app` answers 404, and +no CI workflow runs its pytest suite. A blind rewrite would ship untested +into a path nobody can exercise — the exact failure mode this programme +exists to stop. + +The removal plan, for whoever picks it up: +1. Expose the canonical ER capability as `POST /api/0pt/er/candidates` + (service already exists; only a route + Zod schema are missing). +2. Add a Python CI job that actually runs `services/api/tests`. +3. Replace `resolve.py`'s body with an HTTP adapter over that route, + **fail-closed** when it is unreachable — never falling back to the local + scorer, which would preserve the duplicate under a different name. +4. Port `test_resolve_determinism.py` to assert the adapter's contract, and + delete the fuzzy scorer. diff --git a/app/api/0pt/lineage/route.ts b/app/api/0pt/lineage/route.ts new file mode 100644 index 0000000..8733ad3 --- /dev/null +++ b/app/api/0pt/lineage/route.ts @@ -0,0 +1,47 @@ +/** + * app/api/0pt/lineage/route.ts + * ============================= + * P067 DoD: "source -> output lineage is QUERYABLE". This is the query. + * + * GET /api/0pt/lineage?tenant_id=…&output_name=… -> what fed this output + * GET /api/0pt/lineage?tenant_id=…&input_name=… -> what this input produced + * + * The answer comes from 0pt's local projection of the OpenLineage events it + * emits to AUD0 (src/lineage/lineage-store.ts). AUD0 remains the system of + * record; this endpoint exists because "emitted to AUD0" was never the same + * thing as "answerable from 0pt", and before this route nothing in this repo + * could answer either question. + */ +import { NextRequest, NextResponse } from "next/server"; +import { lineageStore } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest) { + try { + const { searchParams } = new URL(request.url); + const tenantId = searchParams.get("tenant_id"); + const outputName = searchParams.get("output_name") ?? undefined; + const inputName = searchParams.get("input_name") ?? undefined; + if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + if (!outputName && !inputName) { + return NextResponse.json( + { error: "one of output_name or input_name is required — an unfiltered lineage dump is not a query" }, + { status: 400 }, + ); + } + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + const limitRaw = searchParams.get("limit"); + const limit = limitRaw ? Math.min(Math.max(Number(limitRaw), 1), 500) : undefined; + const edges = await lineageStore.query({ + tenant_id: tenantId, + output_name: outputName, + input_name: inputName, + limit, + }); + return NextResponse.json({ edges, count: edges.length }); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/sources/[sourceId]/activate/route.ts b/app/api/0pt/sources/[sourceId]/activate/route.ts new file mode 100644 index 0000000..4a10ef6 --- /dev/null +++ b/app/api/0pt/sources/[sourceId]/activate/route.ts @@ -0,0 +1,47 @@ +/** + * app/api/0pt/sources/[sourceId]/activate/route.ts + * ================================================== + * P060 activation gate over HTTP. POST activates a source ONLY when it is + * classified (owner + unexpired authorization + rights + provenance); + * otherwise it returns 403 AUTHORIZATION_DENIED naming every missing fact. + * GET returns the classification without changing anything, so an operator + * can see what is missing before trying. + */ +import { NextRequest, NextResponse } from "next/server"; +import { activateSourceSchema, parseBody } from "@/src/validation/0pt-schemas"; +import { sourceRegistryService } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id"); + if (!tenantId) { + return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + } + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + return NextResponse.json(await sourceRegistryService.classifySource(sourceId, tenantId)); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function POST(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const parsed = parseBody(activateSourceSchema, await request.json()); + if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status }); + const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id); + if (guard) return guard; + const source = await sourceRegistryService.activateSource( + sourceId, + parsed.data.tenant_id, + parsed.data.actor_id, + ); + return NextResponse.json(source); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/sources/[sourceId]/authorization/route.ts b/app/api/0pt/sources/[sourceId]/authorization/route.ts new file mode 100644 index 0000000..69b1c1f --- /dev/null +++ b/app/api/0pt/sources/[sourceId]/authorization/route.ts @@ -0,0 +1,45 @@ +/** + * app/api/0pt/sources/[sourceId]/authorization/route.ts + * ====================================================== + * P060 "authorization": the recorded decision that 0pt may connect to this + * source at all — separate from rights (what the DATA may be used for). PUT + * is an upsert; a service identity as `granted_by` is rejected (400), because + * a source may not authorize itself. + */ +import { NextRequest, NextResponse } from "next/server"; +import { parseBody, setSourceAuthorizationSchema } from "@/src/validation/0pt-schemas"; +import { sourceRegistryService } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id"); + if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + const auth = await sourceRegistryService.getAuthorization(sourceId, tenantId); + if (!auth) { + return NextResponse.json({ error: `source ${sourceId} has no authorization record` }, { status: 404 }); + } + return NextResponse.json(auth); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function PUT(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const parsed = parseBody(setSourceAuthorizationSchema, await request.json()); + if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status }); + const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id); + if (guard) return guard; + return NextResponse.json( + await sourceRegistryService.setAuthorization({ ...parsed.data, source_id: sourceId }), + ); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/sources/[sourceId]/schedule/route.ts b/app/api/0pt/sources/[sourceId]/schedule/route.ts new file mode 100644 index 0000000..0d2de4f --- /dev/null +++ b/app/api/0pt/sources/[sourceId]/schedule/route.ts @@ -0,0 +1,48 @@ +/** + * app/api/0pt/sources/[sourceId]/schedule/route.ts + * ================================================= + * P060 "schedule": ingestion mode + cadence, recorded on the source. 0pt does + * NOT run cron itself — ACT0 owns scheduling; this row is the declaration a + * scheduler reads, and saying otherwise would be a claim this repo cannot back. + */ +import { NextRequest, NextResponse } from "next/server"; +import { parseBody, setSourceScheduleSchema } from "@/src/validation/0pt-schemas"; +import { sourceRegistryService } from "@/src/services/source-registry"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; + +export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id"); + if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 }); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + const schedule = await sourceRegistryService.getSchedule(sourceId, tenantId); + if (!schedule) { + return NextResponse.json({ error: `source ${sourceId} has no schedule record` }, { status: 404 }); + } + return NextResponse.json(schedule); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function PUT(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) { + try { + const { sourceId } = await params; + const parsed = parseBody(setSourceScheduleSchema, await request.json()); + if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status }); + const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id); + if (guard) return guard; + return NextResponse.json( + await sourceRegistryService.setSchedule({ + ...parsed.data, + source_id: sourceId, + updated_at: new Date().toISOString(), + }), + ); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/docs/P060_INGESTION_PLANE.md b/docs/P060_INGESTION_PLANE.md new file mode 100644 index 0000000..b55d52a --- /dev/null +++ b/docs/P060_INGESTION_PLANE.md @@ -0,0 +1,189 @@ +# 0pt ingestion plane (P060–P068) + +> Written 2026-09-04. Every claim here is backed by a test path or a +> recorded command; anything not verified says so. Read `STATUS.md`'s +> "P060-P068 ingestion plane" section for the classification table and the +> honest-gaps list — this document is the mechanism, that one is the status. + +## The shape + +``` + Source Registry Connector Ingestion Runner + (P060: who, what, (P061: nine (P062: one envelope, + under what rights, operations, one one durable checkpoint) + on what schedule) conformance suite) + │ │ │ + │ classification gate │ discover / validate │ read ─┐ + │ blocks activation │ connect / read │ │ + ▼ ▼ resume / backfill ▼ ▼ + opt_sources records ───────────────▶ IngestionEnvelope + opt_source_rights │ + opt_source_authorization ├─▶ P063 schema gate + opt_source_schedule ├─▶ P064 quality gate + opt_source_schemas ├─▶ P068 ledger (peek) + opt_ingestion_checkpoints ├─▶ SINK (caller-supplied) + ├─▶ P068 ledger (commit) + ├─▶ checkpoint (commit) + └─▶ P067 lineage edge +``` + +## P060 — the registry, and the gate + +A source now carries: `source_id`, `tenant_id`, `project_id`, `source_type`, +`owner_id`, an **authorization** record, a **license/purpose rights** record, +**provenance** (how the data is obtained), a **schedule**, a versioned +**schema**, a durable **checkpoint**, and a `status`. + +`status` starts at **`pending`**. It was `active` before — both repository +implementations wrote `input.status ?? "active"` — which meant an +unclassified source went live the moment it was registered. `activateSource` +is now the only path to `active`, and it refuses unless ALL of these are on +record: + +- an `owner_id` (distinct from `created_by`: who is accountable, not who typed it), +- an **unexpired** authorization granted by an accountable actor + (`svc_*`/`system`/`automation` are rejected — a source may not authorize itself), +- rights with a `license_id` **and** at least one `allowed_purpose`, +- provenance with a sourcing `method`. + +Schedule, schema and checkpoint describe HOW a source runs, not WHETHER it +may, so they are deliberately not part of the gate. + +The default is enforced in two places on purpose: in the repositories, and as +`ALTER TABLE opt_sources ALTER COLUMN status SET DEFAULT 'pending'`, so a raw +`INSERT` from psql cannot produce an unclassified-but-active source either +(proved in the realinfra suite). + +Routes: `GET/POST /api/0pt/sources/{id}/activate` (GET returns the +classification without changing anything), `GET/PUT …/authorization`, +`GET/PUT …/schedule`. + +## P061 — the connector contract + +Nine operations on one interface: `validate` (pure, no IO, fail-closed), +`connect` (idempotent), `discover`, `health` (reports, never throws), `read` +(honours `limit` exactly), `checkpoint`, `resume`, `backfill` (bounded — an +open-ended range is refused), `disconnect` (idempotent, safe after an error). + +`src/connectors/conformance.ts` is a plain async function, not a test file, +so the same suite runs from tests, from an operator script against a real +configured source, and (later) from a pre-activation check. There is no +"skip" verdict: a connector that cannot honestly serve a check FAILS it. + +Two heterogeneous connectors pass it: + +| | `HttpJsonConnector` | `JsonlFileConnector` | +|---|---|---| +| transport | network, real HTTP | local filesystem | +| cursor | opaque server token, with a `cursor_field` fallback | line offset it computes itself | +| exhaustion | server-reported | exact (the file is finite) | +| health | HTTP round trip | `stat` | +| backfill bounds | server-defined (sequence/timestamp) | line numbers | +| CDC deletes | no — it does not claim `cdc` | yes, via `_op` | + +The `cursor_field` fallback exists because the conformance suite caught a +real defect: an API that stops returning `next_cursor` at the end of a page +left the connector with nothing to resume from, so `resume()` silently +restarted at the beginning. + +## P062 — one envelope, one durable checkpoint + +`IngestionEnvelope` is the single shape everything downstream sees: +`mode` (batch/stream/cdc), `operation` (upsert/insert/update/delete), +`partition_key`, `record_key`, `schema_version`, a monotonic `sequence`, +`occurred_at`/`ingested_at`, `payload` (null only for a delete) and +`provenance` (connector id + version + run id + cursor). + +**Ordering is the contract.** Per page: + +``` +read/resume → normalize → schema conformance → quality gate + → ledger PEEK → SINK → ledger COMMIT → checkpoint COMMIT +``` + +Both commits happen after the sink resolves. The two tempting shortcuts are +both data-loss bugs, and one of them was written here first — see STATUS.md's +"A real bug this pass wrote and then caught". + +The resulting guarantee is **at-least-once**, not exactly-once. +`envelope_id` is a deterministic hash of +`(source, partition, record_key, content)` so a dedup-capable sink survives a +re-delivery. + +## P063 — schema evolution + +`RecordSchema` is a small structural shape (field → type + required) plus a +`CompatibilityPolicy` (`backward` / `forward` / `full` / `none`). +`checkCompatibility` answers what breaks, in which direction, and why. It is +deliberately not a second JSON-Schema dialect: the question is "is version +N+1 of an UPSTREAM shape safe for consumers of version N?", which needs a +comparable structure, not a validator. + +Two enforcement points, both upstream of any consumer write: + +1. the runner's **discovery gate** — a shape change that breaks the previous + version's declared policy fails the whole run before the first `read()`; +2. **per-record conformance** — a source that drifted without redeclaring its + schema is caught on the offending record, before the page reaches the sink. + +A producer cannot widen its own policy: the PREVIOUS version's policy governs +whether a change is allowed. + +## P064 — quality with consequences + +`QualityGate` composes the existing `ExpectationEngine` (the Great +Expectations core subset already in this repo) and adds the missing half: a +declared `on_violation` action. + +- `block` — the page fails, and the checkpoint does **not** advance, so the + bad page stays unconsumed rather than being stepped over. +- `quarantine` — failing envelopes are persisted whole to + `opt_quality_quarantine` (recoverable, not merely counted); clean ones + continue and the checkpoint advances. +- `warn` — everything continues, violations are counted and reported. + +There is no default. With no policy the run report says +`quality_policy: "none_declared"` — an undeclared policy must never read as a +passed one. An expectation the local engine cannot execute counts as a +VIOLATION, never a silent pass. + +## P067 — queryable lineage + +`LineageEmitter` keeps emitting OpenLineage to AUD0, and **AUD0 remains the +system of record**. `LineageStore` adds 0pt's own flattened projection — one +row per (run, input, output) with a deterministic `edge_id` — because +"emitted to AUD0" was never the same thing as "answerable from 0pt". + +``` +GET /api/0pt/lineage?tenant_id=…&output_name=0pt:dataset:ds_x # what fed it +GET /api/0pt/lineage?tenant_id=…&input_name=source/src_x # what it produced +``` + +An unfiltered dump is refused (400): that is an export, not a query. + +## P068 — bounded, idempotent backfill + +`opt_ingestion_ledger` holds one row per `(source_id, record_key)` with the +last accepted content hash. A backfill runs through the same envelope, +schema, quality and ledger path as a normal run, and: + +- **never touches the live checkpoint** — a backfill must not rewind or + advance streaming ingestion; +- **re-emits nothing** on a repeat — every record comes back `duplicate`; +- **still re-emits a genuine content change** — idempotency is not amnesia; +- **refuses an unbounded range** at the connector contract level. + +## Running the real-infrastructure proof + +```bash +docker run -d --name opt-ingestion-pgtest \ + -e POSTGRES_USER=opt -e POSTGRES_PASSWORD=opt_test_pw -e POSTGRES_DB=opt_test \ + -p 15434:5432 postgres:16-alpine + +INGESTION_TEST_DATABASE_URL=postgres://opt:opt_test_pw@localhost:15434/opt_test \ + pnpm vitest run tests/p0/ingestion-plane-postgres.realinfra.test.ts +``` + +The suite applies `sql/002_source_registry.sql` and +`sql/007_ingestion_plane.sql` itself (both are idempotent). It skips +honestly, never fabricating success, when the env var is unset. diff --git a/services/api/workers/resolve.py b/services/api/workers/resolve.py index d8131dc..ae50623 100644 --- a/services/api/workers/resolve.py +++ b/services/api/workers/resolve.py @@ -3,6 +3,44 @@ Entity resolution (docs/06): block, score, and link records across sources so th same company from GLEIF + Brreg + CVR collapses to one record with corroborated confidence. rapidfuzz only; deterministic; no external calls. +======================================================================= +P065 — THIS IS THE SECOND ER ENGINE. IT IS NOT THE CANONICAL ONE. +======================================================================= +0pt has TWO entity-resolution implementations, and this is the simplified +one: + + * CANONICAL: src/er/entity-resolution-service.ts — the declared generic ER + capability. Deterministic blocking + an explainable per-pair score, + product-supplied HARD RULES that override the score in both directions, + a configurable review band that hands ambiguous pairs to humans instead + of auto-merging, and delegation of heavy probabilistic linkage (Splink) + to 0pt-motor via the N0D job "0pt.er.batch_resolution". + + * THIS FILE: blocking on (market, first initial) + a single + rapidfuzz token_sort_ratio threshold, first-match-wins, auto-merge with + no review band, no hard-rule hook, no explanation beyond the score. + +P065's definition of done is that no second simplified production engine +remains. It has NOT been met: this file is still what +`workers/pipeline.py` calls (line ~110, `deduped = resolve(records)`). + +WHY IT WAS NOT REPLACED IN THE P060-P068 PASS, stated plainly rather than +quietly deferred: replacing it means turning this module into an ADAPTER +that calls the canonical capability, and that change cannot be verified +from this repo today — + * `services/api` is not deployed by this repo's Dockerfile (which builds + only the Next.js app), and api.0pt.app answers 404, so there is no + running engine to test against; + * no CI workflow runs services/api's pytest suite at all (verified: + `grep -rn "pytest" .gitea/workflows .github/workflows` finds nothing), + so a rewrite here would land with zero automated coverage. +Rewriting it blind and calling P065 done would be exactly the +"claimed complete, broken in operation" failure this programme exists to +stop. The removal plan is recorded in STATUS.md. + +Until then: do NOT add capability to this file. Any new ER behaviour +belongs in the canonical TypeScript service. + Deterministic by construction: for a fixed `records` input, blocking is a pure function of (market, name-first-letter) and matching is first-match-wins within a block, so results do not depend on which source answered first — see diff --git a/sql/007_ingestion_plane.sql b/sql/007_ingestion_plane.sql new file mode 100644 index 0000000..508ae86 --- /dev/null +++ b/sql/007_ingestion_plane.sql @@ -0,0 +1,162 @@ +-- 0pt ingestion plane (P060-P068). +-- +-- Adds the durable state the Source Registry and the ingestion runner need +-- and that sql/002_source_registry.sql did NOT carry: +-- +-- * opt_source_authorization -- P060 "authorization": the recorded decision +-- that this tenant may use this source at all. Separate from rights +-- (license/purpose) on purpose: rights say WHAT the data may be used for, +-- authorization says WHO signed off that 0pt may connect. Both are +-- required to activate. +-- * opt_source_schedule -- P060 "schedule": ingestion mode + cadence. +-- * opt_source_schemas -- P060 "schema" + P063 schema evolution: +-- one row per (source_id, schema_version), plus the compatibility policy +-- that governs the NEXT version. +-- * opt_ingestion_checkpoints -- P060 "checkpoint" + P062: the durable +-- resume point per (source_id, partition_key). This is the table that +-- makes "restart resumes without silent gaps" true rather than aspirational. +-- * opt_ingestion_ledger -- P068: idempotency ledger. One row per +-- (source_id, record_key, payload hash); a repeated backfill that +-- re-delivers the same record is recognised, not re-emitted. +-- * opt_quality_quarantine -- P064: where envelopes go when the declared +-- enforcement policy says "quarantine" instead of "block". +-- * opt_lineage_edges -- P067: the LOCAL, QUERYABLE projection of +-- the OpenLineage events 0pt already emits to AUD0. AUD0 remains the +-- system of record; this table exists because "emitted to AUD0" is not +-- the same thing as "queryable from 0pt", and the P067 DoD asks for the +-- latter. +-- +-- Every statement is IF NOT EXISTS / idempotent so it can be applied to an +-- already-migrated database, matching sql/002_source_registry.sql's contract +-- (tests/p0/*.realinfra.test.ts apply the files themselves). +-- +-- Ordering: requires sql/002_source_registry.sql (opt_sources) to have run. + +-- ── P060 owner + the pre-activation status ──────────────────────────────── +-- `created_by` (002) records who typed the registration; `owner_id` records +-- who is ACCOUNTABLE for the source afterwards. The activation gate requires +-- the latter, so they cannot be collapsed. +ALTER TABLE opt_sources ADD COLUMN IF NOT EXISTS owner_id TEXT; +CREATE INDEX IF NOT EXISTS opt_sources_owner_idx ON opt_sources (owner_id); + +-- New sources are created 'pending' (see src/repositories/*-source-repository.ts). +-- The column default in 002 is 'active'; it is changed here so a plain INSERT +-- that omits `status` -- from any client, including psql -- cannot produce an +-- unclassified-but-active source. Rows that are already 'active' are left +-- exactly as they are: this migration does not retroactively deactivate +-- anything an operator has already approved. +ALTER TABLE opt_sources ALTER COLUMN status SET DEFAULT 'pending'; + +-- ── P060 authorization ──────────────────────────────────────────────────── +-- One row per source. `basis` is the legal/contractual ground; `granted_by` +-- is the accountable actor id (never "system"); `evidence_ref` points at the +-- AUD0 evidence or 1p/contract artifact. `expires_at` is honoured by the +-- activation gate -- an expired authorization does not classify a source. +CREATE TABLE IF NOT EXISTS opt_source_authorization ( + source_id TEXT PRIMARY KEY REFERENCES opt_sources (source_id), + tenant_id TEXT NOT NULL, + basis TEXT NOT NULL, -- contract | public_register | consent | legitimate_interest | internal + granted_by TEXT NOT NULL, -- accountable actor id + granted_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ, + evidence_ref TEXT, -- aud0 evidence ref / contract artifact ref + notes TEXT, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS opt_source_authorization_tenant_idx ON opt_source_authorization (tenant_id); + +-- ── P060 schedule ───────────────────────────────────────────────────────── +CREATE TABLE IF NOT EXISTS opt_source_schedule ( + source_id TEXT PRIMARY KEY REFERENCES opt_sources (source_id), + tenant_id TEXT NOT NULL, + mode TEXT NOT NULL, -- batch | stream | cdc + interval_seconds INT, -- batch/cdc poll cadence; NULL for continuous streams + cron TEXT, -- optional cron expression (documented, not executed here) + enabled BOOLEAN NOT NULL DEFAULT false, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS opt_source_schedule_tenant_idx ON opt_source_schedule (tenant_id); + +-- ── P060 schema + P063 schema evolution ─────────────────────────────────── +-- `fields` is the record schema as JSON: {"": {"type": "...", +-- "required": true|false}}. `compatibility` governs what the NEXT registered +-- version is allowed to change (see src/schema/compatibility.ts). +CREATE TABLE IF NOT EXISTS opt_source_schemas ( + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + schema_version INT NOT NULL, + tenant_id TEXT NOT NULL, + fields JSONB NOT NULL, + compatibility TEXT NOT NULL, -- backward | forward | full | none + registered_at TIMESTAMPTZ NOT NULL DEFAULT now(), + registered_by TEXT, + PRIMARY KEY (source_id, schema_version) +); +CREATE INDEX IF NOT EXISTS opt_source_schemas_tenant_idx ON opt_source_schemas (tenant_id); + +-- ── P060 checkpoint + P062 durable resume ───────────────────────────────── +-- `cursor` is opaque to 0pt -- the connector defines it (an offset, an LSN, +-- a page token). `sequence` is 0pt's own monotonic envelope counter for the +-- partition, used to detect gaps on resume. +CREATE TABLE IF NOT EXISTS opt_ingestion_checkpoints ( + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + partition_key TEXT NOT NULL, + tenant_id TEXT NOT NULL, + cursor TEXT, + sequence BIGINT NOT NULL DEFAULT 0, + records_committed BIGINT NOT NULL DEFAULT 0, + committed_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (source_id, partition_key) +); +CREATE INDEX IF NOT EXISTS opt_ingestion_checkpoints_tenant_idx ON opt_ingestion_checkpoints (tenant_id); + +-- ── P068 backfill idempotency ledger ────────────────────────────────────── +-- A backfill/replay that re-delivers a record 0pt has already accepted must +-- be a no-op, not a duplicate. `payload_hash` distinguishes "same record, +-- same content" (skip) from "same record, new content" (an update, emitted). +CREATE TABLE IF NOT EXISTS opt_ingestion_ledger ( + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + record_key TEXT NOT NULL, + tenant_id TEXT NOT NULL, + payload_hash TEXT NOT NULL, + first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + delivery_count BIGINT NOT NULL DEFAULT 1, + PRIMARY KEY (source_id, record_key) +); +CREATE INDEX IF NOT EXISTS opt_ingestion_ledger_tenant_idx ON opt_ingestion_ledger (tenant_id); + +-- ── P064 quarantine ─────────────────────────────────────────────────────── +CREATE TABLE IF NOT EXISTS opt_quality_quarantine ( + quarantine_id TEXT PRIMARY KEY, + source_id TEXT NOT NULL REFERENCES opt_sources (source_id), + tenant_id TEXT NOT NULL, + run_id TEXT NOT NULL, + record_key TEXT NOT NULL, + violations JSONB NOT NULL, -- [{expectation_type, kwargs, detail}] + envelope JSONB NOT NULL, + quarantined_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS opt_quality_quarantine_source_idx ON opt_quality_quarantine (source_id, quarantined_at DESC); +CREATE INDEX IF NOT EXISTS opt_quality_quarantine_tenant_idx ON opt_quality_quarantine (tenant_id); + +-- ── P067 queryable lineage projection ───────────────────────────────────── +-- One row per (run, input, output) edge. `edge_id` is deterministic +-- (sha256 of the tuple) so a re-emitted idempotent run does not duplicate +-- edges -- the same dedup property LineageEmitter relies on in AUD0. +CREATE TABLE IF NOT EXISTS opt_lineage_edges ( + edge_id TEXT PRIMARY KEY, + tenant_id TEXT NOT NULL, + run_id TEXT NOT NULL, + capability_id TEXT NOT NULL, + input_namespace TEXT NOT NULL, + input_name TEXT NOT NULL, + input_ref TEXT, + output_namespace TEXT NOT NULL, + output_name TEXT NOT NULL, + output_ref TEXT, + event_time TIMESTAMPTZ NOT NULL, + correlation_id TEXT +); +CREATE INDEX IF NOT EXISTS opt_lineage_edges_output_idx ON opt_lineage_edges (tenant_id, output_name); +CREATE INDEX IF NOT EXISTS opt_lineage_edges_input_idx ON opt_lineage_edges (tenant_id, input_name); +CREATE INDEX IF NOT EXISTS opt_lineage_edges_run_idx ON opt_lineage_edges (tenant_id, run_id); diff --git a/src/connectors/conformance.ts b/src/connectors/conformance.ts new file mode 100644 index 0000000..199c863 --- /dev/null +++ b/src/connectors/conformance.ts @@ -0,0 +1,276 @@ +/** + * src/connectors/conformance.ts + * ============================== + * P061 DoD: "two heterogeneous connectors pass the same conformance suite." + * + * This module IS that suite. It is a plain async function returning a report + * — not a vitest file — so it can be run three ways without duplication: + * 1. from tests (tests/p0/connector-conformance.test.ts runs it per connector), + * 2. from an operator script against a real configured source, + * 3. from a future capability route as a pre-activation check. + * + * Each check is a named assertion about the CONTRACT, not about any one + * connector's data. A connector that cannot serve a check honestly (e.g. it + * has no partitions under the given config) FAILS it — there is no "skip" + * verdict, because a skip that reads as a pass is how a conformance suite + * becomes decoration. + */ +import type { + BackfillRequest, + Connector, + ConnectorConfig, + ConnectorContext, +} from "./connector"; + +export interface ConformanceCheck { + name: string; + passed: boolean; + detail: string; +} + +export interface ConformanceReport { + connector_id: string; + connector_version: string; + passed: boolean; + checks: ConformanceCheck[]; +} + +export interface ConformanceFixture { + /** A config the connector must accept. */ + validConfig: ConnectorConfig; + /** A config the connector must REJECT in validate() without doing IO. */ + invalidConfig: ConnectorConfig; + ctx: ConnectorContext; + /** + * A bounded backfill range that overlaps data the connector can serve. + * `from`/`to` are connector-defined (documented per connector). + */ + backfill: Omit & { partition_key?: string }; + /** Total number of records the fixture's first partition holds, so paging can be checked exactly. */ + expectedRecordCount: number; +} + +async function check( + checks: ConformanceCheck[], + name: string, + fn: () => Promise | string, +): Promise { + try { + const detail = await fn(); + checks.push({ name, passed: true, detail }); + } catch (err) { + checks.push({ + name, + passed: false, + detail: err instanceof Error ? err.message : String(err), + }); + } +} + +function assert(condition: boolean, message: string): void { + if (!condition) throw new Error(message); +} + +/** + * Run the full conformance suite against one connector instance. + * The connector is left disconnected on return, pass or fail. + */ +export async function runConnectorConformance( + connector: Connector, + fixture: ConformanceFixture, +): Promise { + const checks: ConformanceCheck[] = []; + + await check(checks, "identity: non-empty id and version", () => { + assert(connector.connector_id.length > 0, "connector_id is empty"); + assert(connector.connector_version.length > 0, "connector_version is empty"); + return `${connector.connector_id}@${connector.connector_version}`; + }); + + await check(checks, "validate: accepts a valid config without IO", () => { + const res = connector.validate(fixture.validConfig); + assert(res.valid, `valid config rejected: ${res.errors.join("; ")}`); + assert(res.errors.length === 0, "valid config produced errors"); + return "valid config accepted"; + }); + + await check(checks, "validate: rejects an invalid config and says why", () => { + const res = connector.validate(fixture.invalidConfig); + assert(!res.valid, "invalid config was accepted (fail-open)"); + assert(res.errors.length > 0, "rejection carried no reason"); + return `rejected with ${res.errors.length} reason(s)`; + }); + + await check(checks, "operations before connect() fail closed", async () => { + let threw = false; + try { + await connector.read({ partition_key: "anything", limit: 1 }); + } catch { + threw = true; + } + assert(threw, "read() succeeded without a session"); + return "read() refused without a session"; + }); + + await check(checks, "connect: idempotent", async () => { + await connector.connect(fixture.validConfig, fixture.ctx); + await connector.connect(fixture.validConfig, fixture.ctx); + return "connect() called twice without error"; + }); + + let partitionKey = fixture.backfill.partition_key ?? ""; + await check(checks, "discover: reports at least one partition with a schema", async () => { + const discovery = await connector.discover(); + assert(discovery.partitions.length > 0, "no partitions discovered"); + assert(discovery.modes.length > 0, "no ingestion modes declared"); + const first = discovery.partitions[0]; + assert(first.partition_key.length > 0, "partition_key is empty"); + assert( + Object.keys(first.schema).length > 0, + `partition "${first.partition_key}" reported an empty schema`, + ); + if (!partitionKey) partitionKey = first.partition_key; + return `${discovery.partitions.length} partition(s), modes: ${discovery.modes.join(",")}`; + }); + + await check(checks, "health: answers with a status instead of throwing", async () => { + const health = await connector.health(); + assert( + ["up", "degraded", "down"].includes(health.status), + `unexpected health status "${health.status}"`, + ); + return `status=${health.status}`; + }); + + await check(checks, "read: honours the limit and advances the cursor", async () => { + const first = await connector.read({ partition_key: partitionKey, limit: 1 }); + assert(first.records.length <= 1, `limit 1 returned ${first.records.length} records`); + assert(first.records.length === 1, "limit 1 returned nothing — fixture has no data"); + assert(first.next_cursor !== undefined, "no cursor returned after a non-empty read"); + const second = await connector.read({ + partition_key: partitionKey, + cursor: first.next_cursor, + limit: 1, + }); + if (second.records.length > 0) { + assert( + second.records[0].record_key !== first.records[0].record_key, + "reading from the returned cursor re-delivered the same record (cursor is not exclusive)", + ); + } + return `cursor advanced past ${first.records[0].record_key}`; + }); + + await check(checks, "read: paging covers every record exactly once", async () => { + const seen: string[] = []; + let cursor: string | undefined; + for (let page = 0; page < 100; page++) { + const res = await connector.read({ partition_key: partitionKey, cursor, limit: 2 }); + for (const r of res.records) seen.push(r.record_key); + cursor = res.next_cursor; + if (res.exhausted || res.records.length === 0) break; + } + assert( + seen.length === fixture.expectedRecordCount, + `paged ${seen.length} records, fixture declares ${fixture.expectedRecordCount}`, + ); + assert(new Set(seen).size === seen.length, "paging delivered a duplicate record_key"); + return `${seen.length} records, no duplicates`; + }); + + await check(checks, "read: an exhausted partition reports exhausted", async () => { + const everything = await connector.read({ partition_key: partitionKey, limit: 1000 }); + const past = await connector.read({ + partition_key: partitionKey, + cursor: everything.next_cursor, + limit: 1000, + }); + assert(past.records.length === 0, "reading past the end returned records"); + assert(past.exhausted, "reading past the end did not report exhausted"); + return "end of partition reported honestly"; + }); + + await check(checks, "checkpoint: reflects the session position", async () => { + const cp = await connector.checkpoint(partitionKey); + assert(cp.partition_key === partitionKey, "checkpoint returned a different partition"); + return `cursor=${cp.cursor ?? ""}`; + }); + + await check(checks, "resume: continues from a committed checkpoint without re-delivery", async () => { + const firstPage = await connector.read({ partition_key: partitionKey, limit: 2 }); + assert(firstPage.records.length > 0, "fixture partition is empty"); + const committed = { + source_id: fixture.ctx.source_id, + tenant_id: fixture.ctx.tenant_id, + partition_key: partitionKey, + cursor: firstPage.next_cursor, + sequence: firstPage.records.length, + records_committed: firstPage.records.length, + committed_at: new Date().toISOString(), + }; + const resumed = await connector.resume(committed, 1000); + const firstKeys = new Set(firstPage.records.map((r) => r.record_key)); + for (const r of resumed.records) { + assert( + !firstKeys.has(r.record_key), + `resume re-delivered already-committed record "${r.record_key}"`, + ); + } + assert( + firstPage.records.length + resumed.records.length === fixture.expectedRecordCount, + `resume covered ${firstPage.records.length + resumed.records.length} of ${fixture.expectedRecordCount} records — a gap`, + ); + return `resumed ${resumed.records.length} remaining records with no gap and no overlap`; + }); + + await check(checks, "backfill: serves a bounded range", async () => { + const res = await connector.backfill({ + partition_key: partitionKey, + from: fixture.backfill.from, + to: fixture.backfill.to, + limit: 1000, + }); + assert(res.records.length > 0, "bounded backfill range returned nothing"); + return `${res.records.length} record(s) in range`; + }); + + await check(checks, "backfill: refuses an unbounded range", async () => { + let threw = false; + try { + await connector.backfill({ + partition_key: partitionKey, + from: fixture.backfill.from, + to: "", + limit: 1000, + }); + } catch { + threw = true; + } + assert(threw, "an unbounded backfill was accepted"); + return "unbounded backfill refused"; + }); + + await check(checks, "disconnect: idempotent and safe", async () => { + await connector.disconnect(); + await connector.disconnect(); + return "disconnect() called twice without error"; + }); + + await check(checks, "operations after disconnect() fail closed", async () => { + let threw = false; + try { + await connector.read({ partition_key: partitionKey, limit: 1 }); + } catch { + threw = true; + } + assert(threw, "read() succeeded after disconnect()"); + return "read() refused after disconnect()"; + }); + + return { + connector_id: connector.connector_id, + connector_version: connector.connector_version, + passed: checks.every((c) => c.passed), + checks, + }; +} diff --git a/src/connectors/connector.ts b/src/connectors/connector.ts new file mode 100644 index 0000000..3734598 --- /dev/null +++ b/src/connectors/connector.ts @@ -0,0 +1,163 @@ +/** + * src/connectors/connector.ts + * ============================ + * P061: THE connector contract. Nine operations, one interface, every + * connector implements all nine: + * + * validate — is this config usable? Pure, no IO, no network. Fail-closed. + * connect — open whatever session the source needs. Must be idempotent. + * discover — what partitions exist, and what shape do their records have? + * health — is the source answering right now? + * read — produce records from a cursor forward, bounded by `limit`. + * checkpoint — the cursor meaning "everything before this is produced". + * resume — continue from a previously committed checkpoint. + * backfill — re-read a bounded historical range. Bounded, never "all". + * disconnect — release the session. Must be idempotent and safe after an error. + * + * WHY AN INTERFACE AND NOT A BASE CLASS: connectors are heterogeneous by + * nature (an HTTP API paginates by opaque token, a file store seeks by byte + * offset, a change feed advances an LSN). The contract fixes the SHAPE of + * the interaction and the meaning of the cursor; it does not pretend the + * mechanics are the same. `src/connectors/conformance.ts` is what proves two + * unrelated implementations really do honour the same shape. + * + * WHAT THIS REPLACES, HONESTLY: services/api/sources/connectors.py has + * per-source `async def fetch_x()` functions and nothing else — no lifecycle, + * no cursor, no resume, no health, no disconnect. Those functions remain the + * Python engine's fetch layer; they are NOT retrofitted here (that engine is + * not deployed — api.0pt.app answers 404 — and rewriting it blind would be + * the exact "build it again" failure this programme forbids). This contract + * is the seam a Python connector is wrapped into when that engine is revived. + */ +import type { IngestionCheckpoint, IngestionMode } from "../types/ingestion"; +import type { RecordSchemaFields } from "../schema/record-schema"; + +/** Config is connector-specific; the contract only requires it be serializable. */ +export type ConnectorConfig = Record; + +export interface ConnectorContext { + tenant_id: string; + project_id: string; + source_id: string; + /** Correlates every operation of one ingestion run. */ + run_id: string; +} + +/** Result of `validate` — never throws, always answers. */ +export interface ValidationResult { + valid: boolean; + /** One entry per problem. Empty iff valid. */ + errors: string[]; +} + +/** One readable partition (a table, a topic, a file, a page family). */ +export interface PartitionDescriptor { + partition_key: string; + /** Human label; may equal partition_key. */ + name: string; + /** Record shape as the connector currently sees it — fed straight into the schema registry (P063). */ + schema: RecordSchemaFields; +} + +export interface DiscoveryResult { + partitions: PartitionDescriptor[]; + /** Modes this connector can actually serve for this config. */ + modes: IngestionMode[]; +} + +export interface ConnectorHealth { + status: "up" | "degraded" | "down"; + latency_ms?: number; + /** Short operator-facing text — never a raw upstream body. */ + detail?: string; +} + +export interface ReadRequest { + partition_key: string; + /** Start AFTER this cursor. Undefined = from the beginning of the partition. */ + cursor?: string; + /** Hard upper bound on records returned. Connectors MUST honour it. */ + limit: number; +} + +/** A record as the connector sees it, before the runner wraps it in an envelope. */ +export interface ConnectorRecord { + /** Stable business key within the partition. */ + record_key: string; + /** Null only for a delete in a change feed. */ + payload: Record | null; + operation?: "upsert" | "insert" | "update" | "delete"; + occurred_at?: string; + /** Cursor value AT this record — the runner records it per envelope. */ + cursor?: string; +} + +export interface ReadResult { + records: ConnectorRecord[]; + /** + * Cursor meaning "everything up to and including `records` has been + * produced". The runner commits this only after the records are accepted. + */ + next_cursor?: string; + /** False when the connector knows more records are immediately available. */ + exhausted: boolean; +} + +/** A bounded historical range. Unbounded backfills are rejected by the contract. */ +export interface BackfillRequest { + partition_key: string; + /** Inclusive lower bound (cursor or ISO timestamp — connector-defined, documented per connector). */ + from: string; + /** Exclusive upper bound. Required: a backfill with no end is a full reload, not a backfill. */ + to: string; + limit: number; +} + +export interface Connector { + readonly connector_id: string; + readonly connector_version: string; + + /** Pure config check. MUST NOT perform IO. MUST NOT throw. */ + validate(config: ConnectorConfig): ValidationResult; + + /** Open a session. Idempotent: calling it twice is a no-op, not an error. */ + connect(config: ConnectorConfig, ctx: ConnectorContext): Promise; + + /** Requires a live session. */ + discover(): Promise; + + /** Requires a live session. Never throws for an unhealthy source — reports it. */ + health(): Promise; + + /** Requires a live session. Honours `limit` exactly. */ + read(req: ReadRequest): Promise; + + /** + * The connector's own view of where it stands for a partition. The DURABLE + * checkpoint is owned by the runner (CheckpointStore); this is the + * in-session position, used to cross-check a resume. + */ + checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }>; + + /** + * Continue from a previously committed checkpoint. Equivalent to + * `read({cursor: checkpoint.cursor})` for cursor-based connectors, but + * explicit so a connector that must re-establish server-side state (a + * replication slot, a scroll context) has a place to do it. + */ + resume(checkpoint: IngestionCheckpoint, limit: number): Promise; + + /** Bounded historical re-read. MUST reject an unbounded range. */ + backfill(req: BackfillRequest): Promise; + + /** Idempotent; safe to call without a session and after an error. */ + disconnect(): Promise; +} + +/** Thrown by connectors when an operation needs a session that was never opened. */ +export class ConnectorNotConnectedError extends Error { + constructor(connectorId: string, operation: string) { + super(`connector ${connectorId}: ${operation} requires connect() first`); + this.name = "ConnectorNotConnectedError"; + } +} diff --git a/src/connectors/http-json-connector.ts b/src/connectors/http-json-connector.ts new file mode 100644 index 0000000..cb11627 --- /dev/null +++ b/src/connectors/http-json-connector.ts @@ -0,0 +1,274 @@ +/** + * src/connectors/http-json-connector.ts + * ====================================== + * P061 conformance connector #1: a real HTTP JSON API source. + * + * Heterogeneity (the point of having two): this connector talks to a REMOTE + * service over the network, pages by an OPAQUE token the server hands back, + * cannot know how many records exist until the server says it is done, and + * has a genuine health check (an HTTP round trip). Its sibling + * (jsonl-file-connector.ts) does none of those things. + * + * COMPLIANCE (CLAUDE.md §1): this connector is a plain, self-identifying + * JSON client. It sends a fixed User-Agent, honours the caller's timeout, and + * treats 401/403/429/451 as a STOP signal for that source — the same posture + * services/api/sources/connectors.py's BLOCKED_STATUSES already takes. There + * is no retry-harder path, no proxy rotation, no header spoofing, and adding + * any would be a compliance violation, not a feature. + * + * `fetchImpl` is injectable so tests drive a real request/response cycle + * without a network. That is a seam for TESTS, not a mock fallback: with no + * override it uses global fetch and talks to the configured URL for real. + */ +import { + ConnectorNotConnectedError, + type BackfillRequest, + type Connector, + type ConnectorConfig, + type ConnectorContext, + type ConnectorHealth, + type ConnectorRecord, + type DiscoveryResult, + type ReadRequest, + type ReadResult, + type ValidationResult, +} from "./connector"; +import type { IngestionCheckpoint } from "../types/ingestion"; +import type { RecordSchemaFields } from "../schema/record-schema"; + +export const HTTP_JSON_CONNECTOR_VERSION = "http-json-connector@1"; + +/** Statuses that mean "stop asking this way" — never "retry harder". */ +const BLOCKED_STATUSES = new Set([401, 403, 429, 451]); + +export interface HttpJsonConnectorConfig extends ConnectorConfig { + /** Absolute https:// (or http:// for a local fixture) base URL. */ + base_url: string; + /** Path serving records, e.g. "/v1/companies". */ + records_path: string; + /** Partition key this endpoint serves. */ + partition_key: string; + /** Declared record shape for the partition — the connector does not infer schemas from data. */ + schema: RecordSchemaFields; + /** JSON field holding the business key of each record. */ + record_key_field: string; + /** + * JSON field holding each record's own cursor value (an id, a sequence, an + * `updated_at`). Optional, but REQUIRED IN PRACTICE for any API that stops + * returning `next_cursor` once a page is exhausted: without it the + * connector has nothing to resume from after reading to the end, and + * resume() would silently restart at the beginning. The conformance suite + * catches exactly that ("read: an exhausted partition reports exhausted"), + * which is how this field came to exist. + */ + cursor_field?: string; + /** Request timeout (ms). Default 15000. */ + timeout_ms?: number; +} + +type FetchImpl = (url: string, init: RequestInit) => Promise; + +/** Wire shape this connector expects from the upstream endpoint. */ +interface UpstreamPage { + records?: Array>; + next_cursor?: string | null; +} + +function isConfig(config: ConnectorConfig): config is HttpJsonConnectorConfig { + return typeof config.base_url === "string" && typeof config.records_path === "string"; +} + +export class HttpJsonConnector implements Connector { + readonly connector_id = "http_json"; + readonly connector_version = HTTP_JSON_CONNECTOR_VERSION; + + private config?: HttpJsonConnectorConfig; + private ctx?: ConnectorContext; + private lastCursor = new Map(); + + constructor(private readonly fetchImpl: FetchImpl = (url, init) => fetch(url, init)) {} + + validate(config: ConnectorConfig): ValidationResult { + const errors: string[] = []; + if (typeof config.base_url !== "string" || config.base_url.length === 0) { + errors.push("base_url is required"); + } else if (!/^https?:\/\//.test(config.base_url)) { + errors.push("base_url must be an absolute http(s) URL"); + } + if (typeof config.records_path !== "string" || !config.records_path.startsWith("/")) { + errors.push("records_path is required and must start with /"); + } + if (typeof config.partition_key !== "string" || config.partition_key.length === 0) { + errors.push("partition_key is required"); + } + if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) { + errors.push("record_key_field is required"); + } + const schema = config.schema; + if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) { + errors.push("schema is required and must declare at least one field"); + } + return { valid: errors.length === 0, errors }; + } + + async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise { + const verdict = this.validate(config); + if (!verdict.valid) { + throw new Error(`http_json connector config invalid: ${verdict.errors.join("; ")}`); + } + if (!isConfig(config)) throw new Error("http_json connector config invalid"); + this.config = config; + this.ctx = ctx; + } + + async discover(): Promise { + const config = this.requireSession("discover"); + return { + partitions: [ + { + partition_key: config.partition_key, + name: `${config.base_url}${config.records_path}`, + schema: config.schema, + }, + ], + // A polled JSON endpoint serves batch and (by repeated polling) stream. + // It is NOT a change feed: it carries no delete events, so "cdc" is not + // declared. Declaring it would be a lie the runner would act on. + modes: ["batch", "stream"], + }; + } + + async health(): Promise { + const config = this.requireSession("health"); + const started = Date.now(); + try { + const res = await this.request(this.url(config.records_path, { limit: "1" }), config); + const latency = Date.now() - started; + if (BLOCKED_STATUSES.has(res.status)) { + return { status: "down", latency_ms: latency, detail: `source signalled a block (HTTP ${res.status})` }; + } + if (!res.ok) return { status: "degraded", latency_ms: latency, detail: `HTTP ${res.status}` }; + return { status: "up", latency_ms: latency }; + } catch (err) { + return { + status: "down", + latency_ms: Date.now() - started, + detail: err instanceof Error ? err.message : "unreachable", + }; + } + } + + async read(req: ReadRequest): Promise { + const config = this.requireSession("read"); + const query: Record = { limit: String(req.limit) }; + if (req.cursor !== undefined) query.cursor = req.cursor; + const res = await this.request(this.url(config.records_path, query), config); + if (BLOCKED_STATUSES.has(res.status)) { + throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`); + } + if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`); + const page = (await res.json()) as UpstreamPage; + const rawRecords = page.records ?? []; + // The contract says limit is a HARD bound; a server that over-delivers is + // truncated here rather than silently widening the caller's page. + const records = rawRecords.slice(0, req.limit).map((row) => this.toRecord(row, config)); + const serverCursor = page.next_cursor ?? undefined; + // When the server stops issuing a cursor at the end of a page, fall back + // to the last record's own cursor value so a later resume() starts AFTER + // the data already produced instead of at the beginning. + const trailing = records.length > 0 ? records[records.length - 1].cursor : undefined; + const nextCursor = serverCursor ?? trailing ?? req.cursor; + if (records.length > 0) this.lastCursor.set(req.partition_key, nextCursor); + return { + records, + next_cursor: nextCursor, + // The SERVER's cursor is what says "there is more"; a fallback cursor + // only preserves position, it never claims more data exists. + exhausted: serverCursor === undefined || records.length === 0, + }; + } + + async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> { + this.requireSession("checkpoint"); + return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) }; + } + + async resume(checkpoint: IngestionCheckpoint, limit: number): Promise { + this.requireSession("resume"); + return this.read({ + partition_key: checkpoint.partition_key, + cursor: checkpoint.cursor, + limit, + }); + } + + async backfill(req: BackfillRequest): Promise { + const config = this.requireSession("backfill"); + if (!req.from || !req.to) { + throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill"); + } + const res = await this.request( + this.url(config.records_path, { from: req.from, to: req.to, limit: String(req.limit) }), + config, + ); + if (BLOCKED_STATUSES.has(res.status)) { + throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`); + } + if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`); + const page = (await res.json()) as UpstreamPage; + const records = (page.records ?? []).slice(0, req.limit).map((row) => this.toRecord(row, config)); + return { records, next_cursor: page.next_cursor ?? undefined, exhausted: true }; + } + + async disconnect(): Promise { + this.config = undefined; + this.ctx = undefined; + this.lastCursor.clear(); + } + + // --- internals ----------------------------------------------------------- + + private toRecord(row: Record, config: HttpJsonConnectorConfig): ConnectorRecord { + const key = row[config.record_key_field]; + if (typeof key !== "string" && typeof key !== "number") { + throw new Error(`record is missing string/number key field "${config.record_key_field}"`); + } + let cursor: string | undefined; + if (config.cursor_field) { + const raw = row[config.cursor_field]; + if (typeof raw === "string" || typeof raw === "number") cursor = String(raw); + } + return { record_key: String(key), payload: row, operation: "upsert", cursor }; + } + + private requireSession(op: string): HttpJsonConnectorConfig { + if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op); + return this.config; + } + + private url(path: string, query: Record): string { + const config = this.config; + if (!config) throw new ConnectorNotConnectedError(this.connector_id, "url"); + const url = new URL(path, config.base_url); + for (const [k, v] of Object.entries(query)) url.searchParams.set(k, v); + return url.toString(); + } + + private async request(url: string, config: HttpJsonConnectorConfig): Promise { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), config.timeout_ms ?? 15_000); + try { + return await this.fetchImpl(url, { + method: "GET", + signal: controller.signal, + headers: { + Accept: "application/json", + "User-Agent": "0pt-connector/1 (+https://0pt.app; SIAX Technology AB)", + ...(this.ctx ? { "X-Correlation-Id": this.ctx.run_id } : {}), + }, + }); + } finally { + clearTimeout(timer); + } + } +} diff --git a/src/connectors/jsonl-file-connector.ts b/src/connectors/jsonl-file-connector.ts new file mode 100644 index 0000000..4386a3a --- /dev/null +++ b/src/connectors/jsonl-file-connector.ts @@ -0,0 +1,224 @@ +/** + * src/connectors/jsonl-file-connector.ts + * ======================================== + * P061 conformance connector #2: a JSONL file drop on local disk. + * + * Heterogeneity vs http-json-connector.ts — these two share nothing but the + * contract: + * - no network, no server, no opaque token; the cursor is a LINE OFFSET the + * connector computes itself, + * - it knows the full record count up front (the file is finite and local), + * so `exhausted` is exact rather than server-reported, + * - health is a filesystem stat, not an HTTP round trip, + * - it serves `cdc`-shaped records: a JSONL line may carry an `_op` field + * ("insert"/"update"/"delete"), so this connector emits real delete + * envelopes with a null payload. The HTTP connector cannot, + * - backfill bounds are line numbers, not timestamps. + * + * That last point is why the conformance suite takes `from`/`to` from a + * fixture: the contract fixes that a backfill is BOUNDED, not what the + * bounds mean. Each connector documents its own bound vocabulary. + */ +import { promises as fs } from "fs"; +import path from "path"; +import { + ConnectorNotConnectedError, + type BackfillRequest, + type Connector, + type ConnectorConfig, + type ConnectorContext, + type ConnectorHealth, + type ConnectorRecord, + type DiscoveryResult, + type ReadRequest, + type ReadResult, + type ValidationResult, +} from "./connector"; +import type { IngestionCheckpoint } from "../types/ingestion"; +import type { RecordSchemaFields } from "../schema/record-schema"; + +export const JSONL_FILE_CONNECTOR_VERSION = "jsonl-file-connector@1"; + +export interface JsonlFileConnectorConfig extends ConnectorConfig { + /** Absolute path to the .jsonl file. */ + file_path: string; + partition_key: string; + schema: RecordSchemaFields; + record_key_field: string; +} + +function isConfig(config: ConnectorConfig): config is JsonlFileConnectorConfig { + return typeof config.file_path === "string" && typeof config.partition_key === "string"; +} + +/** Cursor vocabulary: the decimal count of lines already produced. */ +function parseCursor(cursor: string | undefined): number { + if (cursor === undefined) return 0; + const n = Number.parseInt(cursor, 10); + if (!Number.isInteger(n) || n < 0) throw new Error(`invalid cursor "${cursor}" — expected a line offset`); + return n; +} + +export class JsonlFileConnector implements Connector { + readonly connector_id = "jsonl_file"; + readonly connector_version = JSONL_FILE_CONNECTOR_VERSION; + + private config?: JsonlFileConnectorConfig; + private lastCursor = new Map(); + + validate(config: ConnectorConfig): ValidationResult { + const errors: string[] = []; + if (typeof config.file_path !== "string" || config.file_path.length === 0) { + errors.push("file_path is required"); + } else if (!path.isAbsolute(config.file_path)) { + errors.push("file_path must be absolute"); + } else if (!config.file_path.endsWith(".jsonl")) { + errors.push("file_path must name a .jsonl file"); + } + if (typeof config.partition_key !== "string" || config.partition_key.length === 0) { + errors.push("partition_key is required"); + } + if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) { + errors.push("record_key_field is required"); + } + const schema = config.schema; + if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) { + errors.push("schema is required and must declare at least one field"); + } + return { valid: errors.length === 0, errors }; + } + + async connect(config: ConnectorConfig, _ctx: ConnectorContext): Promise { + const verdict = this.validate(config); + if (!verdict.valid) { + throw new Error(`jsonl_file connector config invalid: ${verdict.errors.join("; ")}`); + } + if (!isConfig(config)) throw new Error("jsonl_file connector config invalid"); + // Fail at connect time, not at first read: a file drop that is not there + // is a connection failure, and the runner must see it as one. + await fs.access(config.file_path); + this.config = config; + } + + async discover(): Promise { + const config = this.requireSession("discover"); + return { + partitions: [ + { partition_key: config.partition_key, name: path.basename(config.file_path), schema: config.schema }, + ], + // A file drop is a batch source; because a line may carry `_op`, it can + // also carry a change log, so cdc is honestly available here. + modes: ["batch", "cdc"], + }; + } + + async health(): Promise { + const config = this.requireSession("health"); + const started = Date.now(); + try { + const stat = await fs.stat(config.file_path); + if (stat.size === 0) { + return { status: "degraded", latency_ms: Date.now() - started, detail: "file is empty" }; + } + return { status: "up", latency_ms: Date.now() - started }; + } catch (err) { + return { + status: "down", + latency_ms: Date.now() - started, + detail: err instanceof Error ? err.message : "stat failed", + }; + } + } + + async read(req: ReadRequest): Promise { + const lines = await this.lines(); + const start = parseCursor(req.cursor); + const slice = lines.slice(start, start + req.limit); + const records = slice.map((line, i) => this.toRecord(line, start + i)); + const nextOffset = start + slice.length; + const cursor = String(nextOffset); + this.lastCursor.set(req.partition_key, cursor); + return { + records, + next_cursor: cursor, + exhausted: nextOffset >= lines.length, + }; + } + + async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> { + this.requireSession("checkpoint"); + return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) }; + } + + async resume(checkpoint: IngestionCheckpoint, limit: number): Promise { + this.requireSession("resume"); + return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit }); + } + + /** Bounds are LINE NUMBERS: `from` inclusive, `to` exclusive. */ + async backfill(req: BackfillRequest): Promise { + this.requireSession("backfill"); + if (!req.from || !req.to) { + throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill"); + } + const from = parseCursor(req.from); + const to = parseCursor(req.to); + if (to <= from) throw new Error(`backfill range is empty or inverted (from=${from}, to=${to})`); + const lines = await this.lines(); + const slice = lines.slice(from, Math.min(to, from + req.limit)); + return { + records: slice.map((line, i) => this.toRecord(line, from + i)), + next_cursor: String(from + slice.length), + exhausted: true, + }; + } + + async disconnect(): Promise { + this.config = undefined; + this.lastCursor.clear(); + } + + // --- internals ----------------------------------------------------------- + + private requireSession(op: string): JsonlFileConnectorConfig { + if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op); + return this.config; + } + + private async lines(): Promise { + const config = this.requireSession("read"); + const raw = await fs.readFile(config.file_path, "utf8"); + return raw.split("\n").filter((l) => l.trim().length > 0); + } + + private toRecord(line: string, lineNumber: number): ConnectorRecord { + const config = this.requireSession("read"); + let parsed: Record; + try { + parsed = JSON.parse(line) as Record; + } catch { + throw new Error(`line ${lineNumber + 1} is not valid JSON`); + } + const key = parsed[config.record_key_field]; + if (typeof key !== "string" && typeof key !== "number") { + throw new Error(`line ${lineNumber + 1} is missing key field "${config.record_key_field}"`); + } + const op = parsed._op; + const operation = + op === "insert" || op === "update" || op === "delete" ? op : ("upsert" as const); + const occurredAt = typeof parsed._occurred_at === "string" ? parsed._occurred_at : undefined; + // `_op`/`_occurred_at` are envelope metadata, not payload fields — leaving + // them in the payload would make every record fail schema conformance for + // an "unregistered field". + const payload: Record = { ...parsed }; + delete payload._op; + delete payload._occurred_at; + return { + record_key: String(key), + payload: operation === "delete" ? null : payload, + operation, + occurred_at: occurredAt, + cursor: String(lineNumber + 1), + }; + } +} diff --git a/src/er/entity-resolution-service.ts b/src/er/entity-resolution-service.ts index 2992a10..913d81f 100644 --- a/src/er/entity-resolution-service.ts +++ b/src/er/entity-resolution-service.ts @@ -23,6 +23,16 @@ * GENERIC means GENERIC: no person-matching logic, no Ratsit/PII pipelines — * records are opaque field maps supplied by the calling product under its * own legal basis; this module implements comparison mechanics only. + * + * P065 STATUS — THIS IS THE CANONICAL ENGINE, AND IT IS NOT YET THE ONLY ONE. + * `services/api/workers/resolve.py` is a second, simplified ER engine (fuzzy + * threshold, auto-merge, no hard rules, no review band) that + * `services/api/workers/pipeline.py` still calls. P065's DoD ("no second + * simplified duplicated production engine remains") is therefore NOT met. + * See that file's header for why it was not rewritten blind in the + * P060-P068 pass and STATUS.md for the removal plan. The ingestion plane + * added in that pass (src/ingestion/*) deliberately does no ER of its own — + * it stops at the sink, so it adds no THIRD engine. */ import crypto from "crypto"; import { SiaxError } from "../platform/errors"; diff --git a/src/ingestion/envelope.ts b/src/ingestion/envelope.ts new file mode 100644 index 0000000..f7038ee --- /dev/null +++ b/src/ingestion/envelope.ts @@ -0,0 +1,87 @@ +/** + * src/ingestion/envelope.ts + * ========================== + * P062: the one place a ConnectorRecord becomes an IngestionEnvelope. Every + * connector, every mode, one function — that is what "normalize all ingestion + * to a common envelope" means operationally. + * + * `hashPayload` is the stable content hash the idempotency ledger (P068) and + * the deterministic `envelope_id` are both built on. Key order must not + * change the hash — two connectors reading the same record can serialize the + * same object with different key order — so keys are sorted recursively. + * Undefined values are dropped (JSON has no undefined); null is kept, because + * "field explicitly null" is different data from "field absent". + */ +import crypto from "crypto"; +import type { ConnectorRecord } from "../connectors/connector"; +import type { EnvelopeOperation, IngestionEnvelope, IngestionMode } from "../types/ingestion"; + +/** Recursively key-sorted JSON — stable across key insertion order. */ +export function stableStringify(value: unknown): string { + if (value === null) return "null"; + if (Array.isArray(value)) return `[${value.map(stableStringify).join(",")}]`; + if (typeof value === "object") { + const entries = Object.entries(value as Record) + .filter(([, v]) => v !== undefined) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); + return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(",")}}`; + } + return JSON.stringify(value) ?? "null"; +} + +/** Content hash of an envelope payload. A delete (null payload) hashes as "deleted". */ +export function hashPayload(payload: Record | null): string { + if (payload === null) return "sha256:deleted"; + return `sha256:${crypto.createHash("sha256").update(stableStringify(payload)).digest("hex")}`; +} + +export interface BuildEnvelopeInput { + record: ConnectorRecord; + source_id: string; + tenant_id: string; + project_id: string; + mode: IngestionMode; + partition_key: string; + schema_version: number; + sequence: number; + connector_id: string; + connector_version: string; + run_id: string; + /** Cursor covering this record, when the connector reports one. */ + cursor?: string; + now?: string; +} + +export function buildEnvelope(input: BuildEnvelopeInput): IngestionEnvelope { + const operation: EnvelopeOperation = input.record.operation ?? "upsert"; + const payload = operation === "delete" ? null : input.record.payload; + const contentHash = hashPayload(payload); + return { + // Deterministic: the same record with the same content re-read after a + // crash produces the same envelope_id, so a downstream sink that dedups + // by id sees a retry, not a new record. + envelope_id: `env_${crypto + .createHash("sha256") + .update(`${input.source_id}|${input.partition_key}|${input.record.record_key}|${contentHash}`) + .digest("hex") + .slice(0, 32)}`, + source_id: input.source_id, + tenant_id: input.tenant_id, + project_id: input.project_id, + mode: input.mode, + operation, + partition_key: input.partition_key, + record_key: input.record.record_key, + schema_version: input.schema_version, + sequence: input.sequence, + occurred_at: input.record.occurred_at, + ingested_at: input.now ?? new Date().toISOString(), + payload, + provenance: { + connector_id: input.connector_id, + connector_version: input.connector_version, + run_id: input.run_id, + cursor: input.cursor ?? input.record.cursor, + }, + }; +} diff --git a/src/ingestion/ingestion-runner.ts b/src/ingestion/ingestion-runner.ts new file mode 100644 index 0000000..73b19a9 --- /dev/null +++ b/src/ingestion/ingestion-runner.ts @@ -0,0 +1,557 @@ +/** + * src/ingestion/ingestion-runner.ts + * ================================== + * The single ingestion highway (P062, P063 gate, P064 gate, P067 emission, + * P068 ledger). One loop, one envelope shape, one durable checkpoint. + * + * ORDERING IS THE WHOLE CONTRACT. Per page: + * + * read/resume -> normalize to envelopes -> schema conformance -> quality gate + * -> ledger PEEK (filter replays) -> SINK -> ledger COMMIT -> checkpoint COMMIT + * + * Both commits happen AFTER the sink resolves. That is what makes "a restart + * resumes without silent gaps" true: a process that dies anywhere before the + * commits leaves the page unconsumed, the next run re-reads it, and nothing is + * lost. The two tempting shortcuts are both data-loss bugs, and one of them + * was actually written here first: + * - committing the CHECKPOINT before the sink skips the page entirely; + * - committing the LEDGER before the sink marks records delivered that the + * sink never received, so the resumed run drops them as duplicates. This + * one shipped in the first draft of this file and was caught by + * tests/p0/ingestion-runner.test.ts's crash/resume case (7 of 10 records + * delivered). The peek/commit split in IngestionRepository exists because + * of it. + * + * The resulting delivery guarantee is AT-LEAST-ONCE, not exactly-once: two + * concurrent runs on the same partition can both peek "new". `envelope_id` is + * deterministic so a dedup-capable sink survives that. A per-(source, + * partition) run lock does not exist yet — see STATUS.md. + * + * A BLOCKED quality page also stops WITHOUT committing: a blocked page must + * stay unconsumed, not be skipped over. + * + * WHAT THIS RUNNER DOES NOT DO, stated plainly: + * - it does not schedule itself (ACT0 owns scheduling; the source's + * `schedule` row is a declaration, not a cron daemon in this repo), + * - it does not write to a lakehouse — the `sink` is supplied by the + * caller, because the Iceberg write boundary is ST0RE's contract + * (src/services/iceberg-write-service.ts) and is owned by another repo + * in this programme (P066), + * - it does not do entity resolution. The canonical ER engine is + * src/er/entity-resolution-service.ts and runs downstream of the sink. + */ +import crypto from "crypto"; +import { SiaxError } from "../platform/errors"; +import { conformanceErrors } from "../schema/compatibility"; +import type { SchemaRegistry } from "../schema/schema-registry"; +import { QualityGate, type QualityEnforcementPolicy } from "../quality/quality-policy"; +import type { IngestionRepository } from "../repositories/ingestion-repository"; +import type { LineageStore } from "../lineage/lineage-store"; +import type { + BackfillRequest, + Connector, + ConnectorConfig, + ReadResult, +} from "../connectors/connector"; +import type { + IngestionCheckpoint, + IngestionEnvelope, + IngestionMode, + QuarantineRecord, +} from "../types/ingestion"; +import { buildEnvelope, hashPayload } from "./envelope"; + +/** Where accepted envelopes go. Supplied by the caller — see the module note. */ +export type IngestionSink = (envelopes: IngestionEnvelope[]) => Promise; + +export interface IngestionRunInput { + source_id: string; + tenant_id: string; + project_id: string; + mode: IngestionMode; + connector: Connector; + config: ConnectorConfig; + sink: IngestionSink; + /** Deterministic run id. Defaults to a random one. */ + run_id?: string; + /** Records per connector read. Default 100. */ + page_size?: number; + /** Safety stop. Default 1000 pages. */ + max_pages?: number; + /** Only this partition; default is every partition discover() reports. */ + partition_key?: string; + /** P064. Omitted = no gate runs, and the report says so explicitly. */ + quality_policy?: QualityEnforcementPolicy; + /** Canonical ref of what the sink produces, for lineage (e.g. "0pt:dataset:ds_x"). */ + output_ref?: string; + correlation_id?: string; +} + +export interface PartitionRunReport { + partition_key: string; + schema_version: number; + pages_read: number; + records_read: number; + envelopes_emitted: number; + duplicates_skipped: number; + changed: number; + quarantined: number; + quality_violations: number; + checkpoint: IngestionCheckpoint; + /** Set when the partition stopped early. */ + stopped_because?: "blocked_by_quality" | "max_pages" | "sink_failed"; +} + +export interface IngestionRunReport { + run_id: string; + source_id: string; + tenant_id: string; + mode: IngestionMode; + status: "succeeded" | "failed"; + /** Honest marker: "none_declared" when the caller supplied no policy at all. */ + quality_policy: "block" | "quarantine" | "warn" | "none_declared"; + partitions: PartitionRunReport[]; + lineage_edges: number; + /** Present when status is "failed". */ + failure?: { code: string; detail: string }; +} + +export class IngestionRunner { + private readonly gate: QualityGate; + + constructor( + private readonly repo: IngestionRepository, + private readonly schemas: SchemaRegistry, + private readonly lineage?: LineageStore, + gate?: QualityGate, + ) { + this.gate = gate ?? new QualityGate(); + } + + async run(input: IngestionRunInput): Promise { + const runId = input.run_id ?? `run_${crypto.randomUUID()}`; + const pageSize = input.page_size ?? 100; + const maxPages = input.max_pages ?? 1000; + const report: IngestionRunReport = { + run_id: runId, + source_id: input.source_id, + tenant_id: input.tenant_id, + mode: input.mode, + status: "succeeded", + quality_policy: input.quality_policy?.on_violation ?? "none_declared", + partitions: [], + lineage_edges: 0, + }; + + const verdict = input.connector.validate(input.config); + if (!verdict.valid) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.ingestion", + `connector config rejected: ${verdict.errors.join("; ")}`, + ); + } + + await input.connector.connect(input.config, { + tenant_id: input.tenant_id, + project_id: input.project_id, + source_id: input.source_id, + run_id: runId, + }); + + try { + const discovery = await input.connector.discover(); + if (!discovery.modes.includes(input.mode)) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.ingestion", + `connector ${input.connector.connector_id} does not serve mode "${input.mode}" for this config (serves: ${discovery.modes.join(", ")})`, + ); + } + const partitions = input.partition_key + ? discovery.partitions.filter((p) => p.partition_key === input.partition_key) + : discovery.partitions; + if (partitions.length === 0) { + throw new SiaxError( + "NOT_FOUND", + "0pt.ingestion", + `partition "${input.partition_key}" was not discovered on source ${input.source_id}`, + ); + } + + // ---- P063: schema gate, BEFORE any record is read ------------------ + // Every partition's shape is checked and registered first. A break + // fails the whole run here, so no envelope — and therefore no consumer + // write — can happen under a broken schema. + const schemaVersions = new Map(); + for (const partition of partitions) { + const evaluated = await this.schemas.evaluate(input.source_id, partition.schema); + if (evaluated.verdict && !evaluated.verdict.compatible) { + throw new SiaxError( + "CONFLICT", + "0pt.ingestion", + `source ${input.source_id} partition "${partition.partition_key}" changed shape in a way that breaks its declared ` + + `"${evaluated.verdict.policy}" compatibility policy — refusing to ingest: ` + + evaluated.verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "), + ); + } + const registered = await this.schemas.register({ + source_id: input.source_id, + tenant_id: input.tenant_id, + fields: partition.schema, + compatibility: evaluated.previous?.compatibility ?? "backward", + registered_by: `run:${runId}`, + }); + schemaVersions.set(partition.partition_key, registered.schema.schema_version); + } + + for (const partition of partitions) { + report.partitions.push( + await this.runPartition(input, { + runId, + pageSize, + maxPages, + partitionKey: partition.partition_key, + schemaVersion: schemaVersions.get(partition.partition_key) ?? 1, + }), + ); + } + + // ---- P067: one lineage event per run, source -> output ------------- + if (this.lineage) { + report.lineage_edges = await this.lineage.record( + { + eventType: "COMPLETE", + eventTime: new Date().toISOString(), + producer: "https://git.siax.io/sax3l/0pt", + run: { runId }, + job: { namespace: "0pt", name: "0pt.ingestion.run.v1" }, + inputs: [ + { + namespace: "0pt", + name: `source/${input.source_id}`, + facets: { siaxResourceRef: { ref: `0pt:source:${input.source_id}` } }, + }, + ], + outputs: [ + { + namespace: "0pt", + name: input.output_ref ?? `ingestion/${input.source_id}`, + ...(input.output_ref ? { facets: { siaxResourceRef: { ref: input.output_ref } } } : {}), + }, + ], + }, + input.tenant_id, + input.correlation_id, + ); + } + + return report; + } finally { + await input.connector.disconnect(); + } + } + + /** + * P068: a bounded historical re-read. Runs through the SAME envelope, + * quality and ledger path as `run`, and deliberately does NOT touch the + * streaming checkpoint — a backfill must not rewind or advance live + * ingestion. Repeating an identical backfill emits nothing: every record + * comes back "duplicate" from the ledger. + */ + async backfill( + input: IngestionRunInput & { range: Omit }, + ): Promise { + const runId = input.run_id ?? `bkf_${crypto.randomUUID()}`; + const pageSize = input.page_size ?? 100; + const verdict = input.connector.validate(input.config); + if (!verdict.valid) { + throw new SiaxError("INVALID_REQUEST", "0pt.ingestion", `connector config rejected: ${verdict.errors.join("; ")}`); + } + await input.connector.connect(input.config, { + tenant_id: input.tenant_id, + project_id: input.project_id, + source_id: input.source_id, + run_id: runId, + }); + try { + const latest = await this.schemas.getLatest(input.source_id); + const schemaVersion = latest?.schema_version ?? 1; + const result = await input.connector.backfill({ ...input.range, limit: pageSize }); + const existing = + (await this.repo.getCheckpoint(input.source_id, input.range.partition_key)) ?? + this.emptyCheckpoint(input, input.range.partition_key); + const outcome = await this.consumePage(input, { + runId, + result, + partitionKey: input.range.partition_key, + schemaVersion, + startSequence: existing.sequence, + schemaFields: latest?.fields, + }); + return { + partition_key: input.range.partition_key, + schema_version: schemaVersion, + pages_read: 1, + records_read: result.records.length, + envelopes_emitted: outcome.emitted, + duplicates_skipped: outcome.duplicates, + changed: outcome.changed, + quarantined: outcome.quarantined, + quality_violations: outcome.violations, + // Unchanged on purpose — a backfill leaves the live checkpoint alone. + checkpoint: existing, + stopped_because: outcome.blocked ? "blocked_by_quality" : undefined, + }; + } finally { + await input.connector.disconnect(); + } + } + + // ------------------------------------------------------------------------- + + private emptyCheckpoint( + input: { source_id: string; tenant_id: string }, + partitionKey: string, + ): IngestionCheckpoint { + return { + source_id: input.source_id, + tenant_id: input.tenant_id, + partition_key: partitionKey, + cursor: undefined, + sequence: 0, + records_committed: 0, + committed_at: new Date(0).toISOString(), + }; + } + + private async runPartition( + input: IngestionRunInput, + opts: { + runId: string; + pageSize: number; + maxPages: number; + partitionKey: string; + schemaVersion: number; + }, + ): Promise { + const schema = await this.schemas.getLatest(input.source_id); + let checkpoint = + (await this.repo.getCheckpoint(input.source_id, opts.partitionKey)) ?? + this.emptyCheckpoint(input, opts.partitionKey); + + const stats = { + pages: 0, + records: 0, + emitted: 0, + duplicates: 0, + changed: 0, + quarantined: 0, + violations: 0, + }; + let stoppedBecause: PartitionRunReport["stopped_because"]; + + for (let page = 0; page < opts.maxPages; page++) { + // First page of a partition that has a durable checkpoint goes through + // resume(), so a connector that must re-establish server-side state has + // its chance; later pages are ordinary reads from the running cursor. + const result: ReadResult = + page === 0 && checkpoint.cursor !== undefined + ? await input.connector.resume(checkpoint, opts.pageSize) + : await input.connector.read({ + partition_key: opts.partitionKey, + cursor: checkpoint.cursor, + limit: opts.pageSize, + }); + + stats.pages++; + stats.records += result.records.length; + if (result.records.length === 0) break; + + const outcome = await this.consumePage(input, { + runId: opts.runId, + result, + partitionKey: opts.partitionKey, + schemaVersion: opts.schemaVersion, + startSequence: checkpoint.sequence, + schemaFields: schema?.fields, + }); + stats.emitted += outcome.emitted; + stats.duplicates += outcome.duplicates; + stats.changed += outcome.changed; + stats.quarantined += outcome.quarantined; + stats.violations += outcome.violations; + + if (outcome.blocked) { + // The checkpoint is NOT advanced: the offending page stays unconsumed + // so the next run sees it again rather than stepping over it. + stoppedBecause = "blocked_by_quality"; + break; + } + + checkpoint = { + source_id: input.source_id, + tenant_id: input.tenant_id, + partition_key: opts.partitionKey, + cursor: result.next_cursor, + sequence: checkpoint.sequence + result.records.length, + records_committed: checkpoint.records_committed + outcome.emitted, + committed_at: new Date().toISOString(), + }; + await this.repo.commitCheckpoint(checkpoint); + + if (result.exhausted) break; + if (page === opts.maxPages - 1) stoppedBecause = "max_pages"; + } + + return { + partition_key: opts.partitionKey, + schema_version: opts.schemaVersion, + pages_read: stats.pages, + records_read: stats.records, + envelopes_emitted: stats.emitted, + duplicates_skipped: stats.duplicates, + changed: stats.changed, + quarantined: stats.quarantined, + quality_violations: stats.violations, + checkpoint, + stopped_because: stoppedBecause, + }; + } + + /** + * Normalize -> schema conformance -> quality gate -> ledger -> sink. + * Throws if the sink throws; the caller must NOT have committed a + * checkpoint by then. + */ + private async consumePage( + input: IngestionRunInput, + opts: { + runId: string; + result: ReadResult; + partitionKey: string; + schemaVersion: number; + startSequence: number; + schemaFields?: Record; + }, + ): Promise<{ + emitted: number; + duplicates: number; + changed: number; + quarantined: number; + violations: number; + blocked: boolean; + }> { + const envelopes: IngestionEnvelope[] = opts.result.records.map((record, i) => + buildEnvelope({ + record, + source_id: input.source_id, + tenant_id: input.tenant_id, + project_id: input.project_id, + mode: input.mode, + partition_key: opts.partitionKey, + schema_version: opts.schemaVersion, + sequence: opts.startSequence + i + 1, + connector_id: input.connector.connector_id, + connector_version: input.connector.connector_version, + run_id: opts.runId, + cursor: opts.result.next_cursor, + }), + ); + + // ---- P063 per-record conformance ------------------------------------ + // A source that silently changed shape without a new discovery is caught + // here, on the record, before the sink sees anything from this page. + if (opts.schemaFields) { + for (const envelope of envelopes) { + if (envelope.payload === null) continue; + const errors = conformanceErrors( + opts.schemaFields as never, + envelope.payload, + ); + if (errors.length > 0) { + throw new SiaxError( + "CONFLICT", + "0pt.ingestion", + `record "${envelope.record_key}" does not conform to registered schema v${opts.schemaVersion} ` + + `for source ${input.source_id}: ${errors.join("; ")}`, + ); + } + } + } + + // ---- P064 quality gate ---------------------------------------------- + let accepted = envelopes; + let quarantinedCount = 0; + let violationCount = 0; + let blocked = false; + if (input.quality_policy) { + const gated = this.gate.evaluate(envelopes, input.quality_policy); + accepted = gated.accepted; + blocked = gated.blocked; + violationCount = gated.violations.length; + for (const item of gated.quarantined) { + const record: QuarantineRecord = { + quarantine_id: `qtn_${crypto + .createHash("sha256") + .update(`${opts.runId}|${item.envelope.envelope_id}`) + .digest("hex") + .slice(0, 32)}`, + source_id: input.source_id, + tenant_id: input.tenant_id, + run_id: opts.runId, + record_key: item.envelope.record_key, + violations: item.violations, + envelope: item.envelope, + quarantined_at: new Date().toISOString(), + }; + await this.repo.quarantine(record); + quarantinedCount++; + } + if (blocked) { + return { + emitted: 0, + duplicates: 0, + changed: 0, + quarantined: quarantinedCount, + violations: violationCount, + blocked: true, + }; + } + } + + // ---- P068 idempotency ledger, PEEK ONLY ------------------------------ + // Read the verdicts, filter replays, and commit NOTHING yet: a record is + // only "delivered" once the sink has actually taken it. Committing here + // would mark records delivered that a crashing sink never received, and + // the resumed run would then skip them as duplicates — silent data loss. + const entries = accepted.map((e) => ({ + record_key: e.record_key, + payload_hash: hashPayload(e.payload), + })); + const decisions = await this.repo.peekDeliveries(input.source_id, entries); + const verdictByKey = new Map(decisions.map((d) => [d.record_key, d.verdict])); + const toEmit = accepted.filter((e) => verdictByKey.get(e.record_key) !== "duplicate"); + const duplicates = accepted.length - toEmit.length; + const changed = decisions.filter((d) => d.verdict === "changed").length; + + // ---- sink (the consumer write) -------------------------------------- + if (toEmit.length > 0) await input.sink(toEmit); + + // ---- P068 ledger COMMIT, after the sink resolved --------------------- + const emitted = new Set(toEmit.map((e) => e.record_key)); + const committed = entries.filter((e) => emitted.has(e.record_key)); + if (committed.length > 0) { + await this.repo.commitDeliveries(input.source_id, input.tenant_id, committed); + } + + return { + emitted: toEmit.length, + duplicates, + changed, + quarantined: quarantinedCount, + violations: violationCount, + blocked: false, + }; + } +} diff --git a/src/lineage/lineage-edge.ts b/src/lineage/lineage-edge.ts new file mode 100644 index 0000000..cf6b3f7 --- /dev/null +++ b/src/lineage/lineage-edge.ts @@ -0,0 +1,42 @@ +/** + * src/lineage/lineage-edge.ts + * ============================ + * P067: the flattened, QUERYABLE form of an OpenLineage run event. + * + * LineageEmitter (src/lineage/lineage-emitter.ts) already emits spec-shaped + * run events to AUD0, and AUD0 remains the SYSTEM OF RECORD for them. But + * "emitted to AUD0" is not "answerable from 0pt": nothing in this repo could + * answer "which sources fed dataset ds_x?" without asking another product. + * These edges are 0pt's own projection — one row per (run, input, output) + * pair — written in the same call that emits, so the question is answerable + * locally and joins back to AUD0 by `run_id`. + * + * The projection is deliberately LOSSY (facets are not stored, only the + * canonical SIAX ref) — it is an index, not a second source of truth. + */ + +export interface LineageEdge { + /** Deterministic sha256 of (run_id, input, output) — a re-emitted idempotent run does not duplicate edges. */ + edge_id: string; + tenant_id: string; + run_id: string; + capability_id: string; + input_namespace: string; + input_name: string; + input_ref?: string; + output_namespace: string; + output_name: string; + output_ref?: string; + event_time: string; + correlation_id?: string; +} + +/** Query one direction of the graph. Exactly one of the two selectors is required. */ +export interface LineageQuery { + tenant_id: string; + /** Everything that fed this output (upstream). */ + output_name?: string; + /** Everything this input produced (downstream). */ + input_name?: string; + limit?: number; +} diff --git a/src/lineage/lineage-store.ts b/src/lineage/lineage-store.ts new file mode 100644 index 0000000..33526f2 --- /dev/null +++ b/src/lineage/lineage-store.ts @@ -0,0 +1,121 @@ +/** + * src/lineage/lineage-store.ts + * ============================= + * P067 DoD: "source -> output lineage is QUERYABLE." + * + * LineageEmitter already emits OpenLineage run events to AUD0 and that does + * not change — AUD0 stays the system of record. This class adds the missing + * half: it flattens the same run event into (input, output) edges, persists + * them, and answers both directions of the question. + * + * `RecordingLineageEmitter` is the composition point: it wraps the existing + * emitter so every existing caller (QualityService, DatasetVersionService, + * the ingestion runner) gets the local projection for free, without any of + * them learning about a second store. Emission to AUD0 keeps its existing + * never-throw contract; a projection write failure is likewise reported, not + * thrown, because lineage is evidence of work that already happened. + */ +import crypto from "crypto"; +import type { IngestionRepository } from "../repositories/ingestion-repository"; +import type { LineageEdge, LineageQuery } from "./lineage-edge"; +import type { LineageEmitInput, LineageEmitResult, LineageEmitter } from "./lineage-emitter"; +import type { OpenLineageRunEvent } from "./openlineage"; + +/** Deterministic edge id: the same run re-emitted produces the same edges. */ +export function lineageEdgeId(runId: string, input: string, output: string): string { + return `led_${crypto.createHash("sha256").update(`${runId}|${input}|${output}`).digest("hex").slice(0, 32)}`; +} + +function refOf(facets: Record | undefined): string | undefined { + const facet = facets?.siaxResourceRef as { ref?: string } | undefined; + return typeof facet?.ref === "string" ? facet.ref : undefined; +} + +/** + * Flatten one OpenLineage run event into edges. A run with N inputs and M + * outputs yields N*M edges; a run with no inputs or no outputs yields none + * (there is no edge to draw), which is why the runner always emits both. + */ +export function edgesFromRunEvent( + event: OpenLineageRunEvent, + tenantId: string, + correlationId?: string, +): LineageEdge[] { + const edges: LineageEdge[] = []; + for (const input of event.inputs ?? []) { + for (const output of event.outputs ?? []) { + edges.push({ + edge_id: lineageEdgeId(event.run.runId, `${input.namespace}/${input.name}`, `${output.namespace}/${output.name}`), + tenant_id: tenantId, + run_id: event.run.runId, + capability_id: event.job.name, + input_namespace: input.namespace, + input_name: input.name, + input_ref: refOf(input.facets), + output_namespace: output.namespace, + output_name: output.name, + output_ref: refOf(output.facets), + event_time: event.eventTime, + correlation_id: correlationId, + }); + } + } + return edges; +} + +export class LineageStore { + constructor(private readonly repo: IngestionRepository) {} + + async record(event: OpenLineageRunEvent, tenantId: string, correlationId?: string): Promise { + const edges = edgesFromRunEvent(event, tenantId, correlationId); + if (edges.length > 0) await this.repo.appendLineageEdges(edges); + return edges.length; + } + + /** Everything that fed `outputName`, newest first. */ + async upstreamOf(tenantId: string, outputName: string, limit?: number): Promise { + return this.repo.queryLineage({ tenant_id: tenantId, output_name: outputName, limit }); + } + + /** Everything `inputName` produced, newest first. */ + async downstreamOf(tenantId: string, inputName: string, limit?: number): Promise { + return this.repo.queryLineage({ tenant_id: tenantId, input_name: inputName, limit }); + } + + async query(query: LineageQuery): Promise { + return this.repo.queryLineage(query); + } +} + +/** + * A LineageEmitter that also writes the local projection. Drop-in for the + * existing emitter: same `emit` signature, same never-throw contract. + */ +export class RecordingLineageEmitter { + constructor( + private readonly inner: LineageEmitter, + private readonly store: LineageStore, + ) {} + + buildRunEvent(input: LineageEmitInput): OpenLineageRunEvent { + return this.inner.buildRunEvent(input); + } + + async emit(input: LineageEmitInput): Promise { + const event = this.inner.buildRunEvent(input); + const result = await this.inner.emit(input); + let projected = 0; + try { + projected = await this.store.record(event, input.tenant_id, input.correlation_id); + } catch (err) { + return { + ...result, + projected: 0, + warning: [result.warning, `lineage projection write failed: ${err instanceof Error ? err.message : String(err)}`] + .filter(Boolean) + .join("; "), + }; + } + return { ...result, projected }; + } +} diff --git a/src/quality/quality-policy.ts b/src/quality/quality-policy.ts new file mode 100644 index 0000000..712d56c --- /dev/null +++ b/src/quality/quality-policy.ts @@ -0,0 +1,129 @@ +/** + * src/quality/quality-policy.ts + * ============================== + * P064: the DECLARED enforcement policy — what happens to a record that fails + * its quality rules — and the gate that applies it on the ingestion path. + * + * WHAT EXISTED BEFORE (and is reused, not rebuilt): QualityService already + * owns the rule registry, runs, results, issues and scores, and + * ExpectationEngine already executes the Great Expectations core subset. What + * neither of them had was a CONSEQUENCE: a failing rule produced an ISSUE for + * a human, and the data flowed on regardless. That is the gap this closes. + * This module deliberately does NOT re-implement rule storage or execution — + * it composes ExpectationEngine and adds the decision. + * + * Three declared actions, and nothing implicit: + * - `block` — the whole page fails; the checkpoint is NOT advanced, so + * the run stops on the bad data instead of stepping over it. + * - `quarantine` — the failing envelopes are persisted to durable quarantine + * and excluded from the output; clean envelopes continue and + * the checkpoint advances past all of them. + * - `warn` — everything continues; violations are counted and reported. + * + * There is no default. A source with no policy gets `block` from the runner + * only if a policy was supplied; with no policy at all the runner skips the + * gate and SAYS SO in the run report (`quality_policy: "none_declared"`) — + * an undeclared policy must never look like a passed one. + */ +import { ExpectationEngine } from "./expectation-engine"; +import type { QualityExpectation } from "./quality-types"; +import type { IngestionEnvelope } from "../types/ingestion"; + +export type QualityEnforcementAction = "block" | "quarantine" | "warn"; + +export interface QualityEnforcementPolicy { + source_id: string; + tenant_id: string; + /** Rules applied per record. Expectations are evaluated against a one-row table. */ + expectations: QualityExpectation[]; + on_violation: QualityEnforcementAction; +} + +export interface EnvelopeViolation { + expectation_type: string; + detail: string; +} + +export interface QualityGateResult { + /** Envelopes that may proceed downstream. */ + accepted: IngestionEnvelope[]; + /** Envelopes routed to quarantine (only under `quarantine`). */ + quarantined: Array<{ envelope: IngestionEnvelope; violations: EnvelopeViolation[] }>; + /** Every violation seen, including under `warn` where nothing was withheld. */ + violations: Array<{ record_key: string; violations: EnvelopeViolation[] }>; + /** True when `block` fired: the caller must NOT advance the checkpoint. */ + blocked: boolean; + action: QualityEnforcementAction; +} + +export class QualityGate { + constructor(private readonly engine: ExpectationEngine = new ExpectationEngine()) {} + + /** + * Evaluate one page of envelopes against the policy. + * + * Delete envelopes (payload null) are passed through untouched: a CDC delete + * carries no columns to check, and failing it against a column expectation + * would quarantine the very deletions a source is asking us to honour. + */ + evaluate(envelopes: IngestionEnvelope[], policy: QualityEnforcementPolicy): QualityGateResult { + const accepted: IngestionEnvelope[] = []; + const quarantined: QualityGateResult["quarantined"] = []; + const allViolations: QualityGateResult["violations"] = []; + + for (const envelope of envelopes) { + if (envelope.payload === null) { + accepted.push(envelope); + continue; + } + const violations = this.violationsFor(envelope.payload, policy.expectations); + if (violations.length === 0) { + accepted.push(envelope); + continue; + } + allViolations.push({ record_key: envelope.record_key, violations }); + if (policy.on_violation === "quarantine") { + quarantined.push({ envelope, violations }); + } else if (policy.on_violation === "warn") { + accepted.push(envelope); + } + // `block`: neither accepted nor quarantined — the caller stops. + } + + const blocked = policy.on_violation === "block" && allViolations.length > 0; + return { + accepted: blocked ? [] : accepted, + quarantined, + violations: allViolations, + blocked, + action: policy.on_violation, + }; + } + + private violationsFor( + payload: Record, + expectations: QualityExpectation[], + ): EnvelopeViolation[] { + const violations: EnvelopeViolation[] = []; + for (const expectation of expectations) { + if (!this.engine.supports(expectation.expectation_type)) { + // An expectation this engine cannot execute is a VIOLATION of the + // policy's executability, not a silent pass. The record is treated as + // failing so an unrunnable rule can never look like a satisfied one. + violations.push({ + expectation_type: expectation.expectation_type, + detail: `expectation is not executable by the local engine; delegate it to the GE backend before declaring it in an enforcement policy`, + }); + continue; + } + const outcome = this.engine.execute(expectation, [payload]); + if (!outcome.success) { + violations.push({ + expectation_type: expectation.expectation_type, + detail: `failed on column "${String(expectation.kwargs.column ?? "")}" (${outcome.unexpected_count}/${outcome.evaluated_count} unexpected)`, + }); + } + } + return violations; + } +} diff --git a/src/repositories/in-memory-ingestion-repository.ts b/src/repositories/in-memory-ingestion-repository.ts new file mode 100644 index 0000000..31202b1 --- /dev/null +++ b/src/repositories/in-memory-ingestion-repository.ts @@ -0,0 +1,120 @@ +/** + * src/repositories/in-memory-ingestion-repository.ts + * ==================================================== + * Non-production implementation of IngestionRepository (hermetic tests, local + * dev). Semantics are kept IDENTICAL to the Postgres implementation on every + * point the runner depends on: + * - checkpoints are upserted per (source, partition), + * - ledger verdicts are new/changed/duplicate against the LAST accepted + * payload hash for the key, + * - quarantine is append-only, newest-first on read, + * - schemas are keyed by (source, version) and "latest" means highest version, + * - lineage edges are deduplicated by edge_id. + * + * Any divergence between the two implementations is a bug in one of them; + * tests/p0/ingestion-repository-parity.test.ts is what keeps them honest for + * the in-memory side, and tests/p0/ingestion-plane-postgres.realinfra.test.ts + * runs the same expectations against real Postgres. + */ +import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge"; +import type { RecordSchema } from "../schema/record-schema"; +import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion"; +import type { IngestionRepository, LedgerEntry } from "./ingestion-repository"; + +function checkpointKey(sourceId: string, partitionKey: string): string { + return `${sourceId}|${partitionKey}`; +} + +export class InMemoryIngestionRepository implements IngestionRepository { + private readonly checkpoints = new Map(); + /** source_id -> record_key -> last accepted payload hash. */ + private readonly ledger = new Map>(); + private readonly quarantined = new Map(); + private readonly schemas = new Map(); + private readonly lineage = new Map(); + + async getCheckpoint(sourceId: string, partitionKey: string): Promise { + return this.checkpoints.get(checkpointKey(sourceId, partitionKey)); + } + + async commitCheckpoint(checkpoint: IngestionCheckpoint): Promise { + this.checkpoints.set(checkpointKey(checkpoint.source_id, checkpoint.partition_key), { ...checkpoint }); + } + + async listCheckpoints(sourceId: string): Promise { + return [...this.checkpoints.values()] + .filter((c) => c.source_id === sourceId) + .sort((a, b) => a.partition_key.localeCompare(b.partition_key)); + } + + async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise { + const keys = this.ledger.get(sourceId); + return entries.map((entry) => { + const prior = keys?.get(entry.record_key); + return { + record_key: entry.record_key, + verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed", + } as LedgerDecision; + }); + } + + async commitDeliveries(sourceId: string, _tenantId: string, entries: LedgerEntry[]): Promise { + let keys = this.ledger.get(sourceId); + if (!keys) { + keys = new Map(); + this.ledger.set(sourceId, keys); + } + for (const entry of entries) keys.set(entry.record_key, entry.payload_hash); + } + + async countLedger(sourceId: string): Promise { + return this.ledger.get(sourceId)?.size ?? 0; + } + + async quarantine(record: QuarantineRecord): Promise { + const list = this.quarantined.get(record.source_id) ?? []; + list.push(record); + this.quarantined.set(record.source_id, list); + } + + async listQuarantine(sourceId: string, limit = 100): Promise { + const list = this.quarantined.get(sourceId) ?? []; + return [...list].reverse().slice(0, limit); + } + + async putSchema(schema: RecordSchema): Promise { + const list = this.schemas.get(schema.source_id) ?? []; + const existing = list.findIndex((s) => s.schema_version === schema.schema_version); + if (existing >= 0) list[existing] = { ...schema }; + else list.push({ ...schema }); + list.sort((a, b) => a.schema_version - b.schema_version); + this.schemas.set(schema.source_id, list); + } + + async getSchema(sourceId: string, version: number): Promise { + return this.schemas.get(sourceId)?.find((s) => s.schema_version === version); + } + + async getLatestSchema(sourceId: string): Promise { + const list = this.schemas.get(sourceId); + return list && list.length > 0 ? list[list.length - 1] : undefined; + } + + async listSchemas(sourceId: string): Promise { + return [...(this.schemas.get(sourceId) ?? [])]; + } + + async appendLineageEdges(edges: LineageEdge[]): Promise { + for (const edge of edges) this.lineage.set(edge.edge_id, { ...edge }); + } + + async queryLineage(query: LineageQuery): Promise { + const limit = query.limit ?? 200; + return [...this.lineage.values()] + .filter((e) => e.tenant_id === query.tenant_id) + .filter((e) => (query.output_name ? e.output_name === query.output_name : true)) + .filter((e) => (query.input_name ? e.input_name === query.input_name : true)) + .sort((a, b) => b.event_time.localeCompare(a.event_time)) + .slice(0, limit); + } +} diff --git a/src/repositories/in-memory-source-registry-repository.ts b/src/repositories/in-memory-source-registry-repository.ts index a0cf046..c5d8deb 100644 --- a/src/repositories/in-memory-source-registry-repository.ts +++ b/src/repositories/in-memory-source-registry-repository.ts @@ -10,7 +10,13 @@ * - 1]` for "most recent" and scanned backward for consecutive failures -- * callers now get the same data pre-ordered instead). */ -import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry"; +import type { + SourceAuthorization, + SourceHealthCheck, + SourceProvenance, + SourceRights, + SourceSchedule, +} from "../types/source-registry"; import type { SourceRegistryRepository } from "./source-registry-repository"; export class InMemorySourceRegistryRepository implements SourceRegistryRepository { @@ -18,6 +24,8 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor private readonly provenance = new Map(); private readonly healthChecks = new Map(); private readonly lastSuccess = new Map(); + private readonly authorizations = new Map(); + private readonly schedules = new Map(); async setRights(rights: SourceRights): Promise { this.rights.set(rights.source_id, rights); @@ -43,6 +51,22 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor return this.lastSuccess.get(sourceId); } + async setAuthorization(auth: SourceAuthorization): Promise { + this.authorizations.set(auth.source_id, auth); + } + + async getAuthorization(sourceId: string): Promise { + return this.authorizations.get(sourceId); + } + + async setSchedule(schedule: SourceSchedule): Promise { + this.schedules.set(schedule.source_id, schedule); + } + + async getSchedule(sourceId: string): Promise { + return this.schedules.get(sourceId); + } + async appendHealthCheck(check: SourceHealthCheck): Promise { const list = this.healthChecks.get(check.source_id) ?? []; list.push(check); diff --git a/src/repositories/in-memory-source-repository.ts b/src/repositories/in-memory-source-repository.ts index 53babf9..b7bfd07 100644 --- a/src/repositories/in-memory-source-repository.ts +++ b/src/repositories/in-memory-source-repository.ts @@ -27,6 +27,7 @@ interface SourceRecord { created_at: string; updated_at: string | null; created_by: string | null; + owner_id: string | null; } export class InMemorySourceRepository implements SourceRepository { @@ -50,11 +51,16 @@ export class InMemorySourceRepository implements SourceRepository { adapter_id: input.adapter_id ?? null, access: input.access ?? null, description: input.description ?? null, - status: input.status ?? "active", + // P060: "pending", not "active". A source with no rights, no provenance + // and no authorization used to be live the instant it was registered; + // reaching "active" now goes through + // SourceRegistryService.activateSource's classification gate. + status: input.status ?? "pending", correlation_id: input.correlation_id ?? null, created_at: now, updated_at: null, created_by: input.created_by ?? null, + owner_id: input.owner_id ?? null, }; this.sources.set(record.source_id, record); return toSource(record); @@ -66,6 +72,14 @@ export class InMemorySourceRepository implements SourceRepository { return toSource(record); } + async updateStatus(sourceId: string, status: SourceStatus): Promise { + const record = this.sources.get(sourceId); + if (!record) throw new Error(`Source ${sourceId} not found`); + record.status = status; + record.updated_at = new Date().toISOString(); + return toSource(record); + } + async list(filter?: SourceListFilter): Promise { let results = Array.from(this.sources.values()); if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id); @@ -91,5 +105,6 @@ function toSource(r: SourceRecord): Source { created_at: r.created_at, updated_at: r.updated_at ?? undefined, created_by: r.created_by ?? undefined, + owner_id: r.owner_id ?? undefined, }; } diff --git a/src/repositories/ingestion-repository.ts b/src/repositories/ingestion-repository.ts new file mode 100644 index 0000000..f7bdfc9 --- /dev/null +++ b/src/repositories/ingestion-repository.ts @@ -0,0 +1,91 @@ +/** + * src/repositories/ingestion-repository.ts + * ========================================== + * The storage seam for the P060-P068 ingestion plane: checkpoints (P060/P062), + * the idempotency ledger (P068), quarantine (P064), the schema registry + * (P060/P063) and the queryable lineage projection (P067). + * + * WHY ONE INTERFACE AND NOT FIVE. These five tables are written by ONE + * caller — IngestionRunner — inside one run, and their correctness is + * relative to each other (a checkpoint that advanced past records the ledger + * never accepted is a silent gap; a quarantined record whose checkpoint moved + * on is a silently dropped record). Splitting them into five factories would + * let a future caller wire four of them to Postgres and one to memory and + * still typecheck, which is exactly the failure this plane must not have. + * They are chosen once, together, by one `resolveStoreMode` call. + * + * Same production posture as every other 0pt store: `resolveStoreMode` fails + * hard in production with no DATABASE_URL rather than silently serving Maps + * (src/platform/dependency-mode.ts, Master-DoD §3/§10). + */ +import { resolveStoreMode } from "../platform/dependency-mode"; +import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge"; +import type { RecordSchema } from "../schema/record-schema"; +import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion"; +import { getRepositorySql, repositoryDatabaseUrl } from "./db"; +import { InMemoryIngestionRepository } from "./in-memory-ingestion-repository"; +import { PostgresIngestionRepository } from "./postgres-ingestion-repository"; + +/** One record offered to the ledger. */ +export interface LedgerEntry { + record_key: string; + /** Stable hash of the payload — see hashPayload() in src/ingestion/ingestion-ledger.ts. */ + payload_hash: string; +} + +export interface IngestionRepository { + // --- checkpoints (P060 registry field, P062 durable resume) -------------- + getCheckpoint(sourceId: string, partitionKey: string): Promise; + commitCheckpoint(checkpoint: IngestionCheckpoint): Promise; + listCheckpoints(sourceId: string): Promise; + + // --- idempotency ledger (P068) ------------------------------------------ + /** + * READ-ONLY verdicts, in input order: + * "new" — this record_key has never been accepted for this source, + * "changed" — it has, but with different content (a real update), + * "duplicate" — it has, with byte-identical content (a replay). + * + * Deliberately separate from {@link commitDeliveries}. An earlier draft of + * this interface combined them into one atomic check-and-record, which was + * WRONG in the way that matters: the runner has to consult the ledger + * BEFORE the sink (to filter replays) and commit AFTER it (so a crash + * between the two re-delivers rather than loses). A combined call marked a + * record "delivered" that the sink never received, and the resumed run then + * skipped it as a duplicate — permanent, silent data loss. That is not a + * hypothetical: tests/p0/ingestion-runner.test.ts caught it, and the + * "resumes with NO gap" case is what pins the fix. + */ + peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise; + + /** + * Record that these records were successfully delivered. Called only after + * the sink resolved. Idempotent: re-committing the same entry bumps the + * delivery count and leaves the verdict unchanged. + */ + commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise; + + countLedger(sourceId: string): Promise; + + // --- quarantine (P064) -------------------------------------------------- + quarantine(record: QuarantineRecord): Promise; + listQuarantine(sourceId: string, limit?: number): Promise; + + // --- schema registry (P060 registry field, P063 evolution) -------------- + putSchema(schema: RecordSchema): Promise; + getSchema(sourceId: string, version: number): Promise; + getLatestSchema(sourceId: string): Promise; + listSchemas(sourceId: string): Promise; + + // --- lineage projection (P067) ------------------------------------------ + appendLineageEdges(edges: LineageEdge[]): Promise; + queryLineage(query: LineageQuery): Promise; +} + +export function createIngestionRepository(databaseUrl?: string): IngestionRepository { + const mode = resolveStoreMode("0pt.ingestion", databaseUrl ?? repositoryDatabaseUrl()); + if (mode.mode === "postgres") { + return new PostgresIngestionRepository(getRepositorySql(mode.url)); + } + return new InMemoryIngestionRepository(); +} diff --git a/src/repositories/postgres-ingestion-repository.ts b/src/repositories/postgres-ingestion-repository.ts new file mode 100644 index 0000000..133096b --- /dev/null +++ b/src/repositories/postgres-ingestion-repository.ts @@ -0,0 +1,308 @@ +/** + * src/repositories/postgres-ingestion-repository.ts + * ================================================== + * Real Postgres backing for IngestionRepository (sql/007_ingestion_plane.sql). + * + * THE LEDGER IS THE INTERESTING ONE (P068), and it is deliberately SPLIT: + * `peekDeliveries` reads verdicts, `commitDeliveries` records them. The + * runner peeks before the sink and commits after it, so a crash in between + * re-delivers on resume instead of losing the record. An earlier version of + * this file combined the two into one atomic check-and-record statement; that + * was measurably wrong (see the interface's doc comment) and the crash/resume + * test in tests/p0/ingestion-runner.test.ts is what proved it. + * + * KNOWN LIMIT, stated rather than papered over: the split means two runs on + * the SAME (source_id, partition) can both peek "new" and both emit. The + * delivery guarantee is therefore AT-LEAST-ONCE, which is what the + * deterministic `envelope_id` (src/ingestion/envelope.ts) exists to make + * survivable downstream. Exactly-once across concurrent runs needs a run lock + * per (source, partition); there is none yet. That is a real open gap, not a + * solved one — do not describe this plane as exactly-once. + */ +import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge"; +import type { + CompatibilityPolicy, + RecordSchema, + RecordSchemaFields, +} from "../schema/record-schema"; +import type { + IngestionCheckpoint, + IngestionEnvelope, + LedgerDecision, + QuarantineRecord, +} from "../types/ingestion"; +import type { RepositorySql } from "./db"; +import type { IngestionRepository, LedgerEntry } from "./ingestion-repository"; + +/** postgres.js's `sql.json()` parameter type, named locally so the two JSONB columns can be passed without importing postgres' internals. */ +type JsonValue = Parameters[0]; + +function toIso(v: string | Date): string { + return v instanceof Date ? v.toISOString() : v; +} + +interface CheckpointRow { + source_id: string; + tenant_id: string; + partition_key: string; + cursor: string | null; + sequence: string | number; + records_committed: string | number; + committed_at: string | Date; +} + +function toCheckpoint(r: CheckpointRow): IngestionCheckpoint { + return { + source_id: r.source_id, + tenant_id: r.tenant_id, + partition_key: r.partition_key, + cursor: r.cursor ?? undefined, + sequence: Number(r.sequence), + records_committed: Number(r.records_committed), + committed_at: toIso(r.committed_at), + }; +} + +export class PostgresIngestionRepository implements IngestionRepository { + constructor(private readonly sql: RepositorySql) {} + + // --- checkpoints --------------------------------------------------------- + + async getCheckpoint(sourceId: string, partitionKey: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at + FROM opt_ingestion_checkpoints + WHERE source_id = ${sourceId} AND partition_key = ${partitionKey} + LIMIT 1`; + return rows[0] ? toCheckpoint(rows[0]) : undefined; + } + + async commitCheckpoint(cp: IngestionCheckpoint): Promise { + await this.sql` + INSERT INTO opt_ingestion_checkpoints + (source_id, partition_key, tenant_id, cursor, sequence, records_committed, committed_at) + VALUES + (${cp.source_id}, ${cp.partition_key}, ${cp.tenant_id}, ${cp.cursor ?? null}, + ${cp.sequence}, ${cp.records_committed}, ${cp.committed_at}) + ON CONFLICT (source_id, partition_key) DO UPDATE + SET cursor = EXCLUDED.cursor, + sequence = EXCLUDED.sequence, + records_committed = EXCLUDED.records_committed, + committed_at = EXCLUDED.committed_at, + tenant_id = EXCLUDED.tenant_id`; + } + + async listCheckpoints(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at + FROM opt_ingestion_checkpoints + WHERE source_id = ${sourceId} + ORDER BY partition_key ASC`; + return rows.map(toCheckpoint); + } + + // --- ledger -------------------------------------------------------------- + + async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise { + if (entries.length === 0) return []; + const keys = entries.map((e) => e.record_key); + const rows = await this.sql<{ record_key: string; payload_hash: string }[]>` + SELECT record_key, payload_hash FROM opt_ingestion_ledger + WHERE source_id = ${sourceId} AND record_key = ANY(${this.sql.array(keys)})`; + const priorByKey = new Map(rows.map((r) => [r.record_key, r.payload_hash])); + return entries.map((entry) => { + const prior = priorByKey.get(entry.record_key); + return { + record_key: entry.record_key, + verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed", + } as LedgerDecision; + }); + } + + async commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise { + for (const entry of entries) { + await this.sql` + INSERT INTO opt_ingestion_ledger (source_id, record_key, tenant_id, payload_hash) + VALUES (${sourceId}, ${entry.record_key}, ${tenantId}, ${entry.payload_hash}) + ON CONFLICT (source_id, record_key) DO UPDATE + SET last_seen_at = now(), + delivery_count = opt_ingestion_ledger.delivery_count + 1, + payload_hash = EXCLUDED.payload_hash`; + } + } + + async countLedger(sourceId: string): Promise { + const rows = await this.sql<{ n: string }[]>` + SELECT count(*)::text AS n FROM opt_ingestion_ledger WHERE source_id = ${sourceId}`; + return Number(rows[0]?.n ?? 0); + } + + // --- quarantine ---------------------------------------------------------- + + async quarantine(record: QuarantineRecord): Promise { + await this.sql` + INSERT INTO opt_quality_quarantine + (quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at) + VALUES + (${record.quarantine_id}, ${record.source_id}, ${record.tenant_id}, ${record.run_id}, + ${record.record_key}, ${this.sql.json(record.violations as unknown as JsonValue)}, + ${this.sql.json(record.envelope as unknown as JsonValue)}, + ${record.quarantined_at}) + ON CONFLICT (quarantine_id) DO NOTHING`; + } + + async listQuarantine(sourceId: string, limit = 100): Promise { + const rows = await this.sql< + { + quarantine_id: string; + source_id: string; + tenant_id: string; + run_id: string; + record_key: string; + violations: QuarantineRecord["violations"]; + envelope: IngestionEnvelope; + quarantined_at: string | Date; + }[] + >` + SELECT quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at + FROM opt_quality_quarantine + WHERE source_id = ${sourceId} + ORDER BY quarantined_at DESC, quarantine_id DESC + LIMIT ${limit}`; + return rows.map((r) => ({ + quarantine_id: r.quarantine_id, + source_id: r.source_id, + tenant_id: r.tenant_id, + run_id: r.run_id, + record_key: r.record_key, + violations: r.violations, + envelope: r.envelope, + quarantined_at: toIso(r.quarantined_at), + })); + } + + // --- schema registry ----------------------------------------------------- + + async putSchema(schema: RecordSchema): Promise { + await this.sql` + INSERT INTO opt_source_schemas + (source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by) + VALUES + (${schema.source_id}, ${schema.schema_version}, ${schema.tenant_id}, + ${this.sql.json(schema.fields as unknown as JsonValue)}, ${schema.compatibility}, + ${schema.registered_at}, ${schema.registered_by ?? null}) + ON CONFLICT (source_id, schema_version) DO UPDATE + SET fields = EXCLUDED.fields, + compatibility = EXCLUDED.compatibility, + registered_at = EXCLUDED.registered_at, + registered_by = EXCLUDED.registered_by`; + } + + async getSchema(sourceId: string, version: number): Promise { + const rows = await this.sql` + SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by + FROM opt_source_schemas + WHERE source_id = ${sourceId} AND schema_version = ${version} + LIMIT 1`; + return rows[0] ? toSchema(rows[0]) : undefined; + } + + async getLatestSchema(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by + FROM opt_source_schemas + WHERE source_id = ${sourceId} + ORDER BY schema_version DESC + LIMIT 1`; + return rows[0] ? toSchema(rows[0]) : undefined; + } + + async listSchemas(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by + FROM opt_source_schemas + WHERE source_id = ${sourceId} + ORDER BY schema_version ASC`; + return rows.map(toSchema); + } + + // --- lineage ------------------------------------------------------------- + + async appendLineageEdges(edges: LineageEdge[]): Promise { + for (const e of edges) { + await this.sql` + INSERT INTO opt_lineage_edges + (edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref, + output_namespace, output_name, output_ref, event_time, correlation_id) + VALUES + (${e.edge_id}, ${e.tenant_id}, ${e.run_id}, ${e.capability_id}, ${e.input_namespace}, + ${e.input_name}, ${e.input_ref ?? null}, ${e.output_namespace}, ${e.output_name}, + ${e.output_ref ?? null}, ${e.event_time}, ${e.correlation_id ?? null}) + ON CONFLICT (edge_id) DO NOTHING`; + } + } + + async queryLineage(query: LineageQuery): Promise { + const limit = query.limit ?? 200; + const rows = await this.sql` + SELECT edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref, + output_namespace, output_name, output_ref, event_time, correlation_id + FROM opt_lineage_edges + WHERE tenant_id = ${query.tenant_id} + ${query.output_name ? this.sql`AND output_name = ${query.output_name}` : this.sql``} + ${query.input_name ? this.sql`AND input_name = ${query.input_name}` : this.sql``} + ORDER BY event_time DESC, edge_id DESC + LIMIT ${limit}`; + return rows.map((r) => ({ + edge_id: r.edge_id, + tenant_id: r.tenant_id, + run_id: r.run_id, + capability_id: r.capability_id, + input_namespace: r.input_namespace, + input_name: r.input_name, + input_ref: r.input_ref ?? undefined, + output_namespace: r.output_namespace, + output_name: r.output_name, + output_ref: r.output_ref ?? undefined, + event_time: toIso(r.event_time), + correlation_id: r.correlation_id ?? undefined, + })); + } +} + +interface SchemaRow { + source_id: string; + schema_version: number; + tenant_id: string; + fields: RecordSchemaFields; + compatibility: string; + registered_at: string | Date; + registered_by: string | null; +} + +function toSchema(r: SchemaRow): RecordSchema { + return { + source_id: r.source_id, + schema_version: Number(r.schema_version), + tenant_id: r.tenant_id, + fields: r.fields, + compatibility: r.compatibility as CompatibilityPolicy, + registered_at: toIso(r.registered_at), + registered_by: r.registered_by ?? undefined, + }; +} + +interface LineageRow { + edge_id: string; + tenant_id: string; + run_id: string; + capability_id: string; + input_namespace: string; + input_name: string; + input_ref: string | null; + output_namespace: string; + output_name: string; + output_ref: string | null; + event_time: string | Date; + correlation_id: string | null; +} diff --git a/src/repositories/postgres-source-registry-repository.ts b/src/repositories/postgres-source-registry-repository.ts index e30abe3..27269d9 100644 --- a/src/repositories/postgres-source-registry-repository.ts +++ b/src/repositories/postgres-source-registry-repository.ts @@ -10,10 +10,13 @@ * is a single upserted "last success" row per source. */ import type { + AuthorizationBasis, + SourceAuthorization, SourceHealthCheck, SourceHealthStatus, SourceProvenance, SourceRights, + SourceSchedule, } from "../types/source-registry"; import type { RepositorySql } from "./db"; import type { SourceRegistryRepository } from "./source-registry-repository"; @@ -167,6 +170,57 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor return rows[0] ? toIsoString(rows[0].last_success_at) : undefined; } + // --- P060 authorization / schedule --------------------------------------- + + async setAuthorization(auth: SourceAuthorization): Promise { + await this.sql` + INSERT INTO opt_source_authorization + (source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes) + VALUES + (${auth.source_id}, ${auth.tenant_id}, ${auth.basis}, ${auth.granted_by}, ${auth.granted_at}, + ${auth.expires_at ?? null}, ${auth.evidence_ref ?? null}, ${auth.notes ?? null}) + ON CONFLICT (source_id) DO UPDATE + SET tenant_id = EXCLUDED.tenant_id, + basis = EXCLUDED.basis, + granted_by = EXCLUDED.granted_by, + granted_at = EXCLUDED.granted_at, + expires_at = EXCLUDED.expires_at, + evidence_ref = EXCLUDED.evidence_ref, + notes = EXCLUDED.notes, + updated_at = now()`; + } + + async getAuthorization(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes + FROM opt_source_authorization WHERE source_id = ${sourceId} LIMIT 1`; + return rows[0] ? toAuthorization(rows[0]) : undefined; + } + + async setSchedule(schedule: SourceSchedule): Promise { + await this.sql` + INSERT INTO opt_source_schedule + (source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at) + VALUES + (${schedule.source_id}, ${schedule.tenant_id}, ${schedule.mode}, + ${schedule.interval_seconds ?? null}, ${schedule.cron ?? null}, ${schedule.enabled}, + ${schedule.updated_at}) + ON CONFLICT (source_id) DO UPDATE + SET tenant_id = EXCLUDED.tenant_id, + mode = EXCLUDED.mode, + interval_seconds = EXCLUDED.interval_seconds, + cron = EXCLUDED.cron, + enabled = EXCLUDED.enabled, + updated_at = EXCLUDED.updated_at`; + } + + async getSchedule(sourceId: string): Promise { + const rows = await this.sql` + SELECT source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at + FROM opt_source_schedule WHERE source_id = ${sourceId} LIMIT 1`; + return rows[0] ? toSchedule(rows[0]) : undefined; + } + async appendHealthCheck(check: SourceHealthCheck): Promise { await this.sql` INSERT INTO opt_source_health_checks (source_id, tenant_id, status, checked_at, latency_ms, detail) @@ -195,3 +249,49 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor return rows.map(toHealthCheck); } } + +interface AuthorizationRow { + source_id: string; + tenant_id: string; + basis: string; + granted_by: string; + granted_at: string | Date; + expires_at: string | Date | null; + evidence_ref: string | null; + notes: string | null; +} + +function toAuthorization(r: AuthorizationRow): SourceAuthorization { + return { + source_id: r.source_id, + tenant_id: r.tenant_id, + basis: r.basis as AuthorizationBasis, + granted_by: r.granted_by, + granted_at: toIsoString(r.granted_at), + expires_at: r.expires_at ? toIsoString(r.expires_at) : undefined, + evidence_ref: r.evidence_ref ?? undefined, + notes: r.notes ?? undefined, + }; +} + +interface ScheduleRow { + source_id: string; + tenant_id: string; + mode: string; + interval_seconds: number | string | null; + cron: string | null; + enabled: boolean; + updated_at: string | Date; +} + +function toSchedule(r: ScheduleRow): SourceSchedule { + return { + source_id: r.source_id, + tenant_id: r.tenant_id, + mode: r.mode as SourceSchedule["mode"], + interval_seconds: r.interval_seconds === null ? undefined : Number(r.interval_seconds), + cron: r.cron ?? undefined, + enabled: r.enabled, + updated_at: toIsoString(r.updated_at), + }; +} diff --git a/src/repositories/postgres-source-repository.ts b/src/repositories/postgres-source-repository.ts index 970e911..fda7f90 100644 --- a/src/repositories/postgres-source-repository.ts +++ b/src/repositories/postgres-source-repository.ts @@ -25,6 +25,7 @@ interface SourceRow { created_at: string | Date; updated_at: string | Date | null; created_by: string | null; + owner_id: string | null; } function toIsoString(v: string | Date): string { @@ -46,6 +47,7 @@ function toSource(r: SourceRow): Source { created_at: toIsoString(r.created_at), updated_at: r.updated_at ? toIsoString(r.updated_at) : undefined, created_by: r.created_by ?? undefined, + owner_id: r.owner_id ?? undefined, }; } @@ -63,29 +65,41 @@ export class PostgresSourceRepository implements SourceRepository { const rows = await this.sql` INSERT INTO opt_sources - (source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by) + (source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by, owner_id) VALUES (${input.source_id}, ${input.tenant_id}, ${input.project_id}, ${input.source_type}, ${input.name}, ${input.adapter_id ?? null}, ${input.access ?? null}, ${input.description ?? null}, - ${input.status ?? "active"}, ${input.correlation_id ?? null}, ${input.created_by ?? null}) + ${input.status ?? "pending"}, ${input.correlation_id ?? null}, ${input.created_by ?? null}, + ${input.owner_id ?? null}) RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, - status, correlation_id, created_at, updated_at, created_by`; + status, correlation_id, created_at, updated_at, created_by, owner_id`; return toSource(rows[0]); } async get(sourceId: string): Promise { const rows = await this.sql` SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, - status, correlation_id, created_at, updated_at, created_by + status, correlation_id, created_at, updated_at, created_by, owner_id FROM opt_sources WHERE source_id = ${sourceId} LIMIT 1`; if (!rows[0]) throw new Error(`Source ${sourceId} not found`); return toSource(rows[0]); } + async updateStatus(sourceId: string, status: SourceStatus): Promise { + const rows = await this.sql` + UPDATE opt_sources + SET status = ${status}, updated_at = now() + WHERE source_id = ${sourceId} + RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, + status, correlation_id, created_at, updated_at, created_by, owner_id`; + if (!rows[0]) throw new Error(`Source ${sourceId} not found`); + return toSource(rows[0]); + } + async list(filter?: SourceListFilter): Promise { const rows = await this.sql` SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, - status, correlation_id, created_at, updated_at, created_by + status, correlation_id, created_at, updated_at, created_by, owner_id FROM opt_sources WHERE 1=1 ${filter?.tenant_id ? this.sql`AND tenant_id = ${filter.tenant_id}` : this.sql``} diff --git a/src/repositories/source-registry-repository.ts b/src/repositories/source-registry-repository.ts index 4560e26..941945b 100644 --- a/src/repositories/source-registry-repository.ts +++ b/src/repositories/source-registry-repository.ts @@ -16,7 +16,13 @@ * layer trusts the caller, same as the Maps did. */ import { resolveStoreMode } from "../platform/dependency-mode"; -import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry"; +import type { + SourceAuthorization, + SourceHealthCheck, + SourceProvenance, + SourceRights, + SourceSchedule, +} from "../types/source-registry"; import { getRepositorySql, repositoryDatabaseUrl } from "./db"; import { InMemorySourceRegistryRepository } from "./in-memory-source-registry-repository"; import { PostgresSourceRegistryRepository } from "./postgres-source-registry-repository"; @@ -31,6 +37,14 @@ export interface SourceRegistryRepository { recordSuccess(sourceId: string, tenantId: string, atIso: string): Promise; getLastSuccess(sourceId: string): Promise; + /** P060 authorization — the recorded decision that 0pt may connect at all. */ + setAuthorization(auth: SourceAuthorization): Promise; + getAuthorization(sourceId: string): Promise; + + /** P060 schedule — ingestion mode + cadence. */ + setSchedule(schedule: SourceSchedule): Promise; + getSchedule(sourceId: string): Promise; + appendHealthCheck(check: SourceHealthCheck): Promise; /** Newest-first, bounded to the latest `limit` observations (default 100) -- health is an operational signal, not an audit log; AUD0 owns history. */ listHealthChecks(sourceId: string, limit?: number): Promise; diff --git a/src/repositories/source-repository.ts b/src/repositories/source-repository.ts index f00e1cf..59b077c 100644 --- a/src/repositories/source-repository.ts +++ b/src/repositories/source-repository.ts @@ -33,6 +33,12 @@ export interface SourceRepository { /** Throws (message contains "not found") when no such source exists -- matches the pre-migration Map behavior every caller already handles. */ get(sourceId: string): Promise; list(filter?: SourceListFilter): Promise; + /** + * P060: the only way a source's lifecycle status changes. Deliberately NOT + * a general `update` -- status is the field the activation gate protects, + * and a generic patch method would be the obvious way around it. + */ + updateStatus(sourceId: string, status: SourceStatus): Promise; } export function createSourceRepository(databaseUrl?: string): SourceRepository { diff --git a/src/schema/compatibility.ts b/src/schema/compatibility.ts new file mode 100644 index 0000000..97a0f8d --- /dev/null +++ b/src/schema/compatibility.ts @@ -0,0 +1,170 @@ +/** + * src/schema/compatibility.ts + * ============================ + * P063: the pure comparison. `checkCompatibility(previous, next, policy)` + * returns every way `next` breaks readers/writers of `previous` under the + * declared policy. No IO, no throwing — the caller decides what a break + * means (SchemaRegistry refuses the registration; the ingestion runner + * refuses the RUN, before a single envelope reaches a consumer). + */ +import type { + CompatibilityPolicy, + CompatibilityVerdict, + RecordSchemaFields, + SchemaBreak, +} from "./record-schema"; + +/** + * BACKWARD breaks: a consumer written against `next` reading data produced + * under `previous` fails when + * - `next` adds a REQUIRED field `previous` never produced, or + * - `next` changes a field's type (old data has the old type), or + * - `next` drops a field that was REQUIRED in `previous` and is therefore + * part of the contract consumers rely on. + */ +function backwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] { + const breaks: SchemaBreak[] = []; + for (const [field, spec] of Object.entries(next)) { + const before = previous[field]; + if (!before) { + if (spec.required) { + breaks.push({ + field, + breaks: "backward", + detail: `new required field "${field}" is absent from every record written under the previous version`, + }); + } + continue; + } + if (before.type !== spec.type) { + breaks.push({ + field, + breaks: "backward", + detail: `field "${field}" changed type ${before.type} -> ${spec.type}; existing data still carries ${before.type}`, + }); + } + } + for (const [field, before] of Object.entries(previous)) { + if (!next[field] && before.required) { + breaks.push({ + field, + breaks: "backward", + detail: `required field "${field}" was removed; consumers of the previous version still depend on it`, + }); + } + } + return breaks; +} + +/** + * FORWARD breaks: a consumer written against `previous` reading data produced + * under `next` fails when + * - `next` removes a field `previous` marked REQUIRED, or + * - `next` changes a field's type. + * Adding a required field is forward-SAFE (old readers ignore it). + */ +function forwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] { + const breaks: SchemaBreak[] = []; + for (const [field, before] of Object.entries(previous)) { + const after = next[field]; + if (!after) { + if (before.required) { + breaks.push({ + field, + breaks: "forward", + detail: `required field "${field}" was removed; readers pinned to the previous version will not find it`, + }); + } + continue; + } + if (before.type !== after.type) { + breaks.push({ + field, + breaks: "forward", + detail: `field "${field}" changed type ${before.type} -> ${after.type}; readers pinned to the previous version expect ${before.type}`, + }); + } + } + return breaks; +} + +/** Deduplicate identical (field, direction, detail) triples produced by both directions under `full`. */ +function dedupe(breaks: SchemaBreak[]): SchemaBreak[] { + const seen = new Set(); + const out: SchemaBreak[] = []; + for (const b of breaks) { + const key = `${b.field}|${b.breaks}|${b.detail}`; + if (seen.has(key)) continue; + seen.add(key); + out.push(b); + } + return out; +} + +export function checkCompatibility( + previous: RecordSchemaFields, + next: RecordSchemaFields, + policy: CompatibilityPolicy, +): CompatibilityVerdict { + if (policy === "none") { + return { compatible: true, policy, breaks: [] }; + } + const breaks: SchemaBreak[] = []; + if (policy === "backward" || policy === "full") breaks.push(...backwardBreaks(previous, next)); + if (policy === "forward" || policy === "full") breaks.push(...forwardBreaks(previous, next)); + const deduped = dedupe(breaks); + return { compatible: deduped.length === 0, policy, breaks: deduped }; +} + +/** + * Does a concrete record conform to a schema? Used by the ingestion runner to + * detect that a source has SILENTLY changed shape — i.e. it is emitting data + * the registered schema does not describe — which is the case P063 exists to + * catch before consumers see it. Returns the problems; empty = conforming. + */ +export function conformanceErrors( + fields: RecordSchemaFields, + payload: Record, +): string[] { + const errors: string[] = []; + for (const [field, spec] of Object.entries(fields)) { + const value = payload[field]; + if (value === undefined || value === null) { + if (spec.required) errors.push(`required field "${field}" is missing`); + continue; + } + if (!matchesType(value, spec.type)) { + errors.push(`field "${field}" is ${describe(value)}, expected ${spec.type}`); + } + } + for (const field of Object.keys(payload)) { + if (!fields[field]) { + errors.push(`unregistered field "${field}" appeared in the payload`); + } + } + return errors; +} + +function matchesType(value: unknown, type: string): boolean { + switch (type) { + case "string": + return typeof value === "string"; + case "number": + return typeof value === "number" && Number.isFinite(value); + case "boolean": + return typeof value === "boolean"; + case "timestamp": + return typeof value === "string" && !Number.isNaN(Date.parse(value)); + case "array": + return Array.isArray(value); + case "object": + return typeof value === "object" && value !== null && !Array.isArray(value); + default: + return false; + } +} + +function describe(value: unknown): string { + if (Array.isArray(value)) return "array"; + return typeof value; +} diff --git a/src/schema/record-schema.ts b/src/schema/record-schema.ts new file mode 100644 index 0000000..e865162 --- /dev/null +++ b/src/schema/record-schema.ts @@ -0,0 +1,65 @@ +/** + * src/schema/record-schema.ts + * ============================ + * P063: the schema 0pt versions per source, and the compatibility policy that + * governs what the next version may change. + * + * Deliberately a SMALL structural schema (field name -> type + required), not + * a second JSON-Schema dialect: 0pt already ships JSON Schema in + * packages/schemas/ for its own contracts, and Zod in src/validation for HTTP + * bodies. Neither of those answers the question P063 actually asks — "is + * version N+1 of an UPSTREAM source's record shape safe for the consumers + * reading version N?" — which needs a comparable, field-level structure, not + * a validator. Anything richer belongs in a real schema registry + * (Confluent/Apicurio) and would be an integration, not a rewrite of this. + */ + +export type FieldType = "string" | "number" | "boolean" | "timestamp" | "object" | "array"; + +export interface FieldSpec { + type: FieldType; + required: boolean; +} + +/** A record shape: field name -> spec. */ +export type RecordSchemaFields = Record; + +/** + * Compatibility policy, using the standard schema-registry vocabulary: + * + * - `backward` — new readers can read data written under the OLD schema. + * Safe: adding an optional field, removing an optional field, relaxing a + * required field to optional. Breaking: adding a REQUIRED field, removing + * a field an old writer still emits as required, changing a type. + * - `forward` — old readers can read data written under the NEW schema. + * Safe: adding a required field, removing an optional field. Breaking: + * removing a REQUIRED field, changing a type. + * - `full` — both of the above. + * - `none` — no check. Recorded, never silently assumed: a source must + * declare `none` explicitly, and the runner logs it as an accepted risk. + */ +export type CompatibilityPolicy = "backward" | "forward" | "full" | "none"; + +export interface RecordSchema { + source_id: string; + tenant_id: string; + schema_version: number; + fields: RecordSchemaFields; + compatibility: CompatibilityPolicy; + registered_at: string; + registered_by?: string; +} + +/** One incompatibility between two schema versions. */ +export interface SchemaBreak { + field: string; + /** Which direction the change breaks. */ + breaks: "backward" | "forward"; + detail: string; +} + +export interface CompatibilityVerdict { + compatible: boolean; + policy: CompatibilityPolicy; + breaks: SchemaBreak[]; +} diff --git a/src/schema/schema-registry.ts b/src/schema/schema-registry.ts new file mode 100644 index 0000000..559f008 --- /dev/null +++ b/src/schema/schema-registry.ts @@ -0,0 +1,144 @@ +/** + * src/schema/schema-registry.ts + * ============================== + * P063: versioning + compatibility policy for a source's record shape. + * + * The DoD is "a breaking schema is detected BEFORE consumer corruption", so + * this registry is placed on the ingestion path in two places, both of them + * upstream of any consumer write: + * + * 1. `registerFromDiscovery` runs when the runner calls connector.discover(), + * i.e. before the first read. A shape change that breaks the declared + * policy makes the RUN fail with the field-level reasons — no envelope is + * produced at all. + * 2. `conformanceErrors` (src/schema/compatibility.ts) runs per envelope, so + * a source that changes shape mid-stream without changing its declared + * schema is caught on the offending record rather than on the next + * discovery, days later. + * + * A source whose policy is "none" opts out of (1) — explicitly, recorded in + * the registry row, never by default. + */ +import { SiaxError } from "../platform/errors"; +import type { IngestionRepository } from "../repositories/ingestion-repository"; +import { checkCompatibility } from "./compatibility"; +import type { + CompatibilityPolicy, + CompatibilityVerdict, + RecordSchema, + RecordSchemaFields, +} from "./record-schema"; + +export interface RegisterSchemaInput { + source_id: string; + tenant_id: string; + fields: RecordSchemaFields; + /** Policy governing THIS and future versions. Required — there is no default. */ + compatibility: CompatibilityPolicy; + registered_by?: string; +} + +export interface RegisterSchemaResult { + schema: RecordSchema; + /** True when the shape was identical to the current version and no new version was created. */ + unchanged: boolean; + /** Present when a comparison ran (i.e. a previous version existed). */ + verdict?: CompatibilityVerdict; +} + +/** Stable comparison of two field maps, order-independent. */ +function sameFields(a: RecordSchemaFields, b: RecordSchemaFields): boolean { + const ka = Object.keys(a).sort(); + const kb = Object.keys(b).sort(); + if (ka.length !== kb.length) return false; + for (let i = 0; i < ka.length; i++) { + if (ka[i] !== kb[i]) return false; + const fa = a[ka[i]]; + const fb = b[kb[i]]; + if (fa.type !== fb.type || fa.required !== fb.required) return false; + } + return true; +} + +export class SchemaRegistry { + constructor(private readonly repo: IngestionRepository) {} + + async getLatest(sourceId: string): Promise { + return this.repo.getLatestSchema(sourceId); + } + + async list(sourceId: string): Promise { + return this.repo.listSchemas(sourceId); + } + + /** + * Register a shape. Throws `SiaxError("CONFLICT")` — naming every break — + * when the new shape violates the previous version's declared policy. + * Nothing is persisted when it throws. + */ + async register(input: RegisterSchemaInput): Promise { + if (Object.keys(input.fields).length === 0) { + throw new SiaxError("INVALID_REQUEST", "0pt.schema_registry", "a schema must declare at least one field"); + } + const previous = await this.repo.getLatestSchema(input.source_id); + if (!previous) { + const schema: RecordSchema = { + source_id: input.source_id, + tenant_id: input.tenant_id, + schema_version: 1, + fields: input.fields, + compatibility: input.compatibility, + registered_at: new Date().toISOString(), + registered_by: input.registered_by, + }; + await this.repo.putSchema(schema); + return { schema, unchanged: false }; + } + + if (sameFields(previous.fields, input.fields) && previous.compatibility === input.compatibility) { + return { schema: previous, unchanged: true }; + } + + // The PREVIOUS version's policy governs whether this change is allowed — + // a producer must not be able to widen its own policy retroactively by + // declaring "none" on the breaking version. + const verdict = checkCompatibility(previous.fields, input.fields, previous.compatibility); + if (!verdict.compatible) { + throw new SiaxError( + "CONFLICT", + "0pt.schema_registry", + `schema v${previous.schema_version + 1} for source ${input.source_id} breaks the declared ` + + `"${previous.compatibility}" compatibility policy: ` + + verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "), + ); + } + + const schema: RecordSchema = { + source_id: input.source_id, + tenant_id: input.tenant_id, + schema_version: previous.schema_version + 1, + fields: input.fields, + compatibility: input.compatibility, + registered_at: new Date().toISOString(), + registered_by: input.registered_by, + }; + await this.repo.putSchema(schema); + return { schema, unchanged: false, verdict }; + } + + /** + * Non-throwing form used by the runner's pre-flight: answers whether the + * shape a connector just discovered would be accepted, without registering + * it. Used to report a break as a RUN failure with a full reason list + * rather than an exception message. + */ + async evaluate( + sourceId: string, + fields: RecordSchemaFields, + ): Promise<{ previous?: RecordSchema; verdict?: CompatibilityVerdict; changed: boolean }> { + const previous = await this.repo.getLatestSchema(sourceId); + if (!previous) return { changed: true }; + if (sameFields(previous.fields, fields)) return { previous, changed: false }; + return { previous, verdict: checkCompatibility(previous.fields, fields, previous.compatibility), changed: true }; + } +} diff --git a/src/services/source-registry-service.ts b/src/services/source-registry-service.ts index e9ec51d..e373ee7 100644 --- a/src/services/source-registry-service.ts +++ b/src/services/source-registry-service.ts @@ -44,6 +44,8 @@ import { SourceService, type SourceServiceOptions } from "./source-service"; import { SourceRatePolicyService } from "./source-rate-policy-service"; import type { CreateSourceInput, Source } from "../types/source"; import type { + SourceAuthorization, + SourceClassification, SourceDescriptor, SourceFreshness, SourceHealth, @@ -51,13 +53,24 @@ import type { SourceHealthStatus, SourceProvenance, SourceRights, + SourceSchedule, } from "../types/source-registry"; export interface RegisterSourceInput extends CreateSourceInput { rights?: Omit; provenance?: Omit; + authorization?: Omit; + schedule?: Omit; } +/** + * Actor ids that may never appear as an authorization's `granted_by`. + * P060 requires an accountable HUMAN decision behind a source going live; + * a service identity granting itself access is exactly the rubber stamp the + * gate exists to prevent. + */ +const NON_ACCOUNTABLE_ACTORS = /^(system|svc_|service:|robot|automation|anonymous)/i; + export class SourceRegistryService { private readonly repo: SourceRegistryRepository; private readonly aud0: Aud0Client; @@ -89,8 +102,16 @@ export class SourceRegistryService { // source.register // ------------------------------------------------------------------------- async registerSource(input: RegisterSourceInput): Promise { - const { rights, provenance, ...base } = input; - const source = await this.sources.createSource(base); + const { rights, provenance, authorization, schedule, ...base } = input; + // P060: registration NEVER activates. A caller asking for status "active" + // gets the source created "pending" and must go through activateSource, + // which checks the classification. Silently honouring `status: "active"` + // here would leave the gate wide open to its most obvious bypass. + const requestedActive = base.status === "active"; + const source = await this.sources.createSource({ + ...base, + status: requestedActive ? "pending" : base.status, + }); if (rights) { await this.setRights({ ...rights, source_id: source.source_id, tenant_id: source.tenant_id }); } @@ -102,9 +123,165 @@ export class SourceRegistryService { updated_at: new Date().toISOString(), }); } + if (authorization) { + await this.setAuthorization({ + ...authorization, + source_id: source.source_id, + tenant_id: source.tenant_id, + }); + } + if (schedule) { + await this.setSchedule({ + ...schedule, + source_id: source.source_id, + tenant_id: source.tenant_id, + updated_at: new Date().toISOString(), + }); + } + // A registration that arrived fully classified AND asked to be active is + // activated here — through the same gate, not around it. + if (requestedActive) { + await this.activateSource(source.source_id, source.tenant_id, base.created_by); + } return this.describeSource(source.source_id, source.tenant_id); } + // ------------------------------------------------------------------------- + // P060 authorization / schedule + // ------------------------------------------------------------------------- + async setAuthorization(auth: SourceAuthorization): Promise { + await this.assertSourceTenant(auth.source_id, auth.tenant_id); + if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.source_registry", + `authorization.granted_by must name an accountable actor, not "${auth.granted_by}"`, + ); + } + await this.repo.setAuthorization(auth); + await this.emit("0pt.source_authorization_set", auth.tenant_id, auth.source_id, { + basis: auth.basis, + granted_by: auth.granted_by, + expires_at: auth.expires_at, + evidence_ref: auth.evidence_ref, + }); + return auth; + } + + async getAuthorization(sourceId: string, tenantId: string): Promise { + await this.assertSourceTenant(sourceId, tenantId); + return this.repo.getAuthorization(sourceId); + } + + async setSchedule(schedule: SourceSchedule): Promise { + await this.assertSourceTenant(schedule.source_id, schedule.tenant_id); + if (schedule.interval_seconds !== undefined && schedule.interval_seconds <= 0) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.source_registry", + "schedule.interval_seconds must be positive", + ); + } + await this.repo.setSchedule(schedule); + await this.emit("0pt.source_schedule_set", schedule.tenant_id, schedule.source_id, { + mode: schedule.mode, + interval_seconds: schedule.interval_seconds, + enabled: schedule.enabled, + }); + return schedule; + } + + async getSchedule(sourceId: string, tenantId: string): Promise { + await this.assertSourceTenant(sourceId, tenantId); + return this.repo.getSchedule(sourceId); + } + + // ------------------------------------------------------------------------- + // P060 classification + activation gate + // ------------------------------------------------------------------------- + /** + * Is this source classified? A source is classified when ALL of the + * following are on record: + * - an accountable owner, + * - an unexpired authorization from an accountable actor, + * - rights: a license id AND at least one allowed purpose, + * - provenance: how the data is obtained. + * + * Everything else in the registry (schedule, schema, checkpoint) describes + * HOW a source runs, not WHETHER it may — those are deliberately not part + * of the gate, and that choice is documented here rather than left implicit. + */ + async classifySource( + sourceId: string, + tenantId: string, + now: Date = new Date(), + ): Promise { + const source = await this.assertSourceTenant(sourceId, tenantId); + const missing: string[] = []; + + if (!source.owner_id) missing.push("owner_id is not set"); + + const auth = await this.repo.getAuthorization(sourceId); + if (!auth) { + missing.push("no authorization on record"); + } else { + if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) { + missing.push(`authorization.granted_by "${auth.granted_by}" is not an accountable actor`); + } + if (auth.expires_at && new Date(auth.expires_at).getTime() <= now.getTime()) { + missing.push(`authorization expired at ${auth.expires_at}`); + } + } + + const rights = await this.repo.getRights(sourceId); + if (!rights) { + missing.push("no rights record (license/purpose) on record"); + } else { + if (!rights.license_id) missing.push("rights.license_id is empty"); + if (rights.allowed_purposes.length === 0) missing.push("rights.allowed_purposes is empty"); + } + + const provenance = await this.repo.getProvenance(sourceId); + if (!provenance) missing.push("no provenance record (sourcing method) on record"); + + return { source_id: sourceId, classified: missing.length === 0, missing }; + } + + /** + * P060 DoD: an UNCLASSIFIED source cannot be activated. Fail-closed — + * every missing fact is named at once, and nothing is written when the + * check fails. + */ + async activateSource(sourceId: string, tenantId: string, actorId?: string): Promise { + const classification = await this.classifySource(sourceId, tenantId); + if (!classification.classified) { + await this.emit("0pt.source_activation_denied", tenantId, sourceId, { + missing: classification.missing, + }); + throw new SiaxError( + "AUTHORIZATION_DENIED", + "0pt.source_registry", + `source ${sourceId} is not classified and cannot be activated: ${classification.missing.join("; ")}`, + ); + } + const source = await this.sources.updateStatus(sourceId, "active", actorId); + await this.emit("0pt.source_activated", tenantId, sourceId, { activated_by: actorId ?? null }); + return source; + } + + /** Take a source out of service. Always allowed — deactivating is never gated. */ + async deactivateSource( + sourceId: string, + tenantId: string, + status: "paused" | "archived" | "deprecated" = "paused", + actorId?: string, + ): Promise { + await this.assertSourceTenant(sourceId, tenantId); + const source = await this.sources.updateStatus(sourceId, status, actorId); + await this.emit("0pt.source_deactivated", tenantId, sourceId, { status }); + return source; + } + // ------------------------------------------------------------------------- // rights / license / purpose metadata // ------------------------------------------------------------------------- @@ -262,13 +439,17 @@ export class SourceRegistryService { async describeSource(sourceId: string, tenantId: string): Promise { const source = await this.assertSourceTenant(sourceId, tenantId); - const [policies, rights, provenance, health, freshness] = await Promise.all([ - this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }), - this.repo.getRights(sourceId), - this.repo.getProvenance(sourceId), - this.getHealth(sourceId, tenantId), - this.getFreshness(sourceId, tenantId), - ]); + const [policies, rights, provenance, health, freshness, authorization, schedule, classification] = + await Promise.all([ + this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }), + this.repo.getRights(sourceId), + this.repo.getProvenance(sourceId), + this.getHealth(sourceId, tenantId), + this.getFreshness(sourceId, tenantId), + this.repo.getAuthorization(sourceId), + this.repo.getSchedule(sourceId), + this.classifySource(sourceId, tenantId), + ]); return { source, rights, @@ -276,6 +457,9 @@ export class SourceRegistryService { health, freshness, rate_policy: policies[0], + authorization, + schedule, + classification, }; } diff --git a/src/services/source-registry.ts b/src/services/source-registry.ts index 50121ff..e68d719 100644 --- a/src/services/source-registry.ts +++ b/src/services/source-registry.ts @@ -28,3 +28,19 @@ export const sourceRegistryService = new SourceRegistryService({ sourceService, ratePolicyService: sourceRatePolicyService, }); + +/** + * P060-P068 ingestion plane singletons. Same reasoning as above: the schema + * registry, checkpoint store, ledger, quarantine and lineage projection all + * share ONE repository so the in-memory dev/test mode is coherent within a + * process, and one `resolveStoreMode` decision in production. + */ +import { createIngestionRepository } from "../repositories/ingestion-repository"; +import { SchemaRegistry } from "../schema/schema-registry"; +import { LineageStore } from "../lineage/lineage-store"; +import { IngestionRunner } from "../ingestion/ingestion-runner"; + +export const ingestionRepository = createIngestionRepository(); +export const schemaRegistry = new SchemaRegistry(ingestionRepository); +export const lineageStore = new LineageStore(ingestionRepository); +export const ingestionRunner = new IngestionRunner(ingestionRepository, schemaRegistry, lineageStore); diff --git a/src/services/source-service.ts b/src/services/source-service.ts index 5eade69..b114f10 100644 --- a/src/services/source-service.ts +++ b/src/services/source-service.ts @@ -77,6 +77,32 @@ export class SourceService { return this.repo.get(sourceId); } + /** + * P060: the ONLY status transition path. Callers do not get a generic + * update method, because status is what the activation gate protects — + * SourceRegistryService.activateSource is the only caller that moves a + * source to "active", and it runs the classification check first. + */ + async updateStatus(sourceId: string, status: SourceStatus, actorId?: string): Promise { + const source = await this.repo.updateStatus(sourceId, status); + await this.aud0.ingestEvent({ + event_id: `evt_0pt_source_status_${source.source_id}_${Date.now()}`, + event_type: "0pt.source_status_changed", + tenant_id: source.tenant_id, + project_id: source.project_id, + actor_id: actorId ?? source.owner_id ?? "system", + app_id: "0pt", + capability_id: "0pt.update_source_status", + resource_type: "source", + resource_id: source.source_id, + risk_level: "L2", + payload: { status: source.status, name: source.name }, + created_at: source.updated_at ?? new Date().toISOString(), + correlation_id: source.correlation_id, + }); + return source; + } + async listSources(filter?: { tenant_id?: string; project_id?: string; diff --git a/src/types/ingestion.ts b/src/types/ingestion.ts new file mode 100644 index 0000000..1f265a8 --- /dev/null +++ b/src/types/ingestion.ts @@ -0,0 +1,111 @@ +/** + * src/types/ingestion.ts + * ======================= + * P062: THE common ingestion envelope. Every record that enters 0pt through + * a connector — batch page, stream message, or change event — is normalized + * into exactly this shape before anything downstream (quality gate, entity + * resolution, dataset write, lineage) sees it. Nothing downstream is allowed + * to special-case "this came from a file" vs "this came from a change feed"; + * that difference is expressed by `mode` + `operation` and nothing else. + * + * HONEST SCOPE NOTE. `mode: "cdc"` and the delete/update operations are part + * of the CONTRACT, and the runner handles them (a delete envelope carries no + * payload and is never quality-gated as if it had one). This repo does NOT + * ship a CDC *source* — there is no Debezium consumer, no replication-slot + * reader, no WAL tailer here, exactly as services/api/tests/test_known_gaps.py + * has stated since P010. What changed with P062 is that a CDC connector can + * now be added without redesigning the pipeline, and that the checkpoint it + * would resume from is durable. A CDC connector itself is still MISSING. + * + * `checkpoint` on a ReadResult is what makes resume real: the connector hands + * back the cursor that means "everything up to and including these envelopes + * has been produced", the runner commits it durably AFTER the envelopes are + * accepted, and a restart passes it straight back into read(). Committing it + * before acceptance would create silent gaps — that ordering is the whole + * point and is asserted in tests/p0/ingestion-runner.test.ts. + */ + +/** How data arrives from the source. */ +export type IngestionMode = "batch" | "stream" | "cdc"; + +/** + * What the record means. Batch/stream sources normally emit "upsert" + * (they carry current state); a change feed distinguishes the three CDC + * operations. "delete" envelopes carry `payload: null`. + */ +export type EnvelopeOperation = "upsert" | "insert" | "update" | "delete"; + +/** Per-envelope provenance — which code, which version, which run produced it. */ +export interface EnvelopeProvenance { + connector_id: string; + connector_version: string; + /** The ingestion run that produced this envelope. Joins to lineage + checkpoints. */ + run_id: string; + /** Opaque connector cursor at which this record was produced. */ + cursor?: string; +} + +/** One normalized record. `payload` is null only for `operation: "delete"`. */ +export interface IngestionEnvelope { + envelope_id: string; + source_id: string; + tenant_id: string; + project_id: string; + mode: IngestionMode; + operation: EnvelopeOperation; + /** Partition/stream within the source (a table, a topic, a file). */ + partition_key: string; + /** Stable business key within the partition — the unit of idempotency (P068). */ + record_key: string; + /** Schema version the payload conforms to (P063). */ + schema_version: number; + /** Monotonic per-(source, partition) sequence assigned by the runner. */ + sequence: number; + /** When the change happened upstream, when known. */ + occurred_at?: string; + /** When 0pt normalized it. */ + ingested_at: string; + payload: Record | null; + provenance: EnvelopeProvenance; +} + +/** + * The durable resume point for one (source, partition). Committed only after + * the envelopes it covers have been accepted downstream. + */ +export interface IngestionCheckpoint { + source_id: string; + tenant_id: string; + partition_key: string; + /** Opaque connector cursor. `undefined` means "never read". */ + cursor?: string; + /** Highest envelope sequence committed for this partition. */ + sequence: number; + records_committed: number; + committed_at: string; +} + +/** + * P064: an envelope the declared quality policy sent to quarantine instead of + * downstream. Quarantine is DURABLE and inspectable — a dropped record that + * only exists in a log line is a silently lost record. + */ +export interface QuarantineRecord { + quarantine_id: string; + source_id: string; + tenant_id: string; + run_id: string; + record_key: string; + /** One entry per failed expectation. */ + violations: Array<{ expectation_type: string; detail: string }>; + envelope: IngestionEnvelope; + quarantined_at: string; +} + +/** P068: what the idempotency ledger decided about one delivered record. */ +export type LedgerVerdict = "new" | "changed" | "duplicate"; + +export interface LedgerDecision { + record_key: string; + verdict: LedgerVerdict; +} diff --git a/src/types/source-registry.ts b/src/types/source-registry.ts index 8fb17c0..65e65d9 100644 --- a/src/types/source-registry.ts +++ b/src/types/source-registry.ts @@ -87,7 +87,11 @@ export interface SourceFreshness { age_hours?: number; } -/** The full `source.describe` document (§10): base + rights + provenance + health + freshness + rate policy. */ +/** + * The full `source.describe` document (§10): base + rights + provenance + + * health + freshness + rate policy, plus the P060 additions (authorization, + * schedule, and the classification verdict the activation gate applies). + */ export interface SourceDescriptor { source: import("./source").Source; rights?: SourceRights; @@ -95,4 +99,67 @@ export interface SourceDescriptor { health: SourceHealth; freshness: SourceFreshness; rate_policy?: import("./source-rate-policy").SourceRatePolicy; + authorization?: SourceAuthorization; + schedule?: SourceSchedule; + /** Why this source may (or may not) be activated. */ + classification?: SourceClassification; +} + +// --------------------------------------------------------------------------- +// P060: the registry facts that were missing from the original §10 pass — +// authorization, schedule, schema reference, checkpoint — plus the +// classification verdict the activation gate produces. +// --------------------------------------------------------------------------- + +/** Legal/contractual ground on which this tenant may connect to the source at all. */ +export type AuthorizationBasis = + | "contract" + | "public_register" + | "consent" + | "legitimate_interest" + | "internal"; + +/** + * The recorded decision that 0pt MAY connect to this source (P060 + * "authorization"). Deliberately separate from {@link SourceRights}: rights + * say what the DATA may be used for, authorization says who signed off that + * the CONNECTION may exist. A source needs both to be classified. + */ +export interface SourceAuthorization { + source_id: string; + tenant_id: string; + basis: AuthorizationBasis; + /** Accountable actor id — never "system"/"svc_*" (the gate rejects those). */ + granted_by: string; + granted_at: string; + /** ISO timestamp; an authorization past this instant does not classify the source. */ + expires_at?: string; + /** AUD0 evidence ref or contract artifact ref backing the decision. */ + evidence_ref?: string; + notes?: string; +} + +/** How often 0pt pulls from the source, and in which ingestion mode (P060 "schedule"). */ +export interface SourceSchedule { + source_id: string; + tenant_id: string; + mode: "batch" | "stream" | "cdc"; + /** Poll cadence in seconds. Omitted for a continuously-open stream. */ + interval_seconds?: number; + /** Optional cron expression. Recorded here; 0pt does not execute cron itself (ACT0 owns scheduling). */ + cron?: string; + enabled: boolean; + updated_at: string; +} + +/** + * Why a source is (not) classified. `activateSource` refuses on + * `classified === false` and names every missing fact, so an operator gets + * the whole list at once instead of one error per round trip. + */ +export interface SourceClassification { + source_id: string; + classified: boolean; + /** Human-readable reasons, one per missing/expired fact. Empty when classified. */ + missing: string[]; } diff --git a/src/types/source.ts b/src/types/source.ts index 2489b40..62f356e 100644 --- a/src/types/source.ts +++ b/src/types/source.ts @@ -34,8 +34,16 @@ export type SourceType = "database" | "file" | "web" | "api" | "stream" | "manua * Union of both status vocabularies. "archived" and "deprecated" both survive * the unification merge -- they were introduced independently and no code * treats either as canonical. + * + * "pending" (P060) is the state every newly registered source starts in. It + * exists because "active" used to be the CREATE default + * (postgres-source-repository.ts / in-memory-source-repository.ts both did + * `input.status ?? "active"`), which meant a source with no rights, no + * provenance and no authorization was live the instant it was registered. + * Reaching "active" now requires passing SourceRegistryService.activateSource's + * classification gate. */ -export type SourceStatus = "active" | "paused" | "archived" | "deprecated"; +export type SourceStatus = "pending" | "active" | "paused" | "archived" | "deprecated"; /** Legal-sourcing vocabulary, mirroring dim_source.access. */ export type SourceAccess = "api" | "bulk" | "authorized_web" | "licensed"; @@ -60,7 +68,14 @@ export interface Source { correlation_id?: string; created_at: string; updated_at?: string; + /** Who registered the source. */ created_by?: string; + /** + * Who is ACCOUNTABLE for the source going forward (P060 "owner"). Distinct + * from `created_by`, which is only who typed the registration. Required by + * the activation gate. + */ + owner_id?: string; } export interface CreateSourceInput { @@ -75,4 +90,5 @@ export interface CreateSourceInput { status?: SourceStatus; correlation_id?: string; created_by?: string; + owner_id?: string; } diff --git a/src/validation/0pt-schemas.ts b/src/validation/0pt-schemas.ts index eb42f9e..a94989e 100644 --- a/src/validation/0pt-schemas.ts +++ b/src/validation/0pt-schemas.ts @@ -127,7 +127,8 @@ export const registerSourceSchema = z.object({ adapter_id: z.string().min(1).optional(), access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(), description: z.string().min(1).optional(), - status: z.enum(['active', 'paused', 'archived', 'deprecated']).optional(), + status: z.enum(['pending', 'active', 'paused', 'archived', 'deprecated']).optional(), + owner_id: z.string().min(1).optional(), correlation_id: z.string().min(1).optional(), created_by: z.string().min(1).optional(), rights: sourceRightsInputSchema.optional(), @@ -232,3 +233,33 @@ export const qualityBridgeRunSchema = z.object({ rows: z.array(jsonObject), correlation_id: z.string().min(1).optional(), }); + +/** Nested `authorization` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/authorization (P060). */ +const sourceAuthorizationInputSchema = z.object({ + basis: z.enum(['contract', 'public_register', 'consent', 'legitimate_interest', 'internal']), + granted_by: z.string().min(1), + granted_at: z.string().min(1), + expires_at: z.string().min(1).optional(), + evidence_ref: z.string().min(1).optional(), + notes: z.string().min(1).optional(), +}); + +/** PUT /api/0pt/sources/[sourceId]/authorization -- source.set_authorization (P060). */ +export const setSourceAuthorizationSchema = sourceAuthorizationInputSchema.extend({ + tenant_id: z.string().min(1), +}); + +/** PUT /api/0pt/sources/[sourceId]/schedule -- source.set_schedule (P060). */ +export const setSourceScheduleSchema = z.object({ + tenant_id: z.string().min(1), + mode: z.enum(['batch', 'stream', 'cdc']), + interval_seconds: z.number().int().positive().optional(), + cron: z.string().min(1).optional(), + enabled: z.boolean(), +}); + +/** POST /api/0pt/sources/[sourceId]/activate -- source.activate (P060 classification gate). */ +export const activateSourceSchema = z.object({ + tenant_id: z.string().min(1), + actor_id: z.string().min(1).optional(), +}); diff --git a/tests/api/0pt-p060-routes.test.ts b/tests/api/0pt-p060-routes.test.ts new file mode 100644 index 0000000..9259bda --- /dev/null +++ b/tests/api/0pt-p060-routes.test.ts @@ -0,0 +1,175 @@ +/** + * HTTP-level tests for the P060/P067 routes added in this pass, in the same + * shape as tests/api/0pt-cross-tenant-routes.test.ts: call the handler + * directly with a hand-built NextRequest carrying `x-0pt-verified-tenant`, + * which is exactly what middleware.ts forwards once a real Bearer token has + * been verified. + * + * Two things are proved here that the service-level tests cannot: + * - the new routes carry the SAME cross-tenant guard as the existing ones + * (a new route with no guard would be the P014 hole reopened), and + * - the activation gate is enforced at the HTTP boundary too — an + * unclassified source gets 403, not a 200 with a quiet no-op. + */ +import { describe, expect, it } from "vitest"; +import { NextRequest } from "next/server"; +import { POST as sourcesPOST } from "@/app/api/0pt/sources/route"; +import { GET as classifyGET, POST as activatePOST } from "@/app/api/0pt/sources/[sourceId]/activate/route"; +import { PUT as authorizationPUT } from "@/app/api/0pt/sources/[sourceId]/authorization/route"; +import { PUT as schedulePUT } from "@/app/api/0pt/sources/[sourceId]/schedule/route"; +import { GET as lineageGET } from "@/app/api/0pt/lineage/route"; + +const VERIFIED = "x-0pt-verified-tenant"; + +function getReq(url: string, verifiedTenant = "t_a"): NextRequest { + return new NextRequest(new URL(url, "http://localhost"), { headers: { [VERIFIED]: verifiedTenant } }); +} +function bodyReq(url: string, method: "POST" | "PUT", body: unknown, verifiedTenant = "t_a"): NextRequest { + return new NextRequest(new URL(url, "http://localhost"), { + method, + headers: { "content-type": "application/json", [VERIFIED]: verifiedTenant }, + body: JSON.stringify(body), + }); +} +const params = (sourceId: string) => ({ params: Promise.resolve({ sourceId }) }); + +describe("P060 routes — cross-tenant guard", () => { + it("POST /activate rejects a foreign tenant_id (403)", async () => { + const res = await activatePOST( + bodyReq("/api/0pt/sources/src_x/activate", "POST", { tenant_id: "t_victim" }), + params("src_x"), + ); + expect(res.status).toBe(403); + expect((await res.json()).error).toBe("OPT_ACCESS_DENIED"); + }); + + it("GET /activate (classification) rejects a foreign tenant_id (403)", async () => { + const res = await classifyGET( + getReq("/api/0pt/sources/src_x/activate?tenant_id=t_victim"), + params("src_x"), + ); + expect(res.status).toBe(403); + }); + + it("PUT /authorization rejects a foreign tenant_id (403)", async () => { + const res = await authorizationPUT( + bodyReq("/api/0pt/sources/src_x/authorization", "PUT", { + tenant_id: "t_victim", + basis: "contract", + granted_by: "usr_attacker", + granted_at: "2026-09-04T00:00:00.000Z", + }), + params("src_x"), + ); + expect(res.status).toBe(403); + }); + + it("PUT /schedule rejects a foreign tenant_id (403)", async () => { + const res = await schedulePUT( + bodyReq("/api/0pt/sources/src_x/schedule", "PUT", { tenant_id: "t_victim", mode: "batch", enabled: true }), + params("src_x"), + ); + expect(res.status).toBe(403); + }); + + it("GET /lineage rejects a foreign tenant_id (403)", async () => { + const res = await lineageGET(getReq("/api/0pt/lineage?tenant_id=t_victim&output_name=0pt:dataset:ds_x")); + expect(res.status).toBe(403); + }); + + it("GET /lineage refuses an unfiltered dump (400)", async () => { + const res = await lineageGET(getReq("/api/0pt/lineage?tenant_id=t_a")); + expect(res.status).toBe(400); + }); +}); + +describe("P060 routes — the activation gate over HTTP", () => { + it("an unclassified source cannot be activated through the API", async () => { + const created = await sourcesPOST( + bodyReq("/api/0pt/sources", "POST", { + source_id: "src_http_gate_1", + tenant_id: "t_a", + project_id: "p_a", + source_type: "api", + name: "Unclassified source", + }), + ); + expect(created.status).toBeLessThan(300); + const body = await created.json(); + expect(body.source.status).toBe("pending"); + + const classification = await classifyGET( + getReq("/api/0pt/sources/src_http_gate_1/activate?tenant_id=t_a"), + params("src_http_gate_1"), + ); + expect(classification.status).toBe(200); + expect((await classification.json()).classified).toBe(false); + + const activated = await activatePOST( + bodyReq("/api/0pt/sources/src_http_gate_1/activate", "POST", { tenant_id: "t_a" }), + params("src_http_gate_1"), + ); + expect(activated.status).toBe(403); + }); + + it("a classified source activates through the API", async () => { + await sourcesPOST( + bodyReq("/api/0pt/sources", "POST", { + source_id: "src_http_gate_2", + tenant_id: "t_a", + project_id: "p_a", + source_type: "api", + name: "Classified source", + owner_id: "usr_simon", + rights: { + license_id: "CC-BY-4.0", + allowed_purposes: ["catalog"], + pii_allowed: false, + redistribution: "internal", + verified_at: "2026-09-04", + verified_by: "usr_simon", + }, + provenance: { method: "official_api" }, + }), + ); + const auth = await authorizationPUT( + bodyReq("/api/0pt/sources/src_http_gate_2/authorization", "PUT", { + tenant_id: "t_a", + basis: "public_register", + granted_by: "usr_simon", + granted_at: "2026-09-01T00:00:00.000Z", + }), + params("src_http_gate_2"), + ); + expect(auth.status).toBe(200); + + const activated = await activatePOST( + bodyReq("/api/0pt/sources/src_http_gate_2/activate", "POST", { tenant_id: "t_a", actor_id: "usr_simon" }), + params("src_http_gate_2"), + ); + expect(activated.status).toBe(200); + expect((await activated.json()).status).toBe("active"); + }); + + it("a service identity cannot be recorded as the authorizing actor (400)", async () => { + await sourcesPOST( + bodyReq("/api/0pt/sources", "POST", { + source_id: "src_http_gate_3", + tenant_id: "t_a", + project_id: "p_a", + source_type: "api", + name: "Self-authorizing source", + }), + ); + const res = await authorizationPUT( + bodyReq("/api/0pt/sources/src_http_gate_3/authorization", "PUT", { + tenant_id: "t_a", + basis: "internal", + granted_by: "svc_0pt", + granted_at: "2026-09-01T00:00:00.000Z", + }), + params("src_http_gate_3"), + ); + expect(res.status).toBe(400); + }); +}); diff --git a/tests/p0/connector-conformance.test.ts b/tests/p0/connector-conformance.test.ts new file mode 100644 index 0000000..ed63b42 --- /dev/null +++ b/tests/p0/connector-conformance.test.ts @@ -0,0 +1,165 @@ +/** + * P061 DoD: "two heterogeneous connectors pass the same conformance suite." + * + * This file is that proof. It runs the SAME function + * (src/connectors/conformance.ts::runConnectorConformance) against two + * connectors that share nothing but the contract: + * + * HttpJsonConnector — a real node:http server on a real socket, opaque + * cursor paging, network health check, timestamp + * backfill bounds. + * JsonlFileConnector — a real file on disk, line-offset cursor, filesystem + * health check, line-number backfill bounds, and CDC + * delete records the HTTP connector cannot produce. + * + * Neither connector is mocked. The HTTP fixture is a genuine server that + * implements paging server-side; the file fixture is a genuine file written + * to a temp dir. A connector that only "passed" against a stub would prove + * nothing about the contract. + */ +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import http from "node:http"; +import type { AddressInfo } from "node:net"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { runConnectorConformance, type ConformanceFixture } from "@/src/connectors/conformance"; +import { HttpJsonConnector } from "@/src/connectors/http-json-connector"; +import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; + +const SCHEMA: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; + +/** 5 records, shared by both fixtures so the suite's counts are comparable. */ +const ROWS = [ + { org_number: "556001-0001", name: "Alfa AB", employees: 10, seq: 1 }, + { org_number: "556002-0002", name: "Beta AB", employees: 20, seq: 2 }, + { org_number: "556003-0003", name: "Gamma AB", employees: 30, seq: 3 }, + { org_number: "556004-0004", name: "Delta AB", employees: 40, seq: 4 }, + { org_number: "556005-0005", name: "Epsilon AB", employees: 50, seq: 5 }, +]; + +// --- fixture 1: a real paged HTTP API --------------------------------------- + +let server: http.Server; +let baseUrl = ""; + +beforeAll(async () => { + server = http.createServer((req, res) => { + const url = new URL(req.url ?? "/", "http://localhost"); + if (url.pathname !== "/v1/records") { + res.writeHead(404).end("{}"); + return; + } + const limit = Number(url.searchParams.get("limit") ?? "100"); + const from = url.searchParams.get("from"); + const to = url.searchParams.get("to"); + if (from !== null && to !== null) { + // Backfill: bounds are `seq` values, from inclusive, to exclusive. + const slice = ROWS.filter((r) => r.seq >= Number(from) && r.seq < Number(to)).slice(0, limit); + res.writeHead(200, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ records: slice, next_cursor: null })); + return; + } + // Cursor is the `seq` of the last record already delivered. + const after = Number(url.searchParams.get("cursor") ?? "0"); + const slice = ROWS.filter((r) => r.seq > after).slice(0, limit); + const next = slice.length > 0 ? String(slice[slice.length - 1].seq) : null; + const more = slice.length > 0 && slice[slice.length - 1].seq < ROWS[ROWS.length - 1].seq; + res.writeHead(200, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ records: slice, next_cursor: more ? next : null })); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + baseUrl = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; +}); + +afterAll(async () => { + await new Promise((resolve) => server.close(() => resolve())); + await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined); +}); + +// --- fixture 2: a real JSONL file ------------------------------------------- + +const tmpDir = path.join(os.tmpdir(), `opt-conformance-${process.pid}`); +const jsonlPath = path.join(tmpDir, "companies.jsonl"); + +beforeAll(async () => { + await fs.mkdir(tmpDir, { recursive: true }); + await fs.writeFile( + jsonlPath, + ROWS.map((r) => JSON.stringify({ org_number: r.org_number, name: r.name, employees: r.employees })).join("\n") + "\n", + "utf8", + ); +}); + +const ctx = { tenant_id: "t_test", project_id: "p_test", source_id: "src_conf", run_id: "run_conf" }; + +describe("P061 connector contract — the same conformance suite, two heterogeneous connectors", () => { + it("HttpJsonConnector passes every check", async () => { + const fixture: ConformanceFixture = { + validConfig: { + base_url: baseUrl, + records_path: "/v1/records", + partition_key: "companies", + schema: SCHEMA, + record_key_field: "org_number", + cursor_field: "seq", + }, + // Not an absolute URL and no schema — must be rejected without any IO. + invalidConfig: { base_url: "not-a-url", records_path: "v1/records", partition_key: "", record_key_field: "" }, + ctx, + backfill: { from: "2", to: "4" }, + expectedRecordCount: ROWS.length, + }; + const report = await runConnectorConformance(new HttpJsonConnector(), fixture); + const failed = report.checks.filter((c) => !c.passed); + expect(failed.map((c) => `${c.name}: ${c.detail}`)).toEqual([]); + expect(report.passed).toBe(true); + expect(report.checks.length).toBeGreaterThanOrEqual(14); + }); + + it("JsonlFileConnector passes every check — the identical suite", async () => { + const fixture: ConformanceFixture = { + validConfig: { + file_path: jsonlPath, + partition_key: "companies", + schema: SCHEMA, + record_key_field: "org_number", + }, + // Relative path, wrong extension, no schema. + invalidConfig: { file_path: "companies.csv", partition_key: "", record_key_field: "" }, + ctx, + // File-connector bounds are LINE NUMBERS — deliberately a different + // vocabulary from the HTTP fixture's timestamps/sequence values, which + // is exactly the heterogeneity the shared contract has to survive. + backfill: { from: "1", to: "3" }, + expectedRecordCount: ROWS.length, + }; + const report = await runConnectorConformance(new JsonlFileConnector(), fixture); + const failed = report.checks.filter((c) => !c.passed); + expect(failed.map((c) => `${c.name}: ${c.detail}`)).toEqual([]); + expect(report.passed).toBe(true); + }); + + it("the suite actually FAILS a connector that breaks the contract (it is not decoration)", async () => { + // A connector that accepts any config at all — the classic fail-open bug. + class FailOpenConnector extends JsonlFileConnector { + override validate() { + return { valid: true, errors: [] }; + } + } + const report = await runConnectorConformance(new FailOpenConnector(), { + validConfig: { file_path: jsonlPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + invalidConfig: { file_path: "nope.csv", partition_key: "", record_key_field: "" }, + ctx, + backfill: { from: "1", to: "3" }, + expectedRecordCount: ROWS.length, + }); + expect(report.passed).toBe(false); + expect(report.checks.find((c) => c.name.startsWith("validate: rejects"))?.passed).toBe(false); + }); +}); diff --git a/tests/p0/ingestion-plane-postgres.realinfra.test.ts b/tests/p0/ingestion-plane-postgres.realinfra.test.ts new file mode 100644 index 0000000..3a947a6 --- /dev/null +++ b/tests/p0/ingestion-plane-postgres.realinfra.test.ts @@ -0,0 +1,310 @@ +/** + * REAL Postgres proof for P060 DoD #1 ("a restart preserves every source") + * and for the durability the rest of the plane depends on (P062 checkpoints, + * P063 schema versions, P064 quarantine, P067 lineage, P068 ledger). + * + * Same honest-degrade convention as tests/p0/source-registry-postgres.realinfra.test.ts: + * gated on an env var, SKIPS HONESTLY when unset, never fabricates success. + * + * docker run -d --name opt-ingestion-pgtest \ + * -e POSTGRES_USER=opt -e POSTGRES_PASSWORD=opt_test_pw -e POSTGRES_DB=opt_test \ + * -p 15434:5432 postgres:16-alpine + * INGESTION_TEST_DATABASE_URL=postgres://opt:opt_test_pw@localhost:15434/opt_test \ + * pnpm vitest run tests/p0/ingestion-plane-postgres.realinfra.test.ts + * + * "RESTART" here is modelled the only way a test can model it without killing + * the runner: every assertion after a write is made through a FRESHLY + * CONSTRUCTED service/repository over a NEW connection, so nothing can be + * answered out of an in-process Map. The stronger proof — stopping and + * starting the Postgres CONTAINER itself and re-reading — was run manually + * during this pass and is recorded in STATUS.md; it is not automated here + * because a test that restarts its own database would need docker access + * from inside CI, which these runners do not grant. + */ +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import fs from "node:fs"; +import path from "node:path"; +import postgres from "postgres"; +import { SourceRegistryService } from "@/src/services/source-registry-service"; +import { SourceService } from "@/src/services/source-service"; +import { PostgresSourceRepository } from "@/src/repositories/postgres-source-repository"; +import { PostgresSourceRegistryRepository } from "@/src/repositories/postgres-source-registry-repository"; +import { PostgresIngestionRepository } from "@/src/repositories/postgres-ingestion-repository"; +import { SchemaRegistry } from "@/src/schema/schema-registry"; +import { LineageStore } from "@/src/lineage/lineage-store"; +import { IngestionRunner } from "@/src/ingestion/ingestion-runner"; +import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector"; +import { SiaxError } from "@/src/platform/errors"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; +import os from "node:os"; +import { promises as fsp } from "node:fs"; + +const TEST_DATABASE_URL = process.env.INGESTION_TEST_DATABASE_URL; +const run = TEST_DATABASE_URL ? describe : describe.skip; + +let adminSql: ReturnType | undefined; +let reachable = false; +const tmpDir = path.join(os.tmpdir(), `opt-realinfra-${process.pid}`); +const filePath = path.join(tmpDir, "companies.jsonl"); + +const SCHEMA: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; +const ROWS = Array.from({ length: 6 }, (_, i) => ({ + org_number: `55611${i}-111${i}`, + name: `Realinfra ${i} AB`, + employees: (i + 1) * 10, +})); + +const TENANT = "t_realinfra"; +const SOURCE = "src_realinfra_1"; + +beforeAll(async () => { + if (!TEST_DATABASE_URL) { + console.warn( + "[realinfra] INGESTION_TEST_DATABASE_URL not set this run -- skipping honestly, not fabricating success.", + ); + return; + } + adminSql = postgres(TEST_DATABASE_URL, { prepare: false, ssl: false, max: 2 }); + await Promise.race([ + adminSql`SELECT 1`, + new Promise((_, reject) => setTimeout(() => reject(new Error("connect timeout")), 5000)), + ]); + reachable = true; + // Only the two migrations this plane needs. sql/001_init.sql is skipped on + // purpose: it requires the `vector` extension (pgvector), which a plain + // postgres:16-alpine does not ship, and none of the tables under test + // depend on it. Same choice tests/p0/source-registry-postgres.realinfra.test.ts + // already makes. + for (const file of ["002_source_registry.sql", "007_ingestion_plane.sql"]) { + const p = path.join(process.cwd(), "sql", file); + if (!fs.existsSync(p)) continue; + await adminSql.unsafe(fs.readFileSync(p, "utf8")); + } + // Clean slate for this suite's ids only — never a blanket TRUNCATE. + await adminSql`DELETE FROM opt_lineage_edges WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_quality_quarantine WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_ingestion_ledger WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_ingestion_checkpoints WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_schemas WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_authorization WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_schedule WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_rights WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_provenance WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_freshness WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_source_health_checks WHERE tenant_id = ${TENANT}`; + await adminSql`DELETE FROM opt_sources WHERE tenant_id = ${TENANT}`; + + await fsp.mkdir(tmpDir, { recursive: true }); + await fsp.writeFile(filePath, ROWS.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8"); +}); + +afterAll(async () => { + await adminSql?.end({ timeout: 5 }); + await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined); +}); + +/** A brand-new service over a brand-new connection — the "restarted process". */ +function freshRegistry(): SourceRegistryService { + const sql = postgres(TEST_DATABASE_URL as string, { prepare: false, ssl: false, max: 2 }); + return new SourceRegistryService({ + repository: new PostgresSourceRegistryRepository(sql), + sourceService: new SourceService({ repository: new PostgresSourceRepository(sql) }), + }); +} + +function freshIngestion(): { + repo: PostgresIngestionRepository; + runner: IngestionRunner; + lineage: LineageStore; +} { + const sql = postgres(TEST_DATABASE_URL as string, { prepare: false, ssl: false, max: 2 }); + const repo = new PostgresIngestionRepository(sql); + const lineage = new LineageStore(repo); + return { repo, runner: new IngestionRunner(repo, new SchemaRegistry(repo), lineage), lineage }; +} + +run("P060 real Postgres — a restart preserves every source and every registry fact", () => { + it("is reachable (otherwise the rest of this suite proves nothing)", () => { + expect(reachable).toBe(true); + }); + + it("registers pending, refuses activation unclassified, then activates — all surviving a fresh process", async () => { + const writer = freshRegistry(); + await writer.registerSource({ + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + source_type: "file", + name: "Realinfra file drop", + owner_id: "usr_simon", + schedule: { mode: "batch", interval_seconds: 900, enabled: true }, + }); + + // A DIFFERENT service instance over a DIFFERENT connection. + const reader = freshRegistry(); + expect((await reader.sources.getSource(SOURCE)).status).toBe("pending"); + await expect(reader.activateSource(SOURCE, TENANT)).rejects.toBeInstanceOf(SiaxError); + + const classifier = freshRegistry(); + await classifier.setRights({ + source_id: SOURCE, + tenant_id: TENANT, + license_id: "proprietary:realinfra", + allowed_purposes: ["catalog"], + pii_allowed: false, + redistribution: "internal", + verified_at: "2026-09-04", + verified_by: "usr_simon", + }); + await classifier.setProvenance({ + source_id: SOURCE, + tenant_id: TENANT, + method: "bulk_download", + updated_at: new Date().toISOString(), + }); + await classifier.setAuthorization({ + source_id: SOURCE, + tenant_id: TENANT, + basis: "contract", + granted_by: "usr_simon", + granted_at: new Date().toISOString(), + }); + + const activator = freshRegistry(); + expect((await activator.classifySource(SOURCE, TENANT)).classified).toBe(true); + expect((await activator.activateSource(SOURCE, TENANT, "usr_simon")).status).toBe("active"); + + // Read back through yet another fresh process AND directly in SQL, so the + // assertion cannot be satisfied by any in-process bookkeeping. + const verifier = freshRegistry(); + const descriptor = await verifier.describeSource(SOURCE, TENANT); + expect(descriptor.source.status).toBe("active"); + expect(descriptor.source.owner_id).toBe("usr_simon"); + expect(descriptor.authorization?.basis).toBe("contract"); + expect(descriptor.schedule?.interval_seconds).toBe(900); + expect(descriptor.classification?.classified).toBe(true); + + const rows = await (adminSql as ReturnType)< + { status: string; owner_id: string | null }[] + >`SELECT status, owner_id FROM opt_sources WHERE source_id = ${SOURCE}`; + expect(rows[0]).toEqual({ status: "active", owner_id: "usr_simon" }); + }); + + it("a raw INSERT that omits status lands PENDING — the gate is in the schema too", async () => { + const sql = adminSql as ReturnType; + await sql` + INSERT INTO opt_sources (source_id, tenant_id, project_id, source_type, name) + VALUES ('src_realinfra_raw', ${TENANT}, 'p_realinfra', 'api', 'Raw insert') + ON CONFLICT (source_id) DO NOTHING`; + const rows = await sql<{ status: string }[]>` + SELECT status FROM opt_sources WHERE source_id = 'src_realinfra_raw'`; + expect(rows[0].status).toBe("pending"); + }); +}); + +run("P062/P063/P064/P067/P068 real Postgres — the durable ingestion plane", () => { + it("runs, crashes, and resumes across fresh processes with no gap and no duplicate", async () => { + const first = freshIngestion(); + const delivered: string[] = []; + let pages = 0; + const input = { + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + mode: "batch" as const, + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + page_size: 2, + output_ref: "0pt:dataset:ds_realinfra", + }; + await expect( + first.runner.run({ + ...input, + sink: async (batch) => { + pages++; + if (pages === 2) throw new Error("simulated process death"); + for (const e of batch) delivered.push(e.record_key); + }, + }), + ).rejects.toThrow(/simulated process death/); + expect(delivered).toHaveLength(2); + + // Fresh everything — only Postgres carries the position forward. + const second = freshIngestion(); + const report = await second.runner.run({ + ...input, + connector: new JsonlFileConnector(), + sink: async (batch) => { + for (const e of batch) delivered.push(e.record_key); + }, + }); + expect(report.status).toBe("succeeded"); + expect(delivered).toHaveLength(ROWS.length); + expect(new Set(delivered).size).toBe(ROWS.length); + + const cp = await second.repo.getCheckpoint(SOURCE, "companies"); + expect(cp?.records_committed).toBe(ROWS.length); + }); + + it("the schema version, ledger, and lineage are all readable from a fresh process", async () => { + const fresh = freshIngestion(); + expect((await new SchemaRegistry(fresh.repo).getLatest(SOURCE))?.schema_version).toBe(1); + expect(await fresh.repo.countLedger(SOURCE)).toBe(ROWS.length); + const upstream = await fresh.lineage.upstreamOf(TENANT, "0pt:dataset:ds_realinfra"); + expect(upstream.length).toBeGreaterThanOrEqual(1); + expect(upstream[0].input_ref).toBe(`0pt:source:${SOURCE}`); + }); + + it("a repeated backfill is idempotent against the durable ledger", async () => { + const fresh = freshIngestion(); + const emitted: string[] = []; + const report = await fresh.runner.backfill({ + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + mode: "batch", + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + sink: async (batch) => { + for (const e of batch) emitted.push(e.record_key); + }, + page_size: 100, + range: { partition_key: "companies", from: "0", to: String(ROWS.length) }, + }); + expect(report.envelopes_emitted).toBe(0); + expect(report.duplicates_skipped).toBe(ROWS.length); + expect(emitted).toHaveLength(0); + }); + + it("quarantine survives to a fresh process with the full envelope", async () => { + const fresh = freshIngestion(); + await fresh.runner.backfill({ + source_id: SOURCE, + tenant_id: TENANT, + project_id: "p_realinfra", + mode: "batch", + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + sink: async () => undefined, + page_size: 100, + quality_policy: { + source_id: SOURCE, + tenant_id: TENANT, + expectations: [ + { expectation_type: "expect_column_values_to_be_between", kwargs: { column: "employees", min_value: 0, max_value: 20 } }, + ], + on_violation: "quarantine", + }, + range: { partition_key: "companies", from: "0", to: String(ROWS.length) }, + }); + const reader = freshIngestion(); + const quarantined = await reader.repo.listQuarantine(SOURCE); + expect(quarantined.length).toBe(ROWS.length - 2); + expect(quarantined[0].envelope.record_key).toBeTruthy(); + expect(quarantined[0].violations[0].expectation_type).toBe("expect_column_values_to_be_between"); + }); +}); diff --git a/tests/p0/ingestion-runner.test.ts b/tests/p0/ingestion-runner.test.ts new file mode 100644 index 0000000..14085e8 --- /dev/null +++ b/tests/p0/ingestion-runner.test.ts @@ -0,0 +1,412 @@ +/** + * The ingestion plane end to end: + * + * P062 — every connector's output becomes the same envelope, and a run that + * DIES mid-page resumes from the durable checkpoint with no gap and + * no double-delivery. + * P063 — a breaking schema change stops the run BEFORE the sink is called. + * P064 — quality failures block or quarantine according to the DECLARED + * policy, and a blocked page does not advance the checkpoint. + * P067 — a run's source -> output lineage is queryable afterwards. + * P068 — a repeated backfill is idempotent. + * + * The connectors here are the real ones (JsonlFileConnector against a real + * file on disk); the "crash" is a sink that throws, which is the closest + * faithful analogue of a process dying between the consumer write and the + * checkpoint commit. + */ +import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { IngestionRunner, type IngestionRunInput } from "@/src/ingestion/ingestion-runner"; +import { InMemoryIngestionRepository } from "@/src/repositories/in-memory-ingestion-repository"; +import { SchemaRegistry } from "@/src/schema/schema-registry"; +import { LineageStore } from "@/src/lineage/lineage-store"; +import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector"; +import { hashPayload } from "@/src/ingestion/envelope"; +import { SiaxError } from "@/src/platform/errors"; +import type { IngestionEnvelope } from "@/src/types/ingestion"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; + +const SCHEMA: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; + +const tmpDir = path.join(os.tmpdir(), `opt-runner-${process.pid}`); +const filePath = path.join(tmpDir, "companies.jsonl"); + +const ROWS = Array.from({ length: 10 }, (_, i) => ({ + org_number: `55600${i}-000${i}`, + name: `Bolag ${i} AB`, + employees: (i + 1) * 10, +})); + +beforeAll(async () => { + await fs.mkdir(tmpDir, { recursive: true }); + await fs.writeFile(filePath, ROWS.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8"); +}); +afterAll(async () => { + await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined); +}); + +let repo: InMemoryIngestionRepository; +let runner: IngestionRunner; +let lineage: LineageStore; + +beforeEach(() => { + repo = new InMemoryIngestionRepository(); + lineage = new LineageStore(repo); + runner = new IngestionRunner(repo, new SchemaRegistry(repo), lineage); +}); + +function baseInput(sink: IngestionRunInput["sink"], overrides: Partial = {}): IngestionRunInput { + return { + source_id: "src_ing", + tenant_id: "t_test", + project_id: "p_test", + mode: "batch", + connector: new JsonlFileConnector(), + config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + sink, + page_size: 3, + output_ref: "0pt:dataset:ds_companies", + ...overrides, + }; +} + +describe("P062 common envelope", () => { + it("normalizes every record into one envelope shape with provenance and a monotonic sequence", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run(baseInput(async (batch) => void seen.push(...batch))); + + expect(report.status).toBe("succeeded"); + expect(seen).toHaveLength(ROWS.length); + expect(seen.map((e) => e.sequence)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]); + for (const envelope of seen) { + expect(envelope.source_id).toBe("src_ing"); + expect(envelope.tenant_id).toBe("t_test"); + expect(envelope.mode).toBe("batch"); + expect(envelope.operation).toBe("upsert"); + expect(envelope.partition_key).toBe("companies"); + expect(envelope.schema_version).toBe(1); + expect(envelope.provenance.connector_id).toBe("jsonl_file"); + expect(envelope.provenance.connector_version).toBe("jsonl-file-connector@1"); + expect(envelope.provenance.run_id).toBe(report.run_id); + expect(envelope.ingested_at).toMatch(/^\d{4}-\d{2}-\d{2}T/); + } + }); + + it("the durable checkpoint advances to the end of the partition", async () => { + await runner.run(baseInput(async () => undefined)); + const checkpoint = await repo.getCheckpoint("src_ing", "companies"); + expect(checkpoint?.cursor).toBe(String(ROWS.length)); + expect(checkpoint?.sequence).toBe(ROWS.length); + expect(checkpoint?.records_committed).toBe(ROWS.length); + }); + + it("a run that dies inside the sink resumes with NO gap and NO double delivery", async () => { + const delivered: string[] = []; + let pages = 0; + + // First run: the sink dies on the third page (records 7-9), AFTER two + // pages have been committed. This is the crash window the ordering exists + // to protect: the checkpoint for page 3 was never committed. + const crashing = baseInput(async (batch) => { + pages++; + if (pages === 3) throw new Error("process died mid-write"); + for (const e of batch) delivered.push(e.record_key); + }); + await expect(runner.run(crashing)).rejects.toThrow(/process died mid-write/); + + expect(delivered).toHaveLength(6); + const afterCrash = await repo.getCheckpoint("src_ing", "companies"); + expect(afterCrash?.cursor).toBe("6"); + + // Second run: a FRESH runner and a FRESH connector, exactly as a restarted + // process would be — only the durable checkpoint carries over. + const resumed = new IngestionRunner(repo, new SchemaRegistry(repo), lineage); + const report = await resumed.run( + baseInput(async (batch) => { + for (const e of batch) delivered.push(e.record_key); + }), + ); + + expect(report.status).toBe("succeeded"); + // Every record exactly once across both runs: no gap, no duplicate. + expect(delivered).toHaveLength(ROWS.length); + expect(new Set(delivered).size).toBe(ROWS.length); + expect([...delivered].sort()).toEqual(ROWS.map((r) => r.org_number).sort()); + }); + + it("refuses a mode the connector does not serve", async () => { + // JsonlFileConnector declares batch + cdc; it never claims "stream". + await expect(runner.run(baseInput(async () => undefined, { mode: "stream" }))).rejects.toThrow( + /does not serve mode "stream"/, + ); + }); +}); + +describe("P063 breaking schema is caught before the consumer sees anything", () => { + it("fails the run BEFORE the sink is called when the shape breaks the declared policy", async () => { + // v1 registered by a clean run. + await runner.run(baseInput(async () => undefined)); + expect((await new SchemaRegistry(repo).getLatest("src_ing"))?.schema_version).toBe(1); + + // The source now claims a new REQUIRED field — backward-breaking. + let sinkCalls = 0; + const breaking = baseInput(async () => void sinkCalls++, { + source_id: "src_ing", + config: { + file_path: filePath, + partition_key: "companies", + schema: { ...SCHEMA, sni_code: { type: "string", required: true } }, + record_key_field: "org_number", + }, + }); + const err = await runner.run(breaking).catch((e: unknown) => e); + expect(err).toBeInstanceOf(SiaxError); + expect((err as SiaxError).code).toBe("CONFLICT"); + expect((err as SiaxError).message).toContain("refusing to ingest"); + // The whole point: no consumer write happened. + expect(sinkCalls).toBe(0); + }); + + it("catches a record that stopped conforming mid-stream, before emitting the page", async () => { + // A file whose DECLARED schema is unchanged but whose DATA drifted: + // `employees` arrives as a string. A distinct source_id so this run starts + // from a clean checkpoint rather than resuming past the file. + const driftPath = path.join(tmpDir, "drift.jsonl"); + await fs.writeFile( + driftPath, + JSON.stringify({ org_number: "556999-9999", name: "Drift AB", employees: "many" }) + "\n", + "utf8", + ); + let sinkCalls = 0; + const drifting = baseInput(async () => void sinkCalls++, { + source_id: "src_drift", + config: { file_path: driftPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + }); + await expect(runner.run(drifting)).rejects.toThrow(/does not conform to registered schema v1/); + expect(sinkCalls).toBe(0); + }); +}); + +describe("P064 quality enforcement follows the DECLARED policy", () => { + const blockingExpectation = { + expectation_type: "expect_column_values_to_be_between", + kwargs: { column: "employees", min_value: 0, max_value: 50 }, + }; + + it("'block' stops the run and does NOT advance the checkpoint past the bad page", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run( + baseInput(async (b) => void seen.push(...b), { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [blockingExpectation], + on_violation: "block", + }, + }), + ); + expect(report.quality_policy).toBe("block"); + const partition = report.partitions[0]; + expect(partition.stopped_because).toBe("blocked_by_quality"); + // Records 0-4 have employees <= 50; the block fires on the page containing + // employees 60, i.e. page 3 (records 6,7,8 -> 70,80,90 ... employees 60 is + // record index 5 on page 2). Whatever the exact page, nothing from the + // blocked page was emitted and the checkpoint stayed behind it. + expect(partition.envelopes_emitted).toBeLessThan(ROWS.length); + const checkpoint = await repo.getCheckpoint("src_ing", "companies"); + expect(Number(checkpoint?.cursor ?? "0")).toBeLessThan(ROWS.length); + expect(seen.every((e) => Number(e.payload?.employees) <= 50)).toBe(true); + }); + + it("'quarantine' persists the failing records, lets the clean ones through, and advances", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run( + baseInput(async (b) => void seen.push(...b), { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [blockingExpectation], + on_violation: "quarantine", + }, + }), + ); + expect(report.quality_policy).toBe("quarantine"); + const partition = report.partitions[0]; + expect(partition.stopped_because).toBeUndefined(); + expect(partition.quarantined).toBe(5); // employees 60..100 + expect(partition.envelopes_emitted).toBe(5); // employees 10..50 + expect(seen.every((e) => Number(e.payload?.employees) <= 50)).toBe(true); + + const quarantined = await repo.listQuarantine("src_ing"); + expect(quarantined).toHaveLength(5); + expect(quarantined[0].violations[0].expectation_type).toBe("expect_column_values_to_be_between"); + // The full envelope is preserved — a quarantined record is recoverable, + // not just counted. + expect(quarantined[0].envelope.payload).toBeTruthy(); + + const checkpoint = await repo.getCheckpoint("src_ing", "companies"); + expect(checkpoint?.cursor).toBe(String(ROWS.length)); + }); + + it("'warn' lets everything through but still reports the violations", async () => { + const seen: IngestionEnvelope[] = []; + const report = await runner.run( + baseInput(async (b) => void seen.push(...b), { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [blockingExpectation], + on_violation: "warn", + }, + }), + ); + expect(seen).toHaveLength(ROWS.length); + expect(report.partitions[0].quality_violations).toBe(5); + expect(report.partitions[0].quarantined).toBe(0); + }); + + it("no declared policy is reported as 'none_declared', never as a pass", async () => { + const report = await runner.run(baseInput(async () => undefined)); + expect(report.quality_policy).toBe("none_declared"); + }); + + it("an expectation the local engine cannot execute is a violation, not a silent pass", async () => { + const report = await runner.run( + baseInput(async () => undefined, { + quality_policy: { + source_id: "src_ing", + tenant_id: "t_test", + expectations: [{ expectation_type: "expect_column_kl_divergence_to_be_less_than", kwargs: { column: "name" } }], + on_violation: "quarantine", + }, + }), + ); + expect(report.partitions[0].quarantined).toBe(ROWS.length); + const quarantined = await repo.listQuarantine("src_ing"); + expect(quarantined[0].violations[0].detail).toContain("not executable by the local engine"); + }); +}); + +describe("P067 lineage is queryable after the run", () => { + it("answers 'which source fed this output' and 'what did this source produce'", async () => { + const report = await runner.run(baseInput(async () => undefined, { correlation_id: "corr-lineage-1" })); + expect(report.lineage_edges).toBe(1); + + const upstream = await lineage.upstreamOf("t_test", "0pt:dataset:ds_companies"); + expect(upstream).toHaveLength(1); + expect(upstream[0].input_name).toBe("source/src_ing"); + expect(upstream[0].input_ref).toBe("0pt:source:src_ing"); + expect(upstream[0].capability_id).toBe("0pt.ingestion.run.v1"); + expect(upstream[0].run_id).toBe(report.run_id); + expect(upstream[0].correlation_id).toBe("corr-lineage-1"); + + const downstream = await lineage.downstreamOf("t_test", "source/src_ing"); + expect(downstream.map((e) => e.output_name)).toEqual(["0pt:dataset:ds_companies"]); + + // Tenant-scoped: another tenant sees nothing. + expect(await lineage.upstreamOf("t_other", "0pt:dataset:ds_companies")).toEqual([]); + }); + + it("re-emitting the same run id does not duplicate edges (deterministic edge ids)", async () => { + await runner.run(baseInput(async () => undefined, { run_id: "run_fixed" })); + // A second identical run of the same id — e.g. an idempotent retry. + const repo2Runner = new IngestionRunner(repo, new SchemaRegistry(repo), lineage); + await repo2Runner.run(baseInput(async () => undefined, { run_id: "run_fixed" })); + const edges = await lineage.upstreamOf("t_test", "0pt:dataset:ds_companies"); + expect(edges).toHaveLength(1); + }); +}); + +describe("P068 backfill is bounded and idempotent", () => { + it("a repeated identical backfill emits nothing the second time", async () => { + const first: IngestionEnvelope[] = []; + const firstReport = await runner.backfill({ + ...baseInput(async (b) => void first.push(...b)), + range: { partition_key: "companies", from: "0", to: "5" }, + page_size: 100, + }); + expect(firstReport.envelopes_emitted).toBe(5); + expect(firstReport.duplicates_skipped).toBe(0); + expect(first).toHaveLength(5); + + const second: IngestionEnvelope[] = []; + const secondReport = await runner.backfill({ + ...baseInput(async (b) => void second.push(...b)), + range: { partition_key: "companies", from: "0", to: "5" }, + page_size: 100, + }); + expect(secondReport.envelopes_emitted).toBe(0); + expect(secondReport.duplicates_skipped).toBe(5); + expect(second).toHaveLength(0); + }); + + it("a backfill does NOT move the live checkpoint", async () => { + await runner.run(baseInput(async () => undefined)); + const before = await repo.getCheckpoint("src_ing", "companies"); + await runner.backfill({ + ...baseInput(async () => undefined), + range: { partition_key: "companies", from: "0", to: "3" }, + }); + const after = await repo.getCheckpoint("src_ing", "companies"); + expect(after).toEqual(before); + }); + + it("a backfill overlapping a completed run re-delivers nothing (the ledger already has it)", async () => { + await runner.run(baseInput(async () => undefined)); + const emitted: IngestionEnvelope[] = []; + const report = await runner.backfill({ + ...baseInput(async (b) => void emitted.push(...b)), + range: { partition_key: "companies", from: "0", to: "10" }, + page_size: 100, + }); + expect(report.duplicates_skipped).toBe(10); + expect(emitted).toHaveLength(0); + }); + + it("a record whose CONTENT changed is re-emitted — idempotency is not amnesia", async () => { + await runner.run(baseInput(async () => undefined)); + const changedPath = path.join(tmpDir, "changed.jsonl"); + await fs.writeFile( + changedPath, + JSON.stringify({ ...ROWS[0], name: "Bolag 0 AB (namnändrad)" }) + "\n", + "utf8", + ); + const emitted: IngestionEnvelope[] = []; + const report = await runner.backfill({ + ...baseInput(async (b) => void emitted.push(...b), { + config: { file_path: changedPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" }, + }), + range: { partition_key: "companies", from: "0", to: "1" }, + page_size: 100, + }); + expect(report.changed).toBe(1); + expect(report.envelopes_emitted).toBe(1); + expect(emitted[0].payload?.name).toBe("Bolag 0 AB (namnändrad)"); + }); + + it("an unbounded backfill is refused by the connector contract", async () => { + await expect( + runner.backfill({ + ...baseInput(async () => undefined), + range: { partition_key: "companies", from: "0", to: "" }, + }), + ).rejects.toThrow(/unbounded backfill|requires both/); + }); +}); + +describe("envelope hashing", () => { + it("is stable across key order and distinguishes a real content change", () => { + expect(hashPayload({ a: 1, b: 2 })).toBe(hashPayload({ b: 2, a: 1 })); + expect(hashPayload({ a: 1, b: 2 })).not.toBe(hashPayload({ a: 1, b: 3 })); + // null and absent are different data, and must hash differently. + expect(hashPayload({ a: 1, b: null })).not.toBe(hashPayload({ a: 1 })); + expect(hashPayload(null)).toBe("sha256:deleted"); + }); +}); diff --git a/tests/p0/schema-evolution.test.ts b/tests/p0/schema-evolution.test.ts new file mode 100644 index 0000000..e8230cc --- /dev/null +++ b/tests/p0/schema-evolution.test.ts @@ -0,0 +1,149 @@ +/** + * P063 DoD: "a breaking schema is detected BEFORE consumer corruption." + * + * Two levels, both tested here: + * - the pure comparison (checkCompatibility) per policy direction, and + * - the registry refusing to record a version that breaks the PREVIOUS + * version's declared policy — including the bypass where a producer tries + * to widen its own policy to "none" on the breaking version. + * + * The end-to-end "no consumer write happened" proof lives in + * tests/p0/ingestion-runner.test.ts, which asserts the sink was never called. + */ +import { describe, expect, it } from "vitest"; +import { checkCompatibility, conformanceErrors } from "@/src/schema/compatibility"; +import { SchemaRegistry } from "@/src/schema/schema-registry"; +import { InMemoryIngestionRepository } from "@/src/repositories/in-memory-ingestion-repository"; +import { SiaxError } from "@/src/platform/errors"; +import type { RecordSchemaFields } from "@/src/schema/record-schema"; + +const V1: RecordSchemaFields = { + org_number: { type: "string", required: true }, + name: { type: "string", required: true }, + employees: { type: "number", required: false }, +}; + +describe("P063 compatibility rules", () => { + it("adding an OPTIONAL field is backward-compatible", () => { + const next = { ...V1, sni_code: { type: "string" as const, required: false } }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(true); + }); + + it("adding a REQUIRED field breaks backward compatibility", () => { + const next = { ...V1, sni_code: { type: "string" as const, required: true } }; + const verdict = checkCompatibility(V1, next, "backward"); + expect(verdict.compatible).toBe(false); + expect(verdict.breaks[0].field).toBe("sni_code"); + expect(verdict.breaks[0].detail).toContain("absent from every record written under the previous version"); + }); + + it("adding a required field is FORWARD-safe — direction matters", () => { + const next = { ...V1, sni_code: { type: "string" as const, required: true } }; + expect(checkCompatibility(V1, next, "forward").compatible).toBe(true); + expect(checkCompatibility(V1, next, "full").compatible).toBe(false); + }); + + it("removing a required field breaks BOTH directions", () => { + const next: RecordSchemaFields = { org_number: V1.org_number, employees: V1.employees }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(false); + expect(checkCompatibility(V1, next, "forward").compatible).toBe(false); + }); + + it("removing an OPTIONAL field is backward-safe but not forward-safe", () => { + const next: RecordSchemaFields = { org_number: V1.org_number, name: V1.name }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(true); + expect(checkCompatibility(V1, next, "forward").compatible).toBe(true); + }); + + it("a type change breaks every policy except an explicit 'none'", () => { + const next: RecordSchemaFields = { ...V1, employees: { type: "string", required: false } }; + expect(checkCompatibility(V1, next, "backward").compatible).toBe(false); + expect(checkCompatibility(V1, next, "forward").compatible).toBe(false); + expect(checkCompatibility(V1, next, "full").compatible).toBe(false); + expect(checkCompatibility(V1, next, "none").compatible).toBe(true); + }); + + it("conformanceErrors catches a record that silently stopped matching its schema", () => { + expect(conformanceErrors(V1, { org_number: "556001-0001", name: "Alfa AB" })).toEqual([]); + expect(conformanceErrors(V1, { org_number: "556001-0001" })).toContain('required field "name" is missing'); + expect(conformanceErrors(V1, { org_number: "556001-0001", name: "Alfa", employees: "ten" })).toContain( + 'field "employees" is string, expected number', + ); + expect(conformanceErrors(V1, { org_number: "1", name: "A", surprise: true })).toContain( + 'unregistered field "surprise" appeared in the payload', + ); + }); +}); + +describe("P063 schema registry — versioning and refusal", () => { + function registry() { + return new SchemaRegistry(new InMemoryIngestionRepository()); + } + + it("the first registration is v1; an identical re-registration does not create v2", async () => { + const reg = registry(); + const first = await reg.register({ source_id: "src_a", tenant_id: "t", fields: V1, compatibility: "backward" }); + expect(first.schema.schema_version).toBe(1); + const again = await reg.register({ source_id: "src_a", tenant_id: "t", fields: V1, compatibility: "backward" }); + expect(again.unchanged).toBe(true); + expect((await reg.list("src_a")).length).toBe(1); + }); + + it("a compatible change becomes v2", async () => { + const reg = registry(); + await reg.register({ source_id: "src_b", tenant_id: "t", fields: V1, compatibility: "backward" }); + const v2 = await reg.register({ + source_id: "src_b", + tenant_id: "t", + fields: { ...V1, sni_code: { type: "string", required: false } }, + compatibility: "backward", + }); + expect(v2.schema.schema_version).toBe(2); + expect((await reg.getLatest("src_b"))?.schema_version).toBe(2); + }); + + it("a BREAKING change is refused and nothing is persisted", async () => { + const reg = registry(); + await reg.register({ source_id: "src_c", tenant_id: "t", fields: V1, compatibility: "backward" }); + const err = await reg + .register({ + source_id: "src_c", + tenant_id: "t", + fields: { ...V1, sni_code: { type: "string", required: true } }, + compatibility: "backward", + }) + .catch((e: unknown) => e); + expect(err).toBeInstanceOf(SiaxError); + expect((err as SiaxError).code).toBe("CONFLICT"); + expect((err as SiaxError).message).toContain("breaks the declared"); + // Still only v1 on record — a refused registration writes nothing. + expect((await reg.list("src_c")).map((s) => s.schema_version)).toEqual([1]); + }); + + it("a producer cannot widen its OWN policy to 'none' on the breaking version", async () => { + const reg = registry(); + await reg.register({ source_id: "src_d", tenant_id: "t", fields: V1, compatibility: "full" }); + // The PREVIOUS version's policy governs — declaring "none" here must not help. + await expect( + reg.register({ + source_id: "src_d", + tenant_id: "t", + fields: { ...V1, employees: { type: "string", required: false } }, + compatibility: "none", + }), + ).rejects.toThrow(/breaks the declared "full" compatibility policy/); + }); + + it("a source that explicitly declared 'none' opts out — recorded, never assumed", async () => { + const reg = registry(); + await reg.register({ source_id: "src_e", tenant_id: "t", fields: V1, compatibility: "none" }); + const v2 = await reg.register({ + source_id: "src_e", + tenant_id: "t", + fields: { ...V1, employees: { type: "string", required: false } }, + compatibility: "none", + }); + expect(v2.schema.schema_version).toBe(2); + expect(v2.schema.compatibility).toBe("none"); + }); +}); diff --git a/tests/p0/source-activation-gate.test.ts b/tests/p0/source-activation-gate.test.ts new file mode 100644 index 0000000..f6f9b88 --- /dev/null +++ b/tests/p0/source-activation-gate.test.ts @@ -0,0 +1,190 @@ +/** + * P060 DoD #2: "an unclassified source can NOT be activated." + * + * Before this pass the opposite was true, and provably so: + * src/repositories/postgres-source-repository.ts wrote + * `${input.status ?? "active"}` and there was no activation operation at all + * — registering a source with no rights, no provenance and no authorization + * made it `active` on the spot. These tests pin the closed gate AND its + * obvious bypasses (asking for status "active" at registration; a service + * identity authorizing itself; an expired authorization). + */ +import { describe, expect, it } from "vitest"; +import { SourceRegistryService } from "@/src/services/source-registry-service"; +import { SourceService } from "@/src/services/source-service"; +import { InMemorySourceRepository } from "@/src/repositories/in-memory-source-repository"; +import { InMemorySourceRegistryRepository } from "@/src/repositories/in-memory-source-registry-repository"; +import { SiaxError } from "@/src/platform/errors"; + +function newService(): SourceRegistryService { + return new SourceRegistryService({ + repository: new InMemorySourceRegistryRepository(), + sourceService: new SourceService({ repository: new InMemorySourceRepository() }), + }); +} + +const TENANT = "t_test"; +const base = { + source_id: "src_gate_1", + tenant_id: TENANT, + project_id: "p_test", + source_type: "api" as const, + name: "Bolagsverket öppna data", + owner_id: "usr_simon", +}; + +const RIGHTS = { + license_id: "CC-BY-4.0", + allowed_purposes: ["catalog", "enrichment"], + pii_allowed: false, + redistribution: "customer_deliverable" as const, + verified_at: "2026-09-04", + verified_by: "usr_simon", +}; + +const PROVENANCE = { method: "official_api" as const, authority: "Bolagsverket" }; + +const AUTHORIZATION = { + basis: "public_register" as const, + granted_by: "usr_simon", + granted_at: "2026-09-01T08:00:00.000Z", +}; + +describe("P060 source registry — the activation gate", () => { + it("a newly registered source is PENDING, never active", async () => { + const svc = newService(); + const descriptor = await svc.registerSource(base); + expect(descriptor.source.status).toBe("pending"); + }); + + it("asking for status:'active' at registration does NOT bypass the gate", async () => { + const svc = newService(); + // The most obvious bypass: just ask for it. + await expect(svc.registerSource({ ...base, status: "active" })).rejects.toBeInstanceOf(SiaxError); + // ...and the source is still on record, still pending — not half-created. + const source = await svc.sources.getSource(base.source_id); + expect(source.status).toBe("pending"); + }); + + it("activating an unclassified source is denied and names every missing fact", async () => { + const svc = newService(); + await svc.registerSource({ ...base, owner_id: undefined }); + const err = await svc.activateSource(base.source_id, TENANT).catch((e: unknown) => e); + expect(err).toBeInstanceOf(SiaxError); + const siax = err as SiaxError; + expect(siax.code).toBe("AUTHORIZATION_DENIED"); + expect(siax.message).toContain("owner_id is not set"); + expect(siax.message).toContain("no authorization on record"); + expect(siax.message).toContain("no rights record"); + expect(siax.message).toContain("no provenance record"); + expect((await svc.sources.getSource(base.source_id)).status).toBe("pending"); + }); + + it("a fully classified source activates, and describe() reports the classification", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + rights: RIGHTS, + provenance: PROVENANCE, + authorization: AUTHORIZATION, + }); + const classification = await svc.classifySource(base.source_id, TENANT); + expect(classification).toEqual({ source_id: base.source_id, classified: true, missing: [] }); + + const activated = await svc.activateSource(base.source_id, TENANT, "usr_simon"); + expect(activated.status).toBe("active"); + + const descriptor = await svc.describeSource(base.source_id, TENANT); + expect(descriptor.classification?.classified).toBe(true); + expect(descriptor.authorization?.basis).toBe("public_register"); + }); + + it("registering fully classified WITH status:'active' activates through the gate, not around it", async () => { + const svc = newService(); + const descriptor = await svc.registerSource({ + ...base, + status: "active", + rights: RIGHTS, + provenance: PROVENANCE, + authorization: AUTHORIZATION, + }); + expect(descriptor.source.status).toBe("active"); + }); + + it("a service identity cannot authorize a source (no self-granted rubber stamp)", async () => { + const svc = newService(); + await svc.registerSource(base); + await expect( + svc.setAuthorization({ + source_id: base.source_id, + tenant_id: TENANT, + basis: "internal", + granted_by: "svc_0pt", + granted_at: "2026-09-01T08:00:00.000Z", + }), + ).rejects.toThrow(/accountable actor/); + }); + + it("an EXPIRED authorization does not classify the source", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + rights: RIGHTS, + provenance: PROVENANCE, + authorization: { ...AUTHORIZATION, expires_at: "2026-01-01T00:00:00.000Z" }, + }); + const classification = await svc.classifySource(base.source_id, TENANT); + expect(classification.classified).toBe(false); + expect(classification.missing.join(" ")).toContain("expired"); + await expect(svc.activateSource(base.source_id, TENANT)).rejects.toThrow(/not classified/); + }); + + it("rights with an empty purpose list do not classify (purpose limitation is not optional)", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + rights: { ...RIGHTS, allowed_purposes: [] }, + provenance: PROVENANCE, + authorization: AUTHORIZATION, + }); + const classification = await svc.classifySource(base.source_id, TENANT); + expect(classification.classified).toBe(false); + expect(classification.missing.join(" ")).toContain("allowed_purposes is empty"); + }); + + it("deactivating is never gated — a source can always be taken out of service", async () => { + const svc = newService(); + await svc.registerSource({ ...base, rights: RIGHTS, provenance: PROVENANCE, authorization: AUTHORIZATION }); + await svc.activateSource(base.source_id, TENANT); + const paused = await svc.deactivateSource(base.source_id, TENANT, "paused"); + expect(paused.status).toBe("paused"); + }); + + it("the gate is tenant-scoped: another tenant cannot activate this source", async () => { + const svc = newService(); + await svc.registerSource({ ...base, rights: RIGHTS, provenance: PROVENANCE, authorization: AUTHORIZATION }); + await expect(svc.activateSource(base.source_id, "t_attacker")).rejects.toThrow(/not found/); + expect((await svc.sources.getSource(base.source_id)).status).toBe("pending"); + }); + + it("schedule is recorded as a registry field and validated", async () => { + const svc = newService(); + await svc.registerSource({ + ...base, + schedule: { mode: "batch", interval_seconds: 3600, enabled: true }, + }); + const schedule = await svc.getSchedule(base.source_id, TENANT); + expect(schedule?.mode).toBe("batch"); + expect(schedule?.interval_seconds).toBe(3600); + await expect( + svc.setSchedule({ + source_id: base.source_id, + tenant_id: TENANT, + mode: "stream", + interval_seconds: 0, + enabled: true, + updated_at: new Date().toISOString(), + }), + ).rejects.toThrow(/interval_seconds must be positive/); + }); +});