P060-P068: durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill #31
@@ -82,3 +82,107 @@ accurate.
|
||||
5. 0pt-motor executors for the delegated jobs
|
||||
(`0pt.quality.ge_run`, `0pt.features.*`, `0pt.er.batch_resolution`).
|
||||
6. Observability: no metrics/traces emitted by the new services (§36).
|
||||
|
||||
---
|
||||
|
||||
## P060-P068 ingestion plane (added 2026-09-04)
|
||||
|
||||
Classification scale unchanged. Every row below states what was actually
|
||||
verified, and by which command.
|
||||
|
||||
| Requirement | BEFORE this pass | Status now | Evidence |
|
||||
|---|---|---|---|
|
||||
| **P060** Source Registry: durable canonical registry; restart preserves sources; an unclassified source cannot be activated | **PARTIAL / BROKEN.** The registry existed with Postgres persistence (`sql/002_source_registry.sql`, `src/repositories/postgres-source-registry-repository.ts`) and carried source_id/tenant/project/type/status/license/purpose/rights. It had NO authorization, NO schedule, NO checkpoint, NO schema, and no `owner` distinct from `created_by`. The activation DoD was inverted: `postgres-source-repository.ts:70` wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered, and no activation operation existed at all | **INTEGRATION_TESTED** | `sql/007_ingestion_plane.sql` (authorization/schedule/schemas/checkpoints + `ALTER … SET DEFAULT 'pending'` + `owner_id`); `SourceRegistryService.classifySource`/`activateSource`; `tests/p0/source-activation-gate.test.ts` (11); `tests/api/0pt-p060-routes.test.ts` (9); `tests/p0/ingestion-plane-postgres.realinfra.test.ts` against a live Postgres 16 |
|
||||
| **P061** Connector contract: discover/validate/connect/checkpoint/read/resume/backfill/health/disconnect; two heterogeneous connectors pass one conformance suite | **MISSING.** `services/api/sources/connectors.py` had per-source `fetch_*()` functions only — no lifecycle, no cursor, no resume, no health, no disconnect, no conformance suite | **INTEGRATION_TESTED** | `src/connectors/connector.ts` (all nine operations); `src/connectors/conformance.ts` (15 checks); `HttpJsonConnector` (real node:http server, opaque cursor) and `JsonlFileConnector` (real file, line-offset cursor, CDC deletes) both pass the identical suite — `tests/p0/connector-conformance.test.ts`, which also proves the suite FAILS a fail-open connector |
|
||||
| **P062** CDC/batch/stream normalized to one envelope; restart resumes from a durable checkpoint without silent gaps | **MISSING**, and documented as such: `services/api/tests/test_known_gaps.py:10-19` skipped the CDC case because no CDC pipeline existed anywhere, and `workers/pipeline.py` had no persistence at all | **INTEGRATION_TESTED** | `src/types/ingestion.ts` (one envelope for all three modes); `src/ingestion/ingestion-runner.ts`; `tests/p0/ingestion-runner.test.ts` kills a run inside the sink and proves the resumed run covers all 10 records exactly once; same proof against real Postgres across fresh processes |
|
||||
| **P063** Schema versioning + compatibility policy; a breaking schema is detected before consumer corruption | **MISSING.** `schema_version` existed as a free-form string on datasets/Iceberg writes; there was no comparison, no policy, no versioned source schema | **INTEGRATION_TESTED** | `src/schema/compatibility.ts`, `src/schema/schema-registry.ts`; `tests/p0/schema-evolution.test.ts` (12); the runner's schema gate runs before the first read — `tests/p0/ingestion-runner.test.ts` asserts `sinkCalls === 0` on a break |
|
||||
| **P064** Quality rules behind the SIAX abstraction; failures block or quarantine per declared policy | **PARTIAL.** `src/quality/*` had rules, runs, results, issues, scores and a GE-compatible engine, but a failing rule only opened an ISSUE — the data flowed on regardless. `grep -ri quarantin src/` found one unrelated hit | **INTEGRATION_TESTED** | `src/quality/quality-policy.ts` composes the EXISTING `ExpectationEngine` (not a second engine) and adds the consequence; block / quarantine / warn each proved in `tests/p0/ingestion-runner.test.ts`; quarantine durability proved against real Postgres |
|
||||
| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **DUPLICATE (confirmed).** `src/er/entity-resolution-service.ts` (canonical: hard rules, review band, Splink delegation) AND `services/api/workers/resolve.py` (simplified fuzzy threshold, auto-merge) — the latter still called by `workers/pipeline.py:110` | **NOT DONE — documented, not built.** The duplication is now stated at BOTH sites in code with the removal plan. The new ingestion plane deliberately performs no ER, so it adds no third engine. See "P065 — why it was not closed" below |
|
||||
| **P066** Iceberg | out of scope — ST0RE is owned by another workflow | **NOT TOUCHED** | Contract read only (`src/services/iceberg-write-service.ts`, `src/clients/st0re-lakehouse-client.ts`) |
|
||||
| **P067** Lineage: OpenLineage/AUD0 events; source → output lineage is queryable | **PARTIAL.** `src/lineage/lineage-emitter.ts` emitted OpenLineage to AUD0, but nothing in this repo could ANSWER a lineage question — no store, no query, no route | **INTEGRATION_TESTED** | `src/lineage/lineage-edge.ts`, `src/lineage/lineage-store.ts` (local queryable projection; AUD0 stays the system of record), `GET /api/0pt/lineage`; `tests/p0/ingestion-runner.test.ts` queries both directions and proves tenant scoping |
|
||||
| **P068** Durable bounded backfill/replay; repeated backfill is idempotent | **MISSING**, documented as such: `services/api/tests/test_known_gaps.py:44-52` skipped it because "no delivery-store persistence layer exists to dedupe a backfill against" | **INTEGRATION_TESTED** | `opt_ingestion_ledger` + `IngestionRunner.backfill`; `tests/p0/ingestion-runner.test.ts` (repeat emits 0, live checkpoint untouched, a real content change still re-emits); same against real Postgres |
|
||||
|
||||
### What was actually run (not "should pass")
|
||||
|
||||
- `pnpm run typecheck` — clean.
|
||||
- `pnpm run test` — **365 passed, 25 skipped** (was 311/18 before this pass).
|
||||
- `pnpm run build` — clean; the four new routes appear in the route table.
|
||||
- `pnpm run lint` — 12 errors / 5 warnings, **all pre-existing** (identical
|
||||
count on the base commit; advisory in CI). This pass adds none.
|
||||
- Real Postgres 16: `tests/p0/ingestion-plane-postgres.realinfra.test.ts`
|
||||
with `INGESTION_TEST_DATABASE_URL` — 7 passed.
|
||||
- **Real container restart**, run manually against that same database:
|
||||
`docker restart` the Postgres container, then re-read through the real
|
||||
repositories. Before and after were identical — 2 sources (one `active`
|
||||
with its owner, one `pending`), 1 checkpoint (cursor 6, 6 committed),
|
||||
6 ledger rows, 1 lineage edge, 4 quarantined records, 1 schema version.
|
||||
That is P060's "a restart preserves all sources", measured rather than
|
||||
asserted.
|
||||
|
||||
### A real bug this pass wrote and then caught
|
||||
|
||||
The first draft of `IngestionRepository` combined the idempotency check and
|
||||
its recording into one atomic call, made BEFORE the sink. The crash/resume
|
||||
test then delivered **7 of 10 records**: the page in flight when the sink
|
||||
died had already been marked delivered, so the resumed run skipped it as a
|
||||
duplicate — permanent, silent data loss. The fix is the `peekDeliveries` /
|
||||
`commitDeliveries` split, with both the ledger commit and the checkpoint
|
||||
commit after the sink resolves. Recorded here because the failure mode is
|
||||
the interesting artifact, not the fix.
|
||||
|
||||
### Honest gaps in the new plane (not hidden)
|
||||
|
||||
1. **Delivery is AT-LEAST-ONCE, not exactly-once.** Two concurrent runs on
|
||||
the same (source, partition) can both peek "new" and both emit.
|
||||
`envelope_id` is deterministic so a dedup-capable sink survives it. There
|
||||
is no per-(source, partition) run lock. Do not describe this plane as
|
||||
exactly-once.
|
||||
2. **No CDC *source* exists.** The envelope, the operations and the runner
|
||||
handle change events, and `JsonlFileConnector` emits real deletes from an
|
||||
`_op` field — but this repo still has no Debezium/WAL/replication-slot
|
||||
consumer. P062's contract is met; a CDC connector is still MISSING.
|
||||
3. **0pt does not schedule itself.** `opt_source_schedule` is a declaration
|
||||
ACT0 would read; there is no cron daemon here.
|
||||
4. **The sink is caller-supplied.** The runner stops at the sink; the
|
||||
lakehouse write boundary is ST0RE's (P066, another workflow's repo).
|
||||
5. **Nothing here is deployed.** See the deployment note below.
|
||||
6. **`QualityService`'s own rule/run/issue registry is still in-process
|
||||
Maps** (`src/quality/quality-service.ts:47-50`). The new *enforcement*
|
||||
path is durable; the older rule registry is not. Unchanged by this pass.
|
||||
7. **Python tests still run nowhere.** `grep -rn "pytest" .gitea/workflows
|
||||
.github/workflows` finds nothing — `services/api/tests/*` (9 files) has
|
||||
never been executed by CI. Flagged, not fixed (it needs a workflow change
|
||||
and a Python toolchain decision).
|
||||
|
||||
### Deployment reality, measured 2026-09-04
|
||||
|
||||
`https://0pt.app/api/health` answers **200** from **Vercel** with
|
||||
`{"db":"not_configured"}`, and `https://0pt.app/api/0pt/sources` answers
|
||||
**404** — the deployed build predates the entire `app/api/0pt/*` capability
|
||||
layer. `https://api.0pt.app/health` answers **404** (the FastAPI engine is
|
||||
not reachable), and `.gitea/workflows/cd.yml` still guards on
|
||||
`COOLIFY_APP_ID = TODO-no-coolify-app-provisioned-yet`. So: none of the
|
||||
capability layer — neither this pass's work nor P010/P014's — is running in
|
||||
production, and P060's durability guarantee cannot be exercised there,
|
||||
because production has no database configured and does not serve the routes.
|
||||
Verified with `curl`; not inferred.
|
||||
|
||||
### P065 — why it was not closed
|
||||
|
||||
Closing P065 means turning `services/api/workers/resolve.py` into an adapter
|
||||
over the canonical `src/er/entity-resolution-service.ts`. That change cannot
|
||||
be verified from this repo today: `services/api` is not built by this repo's
|
||||
Dockerfile (it builds only the Next.js app), `api.0pt.app` answers 404, and
|
||||
no CI workflow runs its pytest suite. A blind rewrite would ship untested
|
||||
into a path nobody can exercise — the exact failure mode this programme
|
||||
exists to stop.
|
||||
|
||||
The removal plan, for whoever picks it up:
|
||||
1. Expose the canonical ER capability as `POST /api/0pt/er/candidates`
|
||||
(service already exists; only a route + Zod schema are missing).
|
||||
2. Add a Python CI job that actually runs `services/api/tests`.
|
||||
3. Replace `resolve.py`'s body with an HTTP adapter over that route,
|
||||
**fail-closed** when it is unreachable — never falling back to the local
|
||||
scorer, which would preserve the duplicate under a different name.
|
||||
4. Port `test_resolve_determinism.py` to assert the adapter's contract, and
|
||||
delete the fuzzy scorer.
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* app/api/0pt/lineage/route.ts
|
||||
* =============================
|
||||
* P067 DoD: "source -> output lineage is QUERYABLE". This is the query.
|
||||
*
|
||||
* GET /api/0pt/lineage?tenant_id=…&output_name=… -> what fed this output
|
||||
* GET /api/0pt/lineage?tenant_id=…&input_name=… -> what this input produced
|
||||
*
|
||||
* The answer comes from 0pt's local projection of the OpenLineage events it
|
||||
* emits to AUD0 (src/lineage/lineage-store.ts). AUD0 remains the system of
|
||||
* record; this endpoint exists because "emitted to AUD0" was never the same
|
||||
* thing as "answerable from 0pt", and before this route nothing in this repo
|
||||
* could answer either question.
|
||||
*/
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { lineageStore } from "@/src/services/source-registry";
|
||||
import { siaxErrorResponse } from "@/src/http/siax-error-response";
|
||||
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
|
||||
|
||||
export async function GET(request: NextRequest) {
|
||||
try {
|
||||
const { searchParams } = new URL(request.url);
|
||||
const tenantId = searchParams.get("tenant_id");
|
||||
const outputName = searchParams.get("output_name") ?? undefined;
|
||||
const inputName = searchParams.get("input_name") ?? undefined;
|
||||
if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 });
|
||||
if (!outputName && !inputName) {
|
||||
return NextResponse.json(
|
||||
{ error: "one of output_name or input_name is required — an unfiltered lineage dump is not a query" },
|
||||
{ status: 400 },
|
||||
);
|
||||
}
|
||||
const guard = assertTenantMatchesVerified(request, tenantId);
|
||||
if (guard) return guard;
|
||||
const limitRaw = searchParams.get("limit");
|
||||
const limit = limitRaw ? Math.min(Math.max(Number(limitRaw), 1), 500) : undefined;
|
||||
const edges = await lineageStore.query({
|
||||
tenant_id: tenantId,
|
||||
output_name: outputName,
|
||||
input_name: inputName,
|
||||
limit,
|
||||
});
|
||||
return NextResponse.json({ edges, count: edges.length });
|
||||
} catch (error: unknown) {
|
||||
return siaxErrorResponse(error);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* app/api/0pt/sources/[sourceId]/activate/route.ts
|
||||
* ==================================================
|
||||
* P060 activation gate over HTTP. POST activates a source ONLY when it is
|
||||
* classified (owner + unexpired authorization + rights + provenance);
|
||||
* otherwise it returns 403 AUTHORIZATION_DENIED naming every missing fact.
|
||||
* GET returns the classification without changing anything, so an operator
|
||||
* can see what is missing before trying.
|
||||
*/
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { activateSourceSchema, parseBody } from "@/src/validation/0pt-schemas";
|
||||
import { sourceRegistryService } from "@/src/services/source-registry";
|
||||
import { siaxErrorResponse } from "@/src/http/siax-error-response";
|
||||
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
|
||||
|
||||
export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) {
|
||||
try {
|
||||
const { sourceId } = await params;
|
||||
const tenantId = new URL(request.url).searchParams.get("tenant_id");
|
||||
if (!tenantId) {
|
||||
return NextResponse.json({ error: "tenant_id is required" }, { status: 400 });
|
||||
}
|
||||
const guard = assertTenantMatchesVerified(request, tenantId);
|
||||
if (guard) return guard;
|
||||
return NextResponse.json(await sourceRegistryService.classifySource(sourceId, tenantId));
|
||||
} catch (error: unknown) {
|
||||
return siaxErrorResponse(error);
|
||||
}
|
||||
}
|
||||
|
||||
export async function POST(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) {
|
||||
try {
|
||||
const { sourceId } = await params;
|
||||
const parsed = parseBody(activateSourceSchema, await request.json());
|
||||
if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status });
|
||||
const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id);
|
||||
if (guard) return guard;
|
||||
const source = await sourceRegistryService.activateSource(
|
||||
sourceId,
|
||||
parsed.data.tenant_id,
|
||||
parsed.data.actor_id,
|
||||
);
|
||||
return NextResponse.json(source);
|
||||
} catch (error: unknown) {
|
||||
return siaxErrorResponse(error);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
/**
|
||||
* app/api/0pt/sources/[sourceId]/authorization/route.ts
|
||||
* ======================================================
|
||||
* P060 "authorization": the recorded decision that 0pt may connect to this
|
||||
* source at all — separate from rights (what the DATA may be used for). PUT
|
||||
* is an upsert; a service identity as `granted_by` is rejected (400), because
|
||||
* a source may not authorize itself.
|
||||
*/
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { parseBody, setSourceAuthorizationSchema } from "@/src/validation/0pt-schemas";
|
||||
import { sourceRegistryService } from "@/src/services/source-registry";
|
||||
import { siaxErrorResponse } from "@/src/http/siax-error-response";
|
||||
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
|
||||
|
||||
export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) {
|
||||
try {
|
||||
const { sourceId } = await params;
|
||||
const tenantId = new URL(request.url).searchParams.get("tenant_id");
|
||||
if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 });
|
||||
const guard = assertTenantMatchesVerified(request, tenantId);
|
||||
if (guard) return guard;
|
||||
const auth = await sourceRegistryService.getAuthorization(sourceId, tenantId);
|
||||
if (!auth) {
|
||||
return NextResponse.json({ error: `source ${sourceId} has no authorization record` }, { status: 404 });
|
||||
}
|
||||
return NextResponse.json(auth);
|
||||
} catch (error: unknown) {
|
||||
return siaxErrorResponse(error);
|
||||
}
|
||||
}
|
||||
|
||||
export async function PUT(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) {
|
||||
try {
|
||||
const { sourceId } = await params;
|
||||
const parsed = parseBody(setSourceAuthorizationSchema, await request.json());
|
||||
if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status });
|
||||
const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id);
|
||||
if (guard) return guard;
|
||||
return NextResponse.json(
|
||||
await sourceRegistryService.setAuthorization({ ...parsed.data, source_id: sourceId }),
|
||||
);
|
||||
} catch (error: unknown) {
|
||||
return siaxErrorResponse(error);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
/**
|
||||
* app/api/0pt/sources/[sourceId]/schedule/route.ts
|
||||
* =================================================
|
||||
* P060 "schedule": ingestion mode + cadence, recorded on the source. 0pt does
|
||||
* NOT run cron itself — ACT0 owns scheduling; this row is the declaration a
|
||||
* scheduler reads, and saying otherwise would be a claim this repo cannot back.
|
||||
*/
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { parseBody, setSourceScheduleSchema } from "@/src/validation/0pt-schemas";
|
||||
import { sourceRegistryService } from "@/src/services/source-registry";
|
||||
import { siaxErrorResponse } from "@/src/http/siax-error-response";
|
||||
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
|
||||
|
||||
export async function GET(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) {
|
||||
try {
|
||||
const { sourceId } = await params;
|
||||
const tenantId = new URL(request.url).searchParams.get("tenant_id");
|
||||
if (!tenantId) return NextResponse.json({ error: "tenant_id is required" }, { status: 400 });
|
||||
const guard = assertTenantMatchesVerified(request, tenantId);
|
||||
if (guard) return guard;
|
||||
const schedule = await sourceRegistryService.getSchedule(sourceId, tenantId);
|
||||
if (!schedule) {
|
||||
return NextResponse.json({ error: `source ${sourceId} has no schedule record` }, { status: 404 });
|
||||
}
|
||||
return NextResponse.json(schedule);
|
||||
} catch (error: unknown) {
|
||||
return siaxErrorResponse(error);
|
||||
}
|
||||
}
|
||||
|
||||
export async function PUT(request: NextRequest, { params }: { params: Promise<{ sourceId: string }> }) {
|
||||
try {
|
||||
const { sourceId } = await params;
|
||||
const parsed = parseBody(setSourceScheduleSchema, await request.json());
|
||||
if (!parsed.ok) return NextResponse.json(parsed.body, { status: parsed.status });
|
||||
const guard = assertTenantMatchesVerified(request, parsed.data.tenant_id);
|
||||
if (guard) return guard;
|
||||
return NextResponse.json(
|
||||
await sourceRegistryService.setSchedule({
|
||||
...parsed.data,
|
||||
source_id: sourceId,
|
||||
updated_at: new Date().toISOString(),
|
||||
}),
|
||||
);
|
||||
} catch (error: unknown) {
|
||||
return siaxErrorResponse(error);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
# 0pt ingestion plane (P060–P068)
|
||||
|
||||
> Written 2026-09-04. Every claim here is backed by a test path or a
|
||||
> recorded command; anything not verified says so. Read `STATUS.md`'s
|
||||
> "P060-P068 ingestion plane" section for the classification table and the
|
||||
> honest-gaps list — this document is the mechanism, that one is the status.
|
||||
|
||||
## The shape
|
||||
|
||||
```
|
||||
Source Registry Connector Ingestion Runner
|
||||
(P060: who, what, (P061: nine (P062: one envelope,
|
||||
under what rights, operations, one one durable checkpoint)
|
||||
on what schedule) conformance suite)
|
||||
│ │ │
|
||||
│ classification gate │ discover / validate │ read ─┐
|
||||
│ blocks activation │ connect / read │ │
|
||||
▼ ▼ resume / backfill ▼ ▼
|
||||
opt_sources records ───────────────▶ IngestionEnvelope
|
||||
opt_source_rights │
|
||||
opt_source_authorization ├─▶ P063 schema gate
|
||||
opt_source_schedule ├─▶ P064 quality gate
|
||||
opt_source_schemas ├─▶ P068 ledger (peek)
|
||||
opt_ingestion_checkpoints ├─▶ SINK (caller-supplied)
|
||||
├─▶ P068 ledger (commit)
|
||||
├─▶ checkpoint (commit)
|
||||
└─▶ P067 lineage edge
|
||||
```
|
||||
|
||||
## P060 — the registry, and the gate
|
||||
|
||||
A source now carries: `source_id`, `tenant_id`, `project_id`, `source_type`,
|
||||
`owner_id`, an **authorization** record, a **license/purpose rights** record,
|
||||
**provenance** (how the data is obtained), a **schedule**, a versioned
|
||||
**schema**, a durable **checkpoint**, and a `status`.
|
||||
|
||||
`status` starts at **`pending`**. It was `active` before — both repository
|
||||
implementations wrote `input.status ?? "active"` — which meant an
|
||||
unclassified source went live the moment it was registered. `activateSource`
|
||||
is now the only path to `active`, and it refuses unless ALL of these are on
|
||||
record:
|
||||
|
||||
- an `owner_id` (distinct from `created_by`: who is accountable, not who typed it),
|
||||
- an **unexpired** authorization granted by an accountable actor
|
||||
(`svc_*`/`system`/`automation` are rejected — a source may not authorize itself),
|
||||
- rights with a `license_id` **and** at least one `allowed_purpose`,
|
||||
- provenance with a sourcing `method`.
|
||||
|
||||
Schedule, schema and checkpoint describe HOW a source runs, not WHETHER it
|
||||
may, so they are deliberately not part of the gate.
|
||||
|
||||
The default is enforced in two places on purpose: in the repositories, and as
|
||||
`ALTER TABLE opt_sources ALTER COLUMN status SET DEFAULT 'pending'`, so a raw
|
||||
`INSERT` from psql cannot produce an unclassified-but-active source either
|
||||
(proved in the realinfra suite).
|
||||
|
||||
Routes: `GET/POST /api/0pt/sources/{id}/activate` (GET returns the
|
||||
classification without changing anything), `GET/PUT …/authorization`,
|
||||
`GET/PUT …/schedule`.
|
||||
|
||||
## P061 — the connector contract
|
||||
|
||||
Nine operations on one interface: `validate` (pure, no IO, fail-closed),
|
||||
`connect` (idempotent), `discover`, `health` (reports, never throws), `read`
|
||||
(honours `limit` exactly), `checkpoint`, `resume`, `backfill` (bounded — an
|
||||
open-ended range is refused), `disconnect` (idempotent, safe after an error).
|
||||
|
||||
`src/connectors/conformance.ts` is a plain async function, not a test file,
|
||||
so the same suite runs from tests, from an operator script against a real
|
||||
configured source, and (later) from a pre-activation check. There is no
|
||||
"skip" verdict: a connector that cannot honestly serve a check FAILS it.
|
||||
|
||||
Two heterogeneous connectors pass it:
|
||||
|
||||
| | `HttpJsonConnector` | `JsonlFileConnector` |
|
||||
|---|---|---|
|
||||
| transport | network, real HTTP | local filesystem |
|
||||
| cursor | opaque server token, with a `cursor_field` fallback | line offset it computes itself |
|
||||
| exhaustion | server-reported | exact (the file is finite) |
|
||||
| health | HTTP round trip | `stat` |
|
||||
| backfill bounds | server-defined (sequence/timestamp) | line numbers |
|
||||
| CDC deletes | no — it does not claim `cdc` | yes, via `_op` |
|
||||
|
||||
The `cursor_field` fallback exists because the conformance suite caught a
|
||||
real defect: an API that stops returning `next_cursor` at the end of a page
|
||||
left the connector with nothing to resume from, so `resume()` silently
|
||||
restarted at the beginning.
|
||||
|
||||
## P062 — one envelope, one durable checkpoint
|
||||
|
||||
`IngestionEnvelope` is the single shape everything downstream sees:
|
||||
`mode` (batch/stream/cdc), `operation` (upsert/insert/update/delete),
|
||||
`partition_key`, `record_key`, `schema_version`, a monotonic `sequence`,
|
||||
`occurred_at`/`ingested_at`, `payload` (null only for a delete) and
|
||||
`provenance` (connector id + version + run id + cursor).
|
||||
|
||||
**Ordering is the contract.** Per page:
|
||||
|
||||
```
|
||||
read/resume → normalize → schema conformance → quality gate
|
||||
→ ledger PEEK → SINK → ledger COMMIT → checkpoint COMMIT
|
||||
```
|
||||
|
||||
Both commits happen after the sink resolves. The two tempting shortcuts are
|
||||
both data-loss bugs, and one of them was written here first — see STATUS.md's
|
||||
"A real bug this pass wrote and then caught".
|
||||
|
||||
The resulting guarantee is **at-least-once**, not exactly-once.
|
||||
`envelope_id` is a deterministic hash of
|
||||
`(source, partition, record_key, content)` so a dedup-capable sink survives a
|
||||
re-delivery.
|
||||
|
||||
## P063 — schema evolution
|
||||
|
||||
`RecordSchema` is a small structural shape (field → type + required) plus a
|
||||
`CompatibilityPolicy` (`backward` / `forward` / `full` / `none`).
|
||||
`checkCompatibility` answers what breaks, in which direction, and why. It is
|
||||
deliberately not a second JSON-Schema dialect: the question is "is version
|
||||
N+1 of an UPSTREAM shape safe for consumers of version N?", which needs a
|
||||
comparable structure, not a validator.
|
||||
|
||||
Two enforcement points, both upstream of any consumer write:
|
||||
|
||||
1. the runner's **discovery gate** — a shape change that breaks the previous
|
||||
version's declared policy fails the whole run before the first `read()`;
|
||||
2. **per-record conformance** — a source that drifted without redeclaring its
|
||||
schema is caught on the offending record, before the page reaches the sink.
|
||||
|
||||
A producer cannot widen its own policy: the PREVIOUS version's policy governs
|
||||
whether a change is allowed.
|
||||
|
||||
## P064 — quality with consequences
|
||||
|
||||
`QualityGate` composes the existing `ExpectationEngine` (the Great
|
||||
Expectations core subset already in this repo) and adds the missing half: a
|
||||
declared `on_violation` action.
|
||||
|
||||
- `block` — the page fails, and the checkpoint does **not** advance, so the
|
||||
bad page stays unconsumed rather than being stepped over.
|
||||
- `quarantine` — failing envelopes are persisted whole to
|
||||
`opt_quality_quarantine` (recoverable, not merely counted); clean ones
|
||||
continue and the checkpoint advances.
|
||||
- `warn` — everything continues, violations are counted and reported.
|
||||
|
||||
There is no default. With no policy the run report says
|
||||
`quality_policy: "none_declared"` — an undeclared policy must never read as a
|
||||
passed one. An expectation the local engine cannot execute counts as a
|
||||
VIOLATION, never a silent pass.
|
||||
|
||||
## P067 — queryable lineage
|
||||
|
||||
`LineageEmitter` keeps emitting OpenLineage to AUD0, and **AUD0 remains the
|
||||
system of record**. `LineageStore` adds 0pt's own flattened projection — one
|
||||
row per (run, input, output) with a deterministic `edge_id` — because
|
||||
"emitted to AUD0" was never the same thing as "answerable from 0pt".
|
||||
|
||||
```
|
||||
GET /api/0pt/lineage?tenant_id=…&output_name=0pt:dataset:ds_x # what fed it
|
||||
GET /api/0pt/lineage?tenant_id=…&input_name=source/src_x # what it produced
|
||||
```
|
||||
|
||||
An unfiltered dump is refused (400): that is an export, not a query.
|
||||
|
||||
## P068 — bounded, idempotent backfill
|
||||
|
||||
`opt_ingestion_ledger` holds one row per `(source_id, record_key)` with the
|
||||
last accepted content hash. A backfill runs through the same envelope,
|
||||
schema, quality and ledger path as a normal run, and:
|
||||
|
||||
- **never touches the live checkpoint** — a backfill must not rewind or
|
||||
advance streaming ingestion;
|
||||
- **re-emits nothing** on a repeat — every record comes back `duplicate`;
|
||||
- **still re-emits a genuine content change** — idempotency is not amnesia;
|
||||
- **refuses an unbounded range** at the connector contract level.
|
||||
|
||||
## Running the real-infrastructure proof
|
||||
|
||||
```bash
|
||||
docker run -d --name opt-ingestion-pgtest \
|
||||
-e POSTGRES_USER=opt -e POSTGRES_PASSWORD=opt_test_pw -e POSTGRES_DB=opt_test \
|
||||
-p 15434:5432 postgres:16-alpine
|
||||
|
||||
INGESTION_TEST_DATABASE_URL=postgres://opt:opt_test_pw@localhost:15434/opt_test \
|
||||
pnpm vitest run tests/p0/ingestion-plane-postgres.realinfra.test.ts
|
||||
```
|
||||
|
||||
The suite applies `sql/002_source_registry.sql` and
|
||||
`sql/007_ingestion_plane.sql` itself (both are idempotent). It skips
|
||||
honestly, never fabricating success, when the env var is unset.
|
||||
@@ -3,6 +3,44 @@ Entity resolution (docs/06): block, score, and link records across sources so th
|
||||
same company from GLEIF + Brreg + CVR collapses to one record with corroborated
|
||||
confidence. rapidfuzz only; deterministic; no external calls.
|
||||
|
||||
=======================================================================
|
||||
P065 — THIS IS THE SECOND ER ENGINE. IT IS NOT THE CANONICAL ONE.
|
||||
=======================================================================
|
||||
0pt has TWO entity-resolution implementations, and this is the simplified
|
||||
one:
|
||||
|
||||
* CANONICAL: src/er/entity-resolution-service.ts — the declared generic ER
|
||||
capability. Deterministic blocking + an explainable per-pair score,
|
||||
product-supplied HARD RULES that override the score in both directions,
|
||||
a configurable review band that hands ambiguous pairs to humans instead
|
||||
of auto-merging, and delegation of heavy probabilistic linkage (Splink)
|
||||
to 0pt-motor via the N0D job "0pt.er.batch_resolution".
|
||||
|
||||
* THIS FILE: blocking on (market, first initial) + a single
|
||||
rapidfuzz token_sort_ratio threshold, first-match-wins, auto-merge with
|
||||
no review band, no hard-rule hook, no explanation beyond the score.
|
||||
|
||||
P065's definition of done is that no second simplified production engine
|
||||
remains. It has NOT been met: this file is still what
|
||||
`workers/pipeline.py` calls (line ~110, `deduped = resolve(records)`).
|
||||
|
||||
WHY IT WAS NOT REPLACED IN THE P060-P068 PASS, stated plainly rather than
|
||||
quietly deferred: replacing it means turning this module into an ADAPTER
|
||||
that calls the canonical capability, and that change cannot be verified
|
||||
from this repo today —
|
||||
* `services/api` is not deployed by this repo's Dockerfile (which builds
|
||||
only the Next.js app), and api.0pt.app answers 404, so there is no
|
||||
running engine to test against;
|
||||
* no CI workflow runs services/api's pytest suite at all (verified:
|
||||
`grep -rn "pytest" .gitea/workflows .github/workflows` finds nothing),
|
||||
so a rewrite here would land with zero automated coverage.
|
||||
Rewriting it blind and calling P065 done would be exactly the
|
||||
"claimed complete, broken in operation" failure this programme exists to
|
||||
stop. The removal plan is recorded in STATUS.md.
|
||||
|
||||
Until then: do NOT add capability to this file. Any new ER behaviour
|
||||
belongs in the canonical TypeScript service.
|
||||
|
||||
Deterministic by construction: for a fixed `records` input, blocking is a pure
|
||||
function of (market, name-first-letter) and matching is first-match-wins
|
||||
within a block, so results do not depend on which source answered first — see
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
-- 0pt ingestion plane (P060-P068).
|
||||
--
|
||||
-- Adds the durable state the Source Registry and the ingestion runner need
|
||||
-- and that sql/002_source_registry.sql did NOT carry:
|
||||
--
|
||||
-- * opt_source_authorization -- P060 "authorization": the recorded decision
|
||||
-- that this tenant may use this source at all. Separate from rights
|
||||
-- (license/purpose) on purpose: rights say WHAT the data may be used for,
|
||||
-- authorization says WHO signed off that 0pt may connect. Both are
|
||||
-- required to activate.
|
||||
-- * opt_source_schedule -- P060 "schedule": ingestion mode + cadence.
|
||||
-- * opt_source_schemas -- P060 "schema" + P063 schema evolution:
|
||||
-- one row per (source_id, schema_version), plus the compatibility policy
|
||||
-- that governs the NEXT version.
|
||||
-- * opt_ingestion_checkpoints -- P060 "checkpoint" + P062: the durable
|
||||
-- resume point per (source_id, partition_key). This is the table that
|
||||
-- makes "restart resumes without silent gaps" true rather than aspirational.
|
||||
-- * opt_ingestion_ledger -- P068: idempotency ledger. One row per
|
||||
-- (source_id, record_key, payload hash); a repeated backfill that
|
||||
-- re-delivers the same record is recognised, not re-emitted.
|
||||
-- * opt_quality_quarantine -- P064: where envelopes go when the declared
|
||||
-- enforcement policy says "quarantine" instead of "block".
|
||||
-- * opt_lineage_edges -- P067: the LOCAL, QUERYABLE projection of
|
||||
-- the OpenLineage events 0pt already emits to AUD0. AUD0 remains the
|
||||
-- system of record; this table exists because "emitted to AUD0" is not
|
||||
-- the same thing as "queryable from 0pt", and the P067 DoD asks for the
|
||||
-- latter.
|
||||
--
|
||||
-- Every statement is IF NOT EXISTS / idempotent so it can be applied to an
|
||||
-- already-migrated database, matching sql/002_source_registry.sql's contract
|
||||
-- (tests/p0/*.realinfra.test.ts apply the files themselves).
|
||||
--
|
||||
-- Ordering: requires sql/002_source_registry.sql (opt_sources) to have run.
|
||||
|
||||
-- ── P060 owner + the pre-activation status ────────────────────────────────
|
||||
-- `created_by` (002) records who typed the registration; `owner_id` records
|
||||
-- who is ACCOUNTABLE for the source afterwards. The activation gate requires
|
||||
-- the latter, so they cannot be collapsed.
|
||||
ALTER TABLE opt_sources ADD COLUMN IF NOT EXISTS owner_id TEXT;
|
||||
CREATE INDEX IF NOT EXISTS opt_sources_owner_idx ON opt_sources (owner_id);
|
||||
|
||||
-- New sources are created 'pending' (see src/repositories/*-source-repository.ts).
|
||||
-- The column default in 002 is 'active'; it is changed here so a plain INSERT
|
||||
-- that omits `status` -- from any client, including psql -- cannot produce an
|
||||
-- unclassified-but-active source. Rows that are already 'active' are left
|
||||
-- exactly as they are: this migration does not retroactively deactivate
|
||||
-- anything an operator has already approved.
|
||||
ALTER TABLE opt_sources ALTER COLUMN status SET DEFAULT 'pending';
|
||||
|
||||
-- ── P060 authorization ────────────────────────────────────────────────────
|
||||
-- One row per source. `basis` is the legal/contractual ground; `granted_by`
|
||||
-- is the accountable actor id (never "system"); `evidence_ref` points at the
|
||||
-- AUD0 evidence or 1p/contract artifact. `expires_at` is honoured by the
|
||||
-- activation gate -- an expired authorization does not classify a source.
|
||||
CREATE TABLE IF NOT EXISTS opt_source_authorization (
|
||||
source_id TEXT PRIMARY KEY REFERENCES opt_sources (source_id),
|
||||
tenant_id TEXT NOT NULL,
|
||||
basis TEXT NOT NULL, -- contract | public_register | consent | legitimate_interest | internal
|
||||
granted_by TEXT NOT NULL, -- accountable actor id
|
||||
granted_at TIMESTAMPTZ NOT NULL,
|
||||
expires_at TIMESTAMPTZ,
|
||||
evidence_ref TEXT, -- aud0 evidence ref / contract artifact ref
|
||||
notes TEXT,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS opt_source_authorization_tenant_idx ON opt_source_authorization (tenant_id);
|
||||
|
||||
-- ── P060 schedule ─────────────────────────────────────────────────────────
|
||||
CREATE TABLE IF NOT EXISTS opt_source_schedule (
|
||||
source_id TEXT PRIMARY KEY REFERENCES opt_sources (source_id),
|
||||
tenant_id TEXT NOT NULL,
|
||||
mode TEXT NOT NULL, -- batch | stream | cdc
|
||||
interval_seconds INT, -- batch/cdc poll cadence; NULL for continuous streams
|
||||
cron TEXT, -- optional cron expression (documented, not executed here)
|
||||
enabled BOOLEAN NOT NULL DEFAULT false,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS opt_source_schedule_tenant_idx ON opt_source_schedule (tenant_id);
|
||||
|
||||
-- ── P060 schema + P063 schema evolution ───────────────────────────────────
|
||||
-- `fields` is the record schema as JSON: {"<field>": {"type": "...",
|
||||
-- "required": true|false}}. `compatibility` governs what the NEXT registered
|
||||
-- version is allowed to change (see src/schema/compatibility.ts).
|
||||
CREATE TABLE IF NOT EXISTS opt_source_schemas (
|
||||
source_id TEXT NOT NULL REFERENCES opt_sources (source_id),
|
||||
schema_version INT NOT NULL,
|
||||
tenant_id TEXT NOT NULL,
|
||||
fields JSONB NOT NULL,
|
||||
compatibility TEXT NOT NULL, -- backward | forward | full | none
|
||||
registered_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
registered_by TEXT,
|
||||
PRIMARY KEY (source_id, schema_version)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS opt_source_schemas_tenant_idx ON opt_source_schemas (tenant_id);
|
||||
|
||||
-- ── P060 checkpoint + P062 durable resume ─────────────────────────────────
|
||||
-- `cursor` is opaque to 0pt -- the connector defines it (an offset, an LSN,
|
||||
-- a page token). `sequence` is 0pt's own monotonic envelope counter for the
|
||||
-- partition, used to detect gaps on resume.
|
||||
CREATE TABLE IF NOT EXISTS opt_ingestion_checkpoints (
|
||||
source_id TEXT NOT NULL REFERENCES opt_sources (source_id),
|
||||
partition_key TEXT NOT NULL,
|
||||
tenant_id TEXT NOT NULL,
|
||||
cursor TEXT,
|
||||
sequence BIGINT NOT NULL DEFAULT 0,
|
||||
records_committed BIGINT NOT NULL DEFAULT 0,
|
||||
committed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (source_id, partition_key)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS opt_ingestion_checkpoints_tenant_idx ON opt_ingestion_checkpoints (tenant_id);
|
||||
|
||||
-- ── P068 backfill idempotency ledger ──────────────────────────────────────
|
||||
-- A backfill/replay that re-delivers a record 0pt has already accepted must
|
||||
-- be a no-op, not a duplicate. `payload_hash` distinguishes "same record,
|
||||
-- same content" (skip) from "same record, new content" (an update, emitted).
|
||||
CREATE TABLE IF NOT EXISTS opt_ingestion_ledger (
|
||||
source_id TEXT NOT NULL REFERENCES opt_sources (source_id),
|
||||
record_key TEXT NOT NULL,
|
||||
tenant_id TEXT NOT NULL,
|
||||
payload_hash TEXT NOT NULL,
|
||||
first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
delivery_count BIGINT NOT NULL DEFAULT 1,
|
||||
PRIMARY KEY (source_id, record_key)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS opt_ingestion_ledger_tenant_idx ON opt_ingestion_ledger (tenant_id);
|
||||
|
||||
-- ── P064 quarantine ───────────────────────────────────────────────────────
|
||||
CREATE TABLE IF NOT EXISTS opt_quality_quarantine (
|
||||
quarantine_id TEXT PRIMARY KEY,
|
||||
source_id TEXT NOT NULL REFERENCES opt_sources (source_id),
|
||||
tenant_id TEXT NOT NULL,
|
||||
run_id TEXT NOT NULL,
|
||||
record_key TEXT NOT NULL,
|
||||
violations JSONB NOT NULL, -- [{expectation_type, kwargs, detail}]
|
||||
envelope JSONB NOT NULL,
|
||||
quarantined_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS opt_quality_quarantine_source_idx ON opt_quality_quarantine (source_id, quarantined_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS opt_quality_quarantine_tenant_idx ON opt_quality_quarantine (tenant_id);
|
||||
|
||||
-- ── P067 queryable lineage projection ─────────────────────────────────────
|
||||
-- One row per (run, input, output) edge. `edge_id` is deterministic
|
||||
-- (sha256 of the tuple) so a re-emitted idempotent run does not duplicate
|
||||
-- edges -- the same dedup property LineageEmitter relies on in AUD0.
|
||||
CREATE TABLE IF NOT EXISTS opt_lineage_edges (
|
||||
edge_id TEXT PRIMARY KEY,
|
||||
tenant_id TEXT NOT NULL,
|
||||
run_id TEXT NOT NULL,
|
||||
capability_id TEXT NOT NULL,
|
||||
input_namespace TEXT NOT NULL,
|
||||
input_name TEXT NOT NULL,
|
||||
input_ref TEXT,
|
||||
output_namespace TEXT NOT NULL,
|
||||
output_name TEXT NOT NULL,
|
||||
output_ref TEXT,
|
||||
event_time TIMESTAMPTZ NOT NULL,
|
||||
correlation_id TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS opt_lineage_edges_output_idx ON opt_lineage_edges (tenant_id, output_name);
|
||||
CREATE INDEX IF NOT EXISTS opt_lineage_edges_input_idx ON opt_lineage_edges (tenant_id, input_name);
|
||||
CREATE INDEX IF NOT EXISTS opt_lineage_edges_run_idx ON opt_lineage_edges (tenant_id, run_id);
|
||||
@@ -0,0 +1,276 @@
|
||||
/**
|
||||
* src/connectors/conformance.ts
|
||||
* ==============================
|
||||
* P061 DoD: "two heterogeneous connectors pass the same conformance suite."
|
||||
*
|
||||
* This module IS that suite. It is a plain async function returning a report
|
||||
* — not a vitest file — so it can be run three ways without duplication:
|
||||
* 1. from tests (tests/p0/connector-conformance.test.ts runs it per connector),
|
||||
* 2. from an operator script against a real configured source,
|
||||
* 3. from a future capability route as a pre-activation check.
|
||||
*
|
||||
* Each check is a named assertion about the CONTRACT, not about any one
|
||||
* connector's data. A connector that cannot serve a check honestly (e.g. it
|
||||
* has no partitions under the given config) FAILS it — there is no "skip"
|
||||
* verdict, because a skip that reads as a pass is how a conformance suite
|
||||
* becomes decoration.
|
||||
*/
|
||||
import type {
|
||||
BackfillRequest,
|
||||
Connector,
|
||||
ConnectorConfig,
|
||||
ConnectorContext,
|
||||
} from "./connector";
|
||||
|
||||
export interface ConformanceCheck {
|
||||
name: string;
|
||||
passed: boolean;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface ConformanceReport {
|
||||
connector_id: string;
|
||||
connector_version: string;
|
||||
passed: boolean;
|
||||
checks: ConformanceCheck[];
|
||||
}
|
||||
|
||||
export interface ConformanceFixture {
|
||||
/** A config the connector must accept. */
|
||||
validConfig: ConnectorConfig;
|
||||
/** A config the connector must REJECT in validate() without doing IO. */
|
||||
invalidConfig: ConnectorConfig;
|
||||
ctx: ConnectorContext;
|
||||
/**
|
||||
* A bounded backfill range that overlaps data the connector can serve.
|
||||
* `from`/`to` are connector-defined (documented per connector).
|
||||
*/
|
||||
backfill: Omit<BackfillRequest, "partition_key" | "limit"> & { partition_key?: string };
|
||||
/** Total number of records the fixture's first partition holds, so paging can be checked exactly. */
|
||||
expectedRecordCount: number;
|
||||
}
|
||||
|
||||
async function check(
|
||||
checks: ConformanceCheck[],
|
||||
name: string,
|
||||
fn: () => Promise<string> | string,
|
||||
): Promise<void> {
|
||||
try {
|
||||
const detail = await fn();
|
||||
checks.push({ name, passed: true, detail });
|
||||
} catch (err) {
|
||||
checks.push({
|
||||
name,
|
||||
passed: false,
|
||||
detail: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function assert(condition: boolean, message: string): void {
|
||||
if (!condition) throw new Error(message);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the full conformance suite against one connector instance.
|
||||
* The connector is left disconnected on return, pass or fail.
|
||||
*/
|
||||
export async function runConnectorConformance(
|
||||
connector: Connector,
|
||||
fixture: ConformanceFixture,
|
||||
): Promise<ConformanceReport> {
|
||||
const checks: ConformanceCheck[] = [];
|
||||
|
||||
await check(checks, "identity: non-empty id and version", () => {
|
||||
assert(connector.connector_id.length > 0, "connector_id is empty");
|
||||
assert(connector.connector_version.length > 0, "connector_version is empty");
|
||||
return `${connector.connector_id}@${connector.connector_version}`;
|
||||
});
|
||||
|
||||
await check(checks, "validate: accepts a valid config without IO", () => {
|
||||
const res = connector.validate(fixture.validConfig);
|
||||
assert(res.valid, `valid config rejected: ${res.errors.join("; ")}`);
|
||||
assert(res.errors.length === 0, "valid config produced errors");
|
||||
return "valid config accepted";
|
||||
});
|
||||
|
||||
await check(checks, "validate: rejects an invalid config and says why", () => {
|
||||
const res = connector.validate(fixture.invalidConfig);
|
||||
assert(!res.valid, "invalid config was accepted (fail-open)");
|
||||
assert(res.errors.length > 0, "rejection carried no reason");
|
||||
return `rejected with ${res.errors.length} reason(s)`;
|
||||
});
|
||||
|
||||
await check(checks, "operations before connect() fail closed", async () => {
|
||||
let threw = false;
|
||||
try {
|
||||
await connector.read({ partition_key: "anything", limit: 1 });
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
assert(threw, "read() succeeded without a session");
|
||||
return "read() refused without a session";
|
||||
});
|
||||
|
||||
await check(checks, "connect: idempotent", async () => {
|
||||
await connector.connect(fixture.validConfig, fixture.ctx);
|
||||
await connector.connect(fixture.validConfig, fixture.ctx);
|
||||
return "connect() called twice without error";
|
||||
});
|
||||
|
||||
let partitionKey = fixture.backfill.partition_key ?? "";
|
||||
await check(checks, "discover: reports at least one partition with a schema", async () => {
|
||||
const discovery = await connector.discover();
|
||||
assert(discovery.partitions.length > 0, "no partitions discovered");
|
||||
assert(discovery.modes.length > 0, "no ingestion modes declared");
|
||||
const first = discovery.partitions[0];
|
||||
assert(first.partition_key.length > 0, "partition_key is empty");
|
||||
assert(
|
||||
Object.keys(first.schema).length > 0,
|
||||
`partition "${first.partition_key}" reported an empty schema`,
|
||||
);
|
||||
if (!partitionKey) partitionKey = first.partition_key;
|
||||
return `${discovery.partitions.length} partition(s), modes: ${discovery.modes.join(",")}`;
|
||||
});
|
||||
|
||||
await check(checks, "health: answers with a status instead of throwing", async () => {
|
||||
const health = await connector.health();
|
||||
assert(
|
||||
["up", "degraded", "down"].includes(health.status),
|
||||
`unexpected health status "${health.status}"`,
|
||||
);
|
||||
return `status=${health.status}`;
|
||||
});
|
||||
|
||||
await check(checks, "read: honours the limit and advances the cursor", async () => {
|
||||
const first = await connector.read({ partition_key: partitionKey, limit: 1 });
|
||||
assert(first.records.length <= 1, `limit 1 returned ${first.records.length} records`);
|
||||
assert(first.records.length === 1, "limit 1 returned nothing — fixture has no data");
|
||||
assert(first.next_cursor !== undefined, "no cursor returned after a non-empty read");
|
||||
const second = await connector.read({
|
||||
partition_key: partitionKey,
|
||||
cursor: first.next_cursor,
|
||||
limit: 1,
|
||||
});
|
||||
if (second.records.length > 0) {
|
||||
assert(
|
||||
second.records[0].record_key !== first.records[0].record_key,
|
||||
"reading from the returned cursor re-delivered the same record (cursor is not exclusive)",
|
||||
);
|
||||
}
|
||||
return `cursor advanced past ${first.records[0].record_key}`;
|
||||
});
|
||||
|
||||
await check(checks, "read: paging covers every record exactly once", async () => {
|
||||
const seen: string[] = [];
|
||||
let cursor: string | undefined;
|
||||
for (let page = 0; page < 100; page++) {
|
||||
const res = await connector.read({ partition_key: partitionKey, cursor, limit: 2 });
|
||||
for (const r of res.records) seen.push(r.record_key);
|
||||
cursor = res.next_cursor;
|
||||
if (res.exhausted || res.records.length === 0) break;
|
||||
}
|
||||
assert(
|
||||
seen.length === fixture.expectedRecordCount,
|
||||
`paged ${seen.length} records, fixture declares ${fixture.expectedRecordCount}`,
|
||||
);
|
||||
assert(new Set(seen).size === seen.length, "paging delivered a duplicate record_key");
|
||||
return `${seen.length} records, no duplicates`;
|
||||
});
|
||||
|
||||
await check(checks, "read: an exhausted partition reports exhausted", async () => {
|
||||
const everything = await connector.read({ partition_key: partitionKey, limit: 1000 });
|
||||
const past = await connector.read({
|
||||
partition_key: partitionKey,
|
||||
cursor: everything.next_cursor,
|
||||
limit: 1000,
|
||||
});
|
||||
assert(past.records.length === 0, "reading past the end returned records");
|
||||
assert(past.exhausted, "reading past the end did not report exhausted");
|
||||
return "end of partition reported honestly";
|
||||
});
|
||||
|
||||
await check(checks, "checkpoint: reflects the session position", async () => {
|
||||
const cp = await connector.checkpoint(partitionKey);
|
||||
assert(cp.partition_key === partitionKey, "checkpoint returned a different partition");
|
||||
return `cursor=${cp.cursor ?? "<none>"}`;
|
||||
});
|
||||
|
||||
await check(checks, "resume: continues from a committed checkpoint without re-delivery", async () => {
|
||||
const firstPage = await connector.read({ partition_key: partitionKey, limit: 2 });
|
||||
assert(firstPage.records.length > 0, "fixture partition is empty");
|
||||
const committed = {
|
||||
source_id: fixture.ctx.source_id,
|
||||
tenant_id: fixture.ctx.tenant_id,
|
||||
partition_key: partitionKey,
|
||||
cursor: firstPage.next_cursor,
|
||||
sequence: firstPage.records.length,
|
||||
records_committed: firstPage.records.length,
|
||||
committed_at: new Date().toISOString(),
|
||||
};
|
||||
const resumed = await connector.resume(committed, 1000);
|
||||
const firstKeys = new Set(firstPage.records.map((r) => r.record_key));
|
||||
for (const r of resumed.records) {
|
||||
assert(
|
||||
!firstKeys.has(r.record_key),
|
||||
`resume re-delivered already-committed record "${r.record_key}"`,
|
||||
);
|
||||
}
|
||||
assert(
|
||||
firstPage.records.length + resumed.records.length === fixture.expectedRecordCount,
|
||||
`resume covered ${firstPage.records.length + resumed.records.length} of ${fixture.expectedRecordCount} records — a gap`,
|
||||
);
|
||||
return `resumed ${resumed.records.length} remaining records with no gap and no overlap`;
|
||||
});
|
||||
|
||||
await check(checks, "backfill: serves a bounded range", async () => {
|
||||
const res = await connector.backfill({
|
||||
partition_key: partitionKey,
|
||||
from: fixture.backfill.from,
|
||||
to: fixture.backfill.to,
|
||||
limit: 1000,
|
||||
});
|
||||
assert(res.records.length > 0, "bounded backfill range returned nothing");
|
||||
return `${res.records.length} record(s) in range`;
|
||||
});
|
||||
|
||||
await check(checks, "backfill: refuses an unbounded range", async () => {
|
||||
let threw = false;
|
||||
try {
|
||||
await connector.backfill({
|
||||
partition_key: partitionKey,
|
||||
from: fixture.backfill.from,
|
||||
to: "",
|
||||
limit: 1000,
|
||||
});
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
assert(threw, "an unbounded backfill was accepted");
|
||||
return "unbounded backfill refused";
|
||||
});
|
||||
|
||||
await check(checks, "disconnect: idempotent and safe", async () => {
|
||||
await connector.disconnect();
|
||||
await connector.disconnect();
|
||||
return "disconnect() called twice without error";
|
||||
});
|
||||
|
||||
await check(checks, "operations after disconnect() fail closed", async () => {
|
||||
let threw = false;
|
||||
try {
|
||||
await connector.read({ partition_key: partitionKey, limit: 1 });
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
assert(threw, "read() succeeded after disconnect()");
|
||||
return "read() refused after disconnect()";
|
||||
});
|
||||
|
||||
return {
|
||||
connector_id: connector.connector_id,
|
||||
connector_version: connector.connector_version,
|
||||
passed: checks.every((c) => c.passed),
|
||||
checks,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
/**
|
||||
* src/connectors/connector.ts
|
||||
* ============================
|
||||
* P061: THE connector contract. Nine operations, one interface, every
|
||||
* connector implements all nine:
|
||||
*
|
||||
* validate — is this config usable? Pure, no IO, no network. Fail-closed.
|
||||
* connect — open whatever session the source needs. Must be idempotent.
|
||||
* discover — what partitions exist, and what shape do their records have?
|
||||
* health — is the source answering right now?
|
||||
* read — produce records from a cursor forward, bounded by `limit`.
|
||||
* checkpoint — the cursor meaning "everything before this is produced".
|
||||
* resume — continue from a previously committed checkpoint.
|
||||
* backfill — re-read a bounded historical range. Bounded, never "all".
|
||||
* disconnect — release the session. Must be idempotent and safe after an error.
|
||||
*
|
||||
* WHY AN INTERFACE AND NOT A BASE CLASS: connectors are heterogeneous by
|
||||
* nature (an HTTP API paginates by opaque token, a file store seeks by byte
|
||||
* offset, a change feed advances an LSN). The contract fixes the SHAPE of
|
||||
* the interaction and the meaning of the cursor; it does not pretend the
|
||||
* mechanics are the same. `src/connectors/conformance.ts` is what proves two
|
||||
* unrelated implementations really do honour the same shape.
|
||||
*
|
||||
* WHAT THIS REPLACES, HONESTLY: services/api/sources/connectors.py has
|
||||
* per-source `async def fetch_x()` functions and nothing else — no lifecycle,
|
||||
* no cursor, no resume, no health, no disconnect. Those functions remain the
|
||||
* Python engine's fetch layer; they are NOT retrofitted here (that engine is
|
||||
* not deployed — api.0pt.app answers 404 — and rewriting it blind would be
|
||||
* the exact "build it again" failure this programme forbids). This contract
|
||||
* is the seam a Python connector is wrapped into when that engine is revived.
|
||||
*/
|
||||
import type { IngestionCheckpoint, IngestionMode } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
|
||||
/** Config is connector-specific; the contract only requires it be serializable. */
|
||||
export type ConnectorConfig = Record<string, unknown>;
|
||||
|
||||
export interface ConnectorContext {
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
source_id: string;
|
||||
/** Correlates every operation of one ingestion run. */
|
||||
run_id: string;
|
||||
}
|
||||
|
||||
/** Result of `validate` — never throws, always answers. */
|
||||
export interface ValidationResult {
|
||||
valid: boolean;
|
||||
/** One entry per problem. Empty iff valid. */
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
/** One readable partition (a table, a topic, a file, a page family). */
|
||||
export interface PartitionDescriptor {
|
||||
partition_key: string;
|
||||
/** Human label; may equal partition_key. */
|
||||
name: string;
|
||||
/** Record shape as the connector currently sees it — fed straight into the schema registry (P063). */
|
||||
schema: RecordSchemaFields;
|
||||
}
|
||||
|
||||
export interface DiscoveryResult {
|
||||
partitions: PartitionDescriptor[];
|
||||
/** Modes this connector can actually serve for this config. */
|
||||
modes: IngestionMode[];
|
||||
}
|
||||
|
||||
export interface ConnectorHealth {
|
||||
status: "up" | "degraded" | "down";
|
||||
latency_ms?: number;
|
||||
/** Short operator-facing text — never a raw upstream body. */
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
export interface ReadRequest {
|
||||
partition_key: string;
|
||||
/** Start AFTER this cursor. Undefined = from the beginning of the partition. */
|
||||
cursor?: string;
|
||||
/** Hard upper bound on records returned. Connectors MUST honour it. */
|
||||
limit: number;
|
||||
}
|
||||
|
||||
/** A record as the connector sees it, before the runner wraps it in an envelope. */
|
||||
export interface ConnectorRecord {
|
||||
/** Stable business key within the partition. */
|
||||
record_key: string;
|
||||
/** Null only for a delete in a change feed. */
|
||||
payload: Record<string, unknown> | null;
|
||||
operation?: "upsert" | "insert" | "update" | "delete";
|
||||
occurred_at?: string;
|
||||
/** Cursor value AT this record — the runner records it per envelope. */
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
export interface ReadResult {
|
||||
records: ConnectorRecord[];
|
||||
/**
|
||||
* Cursor meaning "everything up to and including `records` has been
|
||||
* produced". The runner commits this only after the records are accepted.
|
||||
*/
|
||||
next_cursor?: string;
|
||||
/** False when the connector knows more records are immediately available. */
|
||||
exhausted: boolean;
|
||||
}
|
||||
|
||||
/** A bounded historical range. Unbounded backfills are rejected by the contract. */
|
||||
export interface BackfillRequest {
|
||||
partition_key: string;
|
||||
/** Inclusive lower bound (cursor or ISO timestamp — connector-defined, documented per connector). */
|
||||
from: string;
|
||||
/** Exclusive upper bound. Required: a backfill with no end is a full reload, not a backfill. */
|
||||
to: string;
|
||||
limit: number;
|
||||
}
|
||||
|
||||
export interface Connector {
|
||||
readonly connector_id: string;
|
||||
readonly connector_version: string;
|
||||
|
||||
/** Pure config check. MUST NOT perform IO. MUST NOT throw. */
|
||||
validate(config: ConnectorConfig): ValidationResult;
|
||||
|
||||
/** Open a session. Idempotent: calling it twice is a no-op, not an error. */
|
||||
connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void>;
|
||||
|
||||
/** Requires a live session. */
|
||||
discover(): Promise<DiscoveryResult>;
|
||||
|
||||
/** Requires a live session. Never throws for an unhealthy source — reports it. */
|
||||
health(): Promise<ConnectorHealth>;
|
||||
|
||||
/** Requires a live session. Honours `limit` exactly. */
|
||||
read(req: ReadRequest): Promise<ReadResult>;
|
||||
|
||||
/**
|
||||
* The connector's own view of where it stands for a partition. The DURABLE
|
||||
* checkpoint is owned by the runner (CheckpointStore); this is the
|
||||
* in-session position, used to cross-check a resume.
|
||||
*/
|
||||
checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }>;
|
||||
|
||||
/**
|
||||
* Continue from a previously committed checkpoint. Equivalent to
|
||||
* `read({cursor: checkpoint.cursor})` for cursor-based connectors, but
|
||||
* explicit so a connector that must re-establish server-side state (a
|
||||
* replication slot, a scroll context) has a place to do it.
|
||||
*/
|
||||
resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult>;
|
||||
|
||||
/** Bounded historical re-read. MUST reject an unbounded range. */
|
||||
backfill(req: BackfillRequest): Promise<ReadResult>;
|
||||
|
||||
/** Idempotent; safe to call without a session and after an error. */
|
||||
disconnect(): Promise<void>;
|
||||
}
|
||||
|
||||
/** Thrown by connectors when an operation needs a session that was never opened. */
|
||||
export class ConnectorNotConnectedError extends Error {
|
||||
constructor(connectorId: string, operation: string) {
|
||||
super(`connector ${connectorId}: ${operation} requires connect() first`);
|
||||
this.name = "ConnectorNotConnectedError";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
/**
|
||||
* src/connectors/http-json-connector.ts
|
||||
* ======================================
|
||||
* P061 conformance connector #1: a real HTTP JSON API source.
|
||||
*
|
||||
* Heterogeneity (the point of having two): this connector talks to a REMOTE
|
||||
* service over the network, pages by an OPAQUE token the server hands back,
|
||||
* cannot know how many records exist until the server says it is done, and
|
||||
* has a genuine health check (an HTTP round trip). Its sibling
|
||||
* (jsonl-file-connector.ts) does none of those things.
|
||||
*
|
||||
* COMPLIANCE (CLAUDE.md §1): this connector is a plain, self-identifying
|
||||
* JSON client. It sends a fixed User-Agent, honours the caller's timeout, and
|
||||
* treats 401/403/429/451 as a STOP signal for that source — the same posture
|
||||
* services/api/sources/connectors.py's BLOCKED_STATUSES already takes. There
|
||||
* is no retry-harder path, no proxy rotation, no header spoofing, and adding
|
||||
* any would be a compliance violation, not a feature.
|
||||
*
|
||||
* `fetchImpl` is injectable so tests drive a real request/response cycle
|
||||
* without a network. That is a seam for TESTS, not a mock fallback: with no
|
||||
* override it uses global fetch and talks to the configured URL for real.
|
||||
*/
|
||||
import {
|
||||
ConnectorNotConnectedError,
|
||||
type BackfillRequest,
|
||||
type Connector,
|
||||
type ConnectorConfig,
|
||||
type ConnectorContext,
|
||||
type ConnectorHealth,
|
||||
type ConnectorRecord,
|
||||
type DiscoveryResult,
|
||||
type ReadRequest,
|
||||
type ReadResult,
|
||||
type ValidationResult,
|
||||
} from "./connector";
|
||||
import type { IngestionCheckpoint } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
|
||||
export const HTTP_JSON_CONNECTOR_VERSION = "http-json-connector@1";
|
||||
|
||||
/** Statuses that mean "stop asking this way" — never "retry harder". */
|
||||
const BLOCKED_STATUSES = new Set([401, 403, 429, 451]);
|
||||
|
||||
export interface HttpJsonConnectorConfig extends ConnectorConfig {
|
||||
/** Absolute https:// (or http:// for a local fixture) base URL. */
|
||||
base_url: string;
|
||||
/** Path serving records, e.g. "/v1/companies". */
|
||||
records_path: string;
|
||||
/** Partition key this endpoint serves. */
|
||||
partition_key: string;
|
||||
/** Declared record shape for the partition — the connector does not infer schemas from data. */
|
||||
schema: RecordSchemaFields;
|
||||
/** JSON field holding the business key of each record. */
|
||||
record_key_field: string;
|
||||
/**
|
||||
* JSON field holding each record's own cursor value (an id, a sequence, an
|
||||
* `updated_at`). Optional, but REQUIRED IN PRACTICE for any API that stops
|
||||
* returning `next_cursor` once a page is exhausted: without it the
|
||||
* connector has nothing to resume from after reading to the end, and
|
||||
* resume() would silently restart at the beginning. The conformance suite
|
||||
* catches exactly that ("read: an exhausted partition reports exhausted"),
|
||||
* which is how this field came to exist.
|
||||
*/
|
||||
cursor_field?: string;
|
||||
/** Request timeout (ms). Default 15000. */
|
||||
timeout_ms?: number;
|
||||
}
|
||||
|
||||
type FetchImpl = (url: string, init: RequestInit) => Promise<Response>;
|
||||
|
||||
/** Wire shape this connector expects from the upstream endpoint. */
|
||||
interface UpstreamPage {
|
||||
records?: Array<Record<string, unknown>>;
|
||||
next_cursor?: string | null;
|
||||
}
|
||||
|
||||
function isConfig(config: ConnectorConfig): config is HttpJsonConnectorConfig {
|
||||
return typeof config.base_url === "string" && typeof config.records_path === "string";
|
||||
}
|
||||
|
||||
export class HttpJsonConnector implements Connector {
|
||||
readonly connector_id = "http_json";
|
||||
readonly connector_version = HTTP_JSON_CONNECTOR_VERSION;
|
||||
|
||||
private config?: HttpJsonConnectorConfig;
|
||||
private ctx?: ConnectorContext;
|
||||
private lastCursor = new Map<string, string | undefined>();
|
||||
|
||||
constructor(private readonly fetchImpl: FetchImpl = (url, init) => fetch(url, init)) {}
|
||||
|
||||
validate(config: ConnectorConfig): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (typeof config.base_url !== "string" || config.base_url.length === 0) {
|
||||
errors.push("base_url is required");
|
||||
} else if (!/^https?:\/\//.test(config.base_url)) {
|
||||
errors.push("base_url must be an absolute http(s) URL");
|
||||
}
|
||||
if (typeof config.records_path !== "string" || !config.records_path.startsWith("/")) {
|
||||
errors.push("records_path is required and must start with /");
|
||||
}
|
||||
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
|
||||
errors.push("partition_key is required");
|
||||
}
|
||||
if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) {
|
||||
errors.push("record_key_field is required");
|
||||
}
|
||||
const schema = config.schema;
|
||||
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
|
||||
errors.push("schema is required and must declare at least one field");
|
||||
}
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void> {
|
||||
const verdict = this.validate(config);
|
||||
if (!verdict.valid) {
|
||||
throw new Error(`http_json connector config invalid: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
if (!isConfig(config)) throw new Error("http_json connector config invalid");
|
||||
this.config = config;
|
||||
this.ctx = ctx;
|
||||
}
|
||||
|
||||
async discover(): Promise<DiscoveryResult> {
|
||||
const config = this.requireSession("discover");
|
||||
return {
|
||||
partitions: [
|
||||
{
|
||||
partition_key: config.partition_key,
|
||||
name: `${config.base_url}${config.records_path}`,
|
||||
schema: config.schema,
|
||||
},
|
||||
],
|
||||
// A polled JSON endpoint serves batch and (by repeated polling) stream.
|
||||
// It is NOT a change feed: it carries no delete events, so "cdc" is not
|
||||
// declared. Declaring it would be a lie the runner would act on.
|
||||
modes: ["batch", "stream"],
|
||||
};
|
||||
}
|
||||
|
||||
async health(): Promise<ConnectorHealth> {
|
||||
const config = this.requireSession("health");
|
||||
const started = Date.now();
|
||||
try {
|
||||
const res = await this.request(this.url(config.records_path, { limit: "1" }), config);
|
||||
const latency = Date.now() - started;
|
||||
if (BLOCKED_STATUSES.has(res.status)) {
|
||||
return { status: "down", latency_ms: latency, detail: `source signalled a block (HTTP ${res.status})` };
|
||||
}
|
||||
if (!res.ok) return { status: "degraded", latency_ms: latency, detail: `HTTP ${res.status}` };
|
||||
return { status: "up", latency_ms: latency };
|
||||
} catch (err) {
|
||||
return {
|
||||
status: "down",
|
||||
latency_ms: Date.now() - started,
|
||||
detail: err instanceof Error ? err.message : "unreachable",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async read(req: ReadRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("read");
|
||||
const query: Record<string, string> = { limit: String(req.limit) };
|
||||
if (req.cursor !== undefined) query.cursor = req.cursor;
|
||||
const res = await this.request(this.url(config.records_path, query), config);
|
||||
if (BLOCKED_STATUSES.has(res.status)) {
|
||||
throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`);
|
||||
}
|
||||
if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`);
|
||||
const page = (await res.json()) as UpstreamPage;
|
||||
const rawRecords = page.records ?? [];
|
||||
// The contract says limit is a HARD bound; a server that over-delivers is
|
||||
// truncated here rather than silently widening the caller's page.
|
||||
const records = rawRecords.slice(0, req.limit).map((row) => this.toRecord(row, config));
|
||||
const serverCursor = page.next_cursor ?? undefined;
|
||||
// When the server stops issuing a cursor at the end of a page, fall back
|
||||
// to the last record's own cursor value so a later resume() starts AFTER
|
||||
// the data already produced instead of at the beginning.
|
||||
const trailing = records.length > 0 ? records[records.length - 1].cursor : undefined;
|
||||
const nextCursor = serverCursor ?? trailing ?? req.cursor;
|
||||
if (records.length > 0) this.lastCursor.set(req.partition_key, nextCursor);
|
||||
return {
|
||||
records,
|
||||
next_cursor: nextCursor,
|
||||
// The SERVER's cursor is what says "there is more"; a fallback cursor
|
||||
// only preserves position, it never claims more data exists.
|
||||
exhausted: serverCursor === undefined || records.length === 0,
|
||||
};
|
||||
}
|
||||
|
||||
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
|
||||
this.requireSession("checkpoint");
|
||||
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
|
||||
}
|
||||
|
||||
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
|
||||
this.requireSession("resume");
|
||||
return this.read({
|
||||
partition_key: checkpoint.partition_key,
|
||||
cursor: checkpoint.cursor,
|
||||
limit,
|
||||
});
|
||||
}
|
||||
|
||||
async backfill(req: BackfillRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("backfill");
|
||||
if (!req.from || !req.to) {
|
||||
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
|
||||
}
|
||||
const res = await this.request(
|
||||
this.url(config.records_path, { from: req.from, to: req.to, limit: String(req.limit) }),
|
||||
config,
|
||||
);
|
||||
if (BLOCKED_STATUSES.has(res.status)) {
|
||||
throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`);
|
||||
}
|
||||
if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`);
|
||||
const page = (await res.json()) as UpstreamPage;
|
||||
const records = (page.records ?? []).slice(0, req.limit).map((row) => this.toRecord(row, config));
|
||||
return { records, next_cursor: page.next_cursor ?? undefined, exhausted: true };
|
||||
}
|
||||
|
||||
async disconnect(): Promise<void> {
|
||||
this.config = undefined;
|
||||
this.ctx = undefined;
|
||||
this.lastCursor.clear();
|
||||
}
|
||||
|
||||
// --- internals -----------------------------------------------------------
|
||||
|
||||
private toRecord(row: Record<string, unknown>, config: HttpJsonConnectorConfig): ConnectorRecord {
|
||||
const key = row[config.record_key_field];
|
||||
if (typeof key !== "string" && typeof key !== "number") {
|
||||
throw new Error(`record is missing string/number key field "${config.record_key_field}"`);
|
||||
}
|
||||
let cursor: string | undefined;
|
||||
if (config.cursor_field) {
|
||||
const raw = row[config.cursor_field];
|
||||
if (typeof raw === "string" || typeof raw === "number") cursor = String(raw);
|
||||
}
|
||||
return { record_key: String(key), payload: row, operation: "upsert", cursor };
|
||||
}
|
||||
|
||||
private requireSession(op: string): HttpJsonConnectorConfig {
|
||||
if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
|
||||
return this.config;
|
||||
}
|
||||
|
||||
private url(path: string, query: Record<string, string>): string {
|
||||
const config = this.config;
|
||||
if (!config) throw new ConnectorNotConnectedError(this.connector_id, "url");
|
||||
const url = new URL(path, config.base_url);
|
||||
for (const [k, v] of Object.entries(query)) url.searchParams.set(k, v);
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
private async request(url: string, config: HttpJsonConnectorConfig): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), config.timeout_ms ?? 15_000);
|
||||
try {
|
||||
return await this.fetchImpl(url, {
|
||||
method: "GET",
|
||||
signal: controller.signal,
|
||||
headers: {
|
||||
Accept: "application/json",
|
||||
"User-Agent": "0pt-connector/1 (+https://0pt.app; SIAX Technology AB)",
|
||||
...(this.ctx ? { "X-Correlation-Id": this.ctx.run_id } : {}),
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
/**
|
||||
* src/connectors/jsonl-file-connector.ts
|
||||
* ========================================
|
||||
* P061 conformance connector #2: a JSONL file drop on local disk.
|
||||
*
|
||||
* Heterogeneity vs http-json-connector.ts — these two share nothing but the
|
||||
* contract:
|
||||
* - no network, no server, no opaque token; the cursor is a LINE OFFSET the
|
||||
* connector computes itself,
|
||||
* - it knows the full record count up front (the file is finite and local),
|
||||
* so `exhausted` is exact rather than server-reported,
|
||||
* - health is a filesystem stat, not an HTTP round trip,
|
||||
* - it serves `cdc`-shaped records: a JSONL line may carry an `_op` field
|
||||
* ("insert"/"update"/"delete"), so this connector emits real delete
|
||||
* envelopes with a null payload. The HTTP connector cannot,
|
||||
* - backfill bounds are line numbers, not timestamps.
|
||||
*
|
||||
* That last point is why the conformance suite takes `from`/`to` from a
|
||||
* fixture: the contract fixes that a backfill is BOUNDED, not what the
|
||||
* bounds mean. Each connector documents its own bound vocabulary.
|
||||
*/
|
||||
import { promises as fs } from "fs";
|
||||
import path from "path";
|
||||
import {
|
||||
ConnectorNotConnectedError,
|
||||
type BackfillRequest,
|
||||
type Connector,
|
||||
type ConnectorConfig,
|
||||
type ConnectorContext,
|
||||
type ConnectorHealth,
|
||||
type ConnectorRecord,
|
||||
type DiscoveryResult,
|
||||
type ReadRequest,
|
||||
type ReadResult,
|
||||
type ValidationResult,
|
||||
} from "./connector";
|
||||
import type { IngestionCheckpoint } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
|
||||
export const JSONL_FILE_CONNECTOR_VERSION = "jsonl-file-connector@1";
|
||||
|
||||
export interface JsonlFileConnectorConfig extends ConnectorConfig {
|
||||
/** Absolute path to the .jsonl file. */
|
||||
file_path: string;
|
||||
partition_key: string;
|
||||
schema: RecordSchemaFields;
|
||||
record_key_field: string;
|
||||
}
|
||||
|
||||
function isConfig(config: ConnectorConfig): config is JsonlFileConnectorConfig {
|
||||
return typeof config.file_path === "string" && typeof config.partition_key === "string";
|
||||
}
|
||||
|
||||
/** Cursor vocabulary: the decimal count of lines already produced. */
|
||||
function parseCursor(cursor: string | undefined): number {
|
||||
if (cursor === undefined) return 0;
|
||||
const n = Number.parseInt(cursor, 10);
|
||||
if (!Number.isInteger(n) || n < 0) throw new Error(`invalid cursor "${cursor}" — expected a line offset`);
|
||||
return n;
|
||||
}
|
||||
|
||||
export class JsonlFileConnector implements Connector {
|
||||
readonly connector_id = "jsonl_file";
|
||||
readonly connector_version = JSONL_FILE_CONNECTOR_VERSION;
|
||||
|
||||
private config?: JsonlFileConnectorConfig;
|
||||
private lastCursor = new Map<string, string | undefined>();
|
||||
|
||||
validate(config: ConnectorConfig): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (typeof config.file_path !== "string" || config.file_path.length === 0) {
|
||||
errors.push("file_path is required");
|
||||
} else if (!path.isAbsolute(config.file_path)) {
|
||||
errors.push("file_path must be absolute");
|
||||
} else if (!config.file_path.endsWith(".jsonl")) {
|
||||
errors.push("file_path must name a .jsonl file");
|
||||
}
|
||||
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
|
||||
errors.push("partition_key is required");
|
||||
}
|
||||
if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) {
|
||||
errors.push("record_key_field is required");
|
||||
}
|
||||
const schema = config.schema;
|
||||
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
|
||||
errors.push("schema is required and must declare at least one field");
|
||||
}
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
async connect(config: ConnectorConfig, _ctx: ConnectorContext): Promise<void> {
|
||||
const verdict = this.validate(config);
|
||||
if (!verdict.valid) {
|
||||
throw new Error(`jsonl_file connector config invalid: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
if (!isConfig(config)) throw new Error("jsonl_file connector config invalid");
|
||||
// Fail at connect time, not at first read: a file drop that is not there
|
||||
// is a connection failure, and the runner must see it as one.
|
||||
await fs.access(config.file_path);
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
async discover(): Promise<DiscoveryResult> {
|
||||
const config = this.requireSession("discover");
|
||||
return {
|
||||
partitions: [
|
||||
{ partition_key: config.partition_key, name: path.basename(config.file_path), schema: config.schema },
|
||||
],
|
||||
// A file drop is a batch source; because a line may carry `_op`, it can
|
||||
// also carry a change log, so cdc is honestly available here.
|
||||
modes: ["batch", "cdc"],
|
||||
};
|
||||
}
|
||||
|
||||
async health(): Promise<ConnectorHealth> {
|
||||
const config = this.requireSession("health");
|
||||
const started = Date.now();
|
||||
try {
|
||||
const stat = await fs.stat(config.file_path);
|
||||
if (stat.size === 0) {
|
||||
return { status: "degraded", latency_ms: Date.now() - started, detail: "file is empty" };
|
||||
}
|
||||
return { status: "up", latency_ms: Date.now() - started };
|
||||
} catch (err) {
|
||||
return {
|
||||
status: "down",
|
||||
latency_ms: Date.now() - started,
|
||||
detail: err instanceof Error ? err.message : "stat failed",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async read(req: ReadRequest): Promise<ReadResult> {
|
||||
const lines = await this.lines();
|
||||
const start = parseCursor(req.cursor);
|
||||
const slice = lines.slice(start, start + req.limit);
|
||||
const records = slice.map((line, i) => this.toRecord(line, start + i));
|
||||
const nextOffset = start + slice.length;
|
||||
const cursor = String(nextOffset);
|
||||
this.lastCursor.set(req.partition_key, cursor);
|
||||
return {
|
||||
records,
|
||||
next_cursor: cursor,
|
||||
exhausted: nextOffset >= lines.length,
|
||||
};
|
||||
}
|
||||
|
||||
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
|
||||
this.requireSession("checkpoint");
|
||||
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
|
||||
}
|
||||
|
||||
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
|
||||
this.requireSession("resume");
|
||||
return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit });
|
||||
}
|
||||
|
||||
/** Bounds are LINE NUMBERS: `from` inclusive, `to` exclusive. */
|
||||
async backfill(req: BackfillRequest): Promise<ReadResult> {
|
||||
this.requireSession("backfill");
|
||||
if (!req.from || !req.to) {
|
||||
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
|
||||
}
|
||||
const from = parseCursor(req.from);
|
||||
const to = parseCursor(req.to);
|
||||
if (to <= from) throw new Error(`backfill range is empty or inverted (from=${from}, to=${to})`);
|
||||
const lines = await this.lines();
|
||||
const slice = lines.slice(from, Math.min(to, from + req.limit));
|
||||
return {
|
||||
records: slice.map((line, i) => this.toRecord(line, from + i)),
|
||||
next_cursor: String(from + slice.length),
|
||||
exhausted: true,
|
||||
};
|
||||
}
|
||||
|
||||
async disconnect(): Promise<void> {
|
||||
this.config = undefined;
|
||||
this.lastCursor.clear();
|
||||
}
|
||||
|
||||
// --- internals -----------------------------------------------------------
|
||||
|
||||
private requireSession(op: string): JsonlFileConnectorConfig {
|
||||
if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
|
||||
return this.config;
|
||||
}
|
||||
|
||||
private async lines(): Promise<string[]> {
|
||||
const config = this.requireSession("read");
|
||||
const raw = await fs.readFile(config.file_path, "utf8");
|
||||
return raw.split("\n").filter((l) => l.trim().length > 0);
|
||||
}
|
||||
|
||||
private toRecord(line: string, lineNumber: number): ConnectorRecord {
|
||||
const config = this.requireSession("read");
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
parsed = JSON.parse(line) as Record<string, unknown>;
|
||||
} catch {
|
||||
throw new Error(`line ${lineNumber + 1} is not valid JSON`);
|
||||
}
|
||||
const key = parsed[config.record_key_field];
|
||||
if (typeof key !== "string" && typeof key !== "number") {
|
||||
throw new Error(`line ${lineNumber + 1} is missing key field "${config.record_key_field}"`);
|
||||
}
|
||||
const op = parsed._op;
|
||||
const operation =
|
||||
op === "insert" || op === "update" || op === "delete" ? op : ("upsert" as const);
|
||||
const occurredAt = typeof parsed._occurred_at === "string" ? parsed._occurred_at : undefined;
|
||||
// `_op`/`_occurred_at` are envelope metadata, not payload fields — leaving
|
||||
// them in the payload would make every record fail schema conformance for
|
||||
// an "unregistered field".
|
||||
const payload: Record<string, unknown> = { ...parsed };
|
||||
delete payload._op;
|
||||
delete payload._occurred_at;
|
||||
return {
|
||||
record_key: String(key),
|
||||
payload: operation === "delete" ? null : payload,
|
||||
operation,
|
||||
occurred_at: occurredAt,
|
||||
cursor: String(lineNumber + 1),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -23,6 +23,16 @@
|
||||
* GENERIC means GENERIC: no person-matching logic, no Ratsit/PII pipelines —
|
||||
* records are opaque field maps supplied by the calling product under its
|
||||
* own legal basis; this module implements comparison mechanics only.
|
||||
*
|
||||
* P065 STATUS — THIS IS THE CANONICAL ENGINE, AND IT IS NOT YET THE ONLY ONE.
|
||||
* `services/api/workers/resolve.py` is a second, simplified ER engine (fuzzy
|
||||
* threshold, auto-merge, no hard rules, no review band) that
|
||||
* `services/api/workers/pipeline.py` still calls. P065's DoD ("no second
|
||||
* simplified duplicated production engine remains") is therefore NOT met.
|
||||
* See that file's header for why it was not rewritten blind in the
|
||||
* P060-P068 pass and STATUS.md for the removal plan. The ingestion plane
|
||||
* added in that pass (src/ingestion/*) deliberately does no ER of its own —
|
||||
* it stops at the sink, so it adds no THIRD engine.
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import { SiaxError } from "../platform/errors";
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* src/ingestion/envelope.ts
|
||||
* ==========================
|
||||
* P062: the one place a ConnectorRecord becomes an IngestionEnvelope. Every
|
||||
* connector, every mode, one function — that is what "normalize all ingestion
|
||||
* to a common envelope" means operationally.
|
||||
*
|
||||
* `hashPayload` is the stable content hash the idempotency ledger (P068) and
|
||||
* the deterministic `envelope_id` are both built on. Key order must not
|
||||
* change the hash — two connectors reading the same record can serialize the
|
||||
* same object with different key order — so keys are sorted recursively.
|
||||
* Undefined values are dropped (JSON has no undefined); null is kept, because
|
||||
* "field explicitly null" is different data from "field absent".
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import type { ConnectorRecord } from "../connectors/connector";
|
||||
import type { EnvelopeOperation, IngestionEnvelope, IngestionMode } from "../types/ingestion";
|
||||
|
||||
/** Recursively key-sorted JSON — stable across key insertion order. */
|
||||
export function stableStringify(value: unknown): string {
|
||||
if (value === null) return "null";
|
||||
if (Array.isArray(value)) return `[${value.map(stableStringify).join(",")}]`;
|
||||
if (typeof value === "object") {
|
||||
const entries = Object.entries(value as Record<string, unknown>)
|
||||
.filter(([, v]) => v !== undefined)
|
||||
.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
|
||||
return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value) ?? "null";
|
||||
}
|
||||
|
||||
/** Content hash of an envelope payload. A delete (null payload) hashes as "deleted". */
|
||||
export function hashPayload(payload: Record<string, unknown> | null): string {
|
||||
if (payload === null) return "sha256:deleted";
|
||||
return `sha256:${crypto.createHash("sha256").update(stableStringify(payload)).digest("hex")}`;
|
||||
}
|
||||
|
||||
export interface BuildEnvelopeInput {
|
||||
record: ConnectorRecord;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
mode: IngestionMode;
|
||||
partition_key: string;
|
||||
schema_version: number;
|
||||
sequence: number;
|
||||
connector_id: string;
|
||||
connector_version: string;
|
||||
run_id: string;
|
||||
/** Cursor covering this record, when the connector reports one. */
|
||||
cursor?: string;
|
||||
now?: string;
|
||||
}
|
||||
|
||||
export function buildEnvelope(input: BuildEnvelopeInput): IngestionEnvelope {
|
||||
const operation: EnvelopeOperation = input.record.operation ?? "upsert";
|
||||
const payload = operation === "delete" ? null : input.record.payload;
|
||||
const contentHash = hashPayload(payload);
|
||||
return {
|
||||
// Deterministic: the same record with the same content re-read after a
|
||||
// crash produces the same envelope_id, so a downstream sink that dedups
|
||||
// by id sees a retry, not a new record.
|
||||
envelope_id: `env_${crypto
|
||||
.createHash("sha256")
|
||||
.update(`${input.source_id}|${input.partition_key}|${input.record.record_key}|${contentHash}`)
|
||||
.digest("hex")
|
||||
.slice(0, 32)}`,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
mode: input.mode,
|
||||
operation,
|
||||
partition_key: input.partition_key,
|
||||
record_key: input.record.record_key,
|
||||
schema_version: input.schema_version,
|
||||
sequence: input.sequence,
|
||||
occurred_at: input.record.occurred_at,
|
||||
ingested_at: input.now ?? new Date().toISOString(),
|
||||
payload,
|
||||
provenance: {
|
||||
connector_id: input.connector_id,
|
||||
connector_version: input.connector_version,
|
||||
run_id: input.run_id,
|
||||
cursor: input.cursor ?? input.record.cursor,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,557 @@
|
||||
/**
|
||||
* src/ingestion/ingestion-runner.ts
|
||||
* ==================================
|
||||
* The single ingestion highway (P062, P063 gate, P064 gate, P067 emission,
|
||||
* P068 ledger). One loop, one envelope shape, one durable checkpoint.
|
||||
*
|
||||
* ORDERING IS THE WHOLE CONTRACT. Per page:
|
||||
*
|
||||
* read/resume -> normalize to envelopes -> schema conformance -> quality gate
|
||||
* -> ledger PEEK (filter replays) -> SINK -> ledger COMMIT -> checkpoint COMMIT
|
||||
*
|
||||
* Both commits happen AFTER the sink resolves. That is what makes "a restart
|
||||
* resumes without silent gaps" true: a process that dies anywhere before the
|
||||
* commits leaves the page unconsumed, the next run re-reads it, and nothing is
|
||||
* lost. The two tempting shortcuts are both data-loss bugs, and one of them
|
||||
* was actually written here first:
|
||||
* - committing the CHECKPOINT before the sink skips the page entirely;
|
||||
* - committing the LEDGER before the sink marks records delivered that the
|
||||
* sink never received, so the resumed run drops them as duplicates. This
|
||||
* one shipped in the first draft of this file and was caught by
|
||||
* tests/p0/ingestion-runner.test.ts's crash/resume case (7 of 10 records
|
||||
* delivered). The peek/commit split in IngestionRepository exists because
|
||||
* of it.
|
||||
*
|
||||
* The resulting delivery guarantee is AT-LEAST-ONCE, not exactly-once: two
|
||||
* concurrent runs on the same partition can both peek "new". `envelope_id` is
|
||||
* deterministic so a dedup-capable sink survives that. A per-(source,
|
||||
* partition) run lock does not exist yet — see STATUS.md.
|
||||
*
|
||||
* A BLOCKED quality page also stops WITHOUT committing: a blocked page must
|
||||
* stay unconsumed, not be skipped over.
|
||||
*
|
||||
* WHAT THIS RUNNER DOES NOT DO, stated plainly:
|
||||
* - it does not schedule itself (ACT0 owns scheduling; the source's
|
||||
* `schedule` row is a declaration, not a cron daemon in this repo),
|
||||
* - it does not write to a lakehouse — the `sink` is supplied by the
|
||||
* caller, because the Iceberg write boundary is ST0RE's contract
|
||||
* (src/services/iceberg-write-service.ts) and is owned by another repo
|
||||
* in this programme (P066),
|
||||
* - it does not do entity resolution. The canonical ER engine is
|
||||
* src/er/entity-resolution-service.ts and runs downstream of the sink.
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import { SiaxError } from "../platform/errors";
|
||||
import { conformanceErrors } from "../schema/compatibility";
|
||||
import type { SchemaRegistry } from "../schema/schema-registry";
|
||||
import { QualityGate, type QualityEnforcementPolicy } from "../quality/quality-policy";
|
||||
import type { IngestionRepository } from "../repositories/ingestion-repository";
|
||||
import type { LineageStore } from "../lineage/lineage-store";
|
||||
import type {
|
||||
BackfillRequest,
|
||||
Connector,
|
||||
ConnectorConfig,
|
||||
ReadResult,
|
||||
} from "../connectors/connector";
|
||||
import type {
|
||||
IngestionCheckpoint,
|
||||
IngestionEnvelope,
|
||||
IngestionMode,
|
||||
QuarantineRecord,
|
||||
} from "../types/ingestion";
|
||||
import { buildEnvelope, hashPayload } from "./envelope";
|
||||
|
||||
/** Where accepted envelopes go. Supplied by the caller — see the module note. */
|
||||
export type IngestionSink = (envelopes: IngestionEnvelope[]) => Promise<void>;
|
||||
|
||||
export interface IngestionRunInput {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
mode: IngestionMode;
|
||||
connector: Connector;
|
||||
config: ConnectorConfig;
|
||||
sink: IngestionSink;
|
||||
/** Deterministic run id. Defaults to a random one. */
|
||||
run_id?: string;
|
||||
/** Records per connector read. Default 100. */
|
||||
page_size?: number;
|
||||
/** Safety stop. Default 1000 pages. */
|
||||
max_pages?: number;
|
||||
/** Only this partition; default is every partition discover() reports. */
|
||||
partition_key?: string;
|
||||
/** P064. Omitted = no gate runs, and the report says so explicitly. */
|
||||
quality_policy?: QualityEnforcementPolicy;
|
||||
/** Canonical ref of what the sink produces, for lineage (e.g. "0pt:dataset:ds_x"). */
|
||||
output_ref?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
export interface PartitionRunReport {
|
||||
partition_key: string;
|
||||
schema_version: number;
|
||||
pages_read: number;
|
||||
records_read: number;
|
||||
envelopes_emitted: number;
|
||||
duplicates_skipped: number;
|
||||
changed: number;
|
||||
quarantined: number;
|
||||
quality_violations: number;
|
||||
checkpoint: IngestionCheckpoint;
|
||||
/** Set when the partition stopped early. */
|
||||
stopped_because?: "blocked_by_quality" | "max_pages" | "sink_failed";
|
||||
}
|
||||
|
||||
export interface IngestionRunReport {
|
||||
run_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
mode: IngestionMode;
|
||||
status: "succeeded" | "failed";
|
||||
/** Honest marker: "none_declared" when the caller supplied no policy at all. */
|
||||
quality_policy: "block" | "quarantine" | "warn" | "none_declared";
|
||||
partitions: PartitionRunReport[];
|
||||
lineage_edges: number;
|
||||
/** Present when status is "failed". */
|
||||
failure?: { code: string; detail: string };
|
||||
}
|
||||
|
||||
export class IngestionRunner {
|
||||
private readonly gate: QualityGate;
|
||||
|
||||
constructor(
|
||||
private readonly repo: IngestionRepository,
|
||||
private readonly schemas: SchemaRegistry,
|
||||
private readonly lineage?: LineageStore,
|
||||
gate?: QualityGate,
|
||||
) {
|
||||
this.gate = gate ?? new QualityGate();
|
||||
}
|
||||
|
||||
async run(input: IngestionRunInput): Promise<IngestionRunReport> {
|
||||
const runId = input.run_id ?? `run_${crypto.randomUUID()}`;
|
||||
const pageSize = input.page_size ?? 100;
|
||||
const maxPages = input.max_pages ?? 1000;
|
||||
const report: IngestionRunReport = {
|
||||
run_id: runId,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
mode: input.mode,
|
||||
status: "succeeded",
|
||||
quality_policy: input.quality_policy?.on_violation ?? "none_declared",
|
||||
partitions: [],
|
||||
lineage_edges: 0,
|
||||
};
|
||||
|
||||
const verdict = input.connector.validate(input.config);
|
||||
if (!verdict.valid) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.ingestion",
|
||||
`connector config rejected: ${verdict.errors.join("; ")}`,
|
||||
);
|
||||
}
|
||||
|
||||
await input.connector.connect(input.config, {
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
source_id: input.source_id,
|
||||
run_id: runId,
|
||||
});
|
||||
|
||||
try {
|
||||
const discovery = await input.connector.discover();
|
||||
if (!discovery.modes.includes(input.mode)) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.ingestion",
|
||||
`connector ${input.connector.connector_id} does not serve mode "${input.mode}" for this config (serves: ${discovery.modes.join(", ")})`,
|
||||
);
|
||||
}
|
||||
const partitions = input.partition_key
|
||||
? discovery.partitions.filter((p) => p.partition_key === input.partition_key)
|
||||
: discovery.partitions;
|
||||
if (partitions.length === 0) {
|
||||
throw new SiaxError(
|
||||
"NOT_FOUND",
|
||||
"0pt.ingestion",
|
||||
`partition "${input.partition_key}" was not discovered on source ${input.source_id}`,
|
||||
);
|
||||
}
|
||||
|
||||
// ---- P063: schema gate, BEFORE any record is read ------------------
|
||||
// Every partition's shape is checked and registered first. A break
|
||||
// fails the whole run here, so no envelope — and therefore no consumer
|
||||
// write — can happen under a broken schema.
|
||||
const schemaVersions = new Map<string, number>();
|
||||
for (const partition of partitions) {
|
||||
const evaluated = await this.schemas.evaluate(input.source_id, partition.schema);
|
||||
if (evaluated.verdict && !evaluated.verdict.compatible) {
|
||||
throw new SiaxError(
|
||||
"CONFLICT",
|
||||
"0pt.ingestion",
|
||||
`source ${input.source_id} partition "${partition.partition_key}" changed shape in a way that breaks its declared ` +
|
||||
`"${evaluated.verdict.policy}" compatibility policy — refusing to ingest: ` +
|
||||
evaluated.verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "),
|
||||
);
|
||||
}
|
||||
const registered = await this.schemas.register({
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
fields: partition.schema,
|
||||
compatibility: evaluated.previous?.compatibility ?? "backward",
|
||||
registered_by: `run:${runId}`,
|
||||
});
|
||||
schemaVersions.set(partition.partition_key, registered.schema.schema_version);
|
||||
}
|
||||
|
||||
for (const partition of partitions) {
|
||||
report.partitions.push(
|
||||
await this.runPartition(input, {
|
||||
runId,
|
||||
pageSize,
|
||||
maxPages,
|
||||
partitionKey: partition.partition_key,
|
||||
schemaVersion: schemaVersions.get(partition.partition_key) ?? 1,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
// ---- P067: one lineage event per run, source -> output -------------
|
||||
if (this.lineage) {
|
||||
report.lineage_edges = await this.lineage.record(
|
||||
{
|
||||
eventType: "COMPLETE",
|
||||
eventTime: new Date().toISOString(),
|
||||
producer: "https://git.siax.io/sax3l/0pt",
|
||||
run: { runId },
|
||||
job: { namespace: "0pt", name: "0pt.ingestion.run.v1" },
|
||||
inputs: [
|
||||
{
|
||||
namespace: "0pt",
|
||||
name: `source/${input.source_id}`,
|
||||
facets: { siaxResourceRef: { ref: `0pt:source:${input.source_id}` } },
|
||||
},
|
||||
],
|
||||
outputs: [
|
||||
{
|
||||
namespace: "0pt",
|
||||
name: input.output_ref ?? `ingestion/${input.source_id}`,
|
||||
...(input.output_ref ? { facets: { siaxResourceRef: { ref: input.output_ref } } } : {}),
|
||||
},
|
||||
],
|
||||
},
|
||||
input.tenant_id,
|
||||
input.correlation_id,
|
||||
);
|
||||
}
|
||||
|
||||
return report;
|
||||
} finally {
|
||||
await input.connector.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* P068: a bounded historical re-read. Runs through the SAME envelope,
|
||||
* quality and ledger path as `run`, and deliberately does NOT touch the
|
||||
* streaming checkpoint — a backfill must not rewind or advance live
|
||||
* ingestion. Repeating an identical backfill emits nothing: every record
|
||||
* comes back "duplicate" from the ledger.
|
||||
*/
|
||||
async backfill(
|
||||
input: IngestionRunInput & { range: Omit<BackfillRequest, "limit"> },
|
||||
): Promise<PartitionRunReport> {
|
||||
const runId = input.run_id ?? `bkf_${crypto.randomUUID()}`;
|
||||
const pageSize = input.page_size ?? 100;
|
||||
const verdict = input.connector.validate(input.config);
|
||||
if (!verdict.valid) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.ingestion", `connector config rejected: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
await input.connector.connect(input.config, {
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
source_id: input.source_id,
|
||||
run_id: runId,
|
||||
});
|
||||
try {
|
||||
const latest = await this.schemas.getLatest(input.source_id);
|
||||
const schemaVersion = latest?.schema_version ?? 1;
|
||||
const result = await input.connector.backfill({ ...input.range, limit: pageSize });
|
||||
const existing =
|
||||
(await this.repo.getCheckpoint(input.source_id, input.range.partition_key)) ??
|
||||
this.emptyCheckpoint(input, input.range.partition_key);
|
||||
const outcome = await this.consumePage(input, {
|
||||
runId,
|
||||
result,
|
||||
partitionKey: input.range.partition_key,
|
||||
schemaVersion,
|
||||
startSequence: existing.sequence,
|
||||
schemaFields: latest?.fields,
|
||||
});
|
||||
return {
|
||||
partition_key: input.range.partition_key,
|
||||
schema_version: schemaVersion,
|
||||
pages_read: 1,
|
||||
records_read: result.records.length,
|
||||
envelopes_emitted: outcome.emitted,
|
||||
duplicates_skipped: outcome.duplicates,
|
||||
changed: outcome.changed,
|
||||
quarantined: outcome.quarantined,
|
||||
quality_violations: outcome.violations,
|
||||
// Unchanged on purpose — a backfill leaves the live checkpoint alone.
|
||||
checkpoint: existing,
|
||||
stopped_because: outcome.blocked ? "blocked_by_quality" : undefined,
|
||||
};
|
||||
} finally {
|
||||
await input.connector.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
private emptyCheckpoint(
|
||||
input: { source_id: string; tenant_id: string },
|
||||
partitionKey: string,
|
||||
): IngestionCheckpoint {
|
||||
return {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
partition_key: partitionKey,
|
||||
cursor: undefined,
|
||||
sequence: 0,
|
||||
records_committed: 0,
|
||||
committed_at: new Date(0).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
private async runPartition(
|
||||
input: IngestionRunInput,
|
||||
opts: {
|
||||
runId: string;
|
||||
pageSize: number;
|
||||
maxPages: number;
|
||||
partitionKey: string;
|
||||
schemaVersion: number;
|
||||
},
|
||||
): Promise<PartitionRunReport> {
|
||||
const schema = await this.schemas.getLatest(input.source_id);
|
||||
let checkpoint =
|
||||
(await this.repo.getCheckpoint(input.source_id, opts.partitionKey)) ??
|
||||
this.emptyCheckpoint(input, opts.partitionKey);
|
||||
|
||||
const stats = {
|
||||
pages: 0,
|
||||
records: 0,
|
||||
emitted: 0,
|
||||
duplicates: 0,
|
||||
changed: 0,
|
||||
quarantined: 0,
|
||||
violations: 0,
|
||||
};
|
||||
let stoppedBecause: PartitionRunReport["stopped_because"];
|
||||
|
||||
for (let page = 0; page < opts.maxPages; page++) {
|
||||
// First page of a partition that has a durable checkpoint goes through
|
||||
// resume(), so a connector that must re-establish server-side state has
|
||||
// its chance; later pages are ordinary reads from the running cursor.
|
||||
const result: ReadResult =
|
||||
page === 0 && checkpoint.cursor !== undefined
|
||||
? await input.connector.resume(checkpoint, opts.pageSize)
|
||||
: await input.connector.read({
|
||||
partition_key: opts.partitionKey,
|
||||
cursor: checkpoint.cursor,
|
||||
limit: opts.pageSize,
|
||||
});
|
||||
|
||||
stats.pages++;
|
||||
stats.records += result.records.length;
|
||||
if (result.records.length === 0) break;
|
||||
|
||||
const outcome = await this.consumePage(input, {
|
||||
runId: opts.runId,
|
||||
result,
|
||||
partitionKey: opts.partitionKey,
|
||||
schemaVersion: opts.schemaVersion,
|
||||
startSequence: checkpoint.sequence,
|
||||
schemaFields: schema?.fields,
|
||||
});
|
||||
stats.emitted += outcome.emitted;
|
||||
stats.duplicates += outcome.duplicates;
|
||||
stats.changed += outcome.changed;
|
||||
stats.quarantined += outcome.quarantined;
|
||||
stats.violations += outcome.violations;
|
||||
|
||||
if (outcome.blocked) {
|
||||
// The checkpoint is NOT advanced: the offending page stays unconsumed
|
||||
// so the next run sees it again rather than stepping over it.
|
||||
stoppedBecause = "blocked_by_quality";
|
||||
break;
|
||||
}
|
||||
|
||||
checkpoint = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
partition_key: opts.partitionKey,
|
||||
cursor: result.next_cursor,
|
||||
sequence: checkpoint.sequence + result.records.length,
|
||||
records_committed: checkpoint.records_committed + outcome.emitted,
|
||||
committed_at: new Date().toISOString(),
|
||||
};
|
||||
await this.repo.commitCheckpoint(checkpoint);
|
||||
|
||||
if (result.exhausted) break;
|
||||
if (page === opts.maxPages - 1) stoppedBecause = "max_pages";
|
||||
}
|
||||
|
||||
return {
|
||||
partition_key: opts.partitionKey,
|
||||
schema_version: opts.schemaVersion,
|
||||
pages_read: stats.pages,
|
||||
records_read: stats.records,
|
||||
envelopes_emitted: stats.emitted,
|
||||
duplicates_skipped: stats.duplicates,
|
||||
changed: stats.changed,
|
||||
quarantined: stats.quarantined,
|
||||
quality_violations: stats.violations,
|
||||
checkpoint,
|
||||
stopped_because: stoppedBecause,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize -> schema conformance -> quality gate -> ledger -> sink.
|
||||
* Throws if the sink throws; the caller must NOT have committed a
|
||||
* checkpoint by then.
|
||||
*/
|
||||
private async consumePage(
|
||||
input: IngestionRunInput,
|
||||
opts: {
|
||||
runId: string;
|
||||
result: ReadResult;
|
||||
partitionKey: string;
|
||||
schemaVersion: number;
|
||||
startSequence: number;
|
||||
schemaFields?: Record<string, { type: string; required: boolean }>;
|
||||
},
|
||||
): Promise<{
|
||||
emitted: number;
|
||||
duplicates: number;
|
||||
changed: number;
|
||||
quarantined: number;
|
||||
violations: number;
|
||||
blocked: boolean;
|
||||
}> {
|
||||
const envelopes: IngestionEnvelope[] = opts.result.records.map((record, i) =>
|
||||
buildEnvelope({
|
||||
record,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
mode: input.mode,
|
||||
partition_key: opts.partitionKey,
|
||||
schema_version: opts.schemaVersion,
|
||||
sequence: opts.startSequence + i + 1,
|
||||
connector_id: input.connector.connector_id,
|
||||
connector_version: input.connector.connector_version,
|
||||
run_id: opts.runId,
|
||||
cursor: opts.result.next_cursor,
|
||||
}),
|
||||
);
|
||||
|
||||
// ---- P063 per-record conformance ------------------------------------
|
||||
// A source that silently changed shape without a new discovery is caught
|
||||
// here, on the record, before the sink sees anything from this page.
|
||||
if (opts.schemaFields) {
|
||||
for (const envelope of envelopes) {
|
||||
if (envelope.payload === null) continue;
|
||||
const errors = conformanceErrors(
|
||||
opts.schemaFields as never,
|
||||
envelope.payload,
|
||||
);
|
||||
if (errors.length > 0) {
|
||||
throw new SiaxError(
|
||||
"CONFLICT",
|
||||
"0pt.ingestion",
|
||||
`record "${envelope.record_key}" does not conform to registered schema v${opts.schemaVersion} ` +
|
||||
`for source ${input.source_id}: ${errors.join("; ")}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- P064 quality gate ----------------------------------------------
|
||||
let accepted = envelopes;
|
||||
let quarantinedCount = 0;
|
||||
let violationCount = 0;
|
||||
let blocked = false;
|
||||
if (input.quality_policy) {
|
||||
const gated = this.gate.evaluate(envelopes, input.quality_policy);
|
||||
accepted = gated.accepted;
|
||||
blocked = gated.blocked;
|
||||
violationCount = gated.violations.length;
|
||||
for (const item of gated.quarantined) {
|
||||
const record: QuarantineRecord = {
|
||||
quarantine_id: `qtn_${crypto
|
||||
.createHash("sha256")
|
||||
.update(`${opts.runId}|${item.envelope.envelope_id}`)
|
||||
.digest("hex")
|
||||
.slice(0, 32)}`,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
run_id: opts.runId,
|
||||
record_key: item.envelope.record_key,
|
||||
violations: item.violations,
|
||||
envelope: item.envelope,
|
||||
quarantined_at: new Date().toISOString(),
|
||||
};
|
||||
await this.repo.quarantine(record);
|
||||
quarantinedCount++;
|
||||
}
|
||||
if (blocked) {
|
||||
return {
|
||||
emitted: 0,
|
||||
duplicates: 0,
|
||||
changed: 0,
|
||||
quarantined: quarantinedCount,
|
||||
violations: violationCount,
|
||||
blocked: true,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ---- P068 idempotency ledger, PEEK ONLY ------------------------------
|
||||
// Read the verdicts, filter replays, and commit NOTHING yet: a record is
|
||||
// only "delivered" once the sink has actually taken it. Committing here
|
||||
// would mark records delivered that a crashing sink never received, and
|
||||
// the resumed run would then skip them as duplicates — silent data loss.
|
||||
const entries = accepted.map((e) => ({
|
||||
record_key: e.record_key,
|
||||
payload_hash: hashPayload(e.payload),
|
||||
}));
|
||||
const decisions = await this.repo.peekDeliveries(input.source_id, entries);
|
||||
const verdictByKey = new Map(decisions.map((d) => [d.record_key, d.verdict]));
|
||||
const toEmit = accepted.filter((e) => verdictByKey.get(e.record_key) !== "duplicate");
|
||||
const duplicates = accepted.length - toEmit.length;
|
||||
const changed = decisions.filter((d) => d.verdict === "changed").length;
|
||||
|
||||
// ---- sink (the consumer write) --------------------------------------
|
||||
if (toEmit.length > 0) await input.sink(toEmit);
|
||||
|
||||
// ---- P068 ledger COMMIT, after the sink resolved ---------------------
|
||||
const emitted = new Set(toEmit.map((e) => e.record_key));
|
||||
const committed = entries.filter((e) => emitted.has(e.record_key));
|
||||
if (committed.length > 0) {
|
||||
await this.repo.commitDeliveries(input.source_id, input.tenant_id, committed);
|
||||
}
|
||||
|
||||
return {
|
||||
emitted: toEmit.length,
|
||||
duplicates,
|
||||
changed,
|
||||
quarantined: quarantinedCount,
|
||||
violations: violationCount,
|
||||
blocked: false,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
/**
|
||||
* src/lineage/lineage-edge.ts
|
||||
* ============================
|
||||
* P067: the flattened, QUERYABLE form of an OpenLineage run event.
|
||||
*
|
||||
* LineageEmitter (src/lineage/lineage-emitter.ts) already emits spec-shaped
|
||||
* run events to AUD0, and AUD0 remains the SYSTEM OF RECORD for them. But
|
||||
* "emitted to AUD0" is not "answerable from 0pt": nothing in this repo could
|
||||
* answer "which sources fed dataset ds_x?" without asking another product.
|
||||
* These edges are 0pt's own projection — one row per (run, input, output)
|
||||
* pair — written in the same call that emits, so the question is answerable
|
||||
* locally and joins back to AUD0 by `run_id`.
|
||||
*
|
||||
* The projection is deliberately LOSSY (facets are not stored, only the
|
||||
* canonical SIAX ref) — it is an index, not a second source of truth.
|
||||
*/
|
||||
|
||||
export interface LineageEdge {
|
||||
/** Deterministic sha256 of (run_id, input, output) — a re-emitted idempotent run does not duplicate edges. */
|
||||
edge_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
capability_id: string;
|
||||
input_namespace: string;
|
||||
input_name: string;
|
||||
input_ref?: string;
|
||||
output_namespace: string;
|
||||
output_name: string;
|
||||
output_ref?: string;
|
||||
event_time: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
/** Query one direction of the graph. Exactly one of the two selectors is required. */
|
||||
export interface LineageQuery {
|
||||
tenant_id: string;
|
||||
/** Everything that fed this output (upstream). */
|
||||
output_name?: string;
|
||||
/** Everything this input produced (downstream). */
|
||||
input_name?: string;
|
||||
limit?: number;
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* src/lineage/lineage-store.ts
|
||||
* =============================
|
||||
* P067 DoD: "source -> output lineage is QUERYABLE."
|
||||
*
|
||||
* LineageEmitter already emits OpenLineage run events to AUD0 and that does
|
||||
* not change — AUD0 stays the system of record. This class adds the missing
|
||||
* half: it flattens the same run event into (input, output) edges, persists
|
||||
* them, and answers both directions of the question.
|
||||
*
|
||||
* `RecordingLineageEmitter` is the composition point: it wraps the existing
|
||||
* emitter so every existing caller (QualityService, DatasetVersionService,
|
||||
* the ingestion runner) gets the local projection for free, without any of
|
||||
* them learning about a second store. Emission to AUD0 keeps its existing
|
||||
* never-throw contract; a projection write failure is likewise reported, not
|
||||
* thrown, because lineage is evidence of work that already happened.
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import type { IngestionRepository } from "../repositories/ingestion-repository";
|
||||
import type { LineageEdge, LineageQuery } from "./lineage-edge";
|
||||
import type { LineageEmitInput, LineageEmitResult, LineageEmitter } from "./lineage-emitter";
|
||||
import type { OpenLineageRunEvent } from "./openlineage";
|
||||
|
||||
/** Deterministic edge id: the same run re-emitted produces the same edges. */
|
||||
export function lineageEdgeId(runId: string, input: string, output: string): string {
|
||||
return `led_${crypto.createHash("sha256").update(`${runId}|${input}|${output}`).digest("hex").slice(0, 32)}`;
|
||||
}
|
||||
|
||||
function refOf(facets: Record<string, unknown> | undefined): string | undefined {
|
||||
const facet = facets?.siaxResourceRef as { ref?: string } | undefined;
|
||||
return typeof facet?.ref === "string" ? facet.ref : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Flatten one OpenLineage run event into edges. A run with N inputs and M
|
||||
* outputs yields N*M edges; a run with no inputs or no outputs yields none
|
||||
* (there is no edge to draw), which is why the runner always emits both.
|
||||
*/
|
||||
export function edgesFromRunEvent(
|
||||
event: OpenLineageRunEvent,
|
||||
tenantId: string,
|
||||
correlationId?: string,
|
||||
): LineageEdge[] {
|
||||
const edges: LineageEdge[] = [];
|
||||
for (const input of event.inputs ?? []) {
|
||||
for (const output of event.outputs ?? []) {
|
||||
edges.push({
|
||||
edge_id: lineageEdgeId(event.run.runId, `${input.namespace}/${input.name}`, `${output.namespace}/${output.name}`),
|
||||
tenant_id: tenantId,
|
||||
run_id: event.run.runId,
|
||||
capability_id: event.job.name,
|
||||
input_namespace: input.namespace,
|
||||
input_name: input.name,
|
||||
input_ref: refOf(input.facets),
|
||||
output_namespace: output.namespace,
|
||||
output_name: output.name,
|
||||
output_ref: refOf(output.facets),
|
||||
event_time: event.eventTime,
|
||||
correlation_id: correlationId,
|
||||
});
|
||||
}
|
||||
}
|
||||
return edges;
|
||||
}
|
||||
|
||||
export class LineageStore {
|
||||
constructor(private readonly repo: IngestionRepository) {}
|
||||
|
||||
async record(event: OpenLineageRunEvent, tenantId: string, correlationId?: string): Promise<number> {
|
||||
const edges = edgesFromRunEvent(event, tenantId, correlationId);
|
||||
if (edges.length > 0) await this.repo.appendLineageEdges(edges);
|
||||
return edges.length;
|
||||
}
|
||||
|
||||
/** Everything that fed `outputName`, newest first. */
|
||||
async upstreamOf(tenantId: string, outputName: string, limit?: number): Promise<LineageEdge[]> {
|
||||
return this.repo.queryLineage({ tenant_id: tenantId, output_name: outputName, limit });
|
||||
}
|
||||
|
||||
/** Everything `inputName` produced, newest first. */
|
||||
async downstreamOf(tenantId: string, inputName: string, limit?: number): Promise<LineageEdge[]> {
|
||||
return this.repo.queryLineage({ tenant_id: tenantId, input_name: inputName, limit });
|
||||
}
|
||||
|
||||
async query(query: LineageQuery): Promise<LineageEdge[]> {
|
||||
return this.repo.queryLineage(query);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A LineageEmitter that also writes the local projection. Drop-in for the
|
||||
* existing emitter: same `emit` signature, same never-throw contract.
|
||||
*/
|
||||
export class RecordingLineageEmitter {
|
||||
constructor(
|
||||
private readonly inner: LineageEmitter,
|
||||
private readonly store: LineageStore,
|
||||
) {}
|
||||
|
||||
buildRunEvent(input: LineageEmitInput): OpenLineageRunEvent {
|
||||
return this.inner.buildRunEvent(input);
|
||||
}
|
||||
|
||||
async emit(input: LineageEmitInput): Promise<LineageEmitResult & { projected: number }> {
|
||||
const event = this.inner.buildRunEvent(input);
|
||||
const result = await this.inner.emit(input);
|
||||
let projected = 0;
|
||||
try {
|
||||
projected = await this.store.record(event, input.tenant_id, input.correlation_id);
|
||||
} catch (err) {
|
||||
return {
|
||||
...result,
|
||||
projected: 0,
|
||||
warning: [result.warning, `lineage projection write failed: ${err instanceof Error ? err.message : String(err)}`]
|
||||
.filter(Boolean)
|
||||
.join("; "),
|
||||
};
|
||||
}
|
||||
return { ...result, projected };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* src/quality/quality-policy.ts
|
||||
* ==============================
|
||||
* P064: the DECLARED enforcement policy — what happens to a record that fails
|
||||
* its quality rules — and the gate that applies it on the ingestion path.
|
||||
*
|
||||
* WHAT EXISTED BEFORE (and is reused, not rebuilt): QualityService already
|
||||
* owns the rule registry, runs, results, issues and scores, and
|
||||
* ExpectationEngine already executes the Great Expectations core subset. What
|
||||
* neither of them had was a CONSEQUENCE: a failing rule produced an ISSUE for
|
||||
* a human, and the data flowed on regardless. That is the gap this closes.
|
||||
* This module deliberately does NOT re-implement rule storage or execution —
|
||||
* it composes ExpectationEngine and adds the decision.
|
||||
*
|
||||
* Three declared actions, and nothing implicit:
|
||||
* - `block` — the whole page fails; the checkpoint is NOT advanced, so
|
||||
* the run stops on the bad data instead of stepping over it.
|
||||
* - `quarantine` — the failing envelopes are persisted to durable quarantine
|
||||
* and excluded from the output; clean envelopes continue and
|
||||
* the checkpoint advances past all of them.
|
||||
* - `warn` — everything continues; violations are counted and reported.
|
||||
*
|
||||
* There is no default. A source with no policy gets `block` from the runner
|
||||
* only if a policy was supplied; with no policy at all the runner skips the
|
||||
* gate and SAYS SO in the run report (`quality_policy: "none_declared"`) —
|
||||
* an undeclared policy must never look like a passed one.
|
||||
*/
|
||||
import { ExpectationEngine } from "./expectation-engine";
|
||||
import type { QualityExpectation } from "./quality-types";
|
||||
import type { IngestionEnvelope } from "../types/ingestion";
|
||||
|
||||
export type QualityEnforcementAction = "block" | "quarantine" | "warn";
|
||||
|
||||
export interface QualityEnforcementPolicy {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
/** Rules applied per record. Expectations are evaluated against a one-row table. */
|
||||
expectations: QualityExpectation[];
|
||||
on_violation: QualityEnforcementAction;
|
||||
}
|
||||
|
||||
export interface EnvelopeViolation {
|
||||
expectation_type: string;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface QualityGateResult {
|
||||
/** Envelopes that may proceed downstream. */
|
||||
accepted: IngestionEnvelope[];
|
||||
/** Envelopes routed to quarantine (only under `quarantine`). */
|
||||
quarantined: Array<{ envelope: IngestionEnvelope; violations: EnvelopeViolation[] }>;
|
||||
/** Every violation seen, including under `warn` where nothing was withheld. */
|
||||
violations: Array<{ record_key: string; violations: EnvelopeViolation[] }>;
|
||||
/** True when `block` fired: the caller must NOT advance the checkpoint. */
|
||||
blocked: boolean;
|
||||
action: QualityEnforcementAction;
|
||||
}
|
||||
|
||||
export class QualityGate {
|
||||
constructor(private readonly engine: ExpectationEngine = new ExpectationEngine()) {}
|
||||
|
||||
/**
|
||||
* Evaluate one page of envelopes against the policy.
|
||||
*
|
||||
* Delete envelopes (payload null) are passed through untouched: a CDC delete
|
||||
* carries no columns to check, and failing it against a column expectation
|
||||
* would quarantine the very deletions a source is asking us to honour.
|
||||
*/
|
||||
evaluate(envelopes: IngestionEnvelope[], policy: QualityEnforcementPolicy): QualityGateResult {
|
||||
const accepted: IngestionEnvelope[] = [];
|
||||
const quarantined: QualityGateResult["quarantined"] = [];
|
||||
const allViolations: QualityGateResult["violations"] = [];
|
||||
|
||||
for (const envelope of envelopes) {
|
||||
if (envelope.payload === null) {
|
||||
accepted.push(envelope);
|
||||
continue;
|
||||
}
|
||||
const violations = this.violationsFor(envelope.payload, policy.expectations);
|
||||
if (violations.length === 0) {
|
||||
accepted.push(envelope);
|
||||
continue;
|
||||
}
|
||||
allViolations.push({ record_key: envelope.record_key, violations });
|
||||
if (policy.on_violation === "quarantine") {
|
||||
quarantined.push({ envelope, violations });
|
||||
} else if (policy.on_violation === "warn") {
|
||||
accepted.push(envelope);
|
||||
}
|
||||
// `block`: neither accepted nor quarantined — the caller stops.
|
||||
}
|
||||
|
||||
const blocked = policy.on_violation === "block" && allViolations.length > 0;
|
||||
return {
|
||||
accepted: blocked ? [] : accepted,
|
||||
quarantined,
|
||||
violations: allViolations,
|
||||
blocked,
|
||||
action: policy.on_violation,
|
||||
};
|
||||
}
|
||||
|
||||
private violationsFor(
|
||||
payload: Record<string, unknown>,
|
||||
expectations: QualityExpectation[],
|
||||
): EnvelopeViolation[] {
|
||||
const violations: EnvelopeViolation[] = [];
|
||||
for (const expectation of expectations) {
|
||||
if (!this.engine.supports(expectation.expectation_type)) {
|
||||
// An expectation this engine cannot execute is a VIOLATION of the
|
||||
// policy's executability, not a silent pass. The record is treated as
|
||||
// failing so an unrunnable rule can never look like a satisfied one.
|
||||
violations.push({
|
||||
expectation_type: expectation.expectation_type,
|
||||
detail: `expectation is not executable by the local engine; delegate it to the GE backend before declaring it in an enforcement policy`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const outcome = this.engine.execute(expectation, [payload]);
|
||||
if (!outcome.success) {
|
||||
violations.push({
|
||||
expectation_type: expectation.expectation_type,
|
||||
detail: `failed on column "${String(expectation.kwargs.column ?? "<table>")}" (${outcome.unexpected_count}/${outcome.evaluated_count} unexpected)`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* src/repositories/in-memory-ingestion-repository.ts
|
||||
* ====================================================
|
||||
* Non-production implementation of IngestionRepository (hermetic tests, local
|
||||
* dev). Semantics are kept IDENTICAL to the Postgres implementation on every
|
||||
* point the runner depends on:
|
||||
* - checkpoints are upserted per (source, partition),
|
||||
* - ledger verdicts are new/changed/duplicate against the LAST accepted
|
||||
* payload hash for the key,
|
||||
* - quarantine is append-only, newest-first on read,
|
||||
* - schemas are keyed by (source, version) and "latest" means highest version,
|
||||
* - lineage edges are deduplicated by edge_id.
|
||||
*
|
||||
* Any divergence between the two implementations is a bug in one of them;
|
||||
* tests/p0/ingestion-repository-parity.test.ts is what keeps them honest for
|
||||
* the in-memory side, and tests/p0/ingestion-plane-postgres.realinfra.test.ts
|
||||
* runs the same expectations against real Postgres.
|
||||
*/
|
||||
import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge";
|
||||
import type { RecordSchema } from "../schema/record-schema";
|
||||
import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion";
|
||||
import type { IngestionRepository, LedgerEntry } from "./ingestion-repository";
|
||||
|
||||
function checkpointKey(sourceId: string, partitionKey: string): string {
|
||||
return `${sourceId}|${partitionKey}`;
|
||||
}
|
||||
|
||||
export class InMemoryIngestionRepository implements IngestionRepository {
|
||||
private readonly checkpoints = new Map<string, IngestionCheckpoint>();
|
||||
/** source_id -> record_key -> last accepted payload hash. */
|
||||
private readonly ledger = new Map<string, Map<string, string>>();
|
||||
private readonly quarantined = new Map<string, QuarantineRecord[]>();
|
||||
private readonly schemas = new Map<string, RecordSchema[]>();
|
||||
private readonly lineage = new Map<string, LineageEdge>();
|
||||
|
||||
async getCheckpoint(sourceId: string, partitionKey: string): Promise<IngestionCheckpoint | undefined> {
|
||||
return this.checkpoints.get(checkpointKey(sourceId, partitionKey));
|
||||
}
|
||||
|
||||
async commitCheckpoint(checkpoint: IngestionCheckpoint): Promise<void> {
|
||||
this.checkpoints.set(checkpointKey(checkpoint.source_id, checkpoint.partition_key), { ...checkpoint });
|
||||
}
|
||||
|
||||
async listCheckpoints(sourceId: string): Promise<IngestionCheckpoint[]> {
|
||||
return [...this.checkpoints.values()]
|
||||
.filter((c) => c.source_id === sourceId)
|
||||
.sort((a, b) => a.partition_key.localeCompare(b.partition_key));
|
||||
}
|
||||
|
||||
async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise<LedgerDecision[]> {
|
||||
const keys = this.ledger.get(sourceId);
|
||||
return entries.map((entry) => {
|
||||
const prior = keys?.get(entry.record_key);
|
||||
return {
|
||||
record_key: entry.record_key,
|
||||
verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed",
|
||||
} as LedgerDecision;
|
||||
});
|
||||
}
|
||||
|
||||
async commitDeliveries(sourceId: string, _tenantId: string, entries: LedgerEntry[]): Promise<void> {
|
||||
let keys = this.ledger.get(sourceId);
|
||||
if (!keys) {
|
||||
keys = new Map<string, string>();
|
||||
this.ledger.set(sourceId, keys);
|
||||
}
|
||||
for (const entry of entries) keys.set(entry.record_key, entry.payload_hash);
|
||||
}
|
||||
|
||||
async countLedger(sourceId: string): Promise<number> {
|
||||
return this.ledger.get(sourceId)?.size ?? 0;
|
||||
}
|
||||
|
||||
async quarantine(record: QuarantineRecord): Promise<void> {
|
||||
const list = this.quarantined.get(record.source_id) ?? [];
|
||||
list.push(record);
|
||||
this.quarantined.set(record.source_id, list);
|
||||
}
|
||||
|
||||
async listQuarantine(sourceId: string, limit = 100): Promise<QuarantineRecord[]> {
|
||||
const list = this.quarantined.get(sourceId) ?? [];
|
||||
return [...list].reverse().slice(0, limit);
|
||||
}
|
||||
|
||||
async putSchema(schema: RecordSchema): Promise<void> {
|
||||
const list = this.schemas.get(schema.source_id) ?? [];
|
||||
const existing = list.findIndex((s) => s.schema_version === schema.schema_version);
|
||||
if (existing >= 0) list[existing] = { ...schema };
|
||||
else list.push({ ...schema });
|
||||
list.sort((a, b) => a.schema_version - b.schema_version);
|
||||
this.schemas.set(schema.source_id, list);
|
||||
}
|
||||
|
||||
async getSchema(sourceId: string, version: number): Promise<RecordSchema | undefined> {
|
||||
return this.schemas.get(sourceId)?.find((s) => s.schema_version === version);
|
||||
}
|
||||
|
||||
async getLatestSchema(sourceId: string): Promise<RecordSchema | undefined> {
|
||||
const list = this.schemas.get(sourceId);
|
||||
return list && list.length > 0 ? list[list.length - 1] : undefined;
|
||||
}
|
||||
|
||||
async listSchemas(sourceId: string): Promise<RecordSchema[]> {
|
||||
return [...(this.schemas.get(sourceId) ?? [])];
|
||||
}
|
||||
|
||||
async appendLineageEdges(edges: LineageEdge[]): Promise<void> {
|
||||
for (const edge of edges) this.lineage.set(edge.edge_id, { ...edge });
|
||||
}
|
||||
|
||||
async queryLineage(query: LineageQuery): Promise<LineageEdge[]> {
|
||||
const limit = query.limit ?? 200;
|
||||
return [...this.lineage.values()]
|
||||
.filter((e) => e.tenant_id === query.tenant_id)
|
||||
.filter((e) => (query.output_name ? e.output_name === query.output_name : true))
|
||||
.filter((e) => (query.input_name ? e.input_name === query.input_name : true))
|
||||
.sort((a, b) => b.event_time.localeCompare(a.event_time))
|
||||
.slice(0, limit);
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,13 @@
|
||||
* - 1]` for "most recent" and scanned backward for consecutive failures --
|
||||
* callers now get the same data pre-ordered instead).
|
||||
*/
|
||||
import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry";
|
||||
import type {
|
||||
SourceAuthorization,
|
||||
SourceHealthCheck,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
import type { SourceRegistryRepository } from "./source-registry-repository";
|
||||
|
||||
export class InMemorySourceRegistryRepository implements SourceRegistryRepository {
|
||||
@@ -18,6 +24,8 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor
|
||||
private readonly provenance = new Map<string, SourceProvenance>();
|
||||
private readonly healthChecks = new Map<string, SourceHealthCheck[]>();
|
||||
private readonly lastSuccess = new Map<string, string>();
|
||||
private readonly authorizations = new Map<string, SourceAuthorization>();
|
||||
private readonly schedules = new Map<string, SourceSchedule>();
|
||||
|
||||
async setRights(rights: SourceRights): Promise<void> {
|
||||
this.rights.set(rights.source_id, rights);
|
||||
@@ -43,6 +51,22 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor
|
||||
return this.lastSuccess.get(sourceId);
|
||||
}
|
||||
|
||||
async setAuthorization(auth: SourceAuthorization): Promise<void> {
|
||||
this.authorizations.set(auth.source_id, auth);
|
||||
}
|
||||
|
||||
async getAuthorization(sourceId: string): Promise<SourceAuthorization | undefined> {
|
||||
return this.authorizations.get(sourceId);
|
||||
}
|
||||
|
||||
async setSchedule(schedule: SourceSchedule): Promise<void> {
|
||||
this.schedules.set(schedule.source_id, schedule);
|
||||
}
|
||||
|
||||
async getSchedule(sourceId: string): Promise<SourceSchedule | undefined> {
|
||||
return this.schedules.get(sourceId);
|
||||
}
|
||||
|
||||
async appendHealthCheck(check: SourceHealthCheck): Promise<void> {
|
||||
const list = this.healthChecks.get(check.source_id) ?? [];
|
||||
list.push(check);
|
||||
|
||||
@@ -27,6 +27,7 @@ interface SourceRecord {
|
||||
created_at: string;
|
||||
updated_at: string | null;
|
||||
created_by: string | null;
|
||||
owner_id: string | null;
|
||||
}
|
||||
|
||||
export class InMemorySourceRepository implements SourceRepository {
|
||||
@@ -50,11 +51,16 @@ export class InMemorySourceRepository implements SourceRepository {
|
||||
adapter_id: input.adapter_id ?? null,
|
||||
access: input.access ?? null,
|
||||
description: input.description ?? null,
|
||||
status: input.status ?? "active",
|
||||
// P060: "pending", not "active". A source with no rights, no provenance
|
||||
// and no authorization used to be live the instant it was registered;
|
||||
// reaching "active" now goes through
|
||||
// SourceRegistryService.activateSource's classification gate.
|
||||
status: input.status ?? "pending",
|
||||
correlation_id: input.correlation_id ?? null,
|
||||
created_at: now,
|
||||
updated_at: null,
|
||||
created_by: input.created_by ?? null,
|
||||
owner_id: input.owner_id ?? null,
|
||||
};
|
||||
this.sources.set(record.source_id, record);
|
||||
return toSource(record);
|
||||
@@ -66,6 +72,14 @@ export class InMemorySourceRepository implements SourceRepository {
|
||||
return toSource(record);
|
||||
}
|
||||
|
||||
async updateStatus(sourceId: string, status: SourceStatus): Promise<Source> {
|
||||
const record = this.sources.get(sourceId);
|
||||
if (!record) throw new Error(`Source ${sourceId} not found`);
|
||||
record.status = status;
|
||||
record.updated_at = new Date().toISOString();
|
||||
return toSource(record);
|
||||
}
|
||||
|
||||
async list(filter?: SourceListFilter): Promise<Source[]> {
|
||||
let results = Array.from(this.sources.values());
|
||||
if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id);
|
||||
@@ -91,5 +105,6 @@ function toSource(r: SourceRecord): Source {
|
||||
created_at: r.created_at,
|
||||
updated_at: r.updated_at ?? undefined,
|
||||
created_by: r.created_by ?? undefined,
|
||||
owner_id: r.owner_id ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* src/repositories/ingestion-repository.ts
|
||||
* ==========================================
|
||||
* The storage seam for the P060-P068 ingestion plane: checkpoints (P060/P062),
|
||||
* the idempotency ledger (P068), quarantine (P064), the schema registry
|
||||
* (P060/P063) and the queryable lineage projection (P067).
|
||||
*
|
||||
* WHY ONE INTERFACE AND NOT FIVE. These five tables are written by ONE
|
||||
* caller — IngestionRunner — inside one run, and their correctness is
|
||||
* relative to each other (a checkpoint that advanced past records the ledger
|
||||
* never accepted is a silent gap; a quarantined record whose checkpoint moved
|
||||
* on is a silently dropped record). Splitting them into five factories would
|
||||
* let a future caller wire four of them to Postgres and one to memory and
|
||||
* still typecheck, which is exactly the failure this plane must not have.
|
||||
* They are chosen once, together, by one `resolveStoreMode` call.
|
||||
*
|
||||
* Same production posture as every other 0pt store: `resolveStoreMode` fails
|
||||
* hard in production with no DATABASE_URL rather than silently serving Maps
|
||||
* (src/platform/dependency-mode.ts, Master-DoD §3/§10).
|
||||
*/
|
||||
import { resolveStoreMode } from "../platform/dependency-mode";
|
||||
import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge";
|
||||
import type { RecordSchema } from "../schema/record-schema";
|
||||
import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion";
|
||||
import { getRepositorySql, repositoryDatabaseUrl } from "./db";
|
||||
import { InMemoryIngestionRepository } from "./in-memory-ingestion-repository";
|
||||
import { PostgresIngestionRepository } from "./postgres-ingestion-repository";
|
||||
|
||||
/** One record offered to the ledger. */
|
||||
export interface LedgerEntry {
|
||||
record_key: string;
|
||||
/** Stable hash of the payload — see hashPayload() in src/ingestion/ingestion-ledger.ts. */
|
||||
payload_hash: string;
|
||||
}
|
||||
|
||||
export interface IngestionRepository {
|
||||
// --- checkpoints (P060 registry field, P062 durable resume) --------------
|
||||
getCheckpoint(sourceId: string, partitionKey: string): Promise<IngestionCheckpoint | undefined>;
|
||||
commitCheckpoint(checkpoint: IngestionCheckpoint): Promise<void>;
|
||||
listCheckpoints(sourceId: string): Promise<IngestionCheckpoint[]>;
|
||||
|
||||
// --- idempotency ledger (P068) ------------------------------------------
|
||||
/**
|
||||
* READ-ONLY verdicts, in input order:
|
||||
* "new" — this record_key has never been accepted for this source,
|
||||
* "changed" — it has, but with different content (a real update),
|
||||
* "duplicate" — it has, with byte-identical content (a replay).
|
||||
*
|
||||
* Deliberately separate from {@link commitDeliveries}. An earlier draft of
|
||||
* this interface combined them into one atomic check-and-record, which was
|
||||
* WRONG in the way that matters: the runner has to consult the ledger
|
||||
* BEFORE the sink (to filter replays) and commit AFTER it (so a crash
|
||||
* between the two re-delivers rather than loses). A combined call marked a
|
||||
* record "delivered" that the sink never received, and the resumed run then
|
||||
* skipped it as a duplicate — permanent, silent data loss. That is not a
|
||||
* hypothetical: tests/p0/ingestion-runner.test.ts caught it, and the
|
||||
* "resumes with NO gap" case is what pins the fix.
|
||||
*/
|
||||
peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise<LedgerDecision[]>;
|
||||
|
||||
/**
|
||||
* Record that these records were successfully delivered. Called only after
|
||||
* the sink resolved. Idempotent: re-committing the same entry bumps the
|
||||
* delivery count and leaves the verdict unchanged.
|
||||
*/
|
||||
commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise<void>;
|
||||
|
||||
countLedger(sourceId: string): Promise<number>;
|
||||
|
||||
// --- quarantine (P064) --------------------------------------------------
|
||||
quarantine(record: QuarantineRecord): Promise<void>;
|
||||
listQuarantine(sourceId: string, limit?: number): Promise<QuarantineRecord[]>;
|
||||
|
||||
// --- schema registry (P060 registry field, P063 evolution) --------------
|
||||
putSchema(schema: RecordSchema): Promise<void>;
|
||||
getSchema(sourceId: string, version: number): Promise<RecordSchema | undefined>;
|
||||
getLatestSchema(sourceId: string): Promise<RecordSchema | undefined>;
|
||||
listSchemas(sourceId: string): Promise<RecordSchema[]>;
|
||||
|
||||
// --- lineage projection (P067) ------------------------------------------
|
||||
appendLineageEdges(edges: LineageEdge[]): Promise<void>;
|
||||
queryLineage(query: LineageQuery): Promise<LineageEdge[]>;
|
||||
}
|
||||
|
||||
export function createIngestionRepository(databaseUrl?: string): IngestionRepository {
|
||||
const mode = resolveStoreMode("0pt.ingestion", databaseUrl ?? repositoryDatabaseUrl());
|
||||
if (mode.mode === "postgres") {
|
||||
return new PostgresIngestionRepository(getRepositorySql(mode.url));
|
||||
}
|
||||
return new InMemoryIngestionRepository();
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
/**
|
||||
* src/repositories/postgres-ingestion-repository.ts
|
||||
* ==================================================
|
||||
* Real Postgres backing for IngestionRepository (sql/007_ingestion_plane.sql).
|
||||
*
|
||||
* THE LEDGER IS THE INTERESTING ONE (P068), and it is deliberately SPLIT:
|
||||
* `peekDeliveries` reads verdicts, `commitDeliveries` records them. The
|
||||
* runner peeks before the sink and commits after it, so a crash in between
|
||||
* re-delivers on resume instead of losing the record. An earlier version of
|
||||
* this file combined the two into one atomic check-and-record statement; that
|
||||
* was measurably wrong (see the interface's doc comment) and the crash/resume
|
||||
* test in tests/p0/ingestion-runner.test.ts is what proved it.
|
||||
*
|
||||
* KNOWN LIMIT, stated rather than papered over: the split means two runs on
|
||||
* the SAME (source_id, partition) can both peek "new" and both emit. The
|
||||
* delivery guarantee is therefore AT-LEAST-ONCE, which is what the
|
||||
* deterministic `envelope_id` (src/ingestion/envelope.ts) exists to make
|
||||
* survivable downstream. Exactly-once across concurrent runs needs a run lock
|
||||
* per (source, partition); there is none yet. That is a real open gap, not a
|
||||
* solved one — do not describe this plane as exactly-once.
|
||||
*/
|
||||
import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge";
|
||||
import type {
|
||||
CompatibilityPolicy,
|
||||
RecordSchema,
|
||||
RecordSchemaFields,
|
||||
} from "../schema/record-schema";
|
||||
import type {
|
||||
IngestionCheckpoint,
|
||||
IngestionEnvelope,
|
||||
LedgerDecision,
|
||||
QuarantineRecord,
|
||||
} from "../types/ingestion";
|
||||
import type { RepositorySql } from "./db";
|
||||
import type { IngestionRepository, LedgerEntry } from "./ingestion-repository";
|
||||
|
||||
/** postgres.js's `sql.json()` parameter type, named locally so the two JSONB columns can be passed without importing postgres' internals. */
|
||||
type JsonValue = Parameters<RepositorySql["json"]>[0];
|
||||
|
||||
function toIso(v: string | Date): string {
|
||||
return v instanceof Date ? v.toISOString() : v;
|
||||
}
|
||||
|
||||
interface CheckpointRow {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
partition_key: string;
|
||||
cursor: string | null;
|
||||
sequence: string | number;
|
||||
records_committed: string | number;
|
||||
committed_at: string | Date;
|
||||
}
|
||||
|
||||
function toCheckpoint(r: CheckpointRow): IngestionCheckpoint {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
partition_key: r.partition_key,
|
||||
cursor: r.cursor ?? undefined,
|
||||
sequence: Number(r.sequence),
|
||||
records_committed: Number(r.records_committed),
|
||||
committed_at: toIso(r.committed_at),
|
||||
};
|
||||
}
|
||||
|
||||
export class PostgresIngestionRepository implements IngestionRepository {
|
||||
constructor(private readonly sql: RepositorySql) {}
|
||||
|
||||
// --- checkpoints ---------------------------------------------------------
|
||||
|
||||
async getCheckpoint(sourceId: string, partitionKey: string): Promise<IngestionCheckpoint | undefined> {
|
||||
const rows = await this.sql<CheckpointRow[]>`
|
||||
SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at
|
||||
FROM opt_ingestion_checkpoints
|
||||
WHERE source_id = ${sourceId} AND partition_key = ${partitionKey}
|
||||
LIMIT 1`;
|
||||
return rows[0] ? toCheckpoint(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async commitCheckpoint(cp: IngestionCheckpoint): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_ingestion_checkpoints
|
||||
(source_id, partition_key, tenant_id, cursor, sequence, records_committed, committed_at)
|
||||
VALUES
|
||||
(${cp.source_id}, ${cp.partition_key}, ${cp.tenant_id}, ${cp.cursor ?? null},
|
||||
${cp.sequence}, ${cp.records_committed}, ${cp.committed_at})
|
||||
ON CONFLICT (source_id, partition_key) DO UPDATE
|
||||
SET cursor = EXCLUDED.cursor,
|
||||
sequence = EXCLUDED.sequence,
|
||||
records_committed = EXCLUDED.records_committed,
|
||||
committed_at = EXCLUDED.committed_at,
|
||||
tenant_id = EXCLUDED.tenant_id`;
|
||||
}
|
||||
|
||||
async listCheckpoints(sourceId: string): Promise<IngestionCheckpoint[]> {
|
||||
const rows = await this.sql<CheckpointRow[]>`
|
||||
SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at
|
||||
FROM opt_ingestion_checkpoints
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY partition_key ASC`;
|
||||
return rows.map(toCheckpoint);
|
||||
}
|
||||
|
||||
// --- ledger --------------------------------------------------------------
|
||||
|
||||
async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise<LedgerDecision[]> {
|
||||
if (entries.length === 0) return [];
|
||||
const keys = entries.map((e) => e.record_key);
|
||||
const rows = await this.sql<{ record_key: string; payload_hash: string }[]>`
|
||||
SELECT record_key, payload_hash FROM opt_ingestion_ledger
|
||||
WHERE source_id = ${sourceId} AND record_key = ANY(${this.sql.array(keys)})`;
|
||||
const priorByKey = new Map(rows.map((r) => [r.record_key, r.payload_hash]));
|
||||
return entries.map((entry) => {
|
||||
const prior = priorByKey.get(entry.record_key);
|
||||
return {
|
||||
record_key: entry.record_key,
|
||||
verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed",
|
||||
} as LedgerDecision;
|
||||
});
|
||||
}
|
||||
|
||||
async commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise<void> {
|
||||
for (const entry of entries) {
|
||||
await this.sql`
|
||||
INSERT INTO opt_ingestion_ledger (source_id, record_key, tenant_id, payload_hash)
|
||||
VALUES (${sourceId}, ${entry.record_key}, ${tenantId}, ${entry.payload_hash})
|
||||
ON CONFLICT (source_id, record_key) DO UPDATE
|
||||
SET last_seen_at = now(),
|
||||
delivery_count = opt_ingestion_ledger.delivery_count + 1,
|
||||
payload_hash = EXCLUDED.payload_hash`;
|
||||
}
|
||||
}
|
||||
|
||||
async countLedger(sourceId: string): Promise<number> {
|
||||
const rows = await this.sql<{ n: string }[]>`
|
||||
SELECT count(*)::text AS n FROM opt_ingestion_ledger WHERE source_id = ${sourceId}`;
|
||||
return Number(rows[0]?.n ?? 0);
|
||||
}
|
||||
|
||||
// --- quarantine ----------------------------------------------------------
|
||||
|
||||
async quarantine(record: QuarantineRecord): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_quality_quarantine
|
||||
(quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at)
|
||||
VALUES
|
||||
(${record.quarantine_id}, ${record.source_id}, ${record.tenant_id}, ${record.run_id},
|
||||
${record.record_key}, ${this.sql.json(record.violations as unknown as JsonValue)},
|
||||
${this.sql.json(record.envelope as unknown as JsonValue)},
|
||||
${record.quarantined_at})
|
||||
ON CONFLICT (quarantine_id) DO NOTHING`;
|
||||
}
|
||||
|
||||
async listQuarantine(sourceId: string, limit = 100): Promise<QuarantineRecord[]> {
|
||||
const rows = await this.sql<
|
||||
{
|
||||
quarantine_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
record_key: string;
|
||||
violations: QuarantineRecord["violations"];
|
||||
envelope: IngestionEnvelope;
|
||||
quarantined_at: string | Date;
|
||||
}[]
|
||||
>`
|
||||
SELECT quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at
|
||||
FROM opt_quality_quarantine
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY quarantined_at DESC, quarantine_id DESC
|
||||
LIMIT ${limit}`;
|
||||
return rows.map((r) => ({
|
||||
quarantine_id: r.quarantine_id,
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
run_id: r.run_id,
|
||||
record_key: r.record_key,
|
||||
violations: r.violations,
|
||||
envelope: r.envelope,
|
||||
quarantined_at: toIso(r.quarantined_at),
|
||||
}));
|
||||
}
|
||||
|
||||
// --- schema registry -----------------------------------------------------
|
||||
|
||||
async putSchema(schema: RecordSchema): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_schemas
|
||||
(source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by)
|
||||
VALUES
|
||||
(${schema.source_id}, ${schema.schema_version}, ${schema.tenant_id},
|
||||
${this.sql.json(schema.fields as unknown as JsonValue)}, ${schema.compatibility},
|
||||
${schema.registered_at}, ${schema.registered_by ?? null})
|
||||
ON CONFLICT (source_id, schema_version) DO UPDATE
|
||||
SET fields = EXCLUDED.fields,
|
||||
compatibility = EXCLUDED.compatibility,
|
||||
registered_at = EXCLUDED.registered_at,
|
||||
registered_by = EXCLUDED.registered_by`;
|
||||
}
|
||||
|
||||
async getSchema(sourceId: string, version: number): Promise<RecordSchema | undefined> {
|
||||
const rows = await this.sql<SchemaRow[]>`
|
||||
SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by
|
||||
FROM opt_source_schemas
|
||||
WHERE source_id = ${sourceId} AND schema_version = ${version}
|
||||
LIMIT 1`;
|
||||
return rows[0] ? toSchema(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async getLatestSchema(sourceId: string): Promise<RecordSchema | undefined> {
|
||||
const rows = await this.sql<SchemaRow[]>`
|
||||
SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by
|
||||
FROM opt_source_schemas
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY schema_version DESC
|
||||
LIMIT 1`;
|
||||
return rows[0] ? toSchema(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async listSchemas(sourceId: string): Promise<RecordSchema[]> {
|
||||
const rows = await this.sql<SchemaRow[]>`
|
||||
SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by
|
||||
FROM opt_source_schemas
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY schema_version ASC`;
|
||||
return rows.map(toSchema);
|
||||
}
|
||||
|
||||
// --- lineage -------------------------------------------------------------
|
||||
|
||||
async appendLineageEdges(edges: LineageEdge[]): Promise<void> {
|
||||
for (const e of edges) {
|
||||
await this.sql`
|
||||
INSERT INTO opt_lineage_edges
|
||||
(edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref,
|
||||
output_namespace, output_name, output_ref, event_time, correlation_id)
|
||||
VALUES
|
||||
(${e.edge_id}, ${e.tenant_id}, ${e.run_id}, ${e.capability_id}, ${e.input_namespace},
|
||||
${e.input_name}, ${e.input_ref ?? null}, ${e.output_namespace}, ${e.output_name},
|
||||
${e.output_ref ?? null}, ${e.event_time}, ${e.correlation_id ?? null})
|
||||
ON CONFLICT (edge_id) DO NOTHING`;
|
||||
}
|
||||
}
|
||||
|
||||
async queryLineage(query: LineageQuery): Promise<LineageEdge[]> {
|
||||
const limit = query.limit ?? 200;
|
||||
const rows = await this.sql<LineageRow[]>`
|
||||
SELECT edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref,
|
||||
output_namespace, output_name, output_ref, event_time, correlation_id
|
||||
FROM opt_lineage_edges
|
||||
WHERE tenant_id = ${query.tenant_id}
|
||||
${query.output_name ? this.sql`AND output_name = ${query.output_name}` : this.sql``}
|
||||
${query.input_name ? this.sql`AND input_name = ${query.input_name}` : this.sql``}
|
||||
ORDER BY event_time DESC, edge_id DESC
|
||||
LIMIT ${limit}`;
|
||||
return rows.map((r) => ({
|
||||
edge_id: r.edge_id,
|
||||
tenant_id: r.tenant_id,
|
||||
run_id: r.run_id,
|
||||
capability_id: r.capability_id,
|
||||
input_namespace: r.input_namespace,
|
||||
input_name: r.input_name,
|
||||
input_ref: r.input_ref ?? undefined,
|
||||
output_namespace: r.output_namespace,
|
||||
output_name: r.output_name,
|
||||
output_ref: r.output_ref ?? undefined,
|
||||
event_time: toIso(r.event_time),
|
||||
correlation_id: r.correlation_id ?? undefined,
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
interface SchemaRow {
|
||||
source_id: string;
|
||||
schema_version: number;
|
||||
tenant_id: string;
|
||||
fields: RecordSchemaFields;
|
||||
compatibility: string;
|
||||
registered_at: string | Date;
|
||||
registered_by: string | null;
|
||||
}
|
||||
|
||||
function toSchema(r: SchemaRow): RecordSchema {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
schema_version: Number(r.schema_version),
|
||||
tenant_id: r.tenant_id,
|
||||
fields: r.fields,
|
||||
compatibility: r.compatibility as CompatibilityPolicy,
|
||||
registered_at: toIso(r.registered_at),
|
||||
registered_by: r.registered_by ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
interface LineageRow {
|
||||
edge_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
capability_id: string;
|
||||
input_namespace: string;
|
||||
input_name: string;
|
||||
input_ref: string | null;
|
||||
output_namespace: string;
|
||||
output_name: string;
|
||||
output_ref: string | null;
|
||||
event_time: string | Date;
|
||||
correlation_id: string | null;
|
||||
}
|
||||
@@ -10,10 +10,13 @@
|
||||
* is a single upserted "last success" row per source.
|
||||
*/
|
||||
import type {
|
||||
AuthorizationBasis,
|
||||
SourceAuthorization,
|
||||
SourceHealthCheck,
|
||||
SourceHealthStatus,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
import type { RepositorySql } from "./db";
|
||||
import type { SourceRegistryRepository } from "./source-registry-repository";
|
||||
@@ -167,6 +170,57 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor
|
||||
return rows[0] ? toIsoString(rows[0].last_success_at) : undefined;
|
||||
}
|
||||
|
||||
// --- P060 authorization / schedule ---------------------------------------
|
||||
|
||||
async setAuthorization(auth: SourceAuthorization): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_authorization
|
||||
(source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes)
|
||||
VALUES
|
||||
(${auth.source_id}, ${auth.tenant_id}, ${auth.basis}, ${auth.granted_by}, ${auth.granted_at},
|
||||
${auth.expires_at ?? null}, ${auth.evidence_ref ?? null}, ${auth.notes ?? null})
|
||||
ON CONFLICT (source_id) DO UPDATE
|
||||
SET tenant_id = EXCLUDED.tenant_id,
|
||||
basis = EXCLUDED.basis,
|
||||
granted_by = EXCLUDED.granted_by,
|
||||
granted_at = EXCLUDED.granted_at,
|
||||
expires_at = EXCLUDED.expires_at,
|
||||
evidence_ref = EXCLUDED.evidence_ref,
|
||||
notes = EXCLUDED.notes,
|
||||
updated_at = now()`;
|
||||
}
|
||||
|
||||
async getAuthorization(sourceId: string): Promise<SourceAuthorization | undefined> {
|
||||
const rows = await this.sql<AuthorizationRow[]>`
|
||||
SELECT source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes
|
||||
FROM opt_source_authorization WHERE source_id = ${sourceId} LIMIT 1`;
|
||||
return rows[0] ? toAuthorization(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async setSchedule(schedule: SourceSchedule): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_schedule
|
||||
(source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at)
|
||||
VALUES
|
||||
(${schedule.source_id}, ${schedule.tenant_id}, ${schedule.mode},
|
||||
${schedule.interval_seconds ?? null}, ${schedule.cron ?? null}, ${schedule.enabled},
|
||||
${schedule.updated_at})
|
||||
ON CONFLICT (source_id) DO UPDATE
|
||||
SET tenant_id = EXCLUDED.tenant_id,
|
||||
mode = EXCLUDED.mode,
|
||||
interval_seconds = EXCLUDED.interval_seconds,
|
||||
cron = EXCLUDED.cron,
|
||||
enabled = EXCLUDED.enabled,
|
||||
updated_at = EXCLUDED.updated_at`;
|
||||
}
|
||||
|
||||
async getSchedule(sourceId: string): Promise<SourceSchedule | undefined> {
|
||||
const rows = await this.sql<ScheduleRow[]>`
|
||||
SELECT source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at
|
||||
FROM opt_source_schedule WHERE source_id = ${sourceId} LIMIT 1`;
|
||||
return rows[0] ? toSchedule(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async appendHealthCheck(check: SourceHealthCheck): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_health_checks (source_id, tenant_id, status, checked_at, latency_ms, detail)
|
||||
@@ -195,3 +249,49 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor
|
||||
return rows.map(toHealthCheck);
|
||||
}
|
||||
}
|
||||
|
||||
interface AuthorizationRow {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
basis: string;
|
||||
granted_by: string;
|
||||
granted_at: string | Date;
|
||||
expires_at: string | Date | null;
|
||||
evidence_ref: string | null;
|
||||
notes: string | null;
|
||||
}
|
||||
|
||||
function toAuthorization(r: AuthorizationRow): SourceAuthorization {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
basis: r.basis as AuthorizationBasis,
|
||||
granted_by: r.granted_by,
|
||||
granted_at: toIsoString(r.granted_at),
|
||||
expires_at: r.expires_at ? toIsoString(r.expires_at) : undefined,
|
||||
evidence_ref: r.evidence_ref ?? undefined,
|
||||
notes: r.notes ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
interface ScheduleRow {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
mode: string;
|
||||
interval_seconds: number | string | null;
|
||||
cron: string | null;
|
||||
enabled: boolean;
|
||||
updated_at: string | Date;
|
||||
}
|
||||
|
||||
function toSchedule(r: ScheduleRow): SourceSchedule {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
mode: r.mode as SourceSchedule["mode"],
|
||||
interval_seconds: r.interval_seconds === null ? undefined : Number(r.interval_seconds),
|
||||
cron: r.cron ?? undefined,
|
||||
enabled: r.enabled,
|
||||
updated_at: toIsoString(r.updated_at),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@ interface SourceRow {
|
||||
created_at: string | Date;
|
||||
updated_at: string | Date | null;
|
||||
created_by: string | null;
|
||||
owner_id: string | null;
|
||||
}
|
||||
|
||||
function toIsoString(v: string | Date): string {
|
||||
@@ -46,6 +47,7 @@ function toSource(r: SourceRow): Source {
|
||||
created_at: toIsoString(r.created_at),
|
||||
updated_at: r.updated_at ? toIsoString(r.updated_at) : undefined,
|
||||
created_by: r.created_by ?? undefined,
|
||||
owner_id: r.owner_id ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -63,29 +65,41 @@ export class PostgresSourceRepository implements SourceRepository {
|
||||
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
INSERT INTO opt_sources
|
||||
(source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by)
|
||||
(source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by, owner_id)
|
||||
VALUES
|
||||
(${input.source_id}, ${input.tenant_id}, ${input.project_id}, ${input.source_type}, ${input.name},
|
||||
${input.adapter_id ?? null}, ${input.access ?? null}, ${input.description ?? null},
|
||||
${input.status ?? "active"}, ${input.correlation_id ?? null}, ${input.created_by ?? null})
|
||||
${input.status ?? "pending"}, ${input.correlation_id ?? null}, ${input.created_by ?? null},
|
||||
${input.owner_id ?? null})
|
||||
RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by`;
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id`;
|
||||
return toSource(rows[0]);
|
||||
}
|
||||
|
||||
async get(sourceId: string): Promise<Source> {
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id
|
||||
FROM opt_sources WHERE source_id = ${sourceId} LIMIT 1`;
|
||||
if (!rows[0]) throw new Error(`Source ${sourceId} not found`);
|
||||
return toSource(rows[0]);
|
||||
}
|
||||
|
||||
async updateStatus(sourceId: string, status: SourceStatus): Promise<Source> {
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
UPDATE opt_sources
|
||||
SET status = ${status}, updated_at = now()
|
||||
WHERE source_id = ${sourceId}
|
||||
RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id`;
|
||||
if (!rows[0]) throw new Error(`Source ${sourceId} not found`);
|
||||
return toSource(rows[0]);
|
||||
}
|
||||
|
||||
async list(filter?: SourceListFilter): Promise<Source[]> {
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id
|
||||
FROM opt_sources
|
||||
WHERE 1=1
|
||||
${filter?.tenant_id ? this.sql`AND tenant_id = ${filter.tenant_id}` : this.sql``}
|
||||
|
||||
@@ -16,7 +16,13 @@
|
||||
* layer trusts the caller, same as the Maps did.
|
||||
*/
|
||||
import { resolveStoreMode } from "../platform/dependency-mode";
|
||||
import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry";
|
||||
import type {
|
||||
SourceAuthorization,
|
||||
SourceHealthCheck,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
import { getRepositorySql, repositoryDatabaseUrl } from "./db";
|
||||
import { InMemorySourceRegistryRepository } from "./in-memory-source-registry-repository";
|
||||
import { PostgresSourceRegistryRepository } from "./postgres-source-registry-repository";
|
||||
@@ -31,6 +37,14 @@ export interface SourceRegistryRepository {
|
||||
recordSuccess(sourceId: string, tenantId: string, atIso: string): Promise<void>;
|
||||
getLastSuccess(sourceId: string): Promise<string | undefined>;
|
||||
|
||||
/** P060 authorization — the recorded decision that 0pt may connect at all. */
|
||||
setAuthorization(auth: SourceAuthorization): Promise<void>;
|
||||
getAuthorization(sourceId: string): Promise<SourceAuthorization | undefined>;
|
||||
|
||||
/** P060 schedule — ingestion mode + cadence. */
|
||||
setSchedule(schedule: SourceSchedule): Promise<void>;
|
||||
getSchedule(sourceId: string): Promise<SourceSchedule | undefined>;
|
||||
|
||||
appendHealthCheck(check: SourceHealthCheck): Promise<void>;
|
||||
/** Newest-first, bounded to the latest `limit` observations (default 100) -- health is an operational signal, not an audit log; AUD0 owns history. */
|
||||
listHealthChecks(sourceId: string, limit?: number): Promise<SourceHealthCheck[]>;
|
||||
|
||||
@@ -33,6 +33,12 @@ export interface SourceRepository {
|
||||
/** Throws (message contains "not found") when no such source exists -- matches the pre-migration Map behavior every caller already handles. */
|
||||
get(sourceId: string): Promise<Source>;
|
||||
list(filter?: SourceListFilter): Promise<Source[]>;
|
||||
/**
|
||||
* P060: the only way a source's lifecycle status changes. Deliberately NOT
|
||||
* a general `update` -- status is the field the activation gate protects,
|
||||
* and a generic patch method would be the obvious way around it.
|
||||
*/
|
||||
updateStatus(sourceId: string, status: SourceStatus): Promise<Source>;
|
||||
}
|
||||
|
||||
export function createSourceRepository(databaseUrl?: string): SourceRepository {
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* src/schema/compatibility.ts
|
||||
* ============================
|
||||
* P063: the pure comparison. `checkCompatibility(previous, next, policy)`
|
||||
* returns every way `next` breaks readers/writers of `previous` under the
|
||||
* declared policy. No IO, no throwing — the caller decides what a break
|
||||
* means (SchemaRegistry refuses the registration; the ingestion runner
|
||||
* refuses the RUN, before a single envelope reaches a consumer).
|
||||
*/
|
||||
import type {
|
||||
CompatibilityPolicy,
|
||||
CompatibilityVerdict,
|
||||
RecordSchemaFields,
|
||||
SchemaBreak,
|
||||
} from "./record-schema";
|
||||
|
||||
/**
|
||||
* BACKWARD breaks: a consumer written against `next` reading data produced
|
||||
* under `previous` fails when
|
||||
* - `next` adds a REQUIRED field `previous` never produced, or
|
||||
* - `next` changes a field's type (old data has the old type), or
|
||||
* - `next` drops a field that was REQUIRED in `previous` and is therefore
|
||||
* part of the contract consumers rely on.
|
||||
*/
|
||||
function backwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] {
|
||||
const breaks: SchemaBreak[] = [];
|
||||
for (const [field, spec] of Object.entries(next)) {
|
||||
const before = previous[field];
|
||||
if (!before) {
|
||||
if (spec.required) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "backward",
|
||||
detail: `new required field "${field}" is absent from every record written under the previous version`,
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (before.type !== spec.type) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "backward",
|
||||
detail: `field "${field}" changed type ${before.type} -> ${spec.type}; existing data still carries ${before.type}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
for (const [field, before] of Object.entries(previous)) {
|
||||
if (!next[field] && before.required) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "backward",
|
||||
detail: `required field "${field}" was removed; consumers of the previous version still depend on it`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return breaks;
|
||||
}
|
||||
|
||||
/**
|
||||
* FORWARD breaks: a consumer written against `previous` reading data produced
|
||||
* under `next` fails when
|
||||
* - `next` removes a field `previous` marked REQUIRED, or
|
||||
* - `next` changes a field's type.
|
||||
* Adding a required field is forward-SAFE (old readers ignore it).
|
||||
*/
|
||||
function forwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] {
|
||||
const breaks: SchemaBreak[] = [];
|
||||
for (const [field, before] of Object.entries(previous)) {
|
||||
const after = next[field];
|
||||
if (!after) {
|
||||
if (before.required) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "forward",
|
||||
detail: `required field "${field}" was removed; readers pinned to the previous version will not find it`,
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (before.type !== after.type) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "forward",
|
||||
detail: `field "${field}" changed type ${before.type} -> ${after.type}; readers pinned to the previous version expect ${before.type}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return breaks;
|
||||
}
|
||||
|
||||
/** Deduplicate identical (field, direction, detail) triples produced by both directions under `full`. */
|
||||
function dedupe(breaks: SchemaBreak[]): SchemaBreak[] {
|
||||
const seen = new Set<string>();
|
||||
const out: SchemaBreak[] = [];
|
||||
for (const b of breaks) {
|
||||
const key = `${b.field}|${b.breaks}|${b.detail}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
out.push(b);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function checkCompatibility(
|
||||
previous: RecordSchemaFields,
|
||||
next: RecordSchemaFields,
|
||||
policy: CompatibilityPolicy,
|
||||
): CompatibilityVerdict {
|
||||
if (policy === "none") {
|
||||
return { compatible: true, policy, breaks: [] };
|
||||
}
|
||||
const breaks: SchemaBreak[] = [];
|
||||
if (policy === "backward" || policy === "full") breaks.push(...backwardBreaks(previous, next));
|
||||
if (policy === "forward" || policy === "full") breaks.push(...forwardBreaks(previous, next));
|
||||
const deduped = dedupe(breaks);
|
||||
return { compatible: deduped.length === 0, policy, breaks: deduped };
|
||||
}
|
||||
|
||||
/**
|
||||
* Does a concrete record conform to a schema? Used by the ingestion runner to
|
||||
* detect that a source has SILENTLY changed shape — i.e. it is emitting data
|
||||
* the registered schema does not describe — which is the case P063 exists to
|
||||
* catch before consumers see it. Returns the problems; empty = conforming.
|
||||
*/
|
||||
export function conformanceErrors(
|
||||
fields: RecordSchemaFields,
|
||||
payload: Record<string, unknown>,
|
||||
): string[] {
|
||||
const errors: string[] = [];
|
||||
for (const [field, spec] of Object.entries(fields)) {
|
||||
const value = payload[field];
|
||||
if (value === undefined || value === null) {
|
||||
if (spec.required) errors.push(`required field "${field}" is missing`);
|
||||
continue;
|
||||
}
|
||||
if (!matchesType(value, spec.type)) {
|
||||
errors.push(`field "${field}" is ${describe(value)}, expected ${spec.type}`);
|
||||
}
|
||||
}
|
||||
for (const field of Object.keys(payload)) {
|
||||
if (!fields[field]) {
|
||||
errors.push(`unregistered field "${field}" appeared in the payload`);
|
||||
}
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
function matchesType(value: unknown, type: string): boolean {
|
||||
switch (type) {
|
||||
case "string":
|
||||
return typeof value === "string";
|
||||
case "number":
|
||||
return typeof value === "number" && Number.isFinite(value);
|
||||
case "boolean":
|
||||
return typeof value === "boolean";
|
||||
case "timestamp":
|
||||
return typeof value === "string" && !Number.isNaN(Date.parse(value));
|
||||
case "array":
|
||||
return Array.isArray(value);
|
||||
case "object":
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function describe(value: unknown): string {
|
||||
if (Array.isArray(value)) return "array";
|
||||
return typeof value;
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* src/schema/record-schema.ts
|
||||
* ============================
|
||||
* P063: the schema 0pt versions per source, and the compatibility policy that
|
||||
* governs what the next version may change.
|
||||
*
|
||||
* Deliberately a SMALL structural schema (field name -> type + required), not
|
||||
* a second JSON-Schema dialect: 0pt already ships JSON Schema in
|
||||
* packages/schemas/ for its own contracts, and Zod in src/validation for HTTP
|
||||
* bodies. Neither of those answers the question P063 actually asks — "is
|
||||
* version N+1 of an UPSTREAM source's record shape safe for the consumers
|
||||
* reading version N?" — which needs a comparable, field-level structure, not
|
||||
* a validator. Anything richer belongs in a real schema registry
|
||||
* (Confluent/Apicurio) and would be an integration, not a rewrite of this.
|
||||
*/
|
||||
|
||||
export type FieldType = "string" | "number" | "boolean" | "timestamp" | "object" | "array";
|
||||
|
||||
export interface FieldSpec {
|
||||
type: FieldType;
|
||||
required: boolean;
|
||||
}
|
||||
|
||||
/** A record shape: field name -> spec. */
|
||||
export type RecordSchemaFields = Record<string, FieldSpec>;
|
||||
|
||||
/**
|
||||
* Compatibility policy, using the standard schema-registry vocabulary:
|
||||
*
|
||||
* - `backward` — new readers can read data written under the OLD schema.
|
||||
* Safe: adding an optional field, removing an optional field, relaxing a
|
||||
* required field to optional. Breaking: adding a REQUIRED field, removing
|
||||
* a field an old writer still emits as required, changing a type.
|
||||
* - `forward` — old readers can read data written under the NEW schema.
|
||||
* Safe: adding a required field, removing an optional field. Breaking:
|
||||
* removing a REQUIRED field, changing a type.
|
||||
* - `full` — both of the above.
|
||||
* - `none` — no check. Recorded, never silently assumed: a source must
|
||||
* declare `none` explicitly, and the runner logs it as an accepted risk.
|
||||
*/
|
||||
export type CompatibilityPolicy = "backward" | "forward" | "full" | "none";
|
||||
|
||||
export interface RecordSchema {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
schema_version: number;
|
||||
fields: RecordSchemaFields;
|
||||
compatibility: CompatibilityPolicy;
|
||||
registered_at: string;
|
||||
registered_by?: string;
|
||||
}
|
||||
|
||||
/** One incompatibility between two schema versions. */
|
||||
export interface SchemaBreak {
|
||||
field: string;
|
||||
/** Which direction the change breaks. */
|
||||
breaks: "backward" | "forward";
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface CompatibilityVerdict {
|
||||
compatible: boolean;
|
||||
policy: CompatibilityPolicy;
|
||||
breaks: SchemaBreak[];
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
/**
|
||||
* src/schema/schema-registry.ts
|
||||
* ==============================
|
||||
* P063: versioning + compatibility policy for a source's record shape.
|
||||
*
|
||||
* The DoD is "a breaking schema is detected BEFORE consumer corruption", so
|
||||
* this registry is placed on the ingestion path in two places, both of them
|
||||
* upstream of any consumer write:
|
||||
*
|
||||
* 1. `registerFromDiscovery` runs when the runner calls connector.discover(),
|
||||
* i.e. before the first read. A shape change that breaks the declared
|
||||
* policy makes the RUN fail with the field-level reasons — no envelope is
|
||||
* produced at all.
|
||||
* 2. `conformanceErrors` (src/schema/compatibility.ts) runs per envelope, so
|
||||
* a source that changes shape mid-stream without changing its declared
|
||||
* schema is caught on the offending record rather than on the next
|
||||
* discovery, days later.
|
||||
*
|
||||
* A source whose policy is "none" opts out of (1) — explicitly, recorded in
|
||||
* the registry row, never by default.
|
||||
*/
|
||||
import { SiaxError } from "../platform/errors";
|
||||
import type { IngestionRepository } from "../repositories/ingestion-repository";
|
||||
import { checkCompatibility } from "./compatibility";
|
||||
import type {
|
||||
CompatibilityPolicy,
|
||||
CompatibilityVerdict,
|
||||
RecordSchema,
|
||||
RecordSchemaFields,
|
||||
} from "./record-schema";
|
||||
|
||||
export interface RegisterSchemaInput {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
fields: RecordSchemaFields;
|
||||
/** Policy governing THIS and future versions. Required — there is no default. */
|
||||
compatibility: CompatibilityPolicy;
|
||||
registered_by?: string;
|
||||
}
|
||||
|
||||
export interface RegisterSchemaResult {
|
||||
schema: RecordSchema;
|
||||
/** True when the shape was identical to the current version and no new version was created. */
|
||||
unchanged: boolean;
|
||||
/** Present when a comparison ran (i.e. a previous version existed). */
|
||||
verdict?: CompatibilityVerdict;
|
||||
}
|
||||
|
||||
/** Stable comparison of two field maps, order-independent. */
|
||||
function sameFields(a: RecordSchemaFields, b: RecordSchemaFields): boolean {
|
||||
const ka = Object.keys(a).sort();
|
||||
const kb = Object.keys(b).sort();
|
||||
if (ka.length !== kb.length) return false;
|
||||
for (let i = 0; i < ka.length; i++) {
|
||||
if (ka[i] !== kb[i]) return false;
|
||||
const fa = a[ka[i]];
|
||||
const fb = b[kb[i]];
|
||||
if (fa.type !== fb.type || fa.required !== fb.required) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export class SchemaRegistry {
|
||||
constructor(private readonly repo: IngestionRepository) {}
|
||||
|
||||
async getLatest(sourceId: string): Promise<RecordSchema | undefined> {
|
||||
return this.repo.getLatestSchema(sourceId);
|
||||
}
|
||||
|
||||
async list(sourceId: string): Promise<RecordSchema[]> {
|
||||
return this.repo.listSchemas(sourceId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a shape. Throws `SiaxError("CONFLICT")` — naming every break —
|
||||
* when the new shape violates the previous version's declared policy.
|
||||
* Nothing is persisted when it throws.
|
||||
*/
|
||||
async register(input: RegisterSchemaInput): Promise<RegisterSchemaResult> {
|
||||
if (Object.keys(input.fields).length === 0) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.schema_registry", "a schema must declare at least one field");
|
||||
}
|
||||
const previous = await this.repo.getLatestSchema(input.source_id);
|
||||
if (!previous) {
|
||||
const schema: RecordSchema = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
schema_version: 1,
|
||||
fields: input.fields,
|
||||
compatibility: input.compatibility,
|
||||
registered_at: new Date().toISOString(),
|
||||
registered_by: input.registered_by,
|
||||
};
|
||||
await this.repo.putSchema(schema);
|
||||
return { schema, unchanged: false };
|
||||
}
|
||||
|
||||
if (sameFields(previous.fields, input.fields) && previous.compatibility === input.compatibility) {
|
||||
return { schema: previous, unchanged: true };
|
||||
}
|
||||
|
||||
// The PREVIOUS version's policy governs whether this change is allowed —
|
||||
// a producer must not be able to widen its own policy retroactively by
|
||||
// declaring "none" on the breaking version.
|
||||
const verdict = checkCompatibility(previous.fields, input.fields, previous.compatibility);
|
||||
if (!verdict.compatible) {
|
||||
throw new SiaxError(
|
||||
"CONFLICT",
|
||||
"0pt.schema_registry",
|
||||
`schema v${previous.schema_version + 1} for source ${input.source_id} breaks the declared ` +
|
||||
`"${previous.compatibility}" compatibility policy: ` +
|
||||
verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "),
|
||||
);
|
||||
}
|
||||
|
||||
const schema: RecordSchema = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
schema_version: previous.schema_version + 1,
|
||||
fields: input.fields,
|
||||
compatibility: input.compatibility,
|
||||
registered_at: new Date().toISOString(),
|
||||
registered_by: input.registered_by,
|
||||
};
|
||||
await this.repo.putSchema(schema);
|
||||
return { schema, unchanged: false, verdict };
|
||||
}
|
||||
|
||||
/**
|
||||
* Non-throwing form used by the runner's pre-flight: answers whether the
|
||||
* shape a connector just discovered would be accepted, without registering
|
||||
* it. Used to report a break as a RUN failure with a full reason list
|
||||
* rather than an exception message.
|
||||
*/
|
||||
async evaluate(
|
||||
sourceId: string,
|
||||
fields: RecordSchemaFields,
|
||||
): Promise<{ previous?: RecordSchema; verdict?: CompatibilityVerdict; changed: boolean }> {
|
||||
const previous = await this.repo.getLatestSchema(sourceId);
|
||||
if (!previous) return { changed: true };
|
||||
if (sameFields(previous.fields, fields)) return { previous, changed: false };
|
||||
return { previous, verdict: checkCompatibility(previous.fields, fields, previous.compatibility), changed: true };
|
||||
}
|
||||
}
|
||||
@@ -44,6 +44,8 @@ import { SourceService, type SourceServiceOptions } from "./source-service";
|
||||
import { SourceRatePolicyService } from "./source-rate-policy-service";
|
||||
import type { CreateSourceInput, Source } from "../types/source";
|
||||
import type {
|
||||
SourceAuthorization,
|
||||
SourceClassification,
|
||||
SourceDescriptor,
|
||||
SourceFreshness,
|
||||
SourceHealth,
|
||||
@@ -51,13 +53,24 @@ import type {
|
||||
SourceHealthStatus,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
|
||||
export interface RegisterSourceInput extends CreateSourceInput {
|
||||
rights?: Omit<SourceRights, "source_id" | "tenant_id">;
|
||||
provenance?: Omit<SourceProvenance, "source_id" | "tenant_id" | "updated_at">;
|
||||
authorization?: Omit<SourceAuthorization, "source_id" | "tenant_id">;
|
||||
schedule?: Omit<SourceSchedule, "source_id" | "tenant_id" | "updated_at">;
|
||||
}
|
||||
|
||||
/**
|
||||
* Actor ids that may never appear as an authorization's `granted_by`.
|
||||
* P060 requires an accountable HUMAN decision behind a source going live;
|
||||
* a service identity granting itself access is exactly the rubber stamp the
|
||||
* gate exists to prevent.
|
||||
*/
|
||||
const NON_ACCOUNTABLE_ACTORS = /^(system|svc_|service:|robot|automation|anonymous)/i;
|
||||
|
||||
export class SourceRegistryService {
|
||||
private readonly repo: SourceRegistryRepository;
|
||||
private readonly aud0: Aud0Client;
|
||||
@@ -89,8 +102,16 @@ export class SourceRegistryService {
|
||||
// source.register
|
||||
// -------------------------------------------------------------------------
|
||||
async registerSource(input: RegisterSourceInput): Promise<SourceDescriptor> {
|
||||
const { rights, provenance, ...base } = input;
|
||||
const source = await this.sources.createSource(base);
|
||||
const { rights, provenance, authorization, schedule, ...base } = input;
|
||||
// P060: registration NEVER activates. A caller asking for status "active"
|
||||
// gets the source created "pending" and must go through activateSource,
|
||||
// which checks the classification. Silently honouring `status: "active"`
|
||||
// here would leave the gate wide open to its most obvious bypass.
|
||||
const requestedActive = base.status === "active";
|
||||
const source = await this.sources.createSource({
|
||||
...base,
|
||||
status: requestedActive ? "pending" : base.status,
|
||||
});
|
||||
if (rights) {
|
||||
await this.setRights({ ...rights, source_id: source.source_id, tenant_id: source.tenant_id });
|
||||
}
|
||||
@@ -102,9 +123,165 @@ export class SourceRegistryService {
|
||||
updated_at: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
if (authorization) {
|
||||
await this.setAuthorization({
|
||||
...authorization,
|
||||
source_id: source.source_id,
|
||||
tenant_id: source.tenant_id,
|
||||
});
|
||||
}
|
||||
if (schedule) {
|
||||
await this.setSchedule({
|
||||
...schedule,
|
||||
source_id: source.source_id,
|
||||
tenant_id: source.tenant_id,
|
||||
updated_at: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
// A registration that arrived fully classified AND asked to be active is
|
||||
// activated here — through the same gate, not around it.
|
||||
if (requestedActive) {
|
||||
await this.activateSource(source.source_id, source.tenant_id, base.created_by);
|
||||
}
|
||||
return this.describeSource(source.source_id, source.tenant_id);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// P060 authorization / schedule
|
||||
// -------------------------------------------------------------------------
|
||||
async setAuthorization(auth: SourceAuthorization): Promise<SourceAuthorization> {
|
||||
await this.assertSourceTenant(auth.source_id, auth.tenant_id);
|
||||
if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.source_registry",
|
||||
`authorization.granted_by must name an accountable actor, not "${auth.granted_by}"`,
|
||||
);
|
||||
}
|
||||
await this.repo.setAuthorization(auth);
|
||||
await this.emit("0pt.source_authorization_set", auth.tenant_id, auth.source_id, {
|
||||
basis: auth.basis,
|
||||
granted_by: auth.granted_by,
|
||||
expires_at: auth.expires_at,
|
||||
evidence_ref: auth.evidence_ref,
|
||||
});
|
||||
return auth;
|
||||
}
|
||||
|
||||
async getAuthorization(sourceId: string, tenantId: string): Promise<SourceAuthorization | undefined> {
|
||||
await this.assertSourceTenant(sourceId, tenantId);
|
||||
return this.repo.getAuthorization(sourceId);
|
||||
}
|
||||
|
||||
async setSchedule(schedule: SourceSchedule): Promise<SourceSchedule> {
|
||||
await this.assertSourceTenant(schedule.source_id, schedule.tenant_id);
|
||||
if (schedule.interval_seconds !== undefined && schedule.interval_seconds <= 0) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.source_registry",
|
||||
"schedule.interval_seconds must be positive",
|
||||
);
|
||||
}
|
||||
await this.repo.setSchedule(schedule);
|
||||
await this.emit("0pt.source_schedule_set", schedule.tenant_id, schedule.source_id, {
|
||||
mode: schedule.mode,
|
||||
interval_seconds: schedule.interval_seconds,
|
||||
enabled: schedule.enabled,
|
||||
});
|
||||
return schedule;
|
||||
}
|
||||
|
||||
async getSchedule(sourceId: string, tenantId: string): Promise<SourceSchedule | undefined> {
|
||||
await this.assertSourceTenant(sourceId, tenantId);
|
||||
return this.repo.getSchedule(sourceId);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// P060 classification + activation gate
|
||||
// -------------------------------------------------------------------------
|
||||
/**
|
||||
* Is this source classified? A source is classified when ALL of the
|
||||
* following are on record:
|
||||
* - an accountable owner,
|
||||
* - an unexpired authorization from an accountable actor,
|
||||
* - rights: a license id AND at least one allowed purpose,
|
||||
* - provenance: how the data is obtained.
|
||||
*
|
||||
* Everything else in the registry (schedule, schema, checkpoint) describes
|
||||
* HOW a source runs, not WHETHER it may — those are deliberately not part
|
||||
* of the gate, and that choice is documented here rather than left implicit.
|
||||
*/
|
||||
async classifySource(
|
||||
sourceId: string,
|
||||
tenantId: string,
|
||||
now: Date = new Date(),
|
||||
): Promise<SourceClassification> {
|
||||
const source = await this.assertSourceTenant(sourceId, tenantId);
|
||||
const missing: string[] = [];
|
||||
|
||||
if (!source.owner_id) missing.push("owner_id is not set");
|
||||
|
||||
const auth = await this.repo.getAuthorization(sourceId);
|
||||
if (!auth) {
|
||||
missing.push("no authorization on record");
|
||||
} else {
|
||||
if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) {
|
||||
missing.push(`authorization.granted_by "${auth.granted_by}" is not an accountable actor`);
|
||||
}
|
||||
if (auth.expires_at && new Date(auth.expires_at).getTime() <= now.getTime()) {
|
||||
missing.push(`authorization expired at ${auth.expires_at}`);
|
||||
}
|
||||
}
|
||||
|
||||
const rights = await this.repo.getRights(sourceId);
|
||||
if (!rights) {
|
||||
missing.push("no rights record (license/purpose) on record");
|
||||
} else {
|
||||
if (!rights.license_id) missing.push("rights.license_id is empty");
|
||||
if (rights.allowed_purposes.length === 0) missing.push("rights.allowed_purposes is empty");
|
||||
}
|
||||
|
||||
const provenance = await this.repo.getProvenance(sourceId);
|
||||
if (!provenance) missing.push("no provenance record (sourcing method) on record");
|
||||
|
||||
return { source_id: sourceId, classified: missing.length === 0, missing };
|
||||
}
|
||||
|
||||
/**
|
||||
* P060 DoD: an UNCLASSIFIED source cannot be activated. Fail-closed —
|
||||
* every missing fact is named at once, and nothing is written when the
|
||||
* check fails.
|
||||
*/
|
||||
async activateSource(sourceId: string, tenantId: string, actorId?: string): Promise<Source> {
|
||||
const classification = await this.classifySource(sourceId, tenantId);
|
||||
if (!classification.classified) {
|
||||
await this.emit("0pt.source_activation_denied", tenantId, sourceId, {
|
||||
missing: classification.missing,
|
||||
});
|
||||
throw new SiaxError(
|
||||
"AUTHORIZATION_DENIED",
|
||||
"0pt.source_registry",
|
||||
`source ${sourceId} is not classified and cannot be activated: ${classification.missing.join("; ")}`,
|
||||
);
|
||||
}
|
||||
const source = await this.sources.updateStatus(sourceId, "active", actorId);
|
||||
await this.emit("0pt.source_activated", tenantId, sourceId, { activated_by: actorId ?? null });
|
||||
return source;
|
||||
}
|
||||
|
||||
/** Take a source out of service. Always allowed — deactivating is never gated. */
|
||||
async deactivateSource(
|
||||
sourceId: string,
|
||||
tenantId: string,
|
||||
status: "paused" | "archived" | "deprecated" = "paused",
|
||||
actorId?: string,
|
||||
): Promise<Source> {
|
||||
await this.assertSourceTenant(sourceId, tenantId);
|
||||
const source = await this.sources.updateStatus(sourceId, status, actorId);
|
||||
await this.emit("0pt.source_deactivated", tenantId, sourceId, { status });
|
||||
return source;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// rights / license / purpose metadata
|
||||
// -------------------------------------------------------------------------
|
||||
@@ -262,13 +439,17 @@ export class SourceRegistryService {
|
||||
|
||||
async describeSource(sourceId: string, tenantId: string): Promise<SourceDescriptor> {
|
||||
const source = await this.assertSourceTenant(sourceId, tenantId);
|
||||
const [policies, rights, provenance, health, freshness] = await Promise.all([
|
||||
this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }),
|
||||
this.repo.getRights(sourceId),
|
||||
this.repo.getProvenance(sourceId),
|
||||
this.getHealth(sourceId, tenantId),
|
||||
this.getFreshness(sourceId, tenantId),
|
||||
]);
|
||||
const [policies, rights, provenance, health, freshness, authorization, schedule, classification] =
|
||||
await Promise.all([
|
||||
this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }),
|
||||
this.repo.getRights(sourceId),
|
||||
this.repo.getProvenance(sourceId),
|
||||
this.getHealth(sourceId, tenantId),
|
||||
this.getFreshness(sourceId, tenantId),
|
||||
this.repo.getAuthorization(sourceId),
|
||||
this.repo.getSchedule(sourceId),
|
||||
this.classifySource(sourceId, tenantId),
|
||||
]);
|
||||
return {
|
||||
source,
|
||||
rights,
|
||||
@@ -276,6 +457,9 @@ export class SourceRegistryService {
|
||||
health,
|
||||
freshness,
|
||||
rate_policy: policies[0],
|
||||
authorization,
|
||||
schedule,
|
||||
classification,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -28,3 +28,19 @@ export const sourceRegistryService = new SourceRegistryService({
|
||||
sourceService,
|
||||
ratePolicyService: sourceRatePolicyService,
|
||||
});
|
||||
|
||||
/**
|
||||
* P060-P068 ingestion plane singletons. Same reasoning as above: the schema
|
||||
* registry, checkpoint store, ledger, quarantine and lineage projection all
|
||||
* share ONE repository so the in-memory dev/test mode is coherent within a
|
||||
* process, and one `resolveStoreMode` decision in production.
|
||||
*/
|
||||
import { createIngestionRepository } from "../repositories/ingestion-repository";
|
||||
import { SchemaRegistry } from "../schema/schema-registry";
|
||||
import { LineageStore } from "../lineage/lineage-store";
|
||||
import { IngestionRunner } from "../ingestion/ingestion-runner";
|
||||
|
||||
export const ingestionRepository = createIngestionRepository();
|
||||
export const schemaRegistry = new SchemaRegistry(ingestionRepository);
|
||||
export const lineageStore = new LineageStore(ingestionRepository);
|
||||
export const ingestionRunner = new IngestionRunner(ingestionRepository, schemaRegistry, lineageStore);
|
||||
|
||||
@@ -77,6 +77,32 @@ export class SourceService {
|
||||
return this.repo.get(sourceId);
|
||||
}
|
||||
|
||||
/**
|
||||
* P060: the ONLY status transition path. Callers do not get a generic
|
||||
* update method, because status is what the activation gate protects —
|
||||
* SourceRegistryService.activateSource is the only caller that moves a
|
||||
* source to "active", and it runs the classification check first.
|
||||
*/
|
||||
async updateStatus(sourceId: string, status: SourceStatus, actorId?: string): Promise<Source> {
|
||||
const source = await this.repo.updateStatus(sourceId, status);
|
||||
await this.aud0.ingestEvent({
|
||||
event_id: `evt_0pt_source_status_${source.source_id}_${Date.now()}`,
|
||||
event_type: "0pt.source_status_changed",
|
||||
tenant_id: source.tenant_id,
|
||||
project_id: source.project_id,
|
||||
actor_id: actorId ?? source.owner_id ?? "system",
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.update_source_status",
|
||||
resource_type: "source",
|
||||
resource_id: source.source_id,
|
||||
risk_level: "L2",
|
||||
payload: { status: source.status, name: source.name },
|
||||
created_at: source.updated_at ?? new Date().toISOString(),
|
||||
correlation_id: source.correlation_id,
|
||||
});
|
||||
return source;
|
||||
}
|
||||
|
||||
async listSources(filter?: {
|
||||
tenant_id?: string;
|
||||
project_id?: string;
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
/**
|
||||
* src/types/ingestion.ts
|
||||
* =======================
|
||||
* P062: THE common ingestion envelope. Every record that enters 0pt through
|
||||
* a connector — batch page, stream message, or change event — is normalized
|
||||
* into exactly this shape before anything downstream (quality gate, entity
|
||||
* resolution, dataset write, lineage) sees it. Nothing downstream is allowed
|
||||
* to special-case "this came from a file" vs "this came from a change feed";
|
||||
* that difference is expressed by `mode` + `operation` and nothing else.
|
||||
*
|
||||
* HONEST SCOPE NOTE. `mode: "cdc"` and the delete/update operations are part
|
||||
* of the CONTRACT, and the runner handles them (a delete envelope carries no
|
||||
* payload and is never quality-gated as if it had one). This repo does NOT
|
||||
* ship a CDC *source* — there is no Debezium consumer, no replication-slot
|
||||
* reader, no WAL tailer here, exactly as services/api/tests/test_known_gaps.py
|
||||
* has stated since P010. What changed with P062 is that a CDC connector can
|
||||
* now be added without redesigning the pipeline, and that the checkpoint it
|
||||
* would resume from is durable. A CDC connector itself is still MISSING.
|
||||
*
|
||||
* `checkpoint` on a ReadResult is what makes resume real: the connector hands
|
||||
* back the cursor that means "everything up to and including these envelopes
|
||||
* has been produced", the runner commits it durably AFTER the envelopes are
|
||||
* accepted, and a restart passes it straight back into read(). Committing it
|
||||
* before acceptance would create silent gaps — that ordering is the whole
|
||||
* point and is asserted in tests/p0/ingestion-runner.test.ts.
|
||||
*/
|
||||
|
||||
/** How data arrives from the source. */
|
||||
export type IngestionMode = "batch" | "stream" | "cdc";
|
||||
|
||||
/**
|
||||
* What the record means. Batch/stream sources normally emit "upsert"
|
||||
* (they carry current state); a change feed distinguishes the three CDC
|
||||
* operations. "delete" envelopes carry `payload: null`.
|
||||
*/
|
||||
export type EnvelopeOperation = "upsert" | "insert" | "update" | "delete";
|
||||
|
||||
/** Per-envelope provenance — which code, which version, which run produced it. */
|
||||
export interface EnvelopeProvenance {
|
||||
connector_id: string;
|
||||
connector_version: string;
|
||||
/** The ingestion run that produced this envelope. Joins to lineage + checkpoints. */
|
||||
run_id: string;
|
||||
/** Opaque connector cursor at which this record was produced. */
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
/** One normalized record. `payload` is null only for `operation: "delete"`. */
|
||||
export interface IngestionEnvelope {
|
||||
envelope_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
mode: IngestionMode;
|
||||
operation: EnvelopeOperation;
|
||||
/** Partition/stream within the source (a table, a topic, a file). */
|
||||
partition_key: string;
|
||||
/** Stable business key within the partition — the unit of idempotency (P068). */
|
||||
record_key: string;
|
||||
/** Schema version the payload conforms to (P063). */
|
||||
schema_version: number;
|
||||
/** Monotonic per-(source, partition) sequence assigned by the runner. */
|
||||
sequence: number;
|
||||
/** When the change happened upstream, when known. */
|
||||
occurred_at?: string;
|
||||
/** When 0pt normalized it. */
|
||||
ingested_at: string;
|
||||
payload: Record<string, unknown> | null;
|
||||
provenance: EnvelopeProvenance;
|
||||
}
|
||||
|
||||
/**
|
||||
* The durable resume point for one (source, partition). Committed only after
|
||||
* the envelopes it covers have been accepted downstream.
|
||||
*/
|
||||
export interface IngestionCheckpoint {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
partition_key: string;
|
||||
/** Opaque connector cursor. `undefined` means "never read". */
|
||||
cursor?: string;
|
||||
/** Highest envelope sequence committed for this partition. */
|
||||
sequence: number;
|
||||
records_committed: number;
|
||||
committed_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* P064: an envelope the declared quality policy sent to quarantine instead of
|
||||
* downstream. Quarantine is DURABLE and inspectable — a dropped record that
|
||||
* only exists in a log line is a silently lost record.
|
||||
*/
|
||||
export interface QuarantineRecord {
|
||||
quarantine_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
record_key: string;
|
||||
/** One entry per failed expectation. */
|
||||
violations: Array<{ expectation_type: string; detail: string }>;
|
||||
envelope: IngestionEnvelope;
|
||||
quarantined_at: string;
|
||||
}
|
||||
|
||||
/** P068: what the idempotency ledger decided about one delivered record. */
|
||||
export type LedgerVerdict = "new" | "changed" | "duplicate";
|
||||
|
||||
export interface LedgerDecision {
|
||||
record_key: string;
|
||||
verdict: LedgerVerdict;
|
||||
}
|
||||
@@ -87,7 +87,11 @@ export interface SourceFreshness {
|
||||
age_hours?: number;
|
||||
}
|
||||
|
||||
/** The full `source.describe` document (§10): base + rights + provenance + health + freshness + rate policy. */
|
||||
/**
|
||||
* The full `source.describe` document (§10): base + rights + provenance +
|
||||
* health + freshness + rate policy, plus the P060 additions (authorization,
|
||||
* schedule, and the classification verdict the activation gate applies).
|
||||
*/
|
||||
export interface SourceDescriptor {
|
||||
source: import("./source").Source;
|
||||
rights?: SourceRights;
|
||||
@@ -95,4 +99,67 @@ export interface SourceDescriptor {
|
||||
health: SourceHealth;
|
||||
freshness: SourceFreshness;
|
||||
rate_policy?: import("./source-rate-policy").SourceRatePolicy;
|
||||
authorization?: SourceAuthorization;
|
||||
schedule?: SourceSchedule;
|
||||
/** Why this source may (or may not) be activated. */
|
||||
classification?: SourceClassification;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// P060: the registry facts that were missing from the original §10 pass —
|
||||
// authorization, schedule, schema reference, checkpoint — plus the
|
||||
// classification verdict the activation gate produces.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Legal/contractual ground on which this tenant may connect to the source at all. */
|
||||
export type AuthorizationBasis =
|
||||
| "contract"
|
||||
| "public_register"
|
||||
| "consent"
|
||||
| "legitimate_interest"
|
||||
| "internal";
|
||||
|
||||
/**
|
||||
* The recorded decision that 0pt MAY connect to this source (P060
|
||||
* "authorization"). Deliberately separate from {@link SourceRights}: rights
|
||||
* say what the DATA may be used for, authorization says who signed off that
|
||||
* the CONNECTION may exist. A source needs both to be classified.
|
||||
*/
|
||||
export interface SourceAuthorization {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
basis: AuthorizationBasis;
|
||||
/** Accountable actor id — never "system"/"svc_*" (the gate rejects those). */
|
||||
granted_by: string;
|
||||
granted_at: string;
|
||||
/** ISO timestamp; an authorization past this instant does not classify the source. */
|
||||
expires_at?: string;
|
||||
/** AUD0 evidence ref or contract artifact ref backing the decision. */
|
||||
evidence_ref?: string;
|
||||
notes?: string;
|
||||
}
|
||||
|
||||
/** How often 0pt pulls from the source, and in which ingestion mode (P060 "schedule"). */
|
||||
export interface SourceSchedule {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
mode: "batch" | "stream" | "cdc";
|
||||
/** Poll cadence in seconds. Omitted for a continuously-open stream. */
|
||||
interval_seconds?: number;
|
||||
/** Optional cron expression. Recorded here; 0pt does not execute cron itself (ACT0 owns scheduling). */
|
||||
cron?: string;
|
||||
enabled: boolean;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Why a source is (not) classified. `activateSource` refuses on
|
||||
* `classified === false` and names every missing fact, so an operator gets
|
||||
* the whole list at once instead of one error per round trip.
|
||||
*/
|
||||
export interface SourceClassification {
|
||||
source_id: string;
|
||||
classified: boolean;
|
||||
/** Human-readable reasons, one per missing/expired fact. Empty when classified. */
|
||||
missing: string[];
|
||||
}
|
||||
|
||||
+17
-1
@@ -34,8 +34,16 @@ export type SourceType = "database" | "file" | "web" | "api" | "stream" | "manua
|
||||
* Union of both status vocabularies. "archived" and "deprecated" both survive
|
||||
* the unification merge -- they were introduced independently and no code
|
||||
* treats either as canonical.
|
||||
*
|
||||
* "pending" (P060) is the state every newly registered source starts in. It
|
||||
* exists because "active" used to be the CREATE default
|
||||
* (postgres-source-repository.ts / in-memory-source-repository.ts both did
|
||||
* `input.status ?? "active"`), which meant a source with no rights, no
|
||||
* provenance and no authorization was live the instant it was registered.
|
||||
* Reaching "active" now requires passing SourceRegistryService.activateSource's
|
||||
* classification gate.
|
||||
*/
|
||||
export type SourceStatus = "active" | "paused" | "archived" | "deprecated";
|
||||
export type SourceStatus = "pending" | "active" | "paused" | "archived" | "deprecated";
|
||||
|
||||
/** Legal-sourcing vocabulary, mirroring dim_source.access. */
|
||||
export type SourceAccess = "api" | "bulk" | "authorized_web" | "licensed";
|
||||
@@ -60,7 +68,14 @@ export interface Source {
|
||||
correlation_id?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
/** Who registered the source. */
|
||||
created_by?: string;
|
||||
/**
|
||||
* Who is ACCOUNTABLE for the source going forward (P060 "owner"). Distinct
|
||||
* from `created_by`, which is only who typed the registration. Required by
|
||||
* the activation gate.
|
||||
*/
|
||||
owner_id?: string;
|
||||
}
|
||||
|
||||
export interface CreateSourceInput {
|
||||
@@ -75,4 +90,5 @@ export interface CreateSourceInput {
|
||||
status?: SourceStatus;
|
||||
correlation_id?: string;
|
||||
created_by?: string;
|
||||
owner_id?: string;
|
||||
}
|
||||
|
||||
@@ -127,7 +127,8 @@ export const registerSourceSchema = z.object({
|
||||
adapter_id: z.string().min(1).optional(),
|
||||
access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(),
|
||||
description: z.string().min(1).optional(),
|
||||
status: z.enum(['active', 'paused', 'archived', 'deprecated']).optional(),
|
||||
status: z.enum(['pending', 'active', 'paused', 'archived', 'deprecated']).optional(),
|
||||
owner_id: z.string().min(1).optional(),
|
||||
correlation_id: z.string().min(1).optional(),
|
||||
created_by: z.string().min(1).optional(),
|
||||
rights: sourceRightsInputSchema.optional(),
|
||||
@@ -232,3 +233,33 @@ export const qualityBridgeRunSchema = z.object({
|
||||
rows: z.array(jsonObject),
|
||||
correlation_id: z.string().min(1).optional(),
|
||||
});
|
||||
|
||||
/** Nested `authorization` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/authorization (P060). */
|
||||
const sourceAuthorizationInputSchema = z.object({
|
||||
basis: z.enum(['contract', 'public_register', 'consent', 'legitimate_interest', 'internal']),
|
||||
granted_by: z.string().min(1),
|
||||
granted_at: z.string().min(1),
|
||||
expires_at: z.string().min(1).optional(),
|
||||
evidence_ref: z.string().min(1).optional(),
|
||||
notes: z.string().min(1).optional(),
|
||||
});
|
||||
|
||||
/** PUT /api/0pt/sources/[sourceId]/authorization -- source.set_authorization (P060). */
|
||||
export const setSourceAuthorizationSchema = sourceAuthorizationInputSchema.extend({
|
||||
tenant_id: z.string().min(1),
|
||||
});
|
||||
|
||||
/** PUT /api/0pt/sources/[sourceId]/schedule -- source.set_schedule (P060). */
|
||||
export const setSourceScheduleSchema = z.object({
|
||||
tenant_id: z.string().min(1),
|
||||
mode: z.enum(['batch', 'stream', 'cdc']),
|
||||
interval_seconds: z.number().int().positive().optional(),
|
||||
cron: z.string().min(1).optional(),
|
||||
enabled: z.boolean(),
|
||||
});
|
||||
|
||||
/** POST /api/0pt/sources/[sourceId]/activate -- source.activate (P060 classification gate). */
|
||||
export const activateSourceSchema = z.object({
|
||||
tenant_id: z.string().min(1),
|
||||
actor_id: z.string().min(1).optional(),
|
||||
});
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
/**
|
||||
* HTTP-level tests for the P060/P067 routes added in this pass, in the same
|
||||
* shape as tests/api/0pt-cross-tenant-routes.test.ts: call the handler
|
||||
* directly with a hand-built NextRequest carrying `x-0pt-verified-tenant`,
|
||||
* which is exactly what middleware.ts forwards once a real Bearer token has
|
||||
* been verified.
|
||||
*
|
||||
* Two things are proved here that the service-level tests cannot:
|
||||
* - the new routes carry the SAME cross-tenant guard as the existing ones
|
||||
* (a new route with no guard would be the P014 hole reopened), and
|
||||
* - the activation gate is enforced at the HTTP boundary too — an
|
||||
* unclassified source gets 403, not a 200 with a quiet no-op.
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { NextRequest } from "next/server";
|
||||
import { POST as sourcesPOST } from "@/app/api/0pt/sources/route";
|
||||
import { GET as classifyGET, POST as activatePOST } from "@/app/api/0pt/sources/[sourceId]/activate/route";
|
||||
import { PUT as authorizationPUT } from "@/app/api/0pt/sources/[sourceId]/authorization/route";
|
||||
import { PUT as schedulePUT } from "@/app/api/0pt/sources/[sourceId]/schedule/route";
|
||||
import { GET as lineageGET } from "@/app/api/0pt/lineage/route";
|
||||
|
||||
const VERIFIED = "x-0pt-verified-tenant";
|
||||
|
||||
function getReq(url: string, verifiedTenant = "t_a"): NextRequest {
|
||||
return new NextRequest(new URL(url, "http://localhost"), { headers: { [VERIFIED]: verifiedTenant } });
|
||||
}
|
||||
function bodyReq(url: string, method: "POST" | "PUT", body: unknown, verifiedTenant = "t_a"): NextRequest {
|
||||
return new NextRequest(new URL(url, "http://localhost"), {
|
||||
method,
|
||||
headers: { "content-type": "application/json", [VERIFIED]: verifiedTenant },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
}
|
||||
const params = (sourceId: string) => ({ params: Promise.resolve({ sourceId }) });
|
||||
|
||||
describe("P060 routes — cross-tenant guard", () => {
|
||||
it("POST /activate rejects a foreign tenant_id (403)", async () => {
|
||||
const res = await activatePOST(
|
||||
bodyReq("/api/0pt/sources/src_x/activate", "POST", { tenant_id: "t_victim" }),
|
||||
params("src_x"),
|
||||
);
|
||||
expect(res.status).toBe(403);
|
||||
expect((await res.json()).error).toBe("OPT_ACCESS_DENIED");
|
||||
});
|
||||
|
||||
it("GET /activate (classification) rejects a foreign tenant_id (403)", async () => {
|
||||
const res = await classifyGET(
|
||||
getReq("/api/0pt/sources/src_x/activate?tenant_id=t_victim"),
|
||||
params("src_x"),
|
||||
);
|
||||
expect(res.status).toBe(403);
|
||||
});
|
||||
|
||||
it("PUT /authorization rejects a foreign tenant_id (403)", async () => {
|
||||
const res = await authorizationPUT(
|
||||
bodyReq("/api/0pt/sources/src_x/authorization", "PUT", {
|
||||
tenant_id: "t_victim",
|
||||
basis: "contract",
|
||||
granted_by: "usr_attacker",
|
||||
granted_at: "2026-09-04T00:00:00.000Z",
|
||||
}),
|
||||
params("src_x"),
|
||||
);
|
||||
expect(res.status).toBe(403);
|
||||
});
|
||||
|
||||
it("PUT /schedule rejects a foreign tenant_id (403)", async () => {
|
||||
const res = await schedulePUT(
|
||||
bodyReq("/api/0pt/sources/src_x/schedule", "PUT", { tenant_id: "t_victim", mode: "batch", enabled: true }),
|
||||
params("src_x"),
|
||||
);
|
||||
expect(res.status).toBe(403);
|
||||
});
|
||||
|
||||
it("GET /lineage rejects a foreign tenant_id (403)", async () => {
|
||||
const res = await lineageGET(getReq("/api/0pt/lineage?tenant_id=t_victim&output_name=0pt:dataset:ds_x"));
|
||||
expect(res.status).toBe(403);
|
||||
});
|
||||
|
||||
it("GET /lineage refuses an unfiltered dump (400)", async () => {
|
||||
const res = await lineageGET(getReq("/api/0pt/lineage?tenant_id=t_a"));
|
||||
expect(res.status).toBe(400);
|
||||
});
|
||||
});
|
||||
|
||||
describe("P060 routes — the activation gate over HTTP", () => {
|
||||
it("an unclassified source cannot be activated through the API", async () => {
|
||||
const created = await sourcesPOST(
|
||||
bodyReq("/api/0pt/sources", "POST", {
|
||||
source_id: "src_http_gate_1",
|
||||
tenant_id: "t_a",
|
||||
project_id: "p_a",
|
||||
source_type: "api",
|
||||
name: "Unclassified source",
|
||||
}),
|
||||
);
|
||||
expect(created.status).toBeLessThan(300);
|
||||
const body = await created.json();
|
||||
expect(body.source.status).toBe("pending");
|
||||
|
||||
const classification = await classifyGET(
|
||||
getReq("/api/0pt/sources/src_http_gate_1/activate?tenant_id=t_a"),
|
||||
params("src_http_gate_1"),
|
||||
);
|
||||
expect(classification.status).toBe(200);
|
||||
expect((await classification.json()).classified).toBe(false);
|
||||
|
||||
const activated = await activatePOST(
|
||||
bodyReq("/api/0pt/sources/src_http_gate_1/activate", "POST", { tenant_id: "t_a" }),
|
||||
params("src_http_gate_1"),
|
||||
);
|
||||
expect(activated.status).toBe(403);
|
||||
});
|
||||
|
||||
it("a classified source activates through the API", async () => {
|
||||
await sourcesPOST(
|
||||
bodyReq("/api/0pt/sources", "POST", {
|
||||
source_id: "src_http_gate_2",
|
||||
tenant_id: "t_a",
|
||||
project_id: "p_a",
|
||||
source_type: "api",
|
||||
name: "Classified source",
|
||||
owner_id: "usr_simon",
|
||||
rights: {
|
||||
license_id: "CC-BY-4.0",
|
||||
allowed_purposes: ["catalog"],
|
||||
pii_allowed: false,
|
||||
redistribution: "internal",
|
||||
verified_at: "2026-09-04",
|
||||
verified_by: "usr_simon",
|
||||
},
|
||||
provenance: { method: "official_api" },
|
||||
}),
|
||||
);
|
||||
const auth = await authorizationPUT(
|
||||
bodyReq("/api/0pt/sources/src_http_gate_2/authorization", "PUT", {
|
||||
tenant_id: "t_a",
|
||||
basis: "public_register",
|
||||
granted_by: "usr_simon",
|
||||
granted_at: "2026-09-01T00:00:00.000Z",
|
||||
}),
|
||||
params("src_http_gate_2"),
|
||||
);
|
||||
expect(auth.status).toBe(200);
|
||||
|
||||
const activated = await activatePOST(
|
||||
bodyReq("/api/0pt/sources/src_http_gate_2/activate", "POST", { tenant_id: "t_a", actor_id: "usr_simon" }),
|
||||
params("src_http_gate_2"),
|
||||
);
|
||||
expect(activated.status).toBe(200);
|
||||
expect((await activated.json()).status).toBe("active");
|
||||
});
|
||||
|
||||
it("a service identity cannot be recorded as the authorizing actor (400)", async () => {
|
||||
await sourcesPOST(
|
||||
bodyReq("/api/0pt/sources", "POST", {
|
||||
source_id: "src_http_gate_3",
|
||||
tenant_id: "t_a",
|
||||
project_id: "p_a",
|
||||
source_type: "api",
|
||||
name: "Self-authorizing source",
|
||||
}),
|
||||
);
|
||||
const res = await authorizationPUT(
|
||||
bodyReq("/api/0pt/sources/src_http_gate_3/authorization", "PUT", {
|
||||
tenant_id: "t_a",
|
||||
basis: "internal",
|
||||
granted_by: "svc_0pt",
|
||||
granted_at: "2026-09-01T00:00:00.000Z",
|
||||
}),
|
||||
params("src_http_gate_3"),
|
||||
);
|
||||
expect(res.status).toBe(400);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,165 @@
|
||||
/**
|
||||
* P061 DoD: "two heterogeneous connectors pass the same conformance suite."
|
||||
*
|
||||
* This file is that proof. It runs the SAME function
|
||||
* (src/connectors/conformance.ts::runConnectorConformance) against two
|
||||
* connectors that share nothing but the contract:
|
||||
*
|
||||
* HttpJsonConnector — a real node:http server on a real socket, opaque
|
||||
* cursor paging, network health check, timestamp
|
||||
* backfill bounds.
|
||||
* JsonlFileConnector — a real file on disk, line-offset cursor, filesystem
|
||||
* health check, line-number backfill bounds, and CDC
|
||||
* delete records the HTTP connector cannot produce.
|
||||
*
|
||||
* Neither connector is mocked. The HTTP fixture is a genuine server that
|
||||
* implements paging server-side; the file fixture is a genuine file written
|
||||
* to a temp dir. A connector that only "passed" against a stub would prove
|
||||
* nothing about the contract.
|
||||
*/
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import http from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { runConnectorConformance, type ConformanceFixture } from "@/src/connectors/conformance";
|
||||
import { HttpJsonConnector } from "@/src/connectors/http-json-connector";
|
||||
import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector";
|
||||
import type { RecordSchemaFields } from "@/src/schema/record-schema";
|
||||
|
||||
const SCHEMA: RecordSchemaFields = {
|
||||
org_number: { type: "string", required: true },
|
||||
name: { type: "string", required: true },
|
||||
employees: { type: "number", required: false },
|
||||
};
|
||||
|
||||
/** 5 records, shared by both fixtures so the suite's counts are comparable. */
|
||||
const ROWS = [
|
||||
{ org_number: "556001-0001", name: "Alfa AB", employees: 10, seq: 1 },
|
||||
{ org_number: "556002-0002", name: "Beta AB", employees: 20, seq: 2 },
|
||||
{ org_number: "556003-0003", name: "Gamma AB", employees: 30, seq: 3 },
|
||||
{ org_number: "556004-0004", name: "Delta AB", employees: 40, seq: 4 },
|
||||
{ org_number: "556005-0005", name: "Epsilon AB", employees: 50, seq: 5 },
|
||||
];
|
||||
|
||||
// --- fixture 1: a real paged HTTP API ---------------------------------------
|
||||
|
||||
let server: http.Server;
|
||||
let baseUrl = "";
|
||||
|
||||
beforeAll(async () => {
|
||||
server = http.createServer((req, res) => {
|
||||
const url = new URL(req.url ?? "/", "http://localhost");
|
||||
if (url.pathname !== "/v1/records") {
|
||||
res.writeHead(404).end("{}");
|
||||
return;
|
||||
}
|
||||
const limit = Number(url.searchParams.get("limit") ?? "100");
|
||||
const from = url.searchParams.get("from");
|
||||
const to = url.searchParams.get("to");
|
||||
if (from !== null && to !== null) {
|
||||
// Backfill: bounds are `seq` values, from inclusive, to exclusive.
|
||||
const slice = ROWS.filter((r) => r.seq >= Number(from) && r.seq < Number(to)).slice(0, limit);
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ records: slice, next_cursor: null }));
|
||||
return;
|
||||
}
|
||||
// Cursor is the `seq` of the last record already delivered.
|
||||
const after = Number(url.searchParams.get("cursor") ?? "0");
|
||||
const slice = ROWS.filter((r) => r.seq > after).slice(0, limit);
|
||||
const next = slice.length > 0 ? String(slice[slice.length - 1].seq) : null;
|
||||
const more = slice.length > 0 && slice[slice.length - 1].seq < ROWS[ROWS.length - 1].seq;
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ records: slice, next_cursor: more ? next : null }));
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||
baseUrl = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
});
|
||||
|
||||
// --- fixture 2: a real JSONL file -------------------------------------------
|
||||
|
||||
const tmpDir = path.join(os.tmpdir(), `opt-conformance-${process.pid}`);
|
||||
const jsonlPath = path.join(tmpDir, "companies.jsonl");
|
||||
|
||||
beforeAll(async () => {
|
||||
await fs.mkdir(tmpDir, { recursive: true });
|
||||
await fs.writeFile(
|
||||
jsonlPath,
|
||||
ROWS.map((r) => JSON.stringify({ org_number: r.org_number, name: r.name, employees: r.employees })).join("\n") + "\n",
|
||||
"utf8",
|
||||
);
|
||||
});
|
||||
|
||||
const ctx = { tenant_id: "t_test", project_id: "p_test", source_id: "src_conf", run_id: "run_conf" };
|
||||
|
||||
describe("P061 connector contract — the same conformance suite, two heterogeneous connectors", () => {
|
||||
it("HttpJsonConnector passes every check", async () => {
|
||||
const fixture: ConformanceFixture = {
|
||||
validConfig: {
|
||||
base_url: baseUrl,
|
||||
records_path: "/v1/records",
|
||||
partition_key: "companies",
|
||||
schema: SCHEMA,
|
||||
record_key_field: "org_number",
|
||||
cursor_field: "seq",
|
||||
},
|
||||
// Not an absolute URL and no schema — must be rejected without any IO.
|
||||
invalidConfig: { base_url: "not-a-url", records_path: "v1/records", partition_key: "", record_key_field: "" },
|
||||
ctx,
|
||||
backfill: { from: "2", to: "4" },
|
||||
expectedRecordCount: ROWS.length,
|
||||
};
|
||||
const report = await runConnectorConformance(new HttpJsonConnector(), fixture);
|
||||
const failed = report.checks.filter((c) => !c.passed);
|
||||
expect(failed.map((c) => `${c.name}: ${c.detail}`)).toEqual([]);
|
||||
expect(report.passed).toBe(true);
|
||||
expect(report.checks.length).toBeGreaterThanOrEqual(14);
|
||||
});
|
||||
|
||||
it("JsonlFileConnector passes every check — the identical suite", async () => {
|
||||
const fixture: ConformanceFixture = {
|
||||
validConfig: {
|
||||
file_path: jsonlPath,
|
||||
partition_key: "companies",
|
||||
schema: SCHEMA,
|
||||
record_key_field: "org_number",
|
||||
},
|
||||
// Relative path, wrong extension, no schema.
|
||||
invalidConfig: { file_path: "companies.csv", partition_key: "", record_key_field: "" },
|
||||
ctx,
|
||||
// File-connector bounds are LINE NUMBERS — deliberately a different
|
||||
// vocabulary from the HTTP fixture's timestamps/sequence values, which
|
||||
// is exactly the heterogeneity the shared contract has to survive.
|
||||
backfill: { from: "1", to: "3" },
|
||||
expectedRecordCount: ROWS.length,
|
||||
};
|
||||
const report = await runConnectorConformance(new JsonlFileConnector(), fixture);
|
||||
const failed = report.checks.filter((c) => !c.passed);
|
||||
expect(failed.map((c) => `${c.name}: ${c.detail}`)).toEqual([]);
|
||||
expect(report.passed).toBe(true);
|
||||
});
|
||||
|
||||
it("the suite actually FAILS a connector that breaks the contract (it is not decoration)", async () => {
|
||||
// A connector that accepts any config at all — the classic fail-open bug.
|
||||
class FailOpenConnector extends JsonlFileConnector {
|
||||
override validate() {
|
||||
return { valid: true, errors: [] };
|
||||
}
|
||||
}
|
||||
const report = await runConnectorConformance(new FailOpenConnector(), {
|
||||
validConfig: { file_path: jsonlPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" },
|
||||
invalidConfig: { file_path: "nope.csv", partition_key: "", record_key_field: "" },
|
||||
ctx,
|
||||
backfill: { from: "1", to: "3" },
|
||||
expectedRecordCount: ROWS.length,
|
||||
});
|
||||
expect(report.passed).toBe(false);
|
||||
expect(report.checks.find((c) => c.name.startsWith("validate: rejects"))?.passed).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,310 @@
|
||||
/**
|
||||
* REAL Postgres proof for P060 DoD #1 ("a restart preserves every source")
|
||||
* and for the durability the rest of the plane depends on (P062 checkpoints,
|
||||
* P063 schema versions, P064 quarantine, P067 lineage, P068 ledger).
|
||||
*
|
||||
* Same honest-degrade convention as tests/p0/source-registry-postgres.realinfra.test.ts:
|
||||
* gated on an env var, SKIPS HONESTLY when unset, never fabricates success.
|
||||
*
|
||||
* docker run -d --name opt-ingestion-pgtest \
|
||||
* -e POSTGRES_USER=opt -e POSTGRES_PASSWORD=opt_test_pw -e POSTGRES_DB=opt_test \
|
||||
* -p 15434:5432 postgres:16-alpine
|
||||
* INGESTION_TEST_DATABASE_URL=postgres://opt:opt_test_pw@localhost:15434/opt_test \
|
||||
* pnpm vitest run tests/p0/ingestion-plane-postgres.realinfra.test.ts
|
||||
*
|
||||
* "RESTART" here is modelled the only way a test can model it without killing
|
||||
* the runner: every assertion after a write is made through a FRESHLY
|
||||
* CONSTRUCTED service/repository over a NEW connection, so nothing can be
|
||||
* answered out of an in-process Map. The stronger proof — stopping and
|
||||
* starting the Postgres CONTAINER itself and re-reading — was run manually
|
||||
* during this pass and is recorded in STATUS.md; it is not automated here
|
||||
* because a test that restarts its own database would need docker access
|
||||
* from inside CI, which these runners do not grant.
|
||||
*/
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import postgres from "postgres";
|
||||
import { SourceRegistryService } from "@/src/services/source-registry-service";
|
||||
import { SourceService } from "@/src/services/source-service";
|
||||
import { PostgresSourceRepository } from "@/src/repositories/postgres-source-repository";
|
||||
import { PostgresSourceRegistryRepository } from "@/src/repositories/postgres-source-registry-repository";
|
||||
import { PostgresIngestionRepository } from "@/src/repositories/postgres-ingestion-repository";
|
||||
import { SchemaRegistry } from "@/src/schema/schema-registry";
|
||||
import { LineageStore } from "@/src/lineage/lineage-store";
|
||||
import { IngestionRunner } from "@/src/ingestion/ingestion-runner";
|
||||
import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector";
|
||||
import { SiaxError } from "@/src/platform/errors";
|
||||
import type { RecordSchemaFields } from "@/src/schema/record-schema";
|
||||
import os from "node:os";
|
||||
import { promises as fsp } from "node:fs";
|
||||
|
||||
const TEST_DATABASE_URL = process.env.INGESTION_TEST_DATABASE_URL;
|
||||
const run = TEST_DATABASE_URL ? describe : describe.skip;
|
||||
|
||||
let adminSql: ReturnType<typeof postgres> | undefined;
|
||||
let reachable = false;
|
||||
const tmpDir = path.join(os.tmpdir(), `opt-realinfra-${process.pid}`);
|
||||
const filePath = path.join(tmpDir, "companies.jsonl");
|
||||
|
||||
const SCHEMA: RecordSchemaFields = {
|
||||
org_number: { type: "string", required: true },
|
||||
name: { type: "string", required: true },
|
||||
employees: { type: "number", required: false },
|
||||
};
|
||||
const ROWS = Array.from({ length: 6 }, (_, i) => ({
|
||||
org_number: `55611${i}-111${i}`,
|
||||
name: `Realinfra ${i} AB`,
|
||||
employees: (i + 1) * 10,
|
||||
}));
|
||||
|
||||
const TENANT = "t_realinfra";
|
||||
const SOURCE = "src_realinfra_1";
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!TEST_DATABASE_URL) {
|
||||
console.warn(
|
||||
"[realinfra] INGESTION_TEST_DATABASE_URL not set this run -- skipping honestly, not fabricating success.",
|
||||
);
|
||||
return;
|
||||
}
|
||||
adminSql = postgres(TEST_DATABASE_URL, { prepare: false, ssl: false, max: 2 });
|
||||
await Promise.race([
|
||||
adminSql`SELECT 1`,
|
||||
new Promise((_, reject) => setTimeout(() => reject(new Error("connect timeout")), 5000)),
|
||||
]);
|
||||
reachable = true;
|
||||
// Only the two migrations this plane needs. sql/001_init.sql is skipped on
|
||||
// purpose: it requires the `vector` extension (pgvector), which a plain
|
||||
// postgres:16-alpine does not ship, and none of the tables under test
|
||||
// depend on it. Same choice tests/p0/source-registry-postgres.realinfra.test.ts
|
||||
// already makes.
|
||||
for (const file of ["002_source_registry.sql", "007_ingestion_plane.sql"]) {
|
||||
const p = path.join(process.cwd(), "sql", file);
|
||||
if (!fs.existsSync(p)) continue;
|
||||
await adminSql.unsafe(fs.readFileSync(p, "utf8"));
|
||||
}
|
||||
// Clean slate for this suite's ids only — never a blanket TRUNCATE.
|
||||
await adminSql`DELETE FROM opt_lineage_edges WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_quality_quarantine WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_ingestion_ledger WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_ingestion_checkpoints WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_source_schemas WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_source_authorization WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_source_schedule WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_source_rights WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_source_provenance WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_source_freshness WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_source_health_checks WHERE tenant_id = ${TENANT}`;
|
||||
await adminSql`DELETE FROM opt_sources WHERE tenant_id = ${TENANT}`;
|
||||
|
||||
await fsp.mkdir(tmpDir, { recursive: true });
|
||||
await fsp.writeFile(filePath, ROWS.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8");
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await adminSql?.end({ timeout: 5 });
|
||||
await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
});
|
||||
|
||||
/** A brand-new service over a brand-new connection — the "restarted process". */
|
||||
function freshRegistry(): SourceRegistryService {
|
||||
const sql = postgres(TEST_DATABASE_URL as string, { prepare: false, ssl: false, max: 2 });
|
||||
return new SourceRegistryService({
|
||||
repository: new PostgresSourceRegistryRepository(sql),
|
||||
sourceService: new SourceService({ repository: new PostgresSourceRepository(sql) }),
|
||||
});
|
||||
}
|
||||
|
||||
function freshIngestion(): {
|
||||
repo: PostgresIngestionRepository;
|
||||
runner: IngestionRunner;
|
||||
lineage: LineageStore;
|
||||
} {
|
||||
const sql = postgres(TEST_DATABASE_URL as string, { prepare: false, ssl: false, max: 2 });
|
||||
const repo = new PostgresIngestionRepository(sql);
|
||||
const lineage = new LineageStore(repo);
|
||||
return { repo, runner: new IngestionRunner(repo, new SchemaRegistry(repo), lineage), lineage };
|
||||
}
|
||||
|
||||
run("P060 real Postgres — a restart preserves every source and every registry fact", () => {
|
||||
it("is reachable (otherwise the rest of this suite proves nothing)", () => {
|
||||
expect(reachable).toBe(true);
|
||||
});
|
||||
|
||||
it("registers pending, refuses activation unclassified, then activates — all surviving a fresh process", async () => {
|
||||
const writer = freshRegistry();
|
||||
await writer.registerSource({
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
project_id: "p_realinfra",
|
||||
source_type: "file",
|
||||
name: "Realinfra file drop",
|
||||
owner_id: "usr_simon",
|
||||
schedule: { mode: "batch", interval_seconds: 900, enabled: true },
|
||||
});
|
||||
|
||||
// A DIFFERENT service instance over a DIFFERENT connection.
|
||||
const reader = freshRegistry();
|
||||
expect((await reader.sources.getSource(SOURCE)).status).toBe("pending");
|
||||
await expect(reader.activateSource(SOURCE, TENANT)).rejects.toBeInstanceOf(SiaxError);
|
||||
|
||||
const classifier = freshRegistry();
|
||||
await classifier.setRights({
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
license_id: "proprietary:realinfra",
|
||||
allowed_purposes: ["catalog"],
|
||||
pii_allowed: false,
|
||||
redistribution: "internal",
|
||||
verified_at: "2026-09-04",
|
||||
verified_by: "usr_simon",
|
||||
});
|
||||
await classifier.setProvenance({
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
method: "bulk_download",
|
||||
updated_at: new Date().toISOString(),
|
||||
});
|
||||
await classifier.setAuthorization({
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
basis: "contract",
|
||||
granted_by: "usr_simon",
|
||||
granted_at: new Date().toISOString(),
|
||||
});
|
||||
|
||||
const activator = freshRegistry();
|
||||
expect((await activator.classifySource(SOURCE, TENANT)).classified).toBe(true);
|
||||
expect((await activator.activateSource(SOURCE, TENANT, "usr_simon")).status).toBe("active");
|
||||
|
||||
// Read back through yet another fresh process AND directly in SQL, so the
|
||||
// assertion cannot be satisfied by any in-process bookkeeping.
|
||||
const verifier = freshRegistry();
|
||||
const descriptor = await verifier.describeSource(SOURCE, TENANT);
|
||||
expect(descriptor.source.status).toBe("active");
|
||||
expect(descriptor.source.owner_id).toBe("usr_simon");
|
||||
expect(descriptor.authorization?.basis).toBe("contract");
|
||||
expect(descriptor.schedule?.interval_seconds).toBe(900);
|
||||
expect(descriptor.classification?.classified).toBe(true);
|
||||
|
||||
const rows = await (adminSql as ReturnType<typeof postgres>)<
|
||||
{ status: string; owner_id: string | null }[]
|
||||
>`SELECT status, owner_id FROM opt_sources WHERE source_id = ${SOURCE}`;
|
||||
expect(rows[0]).toEqual({ status: "active", owner_id: "usr_simon" });
|
||||
});
|
||||
|
||||
it("a raw INSERT that omits status lands PENDING — the gate is in the schema too", async () => {
|
||||
const sql = adminSql as ReturnType<typeof postgres>;
|
||||
await sql`
|
||||
INSERT INTO opt_sources (source_id, tenant_id, project_id, source_type, name)
|
||||
VALUES ('src_realinfra_raw', ${TENANT}, 'p_realinfra', 'api', 'Raw insert')
|
||||
ON CONFLICT (source_id) DO NOTHING`;
|
||||
const rows = await sql<{ status: string }[]>`
|
||||
SELECT status FROM opt_sources WHERE source_id = 'src_realinfra_raw'`;
|
||||
expect(rows[0].status).toBe("pending");
|
||||
});
|
||||
});
|
||||
|
||||
run("P062/P063/P064/P067/P068 real Postgres — the durable ingestion plane", () => {
|
||||
it("runs, crashes, and resumes across fresh processes with no gap and no duplicate", async () => {
|
||||
const first = freshIngestion();
|
||||
const delivered: string[] = [];
|
||||
let pages = 0;
|
||||
const input = {
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
project_id: "p_realinfra",
|
||||
mode: "batch" as const,
|
||||
connector: new JsonlFileConnector(),
|
||||
config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" },
|
||||
page_size: 2,
|
||||
output_ref: "0pt:dataset:ds_realinfra",
|
||||
};
|
||||
await expect(
|
||||
first.runner.run({
|
||||
...input,
|
||||
sink: async (batch) => {
|
||||
pages++;
|
||||
if (pages === 2) throw new Error("simulated process death");
|
||||
for (const e of batch) delivered.push(e.record_key);
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow(/simulated process death/);
|
||||
expect(delivered).toHaveLength(2);
|
||||
|
||||
// Fresh everything — only Postgres carries the position forward.
|
||||
const second = freshIngestion();
|
||||
const report = await second.runner.run({
|
||||
...input,
|
||||
connector: new JsonlFileConnector(),
|
||||
sink: async (batch) => {
|
||||
for (const e of batch) delivered.push(e.record_key);
|
||||
},
|
||||
});
|
||||
expect(report.status).toBe("succeeded");
|
||||
expect(delivered).toHaveLength(ROWS.length);
|
||||
expect(new Set(delivered).size).toBe(ROWS.length);
|
||||
|
||||
const cp = await second.repo.getCheckpoint(SOURCE, "companies");
|
||||
expect(cp?.records_committed).toBe(ROWS.length);
|
||||
});
|
||||
|
||||
it("the schema version, ledger, and lineage are all readable from a fresh process", async () => {
|
||||
const fresh = freshIngestion();
|
||||
expect((await new SchemaRegistry(fresh.repo).getLatest(SOURCE))?.schema_version).toBe(1);
|
||||
expect(await fresh.repo.countLedger(SOURCE)).toBe(ROWS.length);
|
||||
const upstream = await fresh.lineage.upstreamOf(TENANT, "0pt:dataset:ds_realinfra");
|
||||
expect(upstream.length).toBeGreaterThanOrEqual(1);
|
||||
expect(upstream[0].input_ref).toBe(`0pt:source:${SOURCE}`);
|
||||
});
|
||||
|
||||
it("a repeated backfill is idempotent against the durable ledger", async () => {
|
||||
const fresh = freshIngestion();
|
||||
const emitted: string[] = [];
|
||||
const report = await fresh.runner.backfill({
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
project_id: "p_realinfra",
|
||||
mode: "batch",
|
||||
connector: new JsonlFileConnector(),
|
||||
config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" },
|
||||
sink: async (batch) => {
|
||||
for (const e of batch) emitted.push(e.record_key);
|
||||
},
|
||||
page_size: 100,
|
||||
range: { partition_key: "companies", from: "0", to: String(ROWS.length) },
|
||||
});
|
||||
expect(report.envelopes_emitted).toBe(0);
|
||||
expect(report.duplicates_skipped).toBe(ROWS.length);
|
||||
expect(emitted).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("quarantine survives to a fresh process with the full envelope", async () => {
|
||||
const fresh = freshIngestion();
|
||||
await fresh.runner.backfill({
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
project_id: "p_realinfra",
|
||||
mode: "batch",
|
||||
connector: new JsonlFileConnector(),
|
||||
config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" },
|
||||
sink: async () => undefined,
|
||||
page_size: 100,
|
||||
quality_policy: {
|
||||
source_id: SOURCE,
|
||||
tenant_id: TENANT,
|
||||
expectations: [
|
||||
{ expectation_type: "expect_column_values_to_be_between", kwargs: { column: "employees", min_value: 0, max_value: 20 } },
|
||||
],
|
||||
on_violation: "quarantine",
|
||||
},
|
||||
range: { partition_key: "companies", from: "0", to: String(ROWS.length) },
|
||||
});
|
||||
const reader = freshIngestion();
|
||||
const quarantined = await reader.repo.listQuarantine(SOURCE);
|
||||
expect(quarantined.length).toBe(ROWS.length - 2);
|
||||
expect(quarantined[0].envelope.record_key).toBeTruthy();
|
||||
expect(quarantined[0].violations[0].expectation_type).toBe("expect_column_values_to_be_between");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,412 @@
|
||||
/**
|
||||
* The ingestion plane end to end:
|
||||
*
|
||||
* P062 — every connector's output becomes the same envelope, and a run that
|
||||
* DIES mid-page resumes from the durable checkpoint with no gap and
|
||||
* no double-delivery.
|
||||
* P063 — a breaking schema change stops the run BEFORE the sink is called.
|
||||
* P064 — quality failures block or quarantine according to the DECLARED
|
||||
* policy, and a blocked page does not advance the checkpoint.
|
||||
* P067 — a run's source -> output lineage is queryable afterwards.
|
||||
* P068 — a repeated backfill is idempotent.
|
||||
*
|
||||
* The connectors here are the real ones (JsonlFileConnector against a real
|
||||
* file on disk); the "crash" is a sink that throws, which is the closest
|
||||
* faithful analogue of a process dying between the consumer write and the
|
||||
* checkpoint commit.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { IngestionRunner, type IngestionRunInput } from "@/src/ingestion/ingestion-runner";
|
||||
import { InMemoryIngestionRepository } from "@/src/repositories/in-memory-ingestion-repository";
|
||||
import { SchemaRegistry } from "@/src/schema/schema-registry";
|
||||
import { LineageStore } from "@/src/lineage/lineage-store";
|
||||
import { JsonlFileConnector } from "@/src/connectors/jsonl-file-connector";
|
||||
import { hashPayload } from "@/src/ingestion/envelope";
|
||||
import { SiaxError } from "@/src/platform/errors";
|
||||
import type { IngestionEnvelope } from "@/src/types/ingestion";
|
||||
import type { RecordSchemaFields } from "@/src/schema/record-schema";
|
||||
|
||||
const SCHEMA: RecordSchemaFields = {
|
||||
org_number: { type: "string", required: true },
|
||||
name: { type: "string", required: true },
|
||||
employees: { type: "number", required: false },
|
||||
};
|
||||
|
||||
const tmpDir = path.join(os.tmpdir(), `opt-runner-${process.pid}`);
|
||||
const filePath = path.join(tmpDir, "companies.jsonl");
|
||||
|
||||
const ROWS = Array.from({ length: 10 }, (_, i) => ({
|
||||
org_number: `55600${i}-000${i}`,
|
||||
name: `Bolag ${i} AB`,
|
||||
employees: (i + 1) * 10,
|
||||
}));
|
||||
|
||||
beforeAll(async () => {
|
||||
await fs.mkdir(tmpDir, { recursive: true });
|
||||
await fs.writeFile(filePath, ROWS.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8");
|
||||
});
|
||||
afterAll(async () => {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
});
|
||||
|
||||
let repo: InMemoryIngestionRepository;
|
||||
let runner: IngestionRunner;
|
||||
let lineage: LineageStore;
|
||||
|
||||
beforeEach(() => {
|
||||
repo = new InMemoryIngestionRepository();
|
||||
lineage = new LineageStore(repo);
|
||||
runner = new IngestionRunner(repo, new SchemaRegistry(repo), lineage);
|
||||
});
|
||||
|
||||
function baseInput(sink: IngestionRunInput["sink"], overrides: Partial<IngestionRunInput> = {}): IngestionRunInput {
|
||||
return {
|
||||
source_id: "src_ing",
|
||||
tenant_id: "t_test",
|
||||
project_id: "p_test",
|
||||
mode: "batch",
|
||||
connector: new JsonlFileConnector(),
|
||||
config: { file_path: filePath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" },
|
||||
sink,
|
||||
page_size: 3,
|
||||
output_ref: "0pt:dataset:ds_companies",
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("P062 common envelope", () => {
|
||||
it("normalizes every record into one envelope shape with provenance and a monotonic sequence", async () => {
|
||||
const seen: IngestionEnvelope[] = [];
|
||||
const report = await runner.run(baseInput(async (batch) => void seen.push(...batch)));
|
||||
|
||||
expect(report.status).toBe("succeeded");
|
||||
expect(seen).toHaveLength(ROWS.length);
|
||||
expect(seen.map((e) => e.sequence)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
|
||||
for (const envelope of seen) {
|
||||
expect(envelope.source_id).toBe("src_ing");
|
||||
expect(envelope.tenant_id).toBe("t_test");
|
||||
expect(envelope.mode).toBe("batch");
|
||||
expect(envelope.operation).toBe("upsert");
|
||||
expect(envelope.partition_key).toBe("companies");
|
||||
expect(envelope.schema_version).toBe(1);
|
||||
expect(envelope.provenance.connector_id).toBe("jsonl_file");
|
||||
expect(envelope.provenance.connector_version).toBe("jsonl-file-connector@1");
|
||||
expect(envelope.provenance.run_id).toBe(report.run_id);
|
||||
expect(envelope.ingested_at).toMatch(/^\d{4}-\d{2}-\d{2}T/);
|
||||
}
|
||||
});
|
||||
|
||||
it("the durable checkpoint advances to the end of the partition", async () => {
|
||||
await runner.run(baseInput(async () => undefined));
|
||||
const checkpoint = await repo.getCheckpoint("src_ing", "companies");
|
||||
expect(checkpoint?.cursor).toBe(String(ROWS.length));
|
||||
expect(checkpoint?.sequence).toBe(ROWS.length);
|
||||
expect(checkpoint?.records_committed).toBe(ROWS.length);
|
||||
});
|
||||
|
||||
it("a run that dies inside the sink resumes with NO gap and NO double delivery", async () => {
|
||||
const delivered: string[] = [];
|
||||
let pages = 0;
|
||||
|
||||
// First run: the sink dies on the third page (records 7-9), AFTER two
|
||||
// pages have been committed. This is the crash window the ordering exists
|
||||
// to protect: the checkpoint for page 3 was never committed.
|
||||
const crashing = baseInput(async (batch) => {
|
||||
pages++;
|
||||
if (pages === 3) throw new Error("process died mid-write");
|
||||
for (const e of batch) delivered.push(e.record_key);
|
||||
});
|
||||
await expect(runner.run(crashing)).rejects.toThrow(/process died mid-write/);
|
||||
|
||||
expect(delivered).toHaveLength(6);
|
||||
const afterCrash = await repo.getCheckpoint("src_ing", "companies");
|
||||
expect(afterCrash?.cursor).toBe("6");
|
||||
|
||||
// Second run: a FRESH runner and a FRESH connector, exactly as a restarted
|
||||
// process would be — only the durable checkpoint carries over.
|
||||
const resumed = new IngestionRunner(repo, new SchemaRegistry(repo), lineage);
|
||||
const report = await resumed.run(
|
||||
baseInput(async (batch) => {
|
||||
for (const e of batch) delivered.push(e.record_key);
|
||||
}),
|
||||
);
|
||||
|
||||
expect(report.status).toBe("succeeded");
|
||||
// Every record exactly once across both runs: no gap, no duplicate.
|
||||
expect(delivered).toHaveLength(ROWS.length);
|
||||
expect(new Set(delivered).size).toBe(ROWS.length);
|
||||
expect([...delivered].sort()).toEqual(ROWS.map((r) => r.org_number).sort());
|
||||
});
|
||||
|
||||
it("refuses a mode the connector does not serve", async () => {
|
||||
// JsonlFileConnector declares batch + cdc; it never claims "stream".
|
||||
await expect(runner.run(baseInput(async () => undefined, { mode: "stream" }))).rejects.toThrow(
|
||||
/does not serve mode "stream"/,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("P063 breaking schema is caught before the consumer sees anything", () => {
|
||||
it("fails the run BEFORE the sink is called when the shape breaks the declared policy", async () => {
|
||||
// v1 registered by a clean run.
|
||||
await runner.run(baseInput(async () => undefined));
|
||||
expect((await new SchemaRegistry(repo).getLatest("src_ing"))?.schema_version).toBe(1);
|
||||
|
||||
// The source now claims a new REQUIRED field — backward-breaking.
|
||||
let sinkCalls = 0;
|
||||
const breaking = baseInput(async () => void sinkCalls++, {
|
||||
source_id: "src_ing",
|
||||
config: {
|
||||
file_path: filePath,
|
||||
partition_key: "companies",
|
||||
schema: { ...SCHEMA, sni_code: { type: "string", required: true } },
|
||||
record_key_field: "org_number",
|
||||
},
|
||||
});
|
||||
const err = await runner.run(breaking).catch((e: unknown) => e);
|
||||
expect(err).toBeInstanceOf(SiaxError);
|
||||
expect((err as SiaxError).code).toBe("CONFLICT");
|
||||
expect((err as SiaxError).message).toContain("refusing to ingest");
|
||||
// The whole point: no consumer write happened.
|
||||
expect(sinkCalls).toBe(0);
|
||||
});
|
||||
|
||||
it("catches a record that stopped conforming mid-stream, before emitting the page", async () => {
|
||||
// A file whose DECLARED schema is unchanged but whose DATA drifted:
|
||||
// `employees` arrives as a string. A distinct source_id so this run starts
|
||||
// from a clean checkpoint rather than resuming past the file.
|
||||
const driftPath = path.join(tmpDir, "drift.jsonl");
|
||||
await fs.writeFile(
|
||||
driftPath,
|
||||
JSON.stringify({ org_number: "556999-9999", name: "Drift AB", employees: "many" }) + "\n",
|
||||
"utf8",
|
||||
);
|
||||
let sinkCalls = 0;
|
||||
const drifting = baseInput(async () => void sinkCalls++, {
|
||||
source_id: "src_drift",
|
||||
config: { file_path: driftPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" },
|
||||
});
|
||||
await expect(runner.run(drifting)).rejects.toThrow(/does not conform to registered schema v1/);
|
||||
expect(sinkCalls).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("P064 quality enforcement follows the DECLARED policy", () => {
|
||||
const blockingExpectation = {
|
||||
expectation_type: "expect_column_values_to_be_between",
|
||||
kwargs: { column: "employees", min_value: 0, max_value: 50 },
|
||||
};
|
||||
|
||||
it("'block' stops the run and does NOT advance the checkpoint past the bad page", async () => {
|
||||
const seen: IngestionEnvelope[] = [];
|
||||
const report = await runner.run(
|
||||
baseInput(async (b) => void seen.push(...b), {
|
||||
quality_policy: {
|
||||
source_id: "src_ing",
|
||||
tenant_id: "t_test",
|
||||
expectations: [blockingExpectation],
|
||||
on_violation: "block",
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(report.quality_policy).toBe("block");
|
||||
const partition = report.partitions[0];
|
||||
expect(partition.stopped_because).toBe("blocked_by_quality");
|
||||
// Records 0-4 have employees <= 50; the block fires on the page containing
|
||||
// employees 60, i.e. page 3 (records 6,7,8 -> 70,80,90 ... employees 60 is
|
||||
// record index 5 on page 2). Whatever the exact page, nothing from the
|
||||
// blocked page was emitted and the checkpoint stayed behind it.
|
||||
expect(partition.envelopes_emitted).toBeLessThan(ROWS.length);
|
||||
const checkpoint = await repo.getCheckpoint("src_ing", "companies");
|
||||
expect(Number(checkpoint?.cursor ?? "0")).toBeLessThan(ROWS.length);
|
||||
expect(seen.every((e) => Number(e.payload?.employees) <= 50)).toBe(true);
|
||||
});
|
||||
|
||||
it("'quarantine' persists the failing records, lets the clean ones through, and advances", async () => {
|
||||
const seen: IngestionEnvelope[] = [];
|
||||
const report = await runner.run(
|
||||
baseInput(async (b) => void seen.push(...b), {
|
||||
quality_policy: {
|
||||
source_id: "src_ing",
|
||||
tenant_id: "t_test",
|
||||
expectations: [blockingExpectation],
|
||||
on_violation: "quarantine",
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(report.quality_policy).toBe("quarantine");
|
||||
const partition = report.partitions[0];
|
||||
expect(partition.stopped_because).toBeUndefined();
|
||||
expect(partition.quarantined).toBe(5); // employees 60..100
|
||||
expect(partition.envelopes_emitted).toBe(5); // employees 10..50
|
||||
expect(seen.every((e) => Number(e.payload?.employees) <= 50)).toBe(true);
|
||||
|
||||
const quarantined = await repo.listQuarantine("src_ing");
|
||||
expect(quarantined).toHaveLength(5);
|
||||
expect(quarantined[0].violations[0].expectation_type).toBe("expect_column_values_to_be_between");
|
||||
// The full envelope is preserved — a quarantined record is recoverable,
|
||||
// not just counted.
|
||||
expect(quarantined[0].envelope.payload).toBeTruthy();
|
||||
|
||||
const checkpoint = await repo.getCheckpoint("src_ing", "companies");
|
||||
expect(checkpoint?.cursor).toBe(String(ROWS.length));
|
||||
});
|
||||
|
||||
it("'warn' lets everything through but still reports the violations", async () => {
|
||||
const seen: IngestionEnvelope[] = [];
|
||||
const report = await runner.run(
|
||||
baseInput(async (b) => void seen.push(...b), {
|
||||
quality_policy: {
|
||||
source_id: "src_ing",
|
||||
tenant_id: "t_test",
|
||||
expectations: [blockingExpectation],
|
||||
on_violation: "warn",
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(seen).toHaveLength(ROWS.length);
|
||||
expect(report.partitions[0].quality_violations).toBe(5);
|
||||
expect(report.partitions[0].quarantined).toBe(0);
|
||||
});
|
||||
|
||||
it("no declared policy is reported as 'none_declared', never as a pass", async () => {
|
||||
const report = await runner.run(baseInput(async () => undefined));
|
||||
expect(report.quality_policy).toBe("none_declared");
|
||||
});
|
||||
|
||||
it("an expectation the local engine cannot execute is a violation, not a silent pass", async () => {
|
||||
const report = await runner.run(
|
||||
baseInput(async () => undefined, {
|
||||
quality_policy: {
|
||||
source_id: "src_ing",
|
||||
tenant_id: "t_test",
|
||||
expectations: [{ expectation_type: "expect_column_kl_divergence_to_be_less_than", kwargs: { column: "name" } }],
|
||||
on_violation: "quarantine",
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(report.partitions[0].quarantined).toBe(ROWS.length);
|
||||
const quarantined = await repo.listQuarantine("src_ing");
|
||||
expect(quarantined[0].violations[0].detail).toContain("not executable by the local engine");
|
||||
});
|
||||
});
|
||||
|
||||
describe("P067 lineage is queryable after the run", () => {
|
||||
it("answers 'which source fed this output' and 'what did this source produce'", async () => {
|
||||
const report = await runner.run(baseInput(async () => undefined, { correlation_id: "corr-lineage-1" }));
|
||||
expect(report.lineage_edges).toBe(1);
|
||||
|
||||
const upstream = await lineage.upstreamOf("t_test", "0pt:dataset:ds_companies");
|
||||
expect(upstream).toHaveLength(1);
|
||||
expect(upstream[0].input_name).toBe("source/src_ing");
|
||||
expect(upstream[0].input_ref).toBe("0pt:source:src_ing");
|
||||
expect(upstream[0].capability_id).toBe("0pt.ingestion.run.v1");
|
||||
expect(upstream[0].run_id).toBe(report.run_id);
|
||||
expect(upstream[0].correlation_id).toBe("corr-lineage-1");
|
||||
|
||||
const downstream = await lineage.downstreamOf("t_test", "source/src_ing");
|
||||
expect(downstream.map((e) => e.output_name)).toEqual(["0pt:dataset:ds_companies"]);
|
||||
|
||||
// Tenant-scoped: another tenant sees nothing.
|
||||
expect(await lineage.upstreamOf("t_other", "0pt:dataset:ds_companies")).toEqual([]);
|
||||
});
|
||||
|
||||
it("re-emitting the same run id does not duplicate edges (deterministic edge ids)", async () => {
|
||||
await runner.run(baseInput(async () => undefined, { run_id: "run_fixed" }));
|
||||
// A second identical run of the same id — e.g. an idempotent retry.
|
||||
const repo2Runner = new IngestionRunner(repo, new SchemaRegistry(repo), lineage);
|
||||
await repo2Runner.run(baseInput(async () => undefined, { run_id: "run_fixed" }));
|
||||
const edges = await lineage.upstreamOf("t_test", "0pt:dataset:ds_companies");
|
||||
expect(edges).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("P068 backfill is bounded and idempotent", () => {
|
||||
it("a repeated identical backfill emits nothing the second time", async () => {
|
||||
const first: IngestionEnvelope[] = [];
|
||||
const firstReport = await runner.backfill({
|
||||
...baseInput(async (b) => void first.push(...b)),
|
||||
range: { partition_key: "companies", from: "0", to: "5" },
|
||||
page_size: 100,
|
||||
});
|
||||
expect(firstReport.envelopes_emitted).toBe(5);
|
||||
expect(firstReport.duplicates_skipped).toBe(0);
|
||||
expect(first).toHaveLength(5);
|
||||
|
||||
const second: IngestionEnvelope[] = [];
|
||||
const secondReport = await runner.backfill({
|
||||
...baseInput(async (b) => void second.push(...b)),
|
||||
range: { partition_key: "companies", from: "0", to: "5" },
|
||||
page_size: 100,
|
||||
});
|
||||
expect(secondReport.envelopes_emitted).toBe(0);
|
||||
expect(secondReport.duplicates_skipped).toBe(5);
|
||||
expect(second).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("a backfill does NOT move the live checkpoint", async () => {
|
||||
await runner.run(baseInput(async () => undefined));
|
||||
const before = await repo.getCheckpoint("src_ing", "companies");
|
||||
await runner.backfill({
|
||||
...baseInput(async () => undefined),
|
||||
range: { partition_key: "companies", from: "0", to: "3" },
|
||||
});
|
||||
const after = await repo.getCheckpoint("src_ing", "companies");
|
||||
expect(after).toEqual(before);
|
||||
});
|
||||
|
||||
it("a backfill overlapping a completed run re-delivers nothing (the ledger already has it)", async () => {
|
||||
await runner.run(baseInput(async () => undefined));
|
||||
const emitted: IngestionEnvelope[] = [];
|
||||
const report = await runner.backfill({
|
||||
...baseInput(async (b) => void emitted.push(...b)),
|
||||
range: { partition_key: "companies", from: "0", to: "10" },
|
||||
page_size: 100,
|
||||
});
|
||||
expect(report.duplicates_skipped).toBe(10);
|
||||
expect(emitted).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("a record whose CONTENT changed is re-emitted — idempotency is not amnesia", async () => {
|
||||
await runner.run(baseInput(async () => undefined));
|
||||
const changedPath = path.join(tmpDir, "changed.jsonl");
|
||||
await fs.writeFile(
|
||||
changedPath,
|
||||
JSON.stringify({ ...ROWS[0], name: "Bolag 0 AB (namnändrad)" }) + "\n",
|
||||
"utf8",
|
||||
);
|
||||
const emitted: IngestionEnvelope[] = [];
|
||||
const report = await runner.backfill({
|
||||
...baseInput(async (b) => void emitted.push(...b), {
|
||||
config: { file_path: changedPath, partition_key: "companies", schema: SCHEMA, record_key_field: "org_number" },
|
||||
}),
|
||||
range: { partition_key: "companies", from: "0", to: "1" },
|
||||
page_size: 100,
|
||||
});
|
||||
expect(report.changed).toBe(1);
|
||||
expect(report.envelopes_emitted).toBe(1);
|
||||
expect(emitted[0].payload?.name).toBe("Bolag 0 AB (namnändrad)");
|
||||
});
|
||||
|
||||
it("an unbounded backfill is refused by the connector contract", async () => {
|
||||
await expect(
|
||||
runner.backfill({
|
||||
...baseInput(async () => undefined),
|
||||
range: { partition_key: "companies", from: "0", to: "" },
|
||||
}),
|
||||
).rejects.toThrow(/unbounded backfill|requires both/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("envelope hashing", () => {
|
||||
it("is stable across key order and distinguishes a real content change", () => {
|
||||
expect(hashPayload({ a: 1, b: 2 })).toBe(hashPayload({ b: 2, a: 1 }));
|
||||
expect(hashPayload({ a: 1, b: 2 })).not.toBe(hashPayload({ a: 1, b: 3 }));
|
||||
// null and absent are different data, and must hash differently.
|
||||
expect(hashPayload({ a: 1, b: null })).not.toBe(hashPayload({ a: 1 }));
|
||||
expect(hashPayload(null)).toBe("sha256:deleted");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,149 @@
|
||||
/**
|
||||
* P063 DoD: "a breaking schema is detected BEFORE consumer corruption."
|
||||
*
|
||||
* Two levels, both tested here:
|
||||
* - the pure comparison (checkCompatibility) per policy direction, and
|
||||
* - the registry refusing to record a version that breaks the PREVIOUS
|
||||
* version's declared policy — including the bypass where a producer tries
|
||||
* to widen its own policy to "none" on the breaking version.
|
||||
*
|
||||
* The end-to-end "no consumer write happened" proof lives in
|
||||
* tests/p0/ingestion-runner.test.ts, which asserts the sink was never called.
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { checkCompatibility, conformanceErrors } from "@/src/schema/compatibility";
|
||||
import { SchemaRegistry } from "@/src/schema/schema-registry";
|
||||
import { InMemoryIngestionRepository } from "@/src/repositories/in-memory-ingestion-repository";
|
||||
import { SiaxError } from "@/src/platform/errors";
|
||||
import type { RecordSchemaFields } from "@/src/schema/record-schema";
|
||||
|
||||
const V1: RecordSchemaFields = {
|
||||
org_number: { type: "string", required: true },
|
||||
name: { type: "string", required: true },
|
||||
employees: { type: "number", required: false },
|
||||
};
|
||||
|
||||
describe("P063 compatibility rules", () => {
|
||||
it("adding an OPTIONAL field is backward-compatible", () => {
|
||||
const next = { ...V1, sni_code: { type: "string" as const, required: false } };
|
||||
expect(checkCompatibility(V1, next, "backward").compatible).toBe(true);
|
||||
});
|
||||
|
||||
it("adding a REQUIRED field breaks backward compatibility", () => {
|
||||
const next = { ...V1, sni_code: { type: "string" as const, required: true } };
|
||||
const verdict = checkCompatibility(V1, next, "backward");
|
||||
expect(verdict.compatible).toBe(false);
|
||||
expect(verdict.breaks[0].field).toBe("sni_code");
|
||||
expect(verdict.breaks[0].detail).toContain("absent from every record written under the previous version");
|
||||
});
|
||||
|
||||
it("adding a required field is FORWARD-safe — direction matters", () => {
|
||||
const next = { ...V1, sni_code: { type: "string" as const, required: true } };
|
||||
expect(checkCompatibility(V1, next, "forward").compatible).toBe(true);
|
||||
expect(checkCompatibility(V1, next, "full").compatible).toBe(false);
|
||||
});
|
||||
|
||||
it("removing a required field breaks BOTH directions", () => {
|
||||
const next: RecordSchemaFields = { org_number: V1.org_number, employees: V1.employees };
|
||||
expect(checkCompatibility(V1, next, "backward").compatible).toBe(false);
|
||||
expect(checkCompatibility(V1, next, "forward").compatible).toBe(false);
|
||||
});
|
||||
|
||||
it("removing an OPTIONAL field is backward-safe but not forward-safe", () => {
|
||||
const next: RecordSchemaFields = { org_number: V1.org_number, name: V1.name };
|
||||
expect(checkCompatibility(V1, next, "backward").compatible).toBe(true);
|
||||
expect(checkCompatibility(V1, next, "forward").compatible).toBe(true);
|
||||
});
|
||||
|
||||
it("a type change breaks every policy except an explicit 'none'", () => {
|
||||
const next: RecordSchemaFields = { ...V1, employees: { type: "string", required: false } };
|
||||
expect(checkCompatibility(V1, next, "backward").compatible).toBe(false);
|
||||
expect(checkCompatibility(V1, next, "forward").compatible).toBe(false);
|
||||
expect(checkCompatibility(V1, next, "full").compatible).toBe(false);
|
||||
expect(checkCompatibility(V1, next, "none").compatible).toBe(true);
|
||||
});
|
||||
|
||||
it("conformanceErrors catches a record that silently stopped matching its schema", () => {
|
||||
expect(conformanceErrors(V1, { org_number: "556001-0001", name: "Alfa AB" })).toEqual([]);
|
||||
expect(conformanceErrors(V1, { org_number: "556001-0001" })).toContain('required field "name" is missing');
|
||||
expect(conformanceErrors(V1, { org_number: "556001-0001", name: "Alfa", employees: "ten" })).toContain(
|
||||
'field "employees" is string, expected number',
|
||||
);
|
||||
expect(conformanceErrors(V1, { org_number: "1", name: "A", surprise: true })).toContain(
|
||||
'unregistered field "surprise" appeared in the payload',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("P063 schema registry — versioning and refusal", () => {
|
||||
function registry() {
|
||||
return new SchemaRegistry(new InMemoryIngestionRepository());
|
||||
}
|
||||
|
||||
it("the first registration is v1; an identical re-registration does not create v2", async () => {
|
||||
const reg = registry();
|
||||
const first = await reg.register({ source_id: "src_a", tenant_id: "t", fields: V1, compatibility: "backward" });
|
||||
expect(first.schema.schema_version).toBe(1);
|
||||
const again = await reg.register({ source_id: "src_a", tenant_id: "t", fields: V1, compatibility: "backward" });
|
||||
expect(again.unchanged).toBe(true);
|
||||
expect((await reg.list("src_a")).length).toBe(1);
|
||||
});
|
||||
|
||||
it("a compatible change becomes v2", async () => {
|
||||
const reg = registry();
|
||||
await reg.register({ source_id: "src_b", tenant_id: "t", fields: V1, compatibility: "backward" });
|
||||
const v2 = await reg.register({
|
||||
source_id: "src_b",
|
||||
tenant_id: "t",
|
||||
fields: { ...V1, sni_code: { type: "string", required: false } },
|
||||
compatibility: "backward",
|
||||
});
|
||||
expect(v2.schema.schema_version).toBe(2);
|
||||
expect((await reg.getLatest("src_b"))?.schema_version).toBe(2);
|
||||
});
|
||||
|
||||
it("a BREAKING change is refused and nothing is persisted", async () => {
|
||||
const reg = registry();
|
||||
await reg.register({ source_id: "src_c", tenant_id: "t", fields: V1, compatibility: "backward" });
|
||||
const err = await reg
|
||||
.register({
|
||||
source_id: "src_c",
|
||||
tenant_id: "t",
|
||||
fields: { ...V1, sni_code: { type: "string", required: true } },
|
||||
compatibility: "backward",
|
||||
})
|
||||
.catch((e: unknown) => e);
|
||||
expect(err).toBeInstanceOf(SiaxError);
|
||||
expect((err as SiaxError).code).toBe("CONFLICT");
|
||||
expect((err as SiaxError).message).toContain("breaks the declared");
|
||||
// Still only v1 on record — a refused registration writes nothing.
|
||||
expect((await reg.list("src_c")).map((s) => s.schema_version)).toEqual([1]);
|
||||
});
|
||||
|
||||
it("a producer cannot widen its OWN policy to 'none' on the breaking version", async () => {
|
||||
const reg = registry();
|
||||
await reg.register({ source_id: "src_d", tenant_id: "t", fields: V1, compatibility: "full" });
|
||||
// The PREVIOUS version's policy governs — declaring "none" here must not help.
|
||||
await expect(
|
||||
reg.register({
|
||||
source_id: "src_d",
|
||||
tenant_id: "t",
|
||||
fields: { ...V1, employees: { type: "string", required: false } },
|
||||
compatibility: "none",
|
||||
}),
|
||||
).rejects.toThrow(/breaks the declared "full" compatibility policy/);
|
||||
});
|
||||
|
||||
it("a source that explicitly declared 'none' opts out — recorded, never assumed", async () => {
|
||||
const reg = registry();
|
||||
await reg.register({ source_id: "src_e", tenant_id: "t", fields: V1, compatibility: "none" });
|
||||
const v2 = await reg.register({
|
||||
source_id: "src_e",
|
||||
tenant_id: "t",
|
||||
fields: { ...V1, employees: { type: "string", required: false } },
|
||||
compatibility: "none",
|
||||
});
|
||||
expect(v2.schema.schema_version).toBe(2);
|
||||
expect(v2.schema.compatibility).toBe("none");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,190 @@
|
||||
/**
|
||||
* P060 DoD #2: "an unclassified source can NOT be activated."
|
||||
*
|
||||
* Before this pass the opposite was true, and provably so:
|
||||
* src/repositories/postgres-source-repository.ts wrote
|
||||
* `${input.status ?? "active"}` and there was no activation operation at all
|
||||
* — registering a source with no rights, no provenance and no authorization
|
||||
* made it `active` on the spot. These tests pin the closed gate AND its
|
||||
* obvious bypasses (asking for status "active" at registration; a service
|
||||
* identity authorizing itself; an expired authorization).
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { SourceRegistryService } from "@/src/services/source-registry-service";
|
||||
import { SourceService } from "@/src/services/source-service";
|
||||
import { InMemorySourceRepository } from "@/src/repositories/in-memory-source-repository";
|
||||
import { InMemorySourceRegistryRepository } from "@/src/repositories/in-memory-source-registry-repository";
|
||||
import { SiaxError } from "@/src/platform/errors";
|
||||
|
||||
function newService(): SourceRegistryService {
|
||||
return new SourceRegistryService({
|
||||
repository: new InMemorySourceRegistryRepository(),
|
||||
sourceService: new SourceService({ repository: new InMemorySourceRepository() }),
|
||||
});
|
||||
}
|
||||
|
||||
const TENANT = "t_test";
|
||||
const base = {
|
||||
source_id: "src_gate_1",
|
||||
tenant_id: TENANT,
|
||||
project_id: "p_test",
|
||||
source_type: "api" as const,
|
||||
name: "Bolagsverket öppna data",
|
||||
owner_id: "usr_simon",
|
||||
};
|
||||
|
||||
const RIGHTS = {
|
||||
license_id: "CC-BY-4.0",
|
||||
allowed_purposes: ["catalog", "enrichment"],
|
||||
pii_allowed: false,
|
||||
redistribution: "customer_deliverable" as const,
|
||||
verified_at: "2026-09-04",
|
||||
verified_by: "usr_simon",
|
||||
};
|
||||
|
||||
const PROVENANCE = { method: "official_api" as const, authority: "Bolagsverket" };
|
||||
|
||||
const AUTHORIZATION = {
|
||||
basis: "public_register" as const,
|
||||
granted_by: "usr_simon",
|
||||
granted_at: "2026-09-01T08:00:00.000Z",
|
||||
};
|
||||
|
||||
describe("P060 source registry — the activation gate", () => {
|
||||
it("a newly registered source is PENDING, never active", async () => {
|
||||
const svc = newService();
|
||||
const descriptor = await svc.registerSource(base);
|
||||
expect(descriptor.source.status).toBe("pending");
|
||||
});
|
||||
|
||||
it("asking for status:'active' at registration does NOT bypass the gate", async () => {
|
||||
const svc = newService();
|
||||
// The most obvious bypass: just ask for it.
|
||||
await expect(svc.registerSource({ ...base, status: "active" })).rejects.toBeInstanceOf(SiaxError);
|
||||
// ...and the source is still on record, still pending — not half-created.
|
||||
const source = await svc.sources.getSource(base.source_id);
|
||||
expect(source.status).toBe("pending");
|
||||
});
|
||||
|
||||
it("activating an unclassified source is denied and names every missing fact", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource({ ...base, owner_id: undefined });
|
||||
const err = await svc.activateSource(base.source_id, TENANT).catch((e: unknown) => e);
|
||||
expect(err).toBeInstanceOf(SiaxError);
|
||||
const siax = err as SiaxError;
|
||||
expect(siax.code).toBe("AUTHORIZATION_DENIED");
|
||||
expect(siax.message).toContain("owner_id is not set");
|
||||
expect(siax.message).toContain("no authorization on record");
|
||||
expect(siax.message).toContain("no rights record");
|
||||
expect(siax.message).toContain("no provenance record");
|
||||
expect((await svc.sources.getSource(base.source_id)).status).toBe("pending");
|
||||
});
|
||||
|
||||
it("a fully classified source activates, and describe() reports the classification", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource({
|
||||
...base,
|
||||
rights: RIGHTS,
|
||||
provenance: PROVENANCE,
|
||||
authorization: AUTHORIZATION,
|
||||
});
|
||||
const classification = await svc.classifySource(base.source_id, TENANT);
|
||||
expect(classification).toEqual({ source_id: base.source_id, classified: true, missing: [] });
|
||||
|
||||
const activated = await svc.activateSource(base.source_id, TENANT, "usr_simon");
|
||||
expect(activated.status).toBe("active");
|
||||
|
||||
const descriptor = await svc.describeSource(base.source_id, TENANT);
|
||||
expect(descriptor.classification?.classified).toBe(true);
|
||||
expect(descriptor.authorization?.basis).toBe("public_register");
|
||||
});
|
||||
|
||||
it("registering fully classified WITH status:'active' activates through the gate, not around it", async () => {
|
||||
const svc = newService();
|
||||
const descriptor = await svc.registerSource({
|
||||
...base,
|
||||
status: "active",
|
||||
rights: RIGHTS,
|
||||
provenance: PROVENANCE,
|
||||
authorization: AUTHORIZATION,
|
||||
});
|
||||
expect(descriptor.source.status).toBe("active");
|
||||
});
|
||||
|
||||
it("a service identity cannot authorize a source (no self-granted rubber stamp)", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource(base);
|
||||
await expect(
|
||||
svc.setAuthorization({
|
||||
source_id: base.source_id,
|
||||
tenant_id: TENANT,
|
||||
basis: "internal",
|
||||
granted_by: "svc_0pt",
|
||||
granted_at: "2026-09-01T08:00:00.000Z",
|
||||
}),
|
||||
).rejects.toThrow(/accountable actor/);
|
||||
});
|
||||
|
||||
it("an EXPIRED authorization does not classify the source", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource({
|
||||
...base,
|
||||
rights: RIGHTS,
|
||||
provenance: PROVENANCE,
|
||||
authorization: { ...AUTHORIZATION, expires_at: "2026-01-01T00:00:00.000Z" },
|
||||
});
|
||||
const classification = await svc.classifySource(base.source_id, TENANT);
|
||||
expect(classification.classified).toBe(false);
|
||||
expect(classification.missing.join(" ")).toContain("expired");
|
||||
await expect(svc.activateSource(base.source_id, TENANT)).rejects.toThrow(/not classified/);
|
||||
});
|
||||
|
||||
it("rights with an empty purpose list do not classify (purpose limitation is not optional)", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource({
|
||||
...base,
|
||||
rights: { ...RIGHTS, allowed_purposes: [] },
|
||||
provenance: PROVENANCE,
|
||||
authorization: AUTHORIZATION,
|
||||
});
|
||||
const classification = await svc.classifySource(base.source_id, TENANT);
|
||||
expect(classification.classified).toBe(false);
|
||||
expect(classification.missing.join(" ")).toContain("allowed_purposes is empty");
|
||||
});
|
||||
|
||||
it("deactivating is never gated — a source can always be taken out of service", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource({ ...base, rights: RIGHTS, provenance: PROVENANCE, authorization: AUTHORIZATION });
|
||||
await svc.activateSource(base.source_id, TENANT);
|
||||
const paused = await svc.deactivateSource(base.source_id, TENANT, "paused");
|
||||
expect(paused.status).toBe("paused");
|
||||
});
|
||||
|
||||
it("the gate is tenant-scoped: another tenant cannot activate this source", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource({ ...base, rights: RIGHTS, provenance: PROVENANCE, authorization: AUTHORIZATION });
|
||||
await expect(svc.activateSource(base.source_id, "t_attacker")).rejects.toThrow(/not found/);
|
||||
expect((await svc.sources.getSource(base.source_id)).status).toBe("pending");
|
||||
});
|
||||
|
||||
it("schedule is recorded as a registry field and validated", async () => {
|
||||
const svc = newService();
|
||||
await svc.registerSource({
|
||||
...base,
|
||||
schedule: { mode: "batch", interval_seconds: 3600, enabled: true },
|
||||
});
|
||||
const schedule = await svc.getSchedule(base.source_id, TENANT);
|
||||
expect(schedule?.mode).toBe("batch");
|
||||
expect(schedule?.interval_seconds).toBe(3600);
|
||||
await expect(
|
||||
svc.setSchedule({
|
||||
source_id: base.source_id,
|
||||
tenant_id: TENANT,
|
||||
mode: "stream",
|
||||
interval_seconds: 0,
|
||||
enabled: true,
|
||||
updated_at: new Date().toISOString(),
|
||||
}),
|
||||
).rejects.toThrow(/interval_seconds must be positive/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user