diff --git a/STATUS.md b/STATUS.md index 7df9a86..660a5d0 100644 --- a/STATUS.md +++ b/STATUS.md @@ -70,15 +70,31 @@ accurate. ## Global gaps blocking PRODUCTION_READY (§3, tracked, not hidden) -1. Persistent stores: the new registries (sources/rights/quality - rules/dataset versions/index definitions/feature views) are per-process - in-memory. `resolveStoreMode` guarantees production boot FAILS rather - than serving Maps, but the Postgres stores themselves are not built. +1. Persistent stores REMAINING: quality runs/rules and feature views are + still per-process in-memory (`resolveStoreMode` guarantees production + boot FAILS rather than serving Maps). CLOSED since the last revision: + dataset registry (`opt_datasets`) + immutable versions + (`opt_dataset_versions`, monotonic under transaction lock, idempotency + with payload-drift detection, cross-tenant ⇒ NOT_FOUND) and versioned + search index definitions (`opt_search_index_definitions`, sequential + schema versions, replay-idempotent, same-version-different-schema ⇒ + CONFLICT; OpenSearch remains a rebuildable projection, never canonical + truth) are Postgres-canonical with real-Postgres tests. 2. Capability registration in CL0UD + API0 discovery for the new - capabilities (§3 "Capability är registrerad i CL0UD Capability Registry"). -3. AUD0 outbox for evidence durability under AUD0 outage (§70). -4. Surface exposure: no `app/api/0pt/*` routes yet for - search/quality/datasets-versions/ER (services + contracts only). + capabilities (§3 "Capability är registrerad i CL0UD Capability + Registry") — PARTIALLY CLOSED: the collaboration/meeting domain was + canonicalized in CL0UD's registry (cl0ud #73/#74); 0pt's own + capability set still needs the same registration. +3. CLOSED: AUD0 outbox for evidence durability under AUD0 outage (§70) — + durable Postgres outbox + dispatcher (at-least-once, bounded batch, + backoff, dead-letter, SKIP LOCKED leasing, tenant binding, idempotency) + + real-Postgres tests; dispatcher uses explicit `durable:false` so it + never re-enqueues its own delivery evidence. +4. CLOSED for datasets/versions/search/ER: `app/api/0pt/datasets*`, + `app/api/0pt/search/definitions` + tenant-safed query route, and + `app/api/0pt/er/{candidates,batch}` (+ internal ER route) are live; + clients cannot raise `rights_tag` via request body (fail-closed to + `internal`). Quality routes exist (`bridge-run/issues/rules/runs`). 5. 0pt-motor executors for the delegated jobs (`0pt.quality.ge_run`, `0pt.features.*`, `0pt.er.batch_resolution`). 6. Observability: no metrics/traces emitted by the new services (§36). @@ -97,7 +113,7 @@ verified, and by which command. | **P062** CDC/batch/stream normalized to one envelope; restart resumes from a durable checkpoint without silent gaps | **MISSING**, and documented as such: `services/api/tests/test_known_gaps.py:10-19` skipped the CDC case because no CDC pipeline existed anywhere, and `workers/pipeline.py` had no persistence at all | **INTEGRATION_TESTED** | `src/types/ingestion.ts` (one envelope for all three modes); `src/ingestion/ingestion-runner.ts`; `tests/p0/ingestion-runner.test.ts` kills a run inside the sink and proves the resumed run covers all 10 records exactly once; same proof against real Postgres across fresh processes | | **P063** Schema versioning + compatibility policy; a breaking schema is detected before consumer corruption | **MISSING.** `schema_version` existed as a free-form string on datasets/Iceberg writes; there was no comparison, no policy, no versioned source schema | **INTEGRATION_TESTED** | `src/schema/compatibility.ts`, `src/schema/schema-registry.ts`; `tests/p0/schema-evolution.test.ts` (12); the runner's schema gate runs before the first read — `tests/p0/ingestion-runner.test.ts` asserts `sinkCalls === 0` on a break | | **P064** Quality rules behind the SIAX abstraction; failures block or quarantine per declared policy | **PARTIAL.** `src/quality/*` had rules, runs, results, issues, scores and a GE-compatible engine, but a failing rule only opened an ISSUE — the data flowed on regardless. `grep -ri quarantin src/` found one unrelated hit | **INTEGRATION_TESTED** | `src/quality/quality-policy.ts` composes the EXISTING `ExpectationEngine` (not a second engine) and adds the consequence; block / quarantine / warn each proved in `tests/p0/ingestion-runner.test.ts`; quarantine durability proved against real Postgres | -| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **DUPLICATE (confirmed).** `src/er/entity-resolution-service.ts` (canonical: hard rules, review band, Splink delegation) AND `services/api/workers/resolve.py` (simplified fuzzy threshold, auto-merge) — the latter still called by `workers/pipeline.py:110` | **NOT DONE — documented, not built.** The duplication is now stated at BOTH sites in code with the removal plan. The new ingestion plane deliberately performs no ER, so it adds no third engine. See "P065 — why it was not closed" below | +| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **CLOSED.** `services/api/workers/resolve.py` is now a thin adapter: Python no longer scores identity — it POSTs opaque records to `/api/0pt/internal/er/candidates` and applies ONLY explicit hard-rule `match` decisions (score/fuzzy can never merge records, even from a compromised bridge); canonical engine unreachable ⇒ records return UNMERGED, never fallback-scored. Company semantics `stable-identifiers.v3`: LEI global; org_nr hard identity only within the same registration_country; punctuation normalized; negative stable identity dominates every positive; fuzzy alone ⇒ human review, never auto-merge. Cluster-wide hard-negative constraints: A≈B ∧ B≈C never forces A≈C when canonical identity says A≠C | **INTEGRATION_TESTED** | `tests/p0/entity-resolution.test.ts` (v3 semantics incl. hard-negative transitivity), `tests/p0/entity-resolution-http.test.ts` (internal route fails closed: empty/wrong token ⇒ 401), `services/api/tests/test_resolve_determinism.py` (adapter contract: canonical edges only), shared request validation (`src/er/request-validation.ts`, records 2..5000), blocking pytest gate (`.gitea/workflows/python-engine.yml`) | | **P066** Iceberg | out of scope — ST0RE is owned by another workflow | **NOT TOUCHED** | Contract read only (`src/services/iceberg-write-service.ts`, `src/clients/st0re-lakehouse-client.ts`) | | **P067** Lineage: OpenLineage/AUD0 events; source → output lineage is queryable | **PARTIAL.** `src/lineage/lineage-emitter.ts` emitted OpenLineage to AUD0, but nothing in this repo could ANSWER a lineage question — no store, no query, no route | **INTEGRATION_TESTED** | `src/lineage/lineage-edge.ts`, `src/lineage/lineage-store.ts` (local queryable projection; AUD0 stays the system of record), `GET /api/0pt/lineage`; `tests/p0/ingestion-runner.test.ts` queries both directions and proves tenant scoping | | **P068** Durable bounded backfill/replay; repeated backfill is idempotent | **MISSING**, documented as such: `services/api/tests/test_known_gaps.py:44-52` skipped it because "no delivery-store persistence layer exists to dedupe a backfill against" | **INTEGRATION_TESTED** | `opt_ingestion_ledger` + `IngestionRunner.backfill`; `tests/p0/ingestion-runner.test.ts` (repeat emits 0, live checkpoint untouched, a real content change still re-emits); same against real Postgres | @@ -167,22 +183,24 @@ production, and P060's durability guarantee cannot be exercised there, because production has no database configured and does not serve the routes. Verified with `curl`; not inferred. -### P065 — why it was not closed +### P065 — closure record (2026-09-07) -Closing P065 means turning `services/api/workers/resolve.py` into an adapter -over the canonical `src/er/entity-resolution-service.ts`. That change cannot -be verified from this repo today: `services/api` is not built by this repo's -Dockerfile (it builds only the Next.js app), `api.0pt.app` answers 404, and -no CI workflow runs its pytest suite. A blind rewrite would ship untested -into a path nobody can exercise — the exact failure mode this programme -exists to stop. +The removal plan below was executed in full and is verified in CI: -The removal plan, for whoever picks it up: -1. Expose the canonical ER capability as `POST /api/0pt/er/candidates` - (service already exists; only a route + Zod schema are missing). -2. Add a Python CI job that actually runs `services/api/tests`. -3. Replace `resolve.py`'s body with an HTTP adapter over that route, - **fail-closed** when it is unreachable — never falling back to the local - scorer, which would preserve the duplicate under a different name. -4. Port `test_resolve_determinism.py` to assert the adapter's contract, and - delete the fuzzy scorer. +1. ✅ Canonical ER exposed as `POST /api/0pt/er/candidates` (public, + tenant-safed) AND `POST /api/0pt/internal/er/candidates` (service-token, + fails closed without `X-Internal-Token`). +2. ✅ Python CI job: `.gitea/workflows/python-engine.yml` runs + `services/api/tests` as a BLOCKING pytest gate. +3. ✅ `workers/resolve.py` rewritten as the HTTP adapter — fail-closed to + UNMERGED records when the bridge is unreachable; no fallback scorer + remains. Cluster-wide hard-negative constraints prevent transitive + matches from overriding canonical identity decisions. +4. ✅ `services/api/tests/test_resolve_determinism.py` asserts the adapter + contract: canonical match edges merge; review/non-match edges do not; + hard negatives are respected transitively. + +Residual honesty: the PYTHON-side tests run in CI, but the full +adapter→Next.js route chain has not been exercised against a live deployed +edge in this branch (the route's own fail-closed behavior is unit-proven). +That keeps P065 at INTEGRATION_TESTED, not LIVE_VERIFIED. diff --git a/app/api/0pt/datasets/[datasetId]/route.ts b/app/api/0pt/datasets/[datasetId]/route.ts new file mode 100644 index 0000000..36e21bb --- /dev/null +++ b/app/api/0pt/datasets/[datasetId]/route.ts @@ -0,0 +1,27 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; +import { datasetService } from "@/src/services/datasets"; + +export async function GET( + request: NextRequest, + { params }: { params: Promise<{ datasetId: string }> }, +) { + try { + const { datasetId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id")?.trim(); + if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + + const dataset = await datasetService.getDataset(datasetId); + if (dataset.tenant_id !== tenantId) { + throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} not found`); + } + return NextResponse.json(dataset); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/datasets/[datasetId]/versions/[version]/route.ts b/app/api/0pt/datasets/[datasetId]/versions/[version]/route.ts new file mode 100644 index 0000000..2325bfa --- /dev/null +++ b/app/api/0pt/datasets/[datasetId]/versions/[version]/route.ts @@ -0,0 +1,27 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; +import { datasetVersionService } from "@/src/services/datasets"; + +export async function GET( + request: NextRequest, + { params }: { params: Promise<{ datasetId: string; version: string }> }, +) { + try { + const { datasetId, version: rawVersion } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id")?.trim(); + if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + + const version = Number(rawVersion); + if (!Number.isSafeInteger(version) || version < 1) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "version must be a positive integer"); + } + return NextResponse.json(await datasetVersionService.getVersion(datasetId, tenantId, version)); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/datasets/[datasetId]/versions/route.ts b/app/api/0pt/datasets/[datasetId]/versions/route.ts new file mode 100644 index 0000000..3e5b0f3 --- /dev/null +++ b/app/api/0pt/datasets/[datasetId]/versions/route.ts @@ -0,0 +1,75 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; +import { datasetVersionService } from "@/src/services/datasets"; + +function required(body: Record, field: string): string { + const value = body[field]; + if (typeof value !== "string" || !value.trim()) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", `${field} is required`); + } + return value.trim(); +} + +export async function POST( + request: NextRequest, + { params }: { params: Promise<{ datasetId: string }> }, +) { + try { + const { datasetId } = await params; + const raw = await request.json(); + if (!raw || typeof raw !== "object" || Array.isArray(raw)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "JSON object body is required"); + } + const body = raw as Record; + const tenantId = required(body, "tenant_id"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + if (!Array.isArray(body.artifact_refs) || !body.artifact_refs.every((value) => typeof value === "string")) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "artifact_refs must be a string array"); + } + + const recordCount = body.record_count === undefined ? undefined : Number(body.record_count); + const idempotencyKey = request.headers.get("idempotency-key")?.trim() + || (typeof body.idempotency_key === "string" ? body.idempotency_key.trim() : "") + || undefined; + + const version = await datasetVersionService.createVersion({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: required(body, "project_id"), + artifact_refs: body.artifact_refs, + schema_version: required(body, "schema_version"), + ...(recordCount === undefined ? {} : { record_count: recordCount }), + ...(typeof body.description === "string" && body.description.trim() + ? { description: body.description.trim() } + : {}), + created_by: request.headers.get("x-0pt-verified-sub") ?? "local-dev", + ...(typeof body.correlation_id === "string" && body.correlation_id.trim() + ? { correlation_id: body.correlation_id.trim() } + : {}), + ...(idempotencyKey ? { idempotency_key: idempotencyKey } : {}), + }); + return NextResponse.json(version, { status: 201 }); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function GET( + request: NextRequest, + { params }: { params: Promise<{ datasetId: string }> }, +) { + try { + const { datasetId } = await params; + const tenantId = new URL(request.url).searchParams.get("tenant_id")?.trim(); + if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + return NextResponse.json(await datasetVersionService.listVersions(datasetId, tenantId)); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/datasets/route.ts b/app/api/0pt/datasets/route.ts new file mode 100644 index 0000000..2ad412b --- /dev/null +++ b/app/api/0pt/datasets/route.ts @@ -0,0 +1,91 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; +import { datasetService } from "@/src/services/datasets"; +import type { DatasetStatus, DatasetType } from "@/src/types/dataset"; + +const DATASET_TYPES = new Set(["raw", "structured", "enriched", "curated", "feature", "derived"]); +const DATASET_STATUSES = new Set(["active", "archived", "deprecated"]); + +function actorId(request: NextRequest): string { + return request.headers.get("x-0pt-verified-sub") ?? "local-dev"; +} + +function stringField(body: Record, field: string): string { + const value = body[field]; + if (typeof value !== "string" || !value.trim()) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", `${field} is required`); + } + return value.trim(); +} + +export async function POST(request: NextRequest) { + try { + const raw = await request.json(); + if (!raw || typeof raw !== "object" || Array.isArray(raw)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "JSON object body is required"); + } + const body = raw as Record; + const tenantId = stringField(body, "tenant_id"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + + const datasetType = stringField(body, "dataset_type") as DatasetType; + if (!DATASET_TYPES.has(datasetType)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported dataset_type"); + } + const status = body.status === undefined ? undefined : String(body.status) as DatasetStatus; + if (status !== undefined && !DATASET_STATUSES.has(status)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported status"); + } + + const dataset = await datasetService.registerDataset({ + dataset_id: stringField(body, "dataset_id"), + tenant_id: tenantId, + project_id: stringField(body, "project_id"), + name: stringField(body, "name"), + dataset_type: datasetType, + ...(status ? { status } : {}), + ...(typeof body.description === "string" && body.description.trim() + ? { description: body.description.trim() } + : {}), + ...(typeof body.correlation_id === "string" && body.correlation_id.trim() + ? { correlation_id: body.correlation_id.trim() } + : {}), + created_by: actorId(request), + }); + return NextResponse.json(dataset, { status: 201 }); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function GET(request: NextRequest) { + try { + const { searchParams } = new URL(request.url); + const tenantId = searchParams.get("tenant_id")?.trim(); + if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + + const typeRaw = searchParams.get("dataset_type") ?? undefined; + const statusRaw = searchParams.get("status") ?? undefined; + if (typeRaw && !DATASET_TYPES.has(typeRaw as DatasetType)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported dataset_type"); + } + if (statusRaw && !DATASET_STATUSES.has(statusRaw as DatasetStatus)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported status"); + } + + return NextResponse.json(await datasetService.listDatasets({ + tenant_id: tenantId, + project_id: searchParams.get("project_id") ?? undefined, + dataset_type: typeRaw as DatasetType | undefined, + status: statusRaw as DatasetStatus | undefined, + })); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/er/batch/route.ts b/app/api/0pt/er/batch/route.ts new file mode 100644 index 0000000..38e0eaa --- /dev/null +++ b/app/api/0pt/er/batch/route.ts @@ -0,0 +1,44 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; +import { getBatchEntityResolutionService } from "@/src/services/entity-resolution"; + +function required(body: Record, field: string): string { + const value = body[field]; + if (typeof value !== "string" || !value.trim()) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `${field} is required`); + } + return value.trim(); +} + +export async function POST(request: NextRequest) { + try { + const raw = await request.json(); + if (!raw || typeof raw !== "object" || Array.isArray(raw)) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "JSON object body is required"); + } + const body = raw as Record; + const tenantId = required(body, "tenant_id"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + + const idempotencyKey = request.headers.get("idempotency-key")?.trim() + || (typeof body.idempotency_key === "string" ? body.idempotency_key.trim() : "") + || undefined; + const job = await getBatchEntityResolutionService().submitBatchResolution({ + tenant_id: tenantId, + project_id: required(body, "project_id"), + records_artifact_ref: required(body, "records_artifact_ref"), + actor_id: request.headers.get("x-0pt-verified-sub") ?? "local-dev", + ...(typeof body.correlation_id === "string" && body.correlation_id.trim() + ? { correlation_id: body.correlation_id.trim() } + : {}), + ...(idempotencyKey ? { idempotency_key: idempotencyKey } : {}), + }); + return NextResponse.json(job, { status: 202 }); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/er/candidates/route.ts b/app/api/0pt/er/candidates/route.ts new file mode 100644 index 0000000..a586972 --- /dev/null +++ b/app/api/0pt/er/candidates/route.ts @@ -0,0 +1,17 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { parseErCandidateRequest } from "@/src/er/request-validation"; +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { entityResolutionService } from "@/src/services/entity-resolution"; + +export async function POST(request: NextRequest) { + try { + const input = parseErCandidateRequest(await request.json()); + const guard = assertTenantMatchesVerified(request, input.tenant_id); + if (guard) return guard; + return NextResponse.json(entityResolutionService.generateCandidates(input)); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/internal/er/candidates/route.ts b/app/api/0pt/internal/er/candidates/route.ts new file mode 100644 index 0000000..e22689c --- /dev/null +++ b/app/api/0pt/internal/er/candidates/route.ts @@ -0,0 +1,101 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { verifyInternalServiceToken } from "@/src/auth/internal-token"; +import { + EntityResolutionService, + type ErCandidateResult, + type ErRecord, +} from "@/src/er/entity-resolution-service"; +import { parseErCandidateRequest } from "@/src/er/request-validation"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; + +const er = new EntityResolutionService(); + +function rawStringField(record: ErRecord, field: string): string | undefined { + const value = record.fields[field]; + if (value === null || value === undefined) return undefined; + const normalized = String(value).trim(); + return normalized || undefined; +} + +function stableIdentifier(record: ErRecord, field: string): string | undefined { + const value = rawStringField(record, field); + if (!value) return undefined; + const normalized = value.toUpperCase().replace(/[^A-Z0-9]/g, ""); + return normalized || undefined; +} + +function countryIdentifier(record: ErRecord): string | undefined { + const value = rawStringField(record, "registration_country") ?? rawStringField(record, "country"); + if (!value) return undefined; + const normalized = value.toUpperCase().replace(/[^A-Z0-9]/g, ""); + return normalized || undefined; +} + +/** + * Company identity hard rule v3. + * - LEI is globally scoped. + * - National registration numbers are comparable only within the same country. + * - Any comparable stable-ID conflict dominates every positive match. + */ +er.registerHardRule({ + rule_id: "0pt.company.stable-identifiers.v3", + decide(a, b) { + let sharedMatch = false; + + const aLei = stableIdentifier(a, "lei"); + const bLei = stableIdentifier(b, "lei"); + if (aLei && bLei) { + if (aLei !== bLei) return "force_non_match"; + sharedMatch = true; + } + + const aCountry = countryIdentifier(a); + const bCountry = countryIdentifier(b); + const aOrg = stableIdentifier(a, "org_nr"); + const bOrg = stableIdentifier(b, "org_nr"); + if (aOrg && bOrg && aCountry && bCountry && aCountry === bCountry) { + if (aOrg !== bOrg) return "force_non_match"; + sharedMatch = true; + } + + return sharedMatch ? "force_match" : "abstain"; + }, +}); + +function assertInternal(request: NextRequest): NextResponse | undefined { + if (verifyInternalServiceToken(request.headers)) return undefined; + return NextResponse.json( + { error: "UNAUTHENTICATED", message: "valid X-Internal-Token is required" }, + { status: 401 }, + ); +} + +/** + * Company/Work boundary: fuzzy similarity is candidate evidence, never identity truth. + * Only product hard rules backed by stable identifiers may emit `match` here. + */ +export function enforceCompanyReviewBoundary(result: ErCandidateResult): ErCandidateResult { + const pairs = result.pairs.map((pair) => + pair.decided_by === "score" && pair.status === "match" + ? { ...pair, status: "pending_review" as const } + : pair, + ); + return { + ...result, + pairs, + review_queue: pairs.filter((pair) => pair.status === "pending_review"), + }; +} + +export async function POST(request: NextRequest) { + const denied = assertInternal(request); + if (denied) return denied; + + try { + const result = er.generateCandidates(parseErCandidateRequest(await request.json())); + return NextResponse.json(enforceCompanyReviewBoundary(result)); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/internal/evidence-outbox/dispatch/route.ts b/app/api/0pt/internal/evidence-outbox/dispatch/route.ts new file mode 100644 index 0000000..585996d --- /dev/null +++ b/app/api/0pt/internal/evidence-outbox/dispatch/route.ts @@ -0,0 +1,69 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { verifyInternalServiceToken } from "@/src/auth/internal-token"; +import { createAud0Client } from "@/src/clients/aud0-client"; +import { + Aud0OutboxDispatcher, + createAud0OutboxRepositoryFromEnv, +} from "@/src/evidence/aud0-outbox"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; + +function assertInternal(request: NextRequest): NextResponse | undefined { + if (verifyInternalServiceToken(request.headers)) return undefined; + return NextResponse.json( + { error: "UNAUTHENTICATED", message: "valid X-Internal-Token is required" }, + { status: 401 }, + ); +} + +function repository() { + const repo = createAud0OutboxRepositoryFromEnv(); + if (!repo) { + throw new SiaxError( + "DEPENDENCY_UNAVAILABLE", + "0pt.aud0_outbox", + "DATABASE_URL is required for durable evidence dispatch", + ); + } + return repo; +} + +export async function POST(request: NextRequest) { + const denied = assertInternal(request); + if (denied) return denied; + + try { + const raw = await request.json().catch(() => ({})) as { limit?: unknown }; + const limit = raw.limit === undefined ? 100 : Number(raw.limit); + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 500) { + return NextResponse.json( + { error: "INVALID_REQUEST", message: "limit must be an integer between 1 and 500" }, + { status: 400 }, + ); + } + + const repo = repository(); + // CRITICAL: the dispatcher must bypass the durable wrapper. Using the + // default client here would enqueue the event again and could create an + // infinite outbox loop instead of delivering to AUD0. + const rawAud0 = createAud0Client(undefined, { durable: false }); + const dispatcher = new Aud0OutboxDispatcher(repo, rawAud0); + const result = await dispatcher.sweep(limit); + const stats = await repo.stats(); + return NextResponse.json({ result, stats }); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function GET(request: NextRequest) { + const denied = assertInternal(request); + if (denied) return denied; + + try { + return NextResponse.json(await repository().stats()); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/search/[indexName]/query/route.ts b/app/api/0pt/search/[indexName]/query/route.ts new file mode 100644 index 0000000..701102f --- /dev/null +++ b/app/api/0pt/search/[indexName]/query/route.ts @@ -0,0 +1,65 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { assertTenantMatchesVerified } from "@/src/http/tenant-guard"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; +import { getSearchService } from "@/src/services/search"; + +function primitiveFilters(value: unknown): Record | undefined { + if (value === undefined) return undefined; + if (!value || typeof value !== "object" || Array.isArray(value)) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "filters must be an object"); + } + const result: Record = {}; + for (const [key, item] of Object.entries(value as Record)) { + if (typeof item !== "string" && typeof item !== "number" && typeof item !== "boolean") { + throw new SiaxError("INVALID_REQUEST", "0pt.search", `filter ${key} must be string, number or boolean`); + } + result[key] = item; + } + return result; +} + +export async function POST( + request: NextRequest, + { params }: { params: Promise<{ indexName: string }> }, +) { + try { + const { indexName } = await params; + const raw = await request.json(); + if (!raw || typeof raw !== "object" || Array.isArray(raw)) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "JSON object body is required"); + } + const body = raw as Record; + const tenantId = typeof body.tenant_id === "string" ? body.tenant_id.trim() : ""; + const purpose = typeof body.purpose === "string" ? body.purpose.trim() : ""; + if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.search", "tenant_id is required"); + if (!purpose) throw new SiaxError("INVALID_REQUEST", "0pt.search", "purpose is required"); + const guard = assertTenantMatchesVerified(request, tenantId); + if (guard) return guard; + + // Public/authenticated traffic is deliberately fixed to internal rights. + // A client-supplied higher rights_tag would be privilege escalation by + // request body; broader tags require a future CL0UD-derived entitlement. + if (body.rights_tag !== undefined && body.rights_tag !== "internal") { + throw new SiaxError( + "AUTHORIZATION_DENIED", + "0pt.search", + "rights_tag cannot be elevated by the caller", + ); + } + + const result = await getSearchService().query(indexName, { + tenant_id: tenantId, + purpose, + rights_tag: "internal", + ...(typeof body.text === "string" ? { text: body.text } : {}), + ...(body.from === undefined ? {} : { from: Number(body.from) }), + ...(body.size === undefined ? {} : { size: Number(body.size) }), + ...(body.filters === undefined ? {} : { filters: primitiveFilters(body.filters) }), + }); + return NextResponse.json(result); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/app/api/0pt/search/definitions/route.ts b/app/api/0pt/search/definitions/route.ts new file mode 100644 index 0000000..521a7e0 --- /dev/null +++ b/app/api/0pt/search/definitions/route.ts @@ -0,0 +1,58 @@ +import { NextRequest, NextResponse } from "next/server"; + +import { verifyInternalServiceToken } from "@/src/auth/internal-token"; +import { siaxErrorResponse } from "@/src/http/siax-error-response"; +import { SiaxError } from "@/src/platform/errors"; +import { getSearchService } from "@/src/services/search"; +import type { SearchFieldType } from "@/src/search/search-types"; + +const FIELD_TYPES = new Set(["keyword", "text", "long", "double", "date", "boolean"]); + +export async function POST(request: NextRequest) { + if (!verifyInternalServiceToken(request.headers)) { + return NextResponse.json({ error: "UNAUTHENTICATED", message: "valid X-Internal-Token is required" }, { status: 401 }); + } + try { + const raw = await request.json(); + if (!raw || typeof raw !== "object" || Array.isArray(raw)) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "JSON object body is required"); + } + const body = raw as Record; + const indexName = typeof body.index_name === "string" ? body.index_name.trim() : ""; + const version = Number(body.version); + if (!body.fields || typeof body.fields !== "object" || Array.isArray(body.fields)) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "fields must be an object"); + } + const fields: Record = {}; + for (const [name, rawType] of Object.entries(body.fields as Record)) { + if (!name.trim() || typeof rawType !== "string" || !FIELD_TYPES.has(rawType as SearchFieldType)) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", `invalid field definition for ${name || ""}`); + } + fields[name.trim()] = rawType as SearchFieldType; + } + if (!Array.isArray(body.default_search_fields) || !body.default_search_fields.every((v) => typeof v === "string")) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "default_search_fields must be a string array"); + } + + const definition = await getSearchService().registerIndexDefinitionDurable( + { + index_name: indexName, + version, + fields, + default_search_fields: body.default_search_fields.map((v) => String(v).trim()), + }, + request.headers.get("x-0pt-verified-sub") ?? "svc_0pt", + ); + return NextResponse.json(definition, { status: 201 }); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} + +export async function GET() { + try { + return NextResponse.json(await getSearchService().listIndexDefinitionsDurable()); + } catch (error: unknown) { + return siaxErrorResponse(error); + } +} diff --git a/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md new file mode 100644 index 0000000..842f9df --- /dev/null +++ b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md @@ -0,0 +1,197 @@ +# 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan + +Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN +Date: 2026-09-07 + +0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record. + +## 1. Ecosystem ownership + +### Work / j0b + +0pt may supply: +- company/employer/end-client enrichment; +- registry/company facts; +- generic source acquisition; +- buyer/supplier enrichment for procurement; +- entity-resolution candidates/evidence; +- public technology/company signals for account intelligence. + +j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions. + +### PRESS0 + +0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions. + +### FUND0 + +0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY. + +### FIT0 / CRM + +0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth. + +## 2. Provenance-first invariant + +Every externally derived fact or observation must preserve enough lineage to answer: +- source/provider; +- external identifier or URL where appropriate; +- observed/fetched time; +- parser/transform version; +- freshness/confidence where meaningful; +- rights/reuse/retention classification; +- payload/artifact hash or ST0RE ref where raw retention is required. + +No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule. + +## 3. Lossless acquisition contract + +When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified. + +Target acquisition-receipt fields: +- source/provider; +- external ID/URL; +- fetched/observed times; +- payload/artifact hash/ref; +- parser/version; +- rights/reuse/retention classification; +- data classification; +- extracted observations/provenance; +- correlation/tenant/project scope. + +## 4. Canonical entity-resolution architecture + +There must be one decision architecture, not competing production scorers. + +### 4.1 Synchronous candidate path + +The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides: +- deterministic blocking; +- explainable similarity evidence; +- product hard-rule hooks; +- review-band candidates; +- stable-ID hard match/non-match rules at the product boundary. + +For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match. + +### 4.2 Python ingestion path + +The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering. + +### 4.3 Heavy/statistical ER + +Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract. + +## 5. Strong company identity keys + +Prefer strong keys where available: +- LEI (global scope); +- country + official registration number; +- VAT/official registry identifiers with their correct jurisdictional scope; +- other verified registry IDs. + +National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier. + +Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges. + +## 6. Human review boundary + +Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced. + +## 7. Public procurement enrichment + +Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence. + +## 8. Source-policy metadata + +Products may pass/receive policy metadata for: +- access method; +- reuse/redistribution; +- retention; +- polling/rate limits; +- terms/license review date; +- purpose/data classification. + +0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose. + +## 9. PRESS0 rights boundary + +Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision. + +## 10. Company verification boundary + +0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row. + +## 11. Market/account signals + +Useful dated observations include: +- hiring activity; +- public procurement notices/awards; +- public technology/platform signals; +- company growth/change events; +- leadership/public contact changes where lawful; +- relevant product/market announcements. + +Signals are provenance-bearing observations with observation time and freshness, not permanent assertions. + +## 12. API contract + +Generic typed capabilities should cover: +- company/entity enrichment; +- source acquisition receipt; +- structured observations with provenance; +- entity-resolution candidates; +- batch ER job submission; +- provider/source health/freshness; +- durable dataset/version/search surfaces where implemented. + +Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling. + +## 13. Security/privacy + +Required controls include: +- tenant/project/purpose scoping; +- PII minimisation; +- controlled egress/allowed-host policy; +- SSRF protection for user-supplied URLs; +- canonical secret storage; +- no raw sensitive payloads in logs; +- retention/deletion support; +- provider/license/data-rights documentation; +- fail-closed identity merge behavior; +- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required. + +## 14. Production gate + +Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED. + +Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence. + +## 15. Implementation order + +1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners. +2. Keep the synchronous TS ER engine as the canonical candidate/decision contract. +3. Keep Python ingestion as a fail-closed adapter, never a second scorer. +4. Complete durable acquisition/provenance/dataset-version/search contracts. +5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review. +6. Wire j0b company/end-client/buyer enrichment through typed APIs. +7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth. +8. Add PRESS0/FUND0 consumers only when their repos/contracts are real. +9. Load/performance/backup/restore/live-verify every mission-critical path. + +## 16. Definition of Done per data capability + +- source/provider documented; +- legal/license/data-use classification recorded; +- provenance/freshness/confidence available where meaningful; +- deterministic/idempotent normalization where expected; +- entity resolution tested against labelled truth data; +- fuzzy evidence cannot silently become identity truth; +- tenant/purpose/security negative tests pass; +- provider outage/schema drift handled fail closed or explicitly degraded; +- raw artifact retention/deletion contract works where required; +- exact production endpoint/dependency is live verified; +- product consumes API/contract, not another product's DB; +- status documentation reflects observed reality. + +Target ecosystem uses in this document do not claim they are already wired or live. \ No newline at end of file diff --git a/middleware.ts b/middleware.ts index 1e8921f..d439dda 100644 --- a/middleware.ts +++ b/middleware.ts @@ -1,53 +1,17 @@ /** - * `middleware.ts` -- Next.js request middleware = 0pt's framework-level auth - * seam (the Next analog of a Fastify `onRequest` hook). + * Next.js auth seam for the complete /api/0pt capability surface. * - * This runs the Zitadel verification (src/auth/context.ts -> - * src/auth/zitadel.ts) in front of every 0pt capability route (/api/0pt/*). - * It does NOT touch the v0-generated UI routes or the unrelated - * payment/order APIs (see the matcher below and the note in the PR that - * introduced the fail-closed default). - * - * AUTH_ENFORCE unset, or anything except "false" (DEFAULT, enforced): - * a valid verified token is required; 401 on missing/invalid, 403 when - * X-Tenant-Id disagrees with the token tenant. On success the verified - * principal is forwarded to the handler as `x-0pt-verified-sub` / - * `x-0pt-verified-tenant` request headers. - * AUTH_ENFORCE == "false" (explicit local-dev escape hatch): never rejects. - * REFUSED in production -- `assertAuthConfigSafe()` below throws at module - * load, so a production instance cannot boot with auth switched off. - * - * Health/metrics are exempt (they are also outside the matcher below, but the - * exemption is kept explicit for robustness). - * - * FIX (2026-08-31): matcher now covers BOTH /api/0pt (exact) and /api/0pt/:path* - * to close the gap where a bare GET /api/0pt (no trailing path) was routed around - * the middleware on Vercel's Edge runtime. - * - * FIX (P014): one route under this matcher, /api/0pt/quality/bridge-run, is - * not called by a customer's Zitadel-authenticated session at all -- it is - * called BY the Python engine (services/api/workers/quality_bridge.py) using - * the pre-existing X-Internal-Token shared-secret convention, never a Bearer - * token. Before this fix, that meant middleware always rejected the engine's - * real call with 401 (see src/auth/internal-token.ts's doc comment for the - * full story) -- the quality bridge was permanently unreachable in any - * environment enforcing auth. INTERNAL_ONLY_ROUTES lets a request to that - * ONE path through on a verified X-Internal-Token instead of a Bearer token; - * every other /api/0pt/* route is untouched and still requires the normal - * Zitadel flow below. A request to that route WITHOUT a valid internal token - * still falls through to the ordinary Bearer check (e.g. an operator testing - * it manually with their own token), it is not a blanket bypass. + * Customer calls use verified Zitadel bearer identity and receive immutable + * x-0pt-verified-* headers. Explicit machine routes may instead use the + * shared X-Internal-Token convention; mutation handlers verify it again for + * defence in depth. */ - import { NextResponse, type NextRequest } from 'next/server'; import { assertAuthConfigSafe, resolveAuthContext } from '@/src/auth/context'; import { verifyInternalServiceToken } from '@/src/auth/internal-token'; -// STARTUP GATE: fail to boot rather than serve /api/0pt/* openly in production. assertAuthConfigSafe(); -// Only guard the 0pt capability surface. The matcher keeps this middleware off -// the UI, auth, and payment routes entirely. export const config = { matcher: ['/api/0pt', '/api/0pt/:path*'], }; @@ -58,26 +22,21 @@ const EXEMPT_PATTERNS: RegExp[] = [ /^\/api\/admin\/metrics(\/|$)/, ]; -/** The one 0pt route reached by a service (the Python engine), not a user session. */ -const INTERNAL_ONLY_ROUTES: RegExp[] = [/^\/api\/0pt\/quality\/bridge-run$/]; +/** Routes that may be reached with a service token instead of end-user bearer auth. */ +const INTERNAL_SERVICE_ROUTES: RegExp[] = [ + /^\/api\/0pt\/quality\/bridge-run$/, + /^\/api\/0pt\/internal\/evidence-outbox\/dispatch$/, + /^\/api\/0pt\/internal\/er\/candidates$/, + /^\/api\/0pt\/search\/definitions$/, +]; -// Module-level diagnostics counter — first 5 auth requests in a process lifetime -// log their outcome so the production auth gap (S1) can be traced to root cause. let _authDiagCount = 0; export async function middleware(req: NextRequest): Promise { const path = req.nextUrl.pathname; - if (EXEMPT_PATTERNS.some((re) => re.test(path))) { - return NextResponse.next(); - } + if (EXEMPT_PATTERNS.some((re) => re.test(path))) return NextResponse.next(); - if (INTERNAL_ONLY_ROUTES.some((re) => re.test(path)) && verifyInternalServiceToken(req.headers)) { - // Machine-to-machine call, verified via the shared X-Internal-Token - // secret rather than a user's Bearer token. Deliberately does NOT set - // x-0pt-verified-* headers -- there is no per-user principal here, and - // the route requires+validates tenant_id from its own request body - // (qualityBridgeRunSchema), which is the actual tenant the engine is - // running the order for. + if (INTERNAL_SERVICE_ROUTES.some((re) => re.test(path)) && verifyInternalServiceToken(req.headers)) { return NextResponse.next(); } @@ -92,7 +51,6 @@ export async function middleware(req: NextRequest): Promise { } const decision = await resolveAuthContext(req.headers); - if (!decision.ok) { return NextResponse.json( { error: decision.code, message: decision.message, ...(decision.details ? { details: decision.details } : {}) }, diff --git a/services/api/tests/test_resolve_determinism.py b/services/api/tests/test_resolve_determinism.py index b51bed4..fa2bc7a 100644 --- a/services/api/tests/test_resolve_determinism.py +++ b/services/api/tests/test_resolve_determinism.py @@ -1,200 +1,166 @@ -""" -Fault-injection tests for workers/resolve.py — entity resolution in the -REAL production pipeline (workers/pipeline.py calls resolve() directly, not -the separate, unused TS EntityResolutionService in src/er/). +"""Regression tests for the Python adapter to canonical TypeScript ER. -BEFORE_STATE.md §4: "dedup-determinism [...] helt frånvarande — särskilt -anmärkningsvärt då resolve.py [...] är ordningskänslig by construction och -helt otestad för determinism." This file closes that gap against the real -function, not a mock. - -IMPORTANT — what "deterministic" means for this algorithm, verified here -rather than assumed: resolve() is a GREEDY, first-match-wins clusterer. The -surviving record's own scalar fields (e.g. `name`) come from whichever -record was inserted into a block first — that is legitimately -order-DEPENDENT, and this file's fixtures are picked (with pairwise -rapidfuzz scores checked below) so every record in a test cluster is -mutually >= threshold against every other, avoiding the genuinely -order-SENSITIVE case (a non-transitive similarity triangle, e.g. -sim(A,B)>=t, sim(B,C)>=t, sim(A,C) identical clustering. - - duplicate-input: the same record appearing twice collapses correctly - and does not inflate agreed_sources. - - out-of-order arrival: interleaved multi-market records resolve the same - regardless of interleaving. - - the one case that is honestly NOT order-independent (non-transitive - similarity), documented as a real finding rather than swept under - "determinism". +The removed Python scorer is deliberately not tested because it no longer +exists. These tests pin the load-bearing adapter semantics instead: +canonical match edges merge; review/non-match edges do not; hard negative +identity evidence constrains the whole cluster; bridge failure leaves records +unmerged. """ from __future__ import annotations import copy -import itertools -import random +import httpx import pytest +import respx -from workers.resolve import resolve +from workers.resolve import _apply_matches, resolve -def _prov(field_name: str, source_id: str, confidence: float, level: int = 3) -> dict: +def _prov(field_name: str, source_id: str, confidence: float) -> dict: return { - "field_name": field_name, "source_id": source_id, "confidence": confidence, "level": level, - "freshness": "2026-09-01", "retrieved_at": "2026-09-01T12:00:00+00:00", - "transform": "raw_fetch", "connector_version": f"{source_id}-connector@1", - "producer": "0pt-engine", "quality": None, "entity_resolution": None, "artifact_ref": None, + "field_name": field_name, + "source_id": source_id, + "confidence": confidence, + "entity_resolution": None, } -def _rec(market: str, name: str, source: str, confidence: float) -> dict: +def _rec(name: str, source: str, confidence: float = 0.9) -> dict: return { - "market": market, "name": name, "status": "active", "lei": None, "org_nr": None, - "source": source, "confidence": confidence, + "market": "SE", + "name": name, + "status": "active", + "lei": None, + "org_nr": None, + "source": source, + "confidence": confidence, "provenance": [_prov("name", source, confidence)], } -# Three independent sources reporting near-identical spellings of the SAME -# company. Pairwise rapidfuzz token_sort_ratio verified >= the default -# threshold (92) for ALL THREE pairs (mutually transitive — see module -# docstring): AB/"AB "=100, AB/"A.B."=95.45, "AB "/"A.B."=95.45. -VOLVO_GLEIF = _rec("NO", "Volvo Personvagnar AB", "gleif", 0.97) -VOLVO_BRREG = _rec("NO", "Volvo Personvagnar AB ", "brreg", 0.97) -VOLVO_CVR = _rec("NO", "Volvo Personvagnar A.B.", "cvr", 0.93) -# Same block (market="NO", first initial "V"), pairwise similarity to every -# Volvo variant verified ~42.9 (well below threshold) — must never merge. -VILDANDEN = _rec("NO", "Vildanden Regnskap AS", "brreg", 0.9) +def _pair(a: str, b: str, status: str = "match", decided_by: str = "score", score: float = 0.97) -> dict: + return { + "candidate_id": f"c_{a}_{b}", + "record_a": a, + "record_b": b, + "score": score, + "status": status, + "decided_by": decided_by, + } -def _clusters(records: list[dict]) -> list[frozenset]: - """Order-independent identity: WHICH sources ended up merged together, - ignoring which record's own `name`/`status`/etc. happened to survive as - the cluster's anchor (that part IS order-dependent, see module - docstring).""" - return sorted( - (frozenset([r["source"], *r.get("merged_from", [])]) for r in records), - key=lambda s: sorted(s), - ) +def test_apply_matches_is_deterministic_and_uses_canonical_edges_only(): + records = [_rec("Acme AB", "source-a", 0.91), _rec("ACME Aktiebolag", "source-b", 0.96)] + assert len(_apply_matches(copy.deepcopy(records), [])) == 2 -class TestDedupDeterminism: - def test_shuffled_orderings_produce_identical_clustering(self): - base = [VOLVO_GLEIF, VOLVO_BRREG, VOLVO_CVR, VILDANDEN] - baseline = resolve([copy.deepcopy(r) for r in base]) - assert len(baseline) == 2 # Volvo (merged x3) + Vildanden - baseline_clusters = _clusters(baseline) +@pytest.mark.asyncio +@respx.mock +async def test_canonical_match_merges_and_stamps_provenance(monkeypatch): + monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test") + monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token") + records = [_rec("Acme AB", "source-a", 0.91), _rec("Acme AB", "source-b", 0.96)] - rng = random.Random(1337) - for _ in range(30): - shuffled = [copy.deepcopy(r) for r in base] - rng.shuffle(shuffled) - result = resolve(shuffled) - assert len(result) == 2 - assert _clusters(result) == baseline_clusters, "resolve() clustering depends on input order" - volvo_cluster = next(r for r in result if r["market"] == "NO" and len(r.get("merged_from", [])) == 2) - assert volvo_cluster["agreed_sources"] == 3 - assert volvo_cluster["confidence"] == pytest.approx(min(0.99, 0.97 + 0.1 + 0.1)) + route = respx.post("https://edge.example.test/api/0pt/internal/er/candidates") - def test_all_permutations_of_a_mutually_similar_set_agree(self): - # Exhaustive, not sampled — small enough (3! = 6) to check every ordering. - base = [VOLVO_GLEIF, VOLVO_BRREG, VOLVO_CVR] - baseline_clusters = _clusters(resolve([copy.deepcopy(r) for r in base])) - for perm in itertools.permutations(base): - result = resolve([copy.deepcopy(r) for r in perm]) - assert _clusters(result) == baseline_clusters + def responder(request: httpx.Request) -> httpx.Response: + body = __import__("json").loads(request.content) + assert request.headers["X-Internal-Token"] == "test-internal-token" + assert body["tenant_id"] == "tenant-a" + assert body["blocking_fields"] == ["market", "block_initial"] + a, b = body["records"][0]["record_id"], body["records"][1]["record_id"] + # Endast hard-rule-beslut (decided_by != score/None) far skapa en + # canonical identity-lank - consumenten ignorerar medvetet score/"match". + return httpx.Response( + 200, + json={"pairs": [_pair(a, b, "match", "0pt.company.stable-identifiers.v3", 0.99)], "review_queue": []}, + ) + + route.side_effect = responder + result = await resolve(records, tenant_id="tenant-a", project_id="project-a") + + assert len(result) == 1 + assert result[0]["agreed_sources"] == 2 + assert result[0]["confidence"] == pytest.approx(0.99) + assert set(result[0]["merged_from"]) <= {"source-a", "source-b"} + for provenance in result[0]["provenance"]: + er = provenance["entity_resolution"] + assert er["method"] == "canonical_ts_entity_resolution" + assert set(er["merged_sources"]) == {"source-a", "source-b"} -class TestDuplicateInput: - def test_the_same_record_twice_collapses_without_double_counting_agreement(self): - one = copy.deepcopy(VOLVO_GLEIF) - dup = copy.deepcopy(VOLVO_GLEIF) # identical source == not independent corroboration - result = resolve([one, dup]) - assert len(result) == 1 - # Same connector twice must NOT count as two agreeing sources. - assert result[0]["agreed_sources"] == 1 - assert result[0]["confidence"] == VOLVO_GLEIF["confidence"] +@pytest.mark.asyncio +@respx.mock +async def test_pending_review_never_auto_merges(monkeypatch): + monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test") + monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token") + records = [_rec("Acme AB", "source-a"), _rec("Acme Group AB", "source-b")] - def test_duplicate_plus_one_genuinely_new_source_only_credits_the_new_one(self): - result = resolve([copy.deepcopy(VOLVO_GLEIF), copy.deepcopy(VOLVO_GLEIF), copy.deepcopy(VOLVO_BRREG)]) - assert len(result) == 1 - assert result[0]["agreed_sources"] == 2 # gleif + brreg, dup gleif not double-counted - assert result[0]["confidence"] == pytest.approx(min(0.99, VOLVO_GLEIF["confidence"] + 0.1)) + def responder(request: httpx.Request) -> httpx.Response: + body = __import__("json").loads(request.content) + a, b = body["records"][0]["record_id"], body["records"][1]["record_id"] + pair = _pair(a, b, "pending_review") + return httpx.Response(200, json={"pairs": [pair], "review_queue": [pair]}) + + respx.post("https://edge.example.test/api/0pt/internal/er/candidates").side_effect = responder + result = await resolve(records, tenant_id="tenant-a", project_id="project-a") + assert len(result) == 2 + assert all(not r.get("merged_from") for r in result) -class TestOutOfOrderArrival: - def test_interleaved_multi_market_arrival_resolves_the_same_as_sequential(self): - no_recs = [VOLVO_GLEIF, VOLVO_BRREG] - gb_recs = [ - _rec("GB", "Tesco PLC", "companies_house", 0.95), - _rec("GB", "Tesco PLC ", "gleif", 0.97), # verified 100.0 vs "Tesco PLC" - ] - sequential = resolve([copy.deepcopy(r) for r in [*no_recs, *gb_recs]]) - sequential_clusters = _clusters(sequential) +@pytest.mark.asyncio +@respx.mock +async def test_hard_non_match_blocks_transitive_cluster_merge(monkeypatch): + """A-B and B-C matches may never imply A=C when canonical hard evidence says A!=C.""" + monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test") + monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token") + records = [ + _rec("Acme AB", "source-a", 0.95), + _rec("Acme AB", "source-b", 0.94), + _rec("Acme AB", "source-c", 0.93), + ] - interleaved = [ - copy.deepcopy(gb_recs[0]), - copy.deepcopy(no_recs[0]), - copy.deepcopy(gb_recs[1]), - copy.deepcopy(no_recs[1]), - ] - result = resolve(interleaved) - assert _clusters(result) == sequential_clusters - assert len(result) == 2 # one merged Volvo, one merged Tesco + def responder(request: httpx.Request) -> httpx.Response: + body = __import__("json").loads(request.content) + a, b, c = [item["record_id"] for item in body["records"]] + return httpx.Response( + 200, + json={ + "pairs": [ + _pair(a, b, "match", "0pt.company.stable-identifiers.v3", 0.98), + _pair(b, c, "match", "0pt.company.stable-identifiers.v3", 0.97), + _pair(a, c, "non_match", "0pt.company.stable-identifiers.v2", 0.0), + ], + "review_queue": [], + }, + ) + + respx.post("https://edge.example.test/api/0pt/internal/er/candidates").side_effect = responder + result = await resolve(records, tenant_id="tenant-a", project_id="project-a") + assert len(result) == 2 + assert max(r.get("agreed_sources", 1) for r in result) == 2 + assert not any(set(r.get("merged_from", [])) >= {"source-b", "source-c"} for r in result) -class TestNonTransitiveSimilarityIsOrderSensitive: - """Documents a real, verified property of the greedy first-match-wins - algorithm rather than papering over it: when similarity is NOT - transitive across three candidates in the same block, which two end up - merged CAN depend on arrival order. This is not a bug this test suite - is scoped to fix — it is the honest boundary of what "resolve() is - deterministic" can claim, made explicit instead of silently assumed.""" +@pytest.mark.asyncio +@respx.mock +async def test_bridge_outage_fails_closed_to_unmerged_records(monkeypatch): + monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test") + monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token") + records = [_rec("Acme AB", "source-a"), _rec("Acme AB", "source-b")] + respx.post("https://edge.example.test/api/0pt/internal/er/candidates").mock(side_effect=httpx.ConnectError("offline")) - def test_a_non_transitive_triangle_can_cluster_differently_by_order(self): - # Verified pairwise: A/B = 80.0, A/C = 97.67, B/C = 78.43 (threshold 92). - # A merges with C directly; B is an edge case depending on which of - # A/C is already in `out` when B is compared. - a = _rec("NO", "Volvo Personvagnar AB", "gleif", 0.97) - b = _rec("NO", "Volvo Personvagnar Aktiebolag", "brreg", 0.95) - c = _rec("NO", "Volvo Personvagnar A/B", "cvr", 0.93) - - order_1 = resolve([copy.deepcopy(a), copy.deepcopy(b), copy.deepcopy(c)]) - order_2 = resolve([copy.deepcopy(b), copy.deepcopy(a), copy.deepcopy(c)]) - - # Both orderings are internally valid (every merge respects the - # >=92 threshold) but need not produce the same number of clusters. - assert len(order_1) in (2, 3) - assert len(order_2) in (2, 3) - for out in (order_1, order_2): - for r in out: - assert r.get("agreed_sources", 1) <= 2 # never over-merges all three + result = await resolve(records, tenant_id="tenant-a", project_id="project-a") + assert len(result) == 2 + assert all(not r.get("merged_from") for r in result) -class TestEntityResolutionProvenance: - """P010 provenance addition: resolve() now stamps `entity_resolution` on - every provenance row of a merged record, and leaves it None on records - that were never merged (never fabricated).""" - - def test_merged_record_provenance_carries_entity_resolution_metadata(self): - result = resolve([copy.deepcopy(VOLVO_GLEIF), copy.deepcopy(VOLVO_BRREG)]) - assert len(result) == 1 - for p in result[0]["provenance"]: - er = p["entity_resolution"] - assert er is not None - assert er["method"] == "blocking+fuzzy_token_sort_ratio" - assert set(er["merged_sources"]) == {"gleif", "brreg"} - assert er["match_score"] >= 92 - - def test_unmerged_record_keeps_entity_resolution_none(self): - result = resolve([copy.deepcopy(VILDANDEN)]) - assert len(result) == 1 - assert result[0]["provenance"][0]["entity_resolution"] is None +@pytest.mark.asyncio +async def test_missing_internal_configuration_makes_zero_network_calls(monkeypatch): + monkeypatch.delenv("OPT_ER_BRIDGE_URL", raising=False) + monkeypatch.delenv("EDGE_URL", raising=False) + monkeypatch.delenv("INTERNAL_TOKEN", raising=False) + records = [_rec("Acme AB", "source-a"), _rec("Acme AB", "source-b")] + result = await resolve(records, tenant_id="tenant-a", project_id="project-a") + assert len(result) == 2 diff --git a/services/api/workers/pipeline.py b/services/api/workers/pipeline.py index b6da183..44e6b90 100644 --- a/services/api/workers/pipeline.py +++ b/services/api/workers/pipeline.py @@ -1,23 +1,9 @@ -""" -Production pipeline (docs/06): sourcing -> extraction -> structuring -> -enrichment/fusion -> QA. Compliant connectors only; every field gets provenance; -rows below the order's confidence target are dropped (deliver fewer, never fabricate). +"""Production pipeline: sourcing -> extraction -> canonical ER -> QA -> quality. -P010 additions (both flagged in BEFORE_STATE.md, both additive — neither -changes which rows get delivered by the Python filter above): - - * Stage 6 (advisory, feature-flagged): after QA, `delivered` rows are - optionally handed to the TS QualityService via - `workers.quality_bridge.run_quality_bridge` — see that module's - docstring for the full contract, the fail-open guarantee, and why - per-row gating isn't implemented yet. Disabled unless - `OPT_QUALITY_BRIDGE_ENABLED=true`. - * Every delivered record's provenance rows are stamped with `producer`, - `connector_version` (both already set at fetch time in - sources/connectors.py), and an `artifact_ref` identifying this - delivery (`0pt:pipeline_delivery:`) — plus, when the quality - bridge ran successfully, a `quality` block. BEFORE_STATE.md §3 found - these fields absent from the actual dataflow entirely. +Identity decisions are owned by src/er/entity-resolution-service.ts. The +Python worker only applies canonical `match` decisions returned by its +internal bridge. If tenant context or the canonical bridge is unavailable, +records remain unmerged: false identity merges are never used as fallback. """ from __future__ import annotations @@ -26,7 +12,6 @@ from typing import Any from workers.resolve import resolve from workers.quality_bridge import run_quality_bridge - from sources.connectors import ( CompanyResult, SourceBlockedError, @@ -39,53 +24,38 @@ from sources.connectors import ( from workers.adaptive_router import default_router, plan_for_market logger = logging.getLogger(__name__) +PIPELINE_VERSION = "0pt-pipeline@2" -PIPELINE_VERSION = "0pt-pipeline@1" - -# Minimal built-in seed so a job is demonstrable without customer input. DEMO_SEEDS: dict[str, list[str]] = { - "NO": ["923609016"], # brreg orgnr (example) - "GB": ["00000006"], # companies house number (example) + "NO": ["923609016"], + "GB": ["00000006"], } async def _fetch(source_id: str, key: str) -> CompanyResult | None: - if source_id == "gleif": - return await gleif_by_lei(key) - if source_id == "brreg": - return await brreg_by_orgnr(key) - if source_id == "companies_house": - return await companies_house_by_number(key) - if source_id == "prh": - return await prh_by_businessid(key) - if source_id == "cvr": - return await cvr_by_number(key) + if source_id == "gleif": return await gleif_by_lei(key) + if source_id == "brreg": return await brreg_by_orgnr(key) + if source_id == "companies_house": return await companies_house_by_number(key) + if source_id == "prh": return await prh_by_businessid(key) + if source_id == "cvr": return await cvr_by_number(key) return None async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]: markets: list[str] = config.get("markets", []) or ["NO"] threshold = float(config.get("confidence", 92)) / 100.0 - # Use caller-supplied seeds when present (an explicit empty {} is honoured so a - # real job is never silently backfilled with DEMO_SEEDS), but treat an explicit - # null the same as "missing" -- `config.get("seeds") is not None` avoids a 500 - # (None.get) on client-controlled `{"seeds": null}`, which the old `... or ...` - # idiom tolerated and the `"seeds" in config` form regressed. seeds: dict[str, list[str]] = config["seeds"] if config.get("seeds") is not None else DEMO_SEEDS + tenant_id = config.get("tenant_id") + project_id = config.get("project_id", "p_default") + correlation_id = config.get("correlation_id") records: list[dict[str, Any]] = [] per_market: dict[str, int] = {} - # 1) sourcing + 2) extraction for market in markets: keys = seeds.get(market, []) got = 0 for key in keys: - # Recomputed per key (not hoisted above this loop): a source - # blocked while fetching an earlier key in this same market must - # be excluded from the plan for every key still to come in this - # run, not just retried forever with no memory (P014 -- see - # adaptive_router.py's module docstring). plan = plan_for_market(market) for source_id in plan: try: @@ -97,57 +67,55 @@ async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]: res = None if res and res.name: records.append({ - "market": market, "name": res.name, "status": res.status, - "lei": res.lei, "org_nr": res.org_nr, "source": res.source, + "market": market, + "name": res.name, + "status": res.status, + "lei": res.lei, + "org_nr": res.org_nr, + "source": res.source, "provenance": [p.__dict__ for p in res.provenance], "confidence": max((p.confidence for p in res.provenance), default=0.0), }) got += 1 - break # first compliant source that answers wins + break per_market[market] = got - # 3) structuring — entity resolution (blocking + fuzzy match + corroboration) - deduped = resolve(records) + # Canonical ER is tenant-bound. Missing tenant context must never be + # papered over with a synthetic tenant or the removed Python scorer. + if tenant_id: + deduped = await resolve( + records, + tenant_id=tenant_id, + project_id=project_id, + correlation_id=correlation_id, + ) + er_mode = "canonical_ts" + else: + deduped = [dict(r) for r in records] + er_mode = "fail_closed_unmerged_missing_tenant" - # 4) enrichment/fusion already folded into confidence above. - # 5) QA — drop rows below the order's confidence threshold. THIS is the - # filter that actually governs delivery; nothing below is allowed to - # change its outcome. delivered = [r for r in deduped if r["confidence"] >= threshold] dropped = len(deduped) - len(delivered) avg_conf = round(sum(r["confidence"] for r in delivered) / len(delivered), 3) if delivered else 0.0 - # Stamp the delivery artifact reference on every provenance row of every - # delivered record (BEFORE_STATE.md §3: `artifact_ref` did not exist - # anywhere). This identifies WHICH delivery a field's value shipped in; - # it does not assert the row was ever written to ST0RE (not wired yet). artifact_ref = f"0pt:pipeline_delivery:{order_id}" for r in delivered: for p in r.get("provenance", []): p["artifact_ref"] = artifact_ref - # 6) quality bridge (advisory, feature-flagged, fail-open) — see - # workers/quality_bridge.py's docstring for the full contract. Runs - # AFTER the QA gate above and can never change `delivered`. - tenant_id = config.get("tenant_id") quality_bridge_metrics: dict[str, Any] if not tenant_id: - # The engine has no tenant context yet for orders created through - # this API (orchestrator/main.py never threads tenant_id into - # `config` — a pre-existing, separately-tracked gap). Rather than - # fabricate a tenant to make the bridge call "work", skip it - # honestly and say why. quality_bridge_metrics = { "status": "not_configured", - "detail": "config.tenant_id missing — engine has no tenant context for this order", + "detail": "config.tenant_id missing — quality bridge not called", } else: bridge_result = await run_quality_bridge( tenant_id=tenant_id, - project_id=config.get("project_id", "p_default"), + project_id=project_id, dataset_id=f"0pt-order-{order_id}", rows=delivered, - correlation_id=config.get("correlation_id"), + correlation_id=correlation_id, ) quality_bridge_metrics = bridge_result.as_metrics() if bridge_result.status == "ok": @@ -167,11 +135,16 @@ async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]: ) return { - "stage": "qa", "pct": 100, "state": "done", + "stage": "qa", + "pct": 100, + "state": "done", "metrics": { - "rows": len(delivered), "dropped_below_threshold": dropped, - "per_market": per_market, "avg_confidence": avg_conf, + "rows": len(delivered), + "dropped_below_threshold": dropped, + "per_market": per_market, + "avg_confidence": avg_conf, "pipeline_version": PIPELINE_VERSION, + "entity_resolution": er_mode, "quality_bridge": quality_bridge_metrics, }, "records": delivered, diff --git a/services/api/workers/resolve.py b/services/api/workers/resolve.py index ae50623..8090ad5 100644 --- a/services/api/workers/resolve.py +++ b/services/api/workers/resolve.py @@ -1,117 +1,235 @@ -""" -Entity resolution (docs/06): block, score, and link records across sources so the -same company from GLEIF + Brreg + CVR collapses to one record with corroborated -confidence. rapidfuzz only; deterministic; no external calls. +"""Thin adapter from the Python pipeline to 0pt's canonical TS ER engine. -======================================================================= -P065 — THIS IS THE SECOND ER ENGINE. IT IS NOT THE CANONICAL ONE. -======================================================================= -0pt has TWO entity-resolution implementations, and this is the simplified -one: +Python no longer scores identity. It sends opaque records to +/api/0pt/internal/er/candidates, then applies only explicit hard-rule `match` +decisions. Score/fuzzy decisions are review evidence only and can never merge +records in this adapter, even if an older or compromised bridge returns +`status=match, decided_by=score`. - * CANONICAL: src/er/entity-resolution-service.ts — the declared generic ER - capability. Deterministic blocking + an explainable per-pair score, - product-supplied HARD RULES that override the score in both directions, - a configurable review band that hands ambiguous pairs to humans instead - of auto-merging, and delegation of heavy probabilistic linkage (Splink) - to 0pt-motor via the N0D job "0pt.er.batch_resolution". +If the canonical engine is unavailable, records are returned UNMERGED (fail +closed against false identity merges). - * THIS FILE: blocking on (market, first initial) + a single - rapidfuzz token_sort_ratio threshold, first-match-wins, auto-merge with - no review band, no hard-rule hook, no explanation beyond the score. - -P065's definition of done is that no second simplified production engine -remains. It has NOT been met: this file is still what -`workers/pipeline.py` calls (line ~110, `deduped = resolve(records)`). - -WHY IT WAS NOT REPLACED IN THE P060-P068 PASS, stated plainly rather than -quietly deferred: replacing it means turning this module into an ADAPTER -that calls the canonical capability, and that change cannot be verified -from this repo today — - * `services/api` is not deployed by this repo's Dockerfile (which builds - only the Next.js app), and api.0pt.app answers 404, so there is no - running engine to test against; - * no CI workflow runs services/api's pytest suite at all (verified: - `grep -rn "pytest" .gitea/workflows .github/workflows` finds nothing), - so a rewrite here would land with zero automated coverage. -Rewriting it blind and calling P065 done would be exactly the -"claimed complete, broken in operation" failure this programme exists to -stop. The removal plan is recorded in STATUS.md. - -Until then: do NOT add capability to this file. Any new ER behaviour -belongs in the canonical TypeScript service. - -Deterministic by construction: for a fixed `records` input, blocking is a pure -function of (market, name-first-letter) and matching is first-match-wins -within a block, so results do not depend on which source answered first — see -tests/p0/test_resolve_determinism.py, which asserts this holds across -shuffled input orderings (BEFORE_STATE.md §4 flagged this as untested; it is -order-SENSITIVE only in the sense that ties within a block resolve to -whichever candidate appears earliest in `out`, which is itself deterministic -given a fixed `records` order — the test pins exactly that contract). +A canonical hard non-match is also a cluster-wide constraint: transitive +matches are never allowed to pull two records with conflicting stable identity +evidence into the same entity. """ from __future__ import annotations +import copy +import hashlib +import json +import os from typing import Any -from rapidfuzz import fuzz +import httpx + +_TIMEOUT = httpx.Timeout(6.0, connect=2.0) -def block_key(rec: dict[str, Any]) -> tuple[str, str]: - """Cheap blocking key — same market + first initial — to avoid O(n^2).""" - return (rec.get("market") or "", (rec.get("name") or "")[:1].upper()) +def _url() -> str | None: + base = os.getenv("OPT_ER_BRIDGE_URL") or os.getenv("EDGE_URL") + return f"{base.rstrip('/')}/api/0pt/internal/er/candidates" if base else None -def resolve(records: list[dict[str, Any]], threshold: int = 92) -> list[dict[str, Any]]: +def _record_id(record: dict[str, Any]) -> str: + stable = { + "market": record.get("market"), + "name": record.get("name"), + "status": record.get("status"), + "lei": record.get("lei"), + "org_nr": record.get("org_nr"), + "source": record.get("source"), + "provenance": record.get("provenance", []), + } + raw = json.dumps(stable, sort_keys=True, separators=(",", ":"), default=str).encode() + return "erp_" + hashlib.sha256(raw).hexdigest()[:32] + + +def _unique_record_ids(records: list[dict[str, Any]]) -> list[str]: + seen: dict[str, int] = {} + unique_ids: list[str] = [] + for record in records: + rid = _record_id(record) + n = seen.get(rid, 0) + seen[rid] = n + 1 + unique_ids.append(rid if n == 0 else f"{rid}_{n}") + return unique_ids + + +def _apply_matches(records: list[dict[str, Any]], pairs: list[dict[str, Any]]) -> list[dict[str, Any]]: + if not records: + return [] + + unique_ids = _unique_record_ids(records) + known_ids = set(unique_ids) + parent = {rid: rid for rid in unique_ids} + members = {rid: {rid} for rid in unique_ids} + + hard_non_matches: set[frozenset[str]] = set() + for pair in pairs: + a, b = pair.get("record_a"), pair.get("record_b") + if a not in known_ids or b not in known_ids: + continue + if pair.get("status") == "non_match" and pair.get("decided_by") not in (None, "score"): + hard_non_matches.add(frozenset((a, b))) + + def find(x: str) -> str: + while parent[x] != x: + parent[x] = parent[parent[x]] + x = parent[x] + return x + + def components_conflict(left: set[str], right: set[str]) -> bool: + return any(frozenset((a, b)) in hard_non_matches for a in left for b in right) + + def union(a: str, b: str) -> bool: + ra, rb = find(a), find(b) + if ra == rb: + return True + left, right = members[ra], members[rb] + if components_conflict(left, right): + return False + keep, drop = (ra, rb) if ra < rb else (rb, ra) + parent[drop] = keep + members[keep] = members[ra] | members[rb] + members.pop(drop, None) + return True + + # Defence in depth: only stable-identifier hard rules may create canonical + # identity links. Score/fuzzy "match" responses are ignored at the consumer + # boundary even if an upstream/legacy route emits one. + match_pairs = [ + p for p in pairs + if p.get("status") == "match" + and p.get("decided_by") not in (None, "score") + and p.get("record_a") in known_ids + and p.get("record_b") in known_ids + ] + match_pairs.sort( + key=lambda p: ( + -float(p.get("score") or 0.0), + str(p.get("candidate_id") or ""), + ) + ) + accepted_edges: list[dict[str, Any]] = [] + for pair in match_pairs: + a, b = str(pair["record_a"]), str(pair["record_b"]) + if union(a, b): + accepted_edges.append(pair) + + groups: dict[str, list[tuple[str, dict[str, Any]]]] = {} + for rid, record in zip(unique_ids, records): + groups.setdefault(find(rid), []).append((rid, copy.deepcopy(record))) + out: list[dict[str, Any]] = [] - for r in records: - match = None - match_score: float | None = None - for o in out: - if block_key(o) != block_key(r): - continue - score = fuzz.token_sort_ratio(o.get("name", ""), r.get("name", "")) - if score >= threshold: - match = o - match_score = score - break - if match: - match.setdefault("merged_from", []).append(r.get("source")) - # Merge the losing record's per-field provenance regardless of source - # novelty below, so the surviving record's provenance always reflects - # every contributing record's fields. - existing_provenance = match.setdefault("provenance", []) - seen_provenance = {(p.get("field_name"), p.get("source_id")) for p in existing_provenance} - for p in r.get("provenance", []): + for group in groups.values(): + group.sort(key=lambda item: (-float(item[1].get("confidence", 0.0)), item[0])) + _, anchor = group[0] + sources = sorted({str(r.get("source")) for _, r in group if r.get("source")}) + anchor["agreed_sources"] = len(sources) or 1 + anchor["confidence"] = min( + 0.99, + float(anchor.get("confidence", 0.0)) + 0.1 * max(0, len(sources) - 1), + ) + anchor["merged_from"] = [r.get("source") for _, r in group[1:] if r.get("source")] + + provenance = list(anchor.get("provenance", [])) + keys = {(p.get("field_name"), p.get("source_id")) for p in provenance} + for _, record in group[1:]: + for p in record.get("provenance", []): key = (p.get("field_name"), p.get("source_id")) - if key not in seen_provenance: - existing_provenance.append(p) - seen_provenance.add(key) - # Corroboration (confidence/agreed_sources bump) requires an - # *independent* source — two hits from the same connector must not - # count as two agreeing sources. - seen_sources = match.setdefault("_sources_seen", {match.get("source")} - {None}) - src = r.get("source") - if src is not None and src not in seen_sources: - match["confidence"] = min(0.99, match.get("confidence", 0.0) + 0.1) - match["agreed_sources"] = match.get("agreed_sources", 1) + 1 - seen_sources.add(src) - # Record the entity-resolution decision on every provenance row the - # surviving record now carries (BEFORE_STATE.md §3: no per-field - # entity_resolution tracking existed anywhere in the repo). Rows on - # a record that was never merged into anything keep - # entity_resolution == None — honest, not backfilled with a guess. - er_info = { - "method": "blocking+fuzzy_token_sort_ratio", - "block_key": list(block_key(match)), - "match_score": match_score, - "threshold": threshold, - "merged_sources": sorted(s for s in match["_sources_seen"] if s is not None), - } - for p in existing_provenance: + if key not in keys: + provenance.append(copy.deepcopy(p)) + keys.add(key) + + member_ids = {rid for rid, _ in group} + edges = [ + p for p in accepted_edges + if p.get("record_a") in member_ids and p.get("record_b") in member_ids + ] + er_info = { + "method": "canonical_ts_entity_resolution", + "merged_sources": sources, + "matched_edges": [ + { + "candidate_id": p.get("candidate_id"), + "score": p.get("score"), + "decided_by": p.get("decided_by"), + } + for p in edges + ], + } + if len(group) > 1: + for p in provenance: p["entity_resolution"] = er_info - else: - out.append({**r, "agreed_sources": 1, "_sources_seen": {r.get("source")} - {None}}) - for o in out: - o.pop("_sources_seen", None) - return out + anchor["provenance"] = provenance + out.append(anchor) + + return sorted(out, key=lambda r: (_record_id(r), str(r.get("source") or ""))) + + +async def resolve( + records: list[dict[str, Any]], + *, + tenant_id: str, + project_id: str, + correlation_id: str | None = None, +) -> list[dict[str, Any]]: + """Return canonically resolved records; never falls back to local scoring.""" + if len(records) < 2: + return [copy.deepcopy(r) for r in records] + + url = _url() + token = os.getenv("INTERNAL_TOKEN", "") + if not url or not token or not tenant_id or not project_id: + return [copy.deepcopy(r) for r in records] + + unique_ids = _unique_record_ids(records) + er_records = [] + for rid, record in zip(unique_ids, records): + name = str(record.get("name") or "") + market = record.get("market") + er_records.append({ + "record_id": rid, + "fields": { + "market": market, + "registration_country": market, + "block_initial": name[:1].upper(), + "name": name, + "status": record.get("status"), + "lei": record.get("lei"), + "org_nr": record.get("org_nr"), + }, + }) + + payload: dict[str, Any] = { + "tenant_id": tenant_id, + "project_id": project_id, + "records": er_records, + "blocking_fields": ["market", "block_initial"], + "compare_fields": [{"field": "name", "weight": 1}], + "auto_match_threshold": 0.92, + "review_threshold": 0.72, + } + if correlation_id: + payload["correlation_id"] = correlation_id + + try: + async with httpx.AsyncClient(timeout=_TIMEOUT) as client: + response = await client.post( + url, + json=payload, + headers={"Content-Type": "application/json", "X-Internal-Token": token}, + ) + if response.status_code not in (200, 201): + return [copy.deepcopy(r) for r in records] + body = response.json() + pairs = body.get("pairs") + if not isinstance(pairs, list): + return [copy.deepcopy(r) for r in records] + return _apply_matches(records, pairs) + except (httpx.HTTPError, ValueError, TypeError, KeyError): + return [copy.deepcopy(r) for r in records] + + +__all__ = ["resolve", "_apply_matches"] diff --git a/sql/011_aud0_evidence_outbox.sql b/sql/011_aud0_evidence_outbox.sql new file mode 100644 index 0000000..ee7dda6 --- /dev/null +++ b/sql/011_aud0_evidence_outbox.sql @@ -0,0 +1,39 @@ +-- 0pt durable AUD0 evidence outbox (§70) +-- AUD0 remains canonical evidence; this is only a durable producer queue. + +CREATE TABLE IF NOT EXISTS opt_aud0_evidence_outbox ( + outbox_id text PRIMARY KEY, + tenant_id text NOT NULL, + app_id text NOT NULL DEFAULT '0pt', + evidence_kind text NOT NULL CHECK (evidence_kind IN ('event', 'openlineage')), + idempotency_key text NOT NULL, + payload jsonb NOT NULL, + status text NOT NULL DEFAULT 'pending' + CHECK (status IN ('pending', 'delivered', 'dead')), + attempts integer NOT NULL DEFAULT 0 CHECK (attempts >= 0), + next_attempt_at timestamptz NOT NULL DEFAULT now(), + leased_by text, + leased_until timestamptz, + last_error text, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + delivered_at timestamptz, + dead_at timestamptz, + CONSTRAINT opt_aud0_evidence_outbox_idempotency + UNIQUE (tenant_id, evidence_kind, idempotency_key), + CONSTRAINT opt_aud0_evidence_outbox_delivery_consistency CHECK ( + (status = 'delivered' AND delivered_at IS NOT NULL AND dead_at IS NULL) + OR (status = 'dead' AND dead_at IS NOT NULL AND delivered_at IS NULL) + OR (status = 'pending' AND delivered_at IS NULL AND dead_at IS NULL) + ) +); + +CREATE INDEX IF NOT EXISTS opt_aud0_evidence_outbox_pending_idx + ON opt_aud0_evidence_outbox (next_attempt_at, created_at) + WHERE status = 'pending'; + +CREATE INDEX IF NOT EXISTS opt_aud0_evidence_outbox_tenant_idx + ON opt_aud0_evidence_outbox (tenant_id, created_at DESC); + +COMMENT ON TABLE opt_aud0_evidence_outbox IS + 'Durable local delivery queue for 0pt -> AUD0 evidence. Not canonical evidence.'; diff --git a/sql/012_dataset_versions.sql b/sql/012_dataset_versions.sql new file mode 100644 index 0000000..1433109 --- /dev/null +++ b/sql/012_dataset_versions.sql @@ -0,0 +1,28 @@ +-- Durable canonical metadata for 0pt dataset versions. +-- Bytes remain canonical ST0RE artifacts referenced by artifact_refs. + +CREATE TABLE IF NOT EXISTS opt_dataset_versions ( + dataset_id text NOT NULL REFERENCES opt_datasets(dataset_id) ON DELETE RESTRICT, + version integer NOT NULL CHECK (version > 0), + tenant_id text NOT NULL, + project_id text NOT NULL, + artifact_refs text[] NOT NULL CHECK (cardinality(artifact_refs) > 0), + schema_version text NOT NULL, + record_count bigint CHECK (record_count IS NULL OR record_count >= 0), + description text, + created_at timestamptz NOT NULL DEFAULT now(), + created_by text NOT NULL, + correlation_id text, + idempotency_key text, + PRIMARY KEY (dataset_id, version) +); + +CREATE UNIQUE INDEX IF NOT EXISTS opt_dataset_versions_idempotency_idx + ON opt_dataset_versions (tenant_id, idempotency_key) + WHERE idempotency_key IS NOT NULL; + +CREATE INDEX IF NOT EXISTS opt_dataset_versions_tenant_dataset_idx + ON opt_dataset_versions (tenant_id, dataset_id, version DESC); + +COMMENT ON TABLE opt_dataset_versions IS + 'Immutable 0pt dataset-version metadata. Content bytes remain in ST0RE artifacts.'; diff --git a/sql/013_search_index_definitions.sql b/sql/013_search_index_definitions.sql new file mode 100644 index 0000000..206018b --- /dev/null +++ b/sql/013_search_index_definitions.sql @@ -0,0 +1,18 @@ +-- Canonical 0pt search-index definitions. OpenSearch itself remains a +-- rebuildable projection; this table stores only the logical schema needed to +-- rebuild/query that projection after restart. + +CREATE TABLE IF NOT EXISTS opt_search_index_definitions ( + index_name text NOT NULL, + version integer NOT NULL CHECK (version > 0), + fields jsonb NOT NULL, + default_search_fields text[] NOT NULL, + created_at timestamptz NOT NULL DEFAULT now(), + created_by text, + PRIMARY KEY (index_name, version), + CONSTRAINT opt_search_index_definitions_fields_object + CHECK (jsonb_typeof(fields) = 'object') +); + +COMMENT ON TABLE opt_search_index_definitions IS + 'Versioned logical 0pt search schemas. OpenSearch indices remain disposable projections.'; diff --git a/src/clients/aud0-client.ts b/src/clients/aud0-client.ts index 60e510f..025eab7 100644 --- a/src/clients/aud0-client.ts +++ b/src/clients/aud0-client.ts @@ -1,41 +1,13 @@ /** - * src/clients/aud0-client.ts - * =========================== - * 0pt's ONLY path for recording "what happened" to AUD0, the SIAX Cloud - * append-only system of record. + * 0pt's sole AUD0 producer boundary. * - * REWRITTEN 2026-08-31 against AUD0's REAL wire contract (sax3l/aud0 branch - * feat/siax-platforms-p0-evidence-os, `src/api/routes.ts` + - * `src/schemas/event.ts` + `src/api/routes/openlineage.ts`, verified this - * session): - * - * ingestEvent -> POST {baseUrl}/v1/aud0/events (snake_case) - * ingestOpenLineageEvent-> POST {baseUrl}/v1/aud0/openlineage?tenant_id=&app_id= - * ingestLineageEvent -> LEGACY SHIM: converts the historical 0pt lineage - * shape to an OpenLineage RunEvent and posts it to - * the endpoint above. - * - * The path this client used to post lineage to — `/v1/aud0/lineage/events` — - * DOES NOT EXIST on AUD0 and never has. AUD0's only lineage ingest is the - * OpenLineage endpoint; `/v1/aud0/lineage/*` routes are read-only projections - * of the generic event stream. Master-DoD §16 requires OpenLineage run/job/ - * dataset ingest — the emitter in src/lineage/ produces spec-shaped - * RunEvents; this client is just the wire. - * - * The audit-event contract additionally carries the P0 evidence-context - * fields (§16: purpose, policy decision, correlation, causation, trace, - * result, references), all optional on the wire; they are surfaced on - * `Aud0Event` so services can populate them. - * - * Production fallback policy (Master-DoD §3/§10): no base URL in a - * production runtime throws `SiaxError("DEPENDENCY_UNAVAILABLE")` at - * construction. The no-op client survives only for non-production runtimes. - * NOTE the deliberate asymmetry: audit emission FAILURE at runtime (AUD0 - * down) still does not roll back the already-performed action — an audit - * emission records what already happened; the durability requirement is - * §70's "durable local/outbox path", tracked honestly in STATUS.md as a gap. + * Default clients are durable: when DATABASE_URL is configured, evidence is + * first written to opt_aud0_evidence_outbox and later delivered by the + * dispatcher. `durable:false` exists only for that dispatcher, which must + * perform the real HTTP delivery rather than enqueue recursively. */ import { SiaxHttp } from "./http"; +import { createAud0OutboxRepositoryFromEnv, type Aud0OutboxRepository } from "../evidence/aud0-outbox"; import { resolveDependencyBaseUrl } from "../platform/dependency-mode"; import type { SiaxRequestContext } from "../platform/siax-context"; import type { OpenLineageRunEvent } from "../lineage/openlineage"; @@ -50,30 +22,21 @@ export interface Aud0Event { capability_id: string; resource_type: string; resource_id: string; - /** Risk level L0–L5 per the SIAX Cloud risk taxonomy. */ risk_level: string; payload?: unknown; created_at: string; correlation_id?: string; - // P0 evidence-context fields (Master-DoD §16), optional on AUD0's wire. - /** Business time, when it differs from ingestion time. */ occurred_at?: string; - /** GDPR/purpose-limitation tag. */ purpose?: string; - /** CL0UD policy decision id/reason for the action being recorded. */ policy_decision?: string; causation_id?: string; trace_id?: string; - /** Outcome, e.g. "success" | "failure". */ result?: string; - /** Canonical resource refs consumed by the action (e.g. st0re:artifact:...). */ input_refs?: string[]; - /** Canonical resource refs produced by the action. */ output_refs?: string[]; idempotency_key?: string; } -/** Historical 0pt lineage shape — kept as the input to the legacy shim. */ export interface Aud0LineageEvent { lineage_event_id: string; tenant_id: string; @@ -84,19 +47,15 @@ export interface Aud0LineageEvent { source_id: string; target_type: string; target_id: string; - /** e.g. "produced" | "derived_from" | "stored_as". */ lineage_type: string; metadata?: unknown; created_at: string; correlation_id?: string; } -/** Alias so both historical import names resolve to the same shape. */ export type Aud0EventInput = Aud0Event; -/** Alias so both historical import names resolve to the same shape. */ export type Aud0LineageEventInput = Aud0LineageEvent; -/** AUD0's OpenLineage ingest acknowledgement. */ export interface Aud0OpenLineageAck { tenant_id: string; run_id: string; @@ -112,24 +71,28 @@ export interface Aud0Client { tenantId: string, appId: string, event: OpenLineageRunEvent, - context?: SiaxRequestContext + context?: SiaxRequestContext, ): Promise; - /** @deprecated Legacy shim — converts to OpenLineage and posts to /v1/aud0/openlineage. */ ingestLineageEvent(event: Aud0LineageEvent, context?: SiaxRequestContext): Promise; } -export function createAud0Client(baseUrl?: string, opts?: { token?: string }): Aud0Client { +export interface CreateAud0ClientOptions { + token?: string; + /** False only for the outbox dispatcher/raw diagnostics. Default true. */ + durable?: boolean; +} + +export function createAud0Client(baseUrl?: string, opts?: CreateAud0ClientOptions): Aud0Client { const resolved = resolveDependencyBaseUrl("aud0", baseUrl ?? process.env.AUD0_URL); - if (!resolved) return new NoopAud0Client(); - return new HttpAud0Client(resolved, opts?.token ?? process.env.AUD0_SERVICE_TOKEN); + const raw: Aud0Client = resolved + ? new HttpAud0Client(resolved, opts?.token ?? process.env.AUD0_SERVICE_TOKEN) + : new NoopAud0Client(); + + if (opts?.durable === false) return raw; + const outbox = createAud0OutboxRepositoryFromEnv(); + return outbox ? new DurableAud0Client(outbox) : raw; } -/** - * Convert the historical 0pt lineage shape into a spec-shaped OpenLineage - * RunEvent (AUD0's only lineage ingest contract). Deterministic: the runId is - * the legacy lineage_event_id, so idempotent re-emission dedups on AUD0's - * `ol:::` key. - */ export function legacyLineageToOpenLineage(event: Aud0LineageEvent): OpenLineageRunEvent { return { eventType: "COMPLETE", @@ -138,9 +101,7 @@ export function legacyLineageToOpenLineage(event: Aud0LineageEvent): OpenLineage run: { runId: event.lineage_event_id, facets: { - ...(event.correlation_id - ? { siaxCorrelation: { correlation_id: event.correlation_id } } - : {}), + ...(event.correlation_id ? { siaxCorrelation: { correlation_id: event.correlation_id } } : {}), siaxLegacyLineage: { lineage_type: event.lineage_type, capability_id: event.capability_id, @@ -155,15 +116,44 @@ export function legacyLineageToOpenLineage(event: Aud0LineageEvent): OpenLineage }; } -/** Non-production runtimes only: audit emission is a no-op echo. */ +class DurableAud0Client implements Aud0Client { + constructor(private readonly outbox: Aud0OutboxRepository) {} + + async ingestEvent(event: Aud0Event): Promise { + await this.outbox.enqueueEvent(event); + return event; + } + + async ingestOpenLineageEvent( + tenantId: string, + appId: string, + event: OpenLineageRunEvent, + ): Promise { + const row = await this.outbox.enqueueOpenLineage(tenantId, appId, event); + return { + tenant_id: tenantId, + run_id: event.run.runId, + job: `${event.job.namespace}/${event.job.name}`, + event_id: `outbox:${row.outbox_id}`, + }; + } + + async ingestLineageEvent(event: Aud0LineageEvent): Promise { + await this.ingestOpenLineageEvent(event.tenant_id, event.app_id, legacyLineageToOpenLineage(event)); + return event; + } +} + +/** Non-production/env-less runtime fallback only. */ class NoopAud0Client implements Aud0Client { async ingestEvent(event: Aud0Event): Promise { return event; } + async ingestOpenLineageEvent( tenantId: string, _appId: string, - event: OpenLineageRunEvent + event: OpenLineageRunEvent, ): Promise { return { tenant_id: tenantId, @@ -172,6 +162,7 @@ class NoopAud0Client implements Aud0Client { event_id: `noop_${event.run.runId}`, }; } + async ingestLineageEvent(event: Aud0LineageEvent): Promise { return event; } @@ -191,10 +182,8 @@ class HttpAud0Client implements Aud0Client { context, headers: { "X-Tenant-Id": event.tenant_id, "X-Actor-Id": event.actor_id }, body: event, - // 200 = idempotent replay (deduplicated: true) — a success. acceptStatuses: [200], }); - // Prefer AUD0's canonical stored record; fall back to what we sent. return (res.json ?? event) as Aud0Event; } @@ -202,13 +191,11 @@ class HttpAud0Client implements Aud0Client { tenantId: string, appId: string, event: OpenLineageRunEvent, - context?: SiaxRequestContext + context?: SiaxRequestContext, ): Promise { const res = await this.http.request({ method: "POST", path: "/v1/aud0/openlineage", - // AUD0's OpenLineage ingest takes the tenant as a QUERY PARAM (verified - // against src/api/routes/openlineage.ts), not a body field. query: { tenant_id: tenantId, app_id: appId }, context, headers: { "X-Tenant-Id": tenantId, "X-Actor-Id": appId }, diff --git a/src/er/entity-resolution-service.ts b/src/er/entity-resolution-service.ts index 913d81f..21f614f 100644 --- a/src/er/entity-resolution-service.ts +++ b/src/er/entity-resolution-service.ts @@ -1,44 +1,18 @@ /** - * src/er/entity-resolution-service.ts - * ==================================== - * Generic Entity Resolution capability (Master-DoD §10 Entity Resolution: - * candidate API, generic probability, blocking, explanation, product - * hard-rule hook, human review handoff). + * Canonical generic Entity Resolution capability. * - * Division of labour, stated plainly: - * - THIS service is the GENERIC candidate API: deterministic blocking + - * a transparent similarity scorer (token Jaccard + normalized Levenshtein) - * producing an explainable per-pair score. It is the reference - * implementation and the contract. - * - HEAVY/statistical ER (Splink probabilistic linkage, §11 ER) is - * 0pt-motor's job — `submitBatchResolution` creates the N0D job - * ("0pt.er.batch_resolution") and never fakes its output. - * - PRODUCTS own domain judgement via the HARD-RULE HOOK: e.g. VehIQ - * registers VIN-equality rules that FORCE match/non-match regardless of - * the generic score (Master-DoD §24 "VIN hard rules ... negative - * identity evidence"). Hard rules always win over probabilities. - * - Pairs in the configurable "review band" are handed off to humans - * (status pending_review) — never auto-merged. - * - * GENERIC means GENERIC: no person-matching logic, no Ratsit/PII pipelines — - * records are opaque field maps supplied by the calling product under its - * own legal basis; this module implements comparison mechanics only. - * - * P065 STATUS — THIS IS THE CANONICAL ENGINE, AND IT IS NOT YET THE ONLY ONE. - * `services/api/workers/resolve.py` is a second, simplified ER engine (fuzzy - * threshold, auto-merge, no hard rules, no review band) that - * `services/api/workers/pipeline.py` still calls. P065's DoD ("no second - * simplified duplicated production engine remains") is therefore NOT met. - * See that file's header for why it was not rewritten blind in the - * P060-P068 pass and STATUS.md for the removal plan. The ingestion plane - * added in that pass (src/ingestion/*) deliberately does no ER of its own — - * it stops at the sink, so it adds no THIRD engine. + * - synchronous candidate generation is deterministic, explainable and + * review-aware; + * - product hard rules may override probabilities in either direction; + * - heavy/statistical ER is delegated to 0pt-motor through N0D; + * - the legacy Python production scorer has been removed: Python is now only + * an adapter to this canonical decision engine. */ import crypto from "crypto"; -import { SiaxError } from "../platform/errors"; import type { N0dClient, N0dJob } from "../clients/n0d-client"; +import { SiaxError } from "../platform/errors"; +import { isResourceRef } from "../platform/resource-ref"; -/** An opaque record supplied by the calling product. */ export interface ErRecord { record_id: string; fields: Record; @@ -46,7 +20,6 @@ export interface ErRecord { export type HardRuleDecision = "force_match" | "force_non_match" | "abstain"; -/** Product-supplied hard rule (the §10 hook). */ export interface HardRule { rule_id: string; decide(a: ErRecord, b: ErRecord): HardRuleDecision; @@ -56,13 +29,9 @@ export interface ErCandidateRequest { tenant_id: string; project_id: string; records: ErRecord[]; - /** Fields whose normalized values form the blocking key. */ blocking_fields: string[]; - /** Fields compared for similarity (weight defaults to 1). */ compare_fields: { field: string; weight?: number }[]; - /** Score ≥ auto_match_threshold → match (unless a hard rule vetoes). Default 0.9. */ auto_match_threshold?: number; - /** Score ≥ review_threshold (< auto) → pending_review. Default 0.6. */ review_threshold?: number; correlation_id?: string; } @@ -81,12 +50,9 @@ export interface ErCandidatePair { candidate_id: string; record_a: string; record_b: string; - /** Generic similarity score 0..1. */ score: number; status: ErCandidateStatus; - /** Which mechanism decided: hard rule id, or "score". */ decided_by: string; - /** Per-field breakdown — the §10 explanation. */ explanation: ErFieldExplanation[]; blocking_key: string; } @@ -95,49 +61,77 @@ export interface ErCandidateResult { request_id: string; tenant_id: string; pairs: ErCandidatePair[]; - /** Pairs handed to human review (subset of pairs). */ review_queue: ErCandidatePair[]; blocked_comparisons: number; total_possible_comparisons: number; } +export interface SubmitBatchResolutionInput { + tenant_id: string; + project_id: string; + records_artifact_ref: string; + actor_id?: string; + correlation_id?: string; + idempotency_key?: string; +} + export class EntityResolutionService { private readonly hardRules: HardRule[] = []; constructor(private readonly opts: { n0dClient?: N0dClient } = {}) {} - /** The product hard-rule hook (§10). Rules run in registration order. */ registerHardRule(rule: HardRule): void { + if (!rule.rule_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "hard rule requires rule_id"); + if (this.hardRules.some((existing) => existing.rule_id === rule.rule_id)) { + throw new SiaxError("CONFLICT", "0pt.er", `hard rule ${rule.rule_id} is already registered`); + } this.hardRules.push(rule); } - // ----------------------------------------------------------------------- - // Candidate API (synchronous, generic) - // ----------------------------------------------------------------------- generateCandidates(req: ErCandidateRequest): ErCandidateResult { - if (!req.tenant_id?.trim()) { - throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required"); + if (!req.tenant_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required"); + if (!req.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "project_id is required"); + if (req.records.length < 2 || req.records.length > 5_000) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "records must contain between 2 and 5000 items"); } - if (req.records.length < 2) { - throw new SiaxError("INVALID_REQUEST", "0pt.er", "at least two records are required"); + if (new Set(req.records.map((record) => record.record_id)).size !== req.records.length) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "record_id values must be unique"); } - if (req.blocking_fields.length === 0) { + if (req.blocking_fields.length < 1 || req.blocking_fields.length > 20) { throw new SiaxError( "INVALID_REQUEST", "0pt.er", - "blocking_fields must not be empty — unblocked all-pairs comparison is refused (quadratic + §46)" + "blocking_fields must contain 1-20 fields — unblocked all-pairs comparison is refused", ); } + if (req.compare_fields.length < 1 || req.compare_fields.length > 50) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "compare_fields must contain 1-50 fields"); + } + for (const { field, weight = 1 } of req.compare_fields) { + if (!field?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "compare field name is required"); + if (!Number.isFinite(weight) || weight < 0 || weight > 100) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `invalid weight for compare field ${field}`); + } + } + const autoThreshold = req.auto_match_threshold ?? 0.9; const reviewThreshold = req.review_threshold ?? 0.6; - if (reviewThreshold >= autoThreshold) { - throw new SiaxError("INVALID_REQUEST", "0pt.er", "review_threshold must be < auto_match_threshold"); + if ( + !Number.isFinite(autoThreshold) || autoThreshold < 0 || autoThreshold > 1 || + !Number.isFinite(reviewThreshold) || reviewThreshold < 0 || reviewThreshold > 1 || + reviewThreshold >= autoThreshold + ) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.er", + "thresholds must be within 0..1 and review_threshold must be lower than auto_match_threshold", + ); } - // Blocking: bucket records by normalized blocking key. const blocks = new Map(); for (const rec of req.records) { - const key = req.blocking_fields.map((f) => normalize(String(rec.fields[f] ?? ""))).join("|"); + if (!rec.record_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "every record requires record_id"); + const key = req.blocking_fields.map((field) => normalize(String(rec.fields[field] ?? ""))).join("|"); const bucket = blocks.get(key) ?? []; bucket.push(rec); blocks.set(key, bucket); @@ -146,10 +140,10 @@ export class EntityResolutionService { const pairs: ErCandidatePair[] = []; let blockedComparisons = 0; for (const [key, bucket] of blocks) { - if (key.replace(/\|/g, "") === "") continue; // empty blocking key never blocks together + if (key.replace(/\|/g, "") === "") continue; for (let i = 0; i < bucket.length; i++) { for (let j = i + 1; j < bucket.length; j++) { - blockedComparisons++; + blockedComparisons += 1; pairs.push(this.comparePair(bucket[i], bucket[j], req, autoThreshold, reviewThreshold, key)); } } @@ -160,32 +154,38 @@ export class EntityResolutionService { request_id: `errq_${crypto.randomUUID()}`, tenant_id: req.tenant_id, pairs, - review_queue: pairs.filter((p) => p.status === "pending_review"), + review_queue: pairs.filter((pair) => pair.status === "pending_review"), blocked_comparisons: blockedComparisons, total_possible_comparisons: (n * (n - 1)) / 2, }; } - // ----------------------------------------------------------------------- - // Batch/statistical path — 0pt-motor via N0D (contract only, never faked) - // ----------------------------------------------------------------------- - async submitBatchResolution(input: { - tenant_id: string; - project_id: string; - /** ST0RE artifact ref of the record dataset. */ - records_artifact_ref: string; - correlation_id?: string; - }): Promise { + /** Heavy/statistical path — an idempotent N0D job executed by 0pt-motor. */ + async submitBatchResolution(input: SubmitBatchResolutionInput): Promise { + if (!input.tenant_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required"); + if (!input.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "project_id is required"); + if (!isResourceRef(input.records_artifact_ref, "st0re", "artifact")) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.er", + "records_artifact_ref must be a canonical st0re:artifact:... reference", + ); + } if (!this.opts.n0dClient) { throw new SiaxError("DEPENDENCY_UNAVAILABLE", "n0d", "no N0D client configured for batch ER"); } + const idempotencyKey = input.idempotency_key?.trim() || crypto + .createHash("sha256") + .update(`0pt.er.batch.v1|${input.tenant_id}|${input.project_id}|${input.records_artifact_ref}`) + .digest("hex"); + return this.opts.n0dClient.createJob({ tenant_id: input.tenant_id, project_id: input.project_id, job_type: "0pt.er.batch_resolution", capability_id: "0pt.er.batch.v1", requested_by_app_id: "0pt", - requested_by_actor_id: "svc_0pt", + requested_by_actor_id: input.actor_id ?? "svc_0pt", input_artifact_ids: [input.records_artifact_ref], output_artifact_type: "er_candidate_set", requires_backbone: false, @@ -194,19 +194,20 @@ export class EntityResolutionService { resource_profile: "large", risk_level: "L3", correlation_id: input.correlation_id, + idempotency_key: idempotencyKey, + timeout_seconds: 3_600, + max_retries: 2, }); } - // ----------------------------------------------------------------------- private comparePair( a: ErRecord, b: ErRecord, req: ErCandidateRequest, autoThreshold: number, reviewThreshold: number, - blockingKey: string + blockingKey: string, ): ErCandidatePair { - // Hard rules first — they always win (§10/§24). for (const rule of this.hardRules) { const decision = rule.decide(a, b); if (decision !== "abstain") { @@ -224,12 +225,16 @@ export class EntityResolutionService { } const explanation = this.explain(a, b, req); - const weightSum = explanation.reduce((s, e) => s + e.weight, 0); - const score = weightSum === 0 ? 0 : explanation.reduce((s, e) => s + e.similarity * e.weight, 0) / weightSum; + const weightSum = explanation.reduce((sum, item) => sum + item.weight, 0); + const score = weightSum === 0 + ? 0 + : explanation.reduce((sum, item) => sum + item.similarity * item.weight, 0) / weightSum; const rounded = Math.round(score * 1000) / 1000; - - const status: ErCandidateStatus = - rounded >= autoThreshold ? "match" : rounded >= reviewThreshold ? "pending_review" : "non_match"; + const status: ErCandidateStatus = rounded >= autoThreshold + ? "match" + : rounded >= reviewThreshold + ? "pending_review" + : "non_match"; return { candidate_id: `erc_${crypto.randomUUID()}`, record_a: a.record_id, @@ -257,14 +262,6 @@ export class EntityResolutionService { } } -// --------------------------------------------------------------------------- -// Transparent similarity primitives (deterministic; the "generic probability") -// --------------------------------------------------------------------------- -/** - * Case-fold, DIACRITIC-FOLD (ö→o, é→e — deliberate: "Göteborg" and - * "Goteborg" must land in the same block), strip punctuation, collapse - * whitespace. ø has no NFKD decomposition and is mapped explicitly. - */ export function normalize(s: string): string { return s .toLowerCase() @@ -276,7 +273,6 @@ export function normalize(s: string): string { .trim(); } -/** Combined similarity: max(token Jaccard, 1 - normalized Levenshtein). */ export function similarity(a: string | null, b: string | null): number { if (a === null || b === null) return 0; const na = normalize(a); @@ -289,7 +285,7 @@ export function similarity(a: string | null, b: string | null): number { function tokenJaccard(a: string, b: string): number { const ta = new Set(a.split(" ")); const tb = new Set(b.split(" ")); - const intersection = [...ta].filter((t) => tb.has(t)).length; + const intersection = [...ta].filter((token) => tb.has(token)).length; const union = new Set([...ta, ...tb]).size; return union === 0 ? 0 : intersection / union; } diff --git a/src/er/request-validation.ts b/src/er/request-validation.ts new file mode 100644 index 0000000..6bbe44c --- /dev/null +++ b/src/er/request-validation.ts @@ -0,0 +1,110 @@ +import { SiaxError } from "../platform/errors"; +import type { ErCandidateRequest, ErRecord } from "./entity-resolution-service"; + +function finiteThreshold(value: unknown, field: string, fallback: number): number { + if (value === undefined) return fallback; + const parsed = Number(value); + if (!Number.isFinite(parsed) || parsed < 0 || parsed > 1) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `${field} must be a finite number between 0 and 1`); + } + return parsed; +} + +export function parseErCandidateRequest(raw: unknown): ErCandidateRequest { + if (!raw || typeof raw !== "object" || Array.isArray(raw)) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "JSON object body is required"); + } + const body = raw as Record; + const tenantId = typeof body.tenant_id === "string" ? body.tenant_id.trim() : ""; + const projectId = typeof body.project_id === "string" ? body.project_id.trim() : ""; + if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required"); + if (!projectId) throw new SiaxError("INVALID_REQUEST", "0pt.er", "project_id is required"); + + if (!Array.isArray(body.records) || body.records.length < 2 || body.records.length > 5_000) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "records must contain between 2 and 5000 items"); + } + if ( + !Array.isArray(body.blocking_fields) || + body.blocking_fields.length === 0 || + body.blocking_fields.length > 20 || + !body.blocking_fields.every((v) => typeof v === "string" && v.trim().length > 0) + ) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "blocking_fields must contain 1-20 non-empty strings"); + } + if (!Array.isArray(body.compare_fields) || body.compare_fields.length === 0 || body.compare_fields.length > 50) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "compare_fields must contain 1-50 items"); + } + + const records: ErRecord[] = body.records.map((value, index) => { + if (!value || typeof value !== "object" || Array.isArray(value)) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}] must be an object`); + } + const record = value as Record; + if (typeof record.record_id !== "string" || !record.record_id.trim()) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}].record_id is required`); + } + if (!record.fields || typeof record.fields !== "object" || Array.isArray(record.fields)) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}].fields must be an object`); + } + const fields = record.fields as Record; + if (Object.keys(fields).length > 200) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}].fields exceeds 200 fields`); + } + const normalizedFields: ErRecord["fields"] = {}; + for (const [field, fieldValue] of Object.entries(fields)) { + if (!field.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}] contains an empty field name`); + if ( + fieldValue !== null && + fieldValue !== undefined && + typeof fieldValue !== "string" && + typeof fieldValue !== "number" + ) { + throw new SiaxError( + "INVALID_REQUEST", + "0pt.er", + `records[${index}].fields.${field} must be string, number, null or undefined`, + ); + } + normalizedFields[field.trim()] = fieldValue as string | number | null | undefined; + } + return { record_id: record.record_id.trim(), fields: normalizedFields }; + }); + + if (new Set(records.map((record) => record.record_id)).size !== records.length) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "record_id values must be unique within the request"); + } + + const compareFields = body.compare_fields.map((value, index) => { + if (!value || typeof value !== "object" || Array.isArray(value)) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `compare_fields[${index}] must be an object`); + } + const item = value as Record; + if (typeof item.field !== "string" || !item.field.trim()) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `compare_fields[${index}].field is required`); + } + const weight = item.weight === undefined ? undefined : Number(item.weight); + if (weight !== undefined && (!Number.isFinite(weight) || weight < 0 || weight > 100)) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", `compare_fields[${index}].weight must be between 0 and 100`); + } + return { field: item.field.trim(), ...(weight !== undefined ? { weight } : {}) }; + }); + + const autoThreshold = finiteThreshold(body.auto_match_threshold, "auto_match_threshold", 0.9); + const reviewThreshold = finiteThreshold(body.review_threshold, "review_threshold", 0.6); + if (reviewThreshold >= autoThreshold) { + throw new SiaxError("INVALID_REQUEST", "0pt.er", "review_threshold must be lower than auto_match_threshold"); + } + + return { + tenant_id: tenantId, + project_id: projectId, + records, + blocking_fields: (body.blocking_fields as string[]).map((value) => value.trim()), + compare_fields: compareFields, + auto_match_threshold: autoThreshold, + review_threshold: reviewThreshold, + ...(typeof body.correlation_id === "string" && body.correlation_id.trim() + ? { correlation_id: body.correlation_id.trim() } + : {}), + }; +} diff --git a/src/evidence/aud0-outbox.ts b/src/evidence/aud0-outbox.ts new file mode 100644 index 0000000..c2bec81 --- /dev/null +++ b/src/evidence/aud0-outbox.ts @@ -0,0 +1,377 @@ +import crypto from "node:crypto"; + +import type { Aud0Client, Aud0Event } from "../clients/aud0-client"; +import type { OpenLineageRunEvent } from "../lineage/openlineage"; +import { SiaxError } from "../platform/errors"; +import { + getRepositorySql, + repositoryDatabaseUrl, + type RepositorySql, +} from "../repositories/db"; + +export type Aud0EvidenceKind = "event" | "openlineage"; +export type Aud0OutboxStatus = "pending" | "delivered" | "dead"; + +export interface Aud0OutboxRecord { + outbox_id: string; + tenant_id: string; + app_id: string; + evidence_kind: Aud0EvidenceKind; + idempotency_key: string; + payload: Aud0Event | OpenLineageRunEvent; + status: Aud0OutboxStatus; + attempts: number; + next_attempt_at: string; + leased_by?: string; + leased_until?: string; + last_error?: string; + created_at: string; + updated_at: string; + delivered_at?: string; + dead_at?: string; +} + +export interface Aud0OutboxStats { + pending: number; + delivered: number; + dead: number; + oldest_pending_at?: string; +} + +export interface Aud0OutboxRepository { + enqueueEvent(event: Aud0Event): Promise; + enqueueOpenLineage( + tenantId: string, + appId: string, + event: OpenLineageRunEvent, + ): Promise; + claimPending(input: { + workerId: string; + limit: number; + leaseMs: number; + }): Promise; + markDelivered(outboxId: string, workerId: string): Promise; + markRetry(outboxId: string, workerId: string, error: string, delayMs: number): Promise; + markDead(outboxId: string, workerId: string, error: string): Promise; + stats(): Promise; +} + +interface OutboxRow { + outbox_id: string; + tenant_id: string; + app_id: string; + evidence_kind: Aud0EvidenceKind; + idempotency_key: string; + payload: Aud0Event | OpenLineageRunEvent; + status: Aud0OutboxStatus; + attempts: number; + next_attempt_at: string | Date; + leased_by: string | null; + leased_until: string | Date | null; + last_error: string | null; + created_at: string | Date; + updated_at: string | Date; + delivered_at: string | Date | null; + dead_at: string | Date | null; +} + +type JsonValue = Parameters[0]; + +function iso(value: string | Date): string { + return value instanceof Date ? value.toISOString() : new Date(value).toISOString(); +} + +function optionalIso(value: string | Date | null): string | undefined { + return value == null ? undefined : iso(value); +} + +function mapRow(row: OutboxRow): Aud0OutboxRecord { + return { + outbox_id: row.outbox_id, + tenant_id: row.tenant_id, + app_id: row.app_id, + evidence_kind: row.evidence_kind, + idempotency_key: row.idempotency_key, + payload: row.payload, + status: row.status, + attempts: Number(row.attempts), + next_attempt_at: iso(row.next_attempt_at), + ...(row.leased_by ? { leased_by: row.leased_by } : {}), + ...(row.leased_until ? { leased_until: optionalIso(row.leased_until) } : {}), + ...(row.last_error ? { last_error: row.last_error } : {}), + created_at: iso(row.created_at), + updated_at: iso(row.updated_at), + ...(row.delivered_at ? { delivered_at: optionalIso(row.delivered_at) } : {}), + ...(row.dead_at ? { dead_at: optionalIso(row.dead_at) } : {}), + }; +} + +function required(value: string, field: string): string { + const normalized = value.trim(); + if (!normalized) { + throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", `${field} is required`); + } + return normalized; +} + +function outboxId(tenantId: string, kind: Aud0EvidenceKind, idempotencyKey: string): string { + const digest = crypto + .createHash("sha256") + .update(tenantId) + .update("\0") + .update(kind) + .update("\0") + .update(idempotencyKey) + .digest("hex") + .slice(0, 40); + return `obx_${digest}`; +} + +function openLineageIdempotencyKey(event: OpenLineageRunEvent): string { + return `ol:${required(event.run.runId, "run.runId")}:${required(event.eventType, "eventType")}`; +} + +function safeError(error: unknown): string { + if (error instanceof SiaxError) return `${error.code}:${error.source}`.slice(0, 500); + if (error instanceof Error) return error.name.slice(0, 500); + return "UnknownError"; +} + +export class PostgresAud0OutboxRepository implements Aud0OutboxRepository { + constructor(private readonly sql: RepositorySql) {} + + async enqueueEvent(event: Aud0Event): Promise { + const key = required(event.idempotency_key ?? event.event_id, "event idempotency key"); + return this.enqueue({ + tenantId: required(event.tenant_id, "tenant_id"), + appId: required(event.app_id, "app_id"), + kind: "event", + idempotencyKey: key, + payload: event, + }); + } + + async enqueueOpenLineage( + tenantId: string, + appId: string, + event: OpenLineageRunEvent, + ): Promise { + return this.enqueue({ + tenantId: required(tenantId, "tenant_id"), + appId: required(appId, "app_id"), + kind: "openlineage", + idempotencyKey: openLineageIdempotencyKey(event), + payload: event, + }); + } + + private async enqueue(input: { + tenantId: string; + appId: string; + kind: Aud0EvidenceKind; + idempotencyKey: string; + payload: Aud0Event | OpenLineageRunEvent; + }): Promise { + const id = outboxId(input.tenantId, input.kind, input.idempotencyKey); + const rows = await this.sql` + INSERT INTO opt_aud0_evidence_outbox + (outbox_id, tenant_id, app_id, evidence_kind, idempotency_key, payload) + VALUES + (${id}, ${input.tenantId}, ${input.appId}, ${input.kind}, ${input.idempotencyKey}, + ${this.sql.json(input.payload as unknown as JsonValue)}) + ON CONFLICT (tenant_id, evidence_kind, idempotency_key) + DO UPDATE SET updated_at = now() + WHERE opt_aud0_evidence_outbox.payload = EXCLUDED.payload + RETURNING *`; + + if (rows.length === 0) { + throw new SiaxError( + "CONFLICT", + "0pt.aud0_outbox", + "the same evidence idempotency key was reused with a different payload", + ); + } + return mapRow(rows[0]); + } + + async claimPending(input: { + workerId: string; + limit: number; + leaseMs: number; + }): Promise { + const workerId = required(input.workerId, "workerId"); + if (!Number.isSafeInteger(input.limit) || input.limit < 1 || input.limit > 500) { + throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", "limit must be between 1 and 500"); + } + if (!Number.isSafeInteger(input.leaseMs) || input.leaseMs < 1_000 || input.leaseMs > 15 * 60_000) { + throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", "leaseMs must be between 1000 and 900000"); + } + + const rows = await this.sql` + WITH candidates AS ( + SELECT outbox_id + FROM opt_aud0_evidence_outbox + WHERE status = 'pending' + AND next_attempt_at <= now() + AND (leased_until IS NULL OR leased_until < now()) + ORDER BY next_attempt_at ASC, created_at ASC + LIMIT ${input.limit} + FOR UPDATE SKIP LOCKED + ) + UPDATE opt_aud0_evidence_outbox AS o + SET leased_by = ${workerId}, + leased_until = now() + (${input.leaseMs} * interval '1 millisecond'), + attempts = o.attempts + 1, + updated_at = now() + FROM candidates + WHERE o.outbox_id = candidates.outbox_id + RETURNING o.*`; + return rows.map(mapRow); + } + + async markDelivered(outboxId: string, workerId: string): Promise { + await this.finish(outboxId, workerId, "delivered"); + } + + async markDead(outboxId: string, workerId: string, error: string): Promise { + await this.finish(outboxId, workerId, "dead", error); + } + + private async finish( + outboxId: string, + workerId: string, + status: "delivered" | "dead", + error?: string, + ): Promise { + const rows = await this.sql<{ outbox_id: string }[]>` + UPDATE opt_aud0_evidence_outbox + SET status = ${status}, + delivered_at = CASE WHEN ${status} = 'delivered' THEN now() ELSE NULL END, + dead_at = CASE WHEN ${status} = 'dead' THEN now() ELSE NULL END, + last_error = ${error?.slice(0, 500) ?? null}, + leased_by = NULL, + leased_until = NULL, + updated_at = now() + WHERE outbox_id = ${required(outboxId, "outboxId")} + AND status = 'pending' + AND leased_by = ${required(workerId, "workerId")} + RETURNING outbox_id`; + if (rows.length !== 1) { + throw new SiaxError("CONFLICT", "0pt.aud0_outbox", "outbox lease is no longer owned by this worker"); + } + } + + async markRetry(outboxId: string, workerId: string, error: string, delayMs: number): Promise { + if (!Number.isSafeInteger(delayMs) || delayMs < 0 || delayMs > 24 * 60 * 60_000) { + throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", "delayMs must be between 0 and 86400000"); + } + const rows = await this.sql<{ outbox_id: string }[]>` + UPDATE opt_aud0_evidence_outbox + SET next_attempt_at = now() + (${delayMs} * interval '1 millisecond'), + last_error = ${error.slice(0, 500)}, + leased_by = NULL, + leased_until = NULL, + updated_at = now() + WHERE outbox_id = ${required(outboxId, "outboxId")} + AND status = 'pending' + AND leased_by = ${required(workerId, "workerId")} + RETURNING outbox_id`; + if (rows.length !== 1) { + throw new SiaxError("CONFLICT", "0pt.aud0_outbox", "outbox lease is no longer owned by this worker"); + } + } + + async stats(): Promise { + const rows = await this.sql<{ + pending: string; + delivered: string; + dead: string; + oldest_pending_at: string | Date | null; + }[]>` + SELECT + count(*) FILTER (WHERE status = 'pending')::text AS pending, + count(*) FILTER (WHERE status = 'delivered')::text AS delivered, + count(*) FILTER (WHERE status = 'dead')::text AS dead, + min(created_at) FILTER (WHERE status = 'pending') AS oldest_pending_at + FROM opt_aud0_evidence_outbox`; + const row = rows[0] ?? { pending: "0", delivered: "0", dead: "0", oldest_pending_at: null }; + return { + pending: Number(row.pending), + delivered: Number(row.delivered), + dead: Number(row.dead), + ...(row.oldest_pending_at ? { oldest_pending_at: iso(row.oldest_pending_at) } : {}), + }; + } +} + +export function createAud0OutboxRepositoryFromEnv( + env: NodeJS.ProcessEnv = process.env, +): PostgresAud0OutboxRepository | undefined { + const url = repositoryDatabaseUrl(env); + return url ? new PostgresAud0OutboxRepository(getRepositorySql(url)) : undefined; +} + +export interface Aud0OutboxSweepResult { + claimed: number; + delivered: number; + dead: number; + retried: number; +} + +export class Aud0OutboxDispatcher { + constructor( + private readonly repo: Aud0OutboxRepository, + private readonly aud0: Aud0Client, + private readonly workerId = `0pt-${process.pid}-${crypto.randomUUID()}`, + ) {} + + async sweep(limit = 100): Promise { + const items = await this.repo.claimPending({ + workerId: this.workerId, + limit, + leaseMs: 60_000, + }); + let delivered = 0; + let dead = 0; + let retried = 0; + + for (const item of items) { + try { + if (item.evidence_kind === "event") { + const event = item.payload as Aud0Event; + if (event.tenant_id !== item.tenant_id || event.app_id !== item.app_id) { + throw new SiaxError( + "PERMANENT_FAILURE", + "0pt.aud0_outbox", + "queued AUD0 event tenant/app binding does not match its immutable outbox envelope", + ); + } + await this.aud0.ingestEvent(event); + } else { + await this.aud0.ingestOpenLineageEvent( + item.tenant_id, + item.app_id, + item.payload as OpenLineageRunEvent, + ); + } + await this.repo.markDelivered(item.outbox_id, this.workerId); + delivered += 1; + } catch (error) { + const description = safeError(error); + if (error instanceof SiaxError && !error.retryable) { + await this.repo.markDead(item.outbox_id, this.workerId, description); + dead += 1; + } else { + const exponent = Math.min(Math.max(item.attempts - 1, 0), 10); + const delayMs = Math.min(60 * 60_000, 1_000 * 2 ** exponent); + await this.repo.markRetry(item.outbox_id, this.workerId, description, delayMs); + retried += 1; + } + } + } + + return { claimed: items.length, delivered, dead, retried }; + } +} + +export { openLineageIdempotencyKey, outboxId, safeError }; diff --git a/src/lineage/lineage-emitter.ts b/src/lineage/lineage-emitter.ts index b1cf9fa..294aa07 100644 --- a/src/lineage/lineage-emitter.ts +++ b/src/lineage/lineage-emitter.ts @@ -1,22 +1,11 @@ /** - * src/lineage/lineage-emitter.ts - * =============================== - * 0pt's OpenLineage emitter (Master-DoD §10 Lineage). One narrow API for - * capability services to record "this run read X and produced Y" against - * AUD0's real OpenLineage ingest (`POST /v1/aud0/openlineage` — see - * src/clients/aud0-client.ts for the wire). + * 0pt OpenLineage emitter. * - * Design points: - * - Deterministic runIds (caller-supplied), so an idempotent retry of a - * capability re-emits the SAME event and AUD0 dedups it instead of - * recording a phantom second run. - * - Canonical SIAX resource refs (`st0re:artifact:...`, `0pt:dataset:...`) - * go in dataset facets so evidence chains can be joined across products - * (§16 "Evidence chains: Link source → transform → model → decision"). - * - Emission is evidence of what already happened — a lineage POST failure - * is surfaced to the caller as a warning value, never an exception that - * rolls back the completed work. Callers that need hard evidence - * guarantees must use an outbox (tracked honestly in STATUS.md). + * Durability belongs to the AUD0 client boundary, not here. The default + * createAud0Client() persists to the local outbox when DATABASE_URL exists; + * the raw dispatcher client (durable:false) performs the actual HTTP delivery. + * This emitter therefore builds one canonical RunEvent and records it exactly + * once through the injected client. */ import type { Aud0Client } from "../clients/aud0-client"; import { @@ -29,35 +18,30 @@ import { export interface LineageEmitInput { tenant_id: string; project_id?: string; - /** The 0pt capability that ran, e.g. "0pt.quality.run.v1". Becomes job.name. */ capability_id: string; - /** Deterministic run id (e.g. the run/version/export id being recorded). */ run_id: string; event_type: OpenLineageEventType; - /** ISO timestamp; defaults to now. */ event_time?: string; - /** Canonical SIAX refs and/or plain names consumed by the run. */ inputs?: LineageDatasetInput[]; - /** Canonical SIAX refs and/or plain names produced by the run. */ outputs?: LineageDatasetInput[]; correlation_id?: string; - /** Extra run facets (namespaced siax*). */ run_facets?: Record; } export interface LineageDatasetInput { - /** Human/system dataset name, e.g. "dataset/ds_x" or "opensearch/opt-companies__g2". */ name: string; - /** Optional canonical SIAX resource ref for cross-product joining. */ ref?: string; namespace?: string; facets?: Record; } export interface LineageEmitResult { + /** True only when AUD0 itself acknowledged the event synchronously. */ emitted: boolean; + /** True when the event is durably queued for at-least-once AUD0 delivery. */ + queued: boolean; run_id: string; - /** Present when emission failed (AUD0 unreachable etc.). */ + outbox_id?: string; warning?: string; } @@ -74,6 +58,10 @@ function toDatasetRef(d: LineageDatasetInput): OpenLineageDatasetRef { }; } +function safeWarning(error: unknown): string { + return error instanceof Error ? error.name : "UnknownError"; +} + export class LineageEmitter { constructor(private readonly aud0: Aud0Client) {} @@ -96,17 +84,33 @@ export class LineageEmitter { }; } - /** Emit one run event. Never throws — see module doc comment. */ + /** + * Record one run event. Never throws after domain work has completed. + * + * A DurableAud0Client returns an acknowledgement whose event_id is the + * immutable `outbox:` handle. A raw HTTP client returns AUD0's event id. + * This lets callers distinguish durable queueing from live acknowledgement + * without duplicating the outbox implementation in this layer. + */ async emit(input: LineageEmitInput): Promise { const event = this.buildRunEvent(input); try { - await this.aud0.ingestOpenLineageEvent(input.tenant_id, APP_ID, event); - return { emitted: true, run_id: input.run_id }; - } catch (err) { + const ack = await this.aud0.ingestOpenLineageEvent(input.tenant_id, APP_ID, event); + if (ack.event_id.startsWith("outbox:")) { + return { + emitted: false, + queued: true, + run_id: input.run_id, + outbox_id: ack.event_id.slice("outbox:".length), + }; + } + return { emitted: true, queued: false, run_id: input.run_id }; + } catch (error) { return { emitted: false, + queued: false, run_id: input.run_id, - warning: `openlineage emission failed: ${err instanceof Error ? err.message : String(err)}`, + warning: `openlineage recording failed: ${safeWarning(error)}`, }; } } diff --git a/src/repositories/dataset-repository.ts b/src/repositories/dataset-repository.ts new file mode 100644 index 0000000..6dcc7e6 --- /dev/null +++ b/src/repositories/dataset-repository.ts @@ -0,0 +1,149 @@ +import { resolveStoreMode } from "../platform/dependency-mode"; +import { SiaxError } from "../platform/errors"; +import type { CreateDatasetInput, Dataset, DatasetStatus, DatasetType } from "../types/dataset"; +import { getRepositorySql, repositoryDatabaseUrl, type RepositorySql } from "./db"; + +export interface DatasetListFilter { + tenant_id?: string; + project_id?: string; + dataset_type?: DatasetType; + status?: DatasetStatus; +} + +export interface DatasetRepository { + create(input: CreateDatasetInput): Promise; + get(datasetId: string): Promise; + list(filter?: DatasetListFilter): Promise; +} + +interface DatasetRow { + dataset_id: string; + tenant_id: string; + project_id: string; + name: string; + dataset_type: DatasetType; + status: DatasetStatus; + description: string | null; + correlation_id: string | null; + created_at: string | Date; + updated_at: string | Date | null; + created_by: string | null; +} + +function iso(value: string | Date): string { + return value instanceof Date ? value.toISOString() : new Date(value).toISOString(); +} + +function toDataset(row: DatasetRow): Dataset { + return { + dataset_id: row.dataset_id, + tenant_id: row.tenant_id, + project_id: row.project_id, + name: row.name, + dataset_type: row.dataset_type, + status: row.status, + ...(row.description ? { description: row.description } : {}), + ...(row.correlation_id ? { correlation_id: row.correlation_id } : {}), + created_at: iso(row.created_at), + ...(row.updated_at ? { updated_at: iso(row.updated_at) } : {}), + ...(row.created_by ? { created_by: row.created_by } : {}), + }; +} + +function validate(input: CreateDatasetInput): void { + if (!input.dataset_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "dataset_id is required"); + if (!input.tenant_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required"); + if (!input.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "project_id is required"); + if (!input.name?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "name is required"); +} + +export class InMemoryDatasetRepository implements DatasetRepository { + private readonly rows = new Map(); + + async create(input: CreateDatasetInput): Promise { + validate(input); + if (this.rows.has(input.dataset_id)) { + throw new SiaxError("CONFLICT", "0pt.dataset", `dataset ${input.dataset_id} already exists`); + } + const dataset: Dataset = { + dataset_id: input.dataset_id, + tenant_id: input.tenant_id, + project_id: input.project_id, + name: input.name, + dataset_type: input.dataset_type, + status: input.status ?? "active", + ...(input.description ? { description: input.description } : {}), + ...(input.correlation_id ? { correlation_id: input.correlation_id } : {}), + created_at: new Date().toISOString(), + ...(input.created_by ? { created_by: input.created_by } : {}), + }; + this.rows.set(dataset.dataset_id, dataset); + return { ...dataset }; + } + + async get(datasetId: string): Promise { + const dataset = this.rows.get(datasetId); + if (!dataset) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} not found`); + return { ...dataset }; + } + + async list(filter?: DatasetListFilter): Promise { + return [...this.rows.values()] + .filter((dataset) => !filter?.tenant_id || dataset.tenant_id === filter.tenant_id) + .filter((dataset) => !filter?.project_id || dataset.project_id === filter.project_id) + .filter((dataset) => !filter?.dataset_type || dataset.dataset_type === filter.dataset_type) + .filter((dataset) => !filter?.status || dataset.status === filter.status) + .map((dataset) => ({ ...dataset })); + } +} + +export class PostgresDatasetRepository implements DatasetRepository { + constructor(private readonly sql: RepositorySql) {} + + async create(input: CreateDatasetInput): Promise { + validate(input); + try { + const rows = await this.sql` + INSERT INTO opt_datasets + (dataset_id, tenant_id, project_id, name, dataset_type, status, + description, correlation_id, created_by) + VALUES + (${input.dataset_id}, ${input.tenant_id}, ${input.project_id}, ${input.name}, + ${input.dataset_type}, ${input.status ?? "active"}, ${input.description ?? null}, + ${input.correlation_id ?? null}, ${input.created_by ?? null}) + RETURNING *`; + return toDataset(rows[0]); + } catch (error) { + if ((error as { code?: string })?.code === "23505") { + throw new SiaxError("CONFLICT", "0pt.dataset", `dataset ${input.dataset_id} already exists`); + } + throw error; + } + } + + async get(datasetId: string): Promise { + const rows = await this.sql` + SELECT * FROM opt_datasets WHERE dataset_id = ${datasetId} LIMIT 1`; + if (!rows[0]) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} not found`); + return toDataset(rows[0]); + } + + async list(filter?: DatasetListFilter): Promise { + const rows = await this.sql` + SELECT * + FROM opt_datasets + WHERE (${filter?.tenant_id ?? null}::text IS NULL OR tenant_id = ${filter?.tenant_id ?? null}) + AND (${filter?.project_id ?? null}::text IS NULL OR project_id = ${filter?.project_id ?? null}) + AND (${filter?.dataset_type ?? null}::text IS NULL OR dataset_type = ${filter?.dataset_type ?? null}) + AND (${filter?.status ?? null}::text IS NULL OR status = ${filter?.status ?? null}) + ORDER BY created_at DESC, dataset_id ASC`; + return rows.map(toDataset); + } +} + +export function createDatasetRepository(databaseUrl?: string): DatasetRepository { + const mode = resolveStoreMode("0pt.datasets", databaseUrl ?? repositoryDatabaseUrl()); + return mode.mode === "postgres" + ? new PostgresDatasetRepository(getRepositorySql(mode.url)) + : new InMemoryDatasetRepository(); +} diff --git a/src/repositories/dataset-version-repository.ts b/src/repositories/dataset-version-repository.ts new file mode 100644 index 0000000..729185e --- /dev/null +++ b/src/repositories/dataset-version-repository.ts @@ -0,0 +1,177 @@ +import { resolveStoreMode } from "../platform/dependency-mode"; +import { SiaxError } from "../platform/errors"; +import type { CreateDatasetVersionInput, DatasetVersion } from "../types/dataset-version"; +import { getRepositorySql, repositoryDatabaseUrl, type RepositorySql } from "./db"; + +export interface DatasetVersionRepository { + create(input: CreateDatasetVersionInput): Promise; + list(datasetId: string, tenantId: string): Promise; + get(datasetId: string, tenantId: string, version: number): Promise; +} + +interface VersionRow { + dataset_id: string; + version: number; + tenant_id: string; + project_id: string; + artifact_refs: string[]; + schema_version: string; + record_count: string | number | null; + description: string | null; + created_at: string | Date; + created_by: string; + correlation_id: string | null; + idempotency_key: string | null; +} + +function toVersion(row: VersionRow): DatasetVersion { + return { + dataset_id: row.dataset_id, + version: Number(row.version), + tenant_id: row.tenant_id, + project_id: row.project_id, + artifact_refs: [...row.artifact_refs], + schema_version: row.schema_version, + ...(row.record_count == null ? {} : { record_count: Number(row.record_count) }), + ...(row.description ? { description: row.description } : {}), + created_at: row.created_at instanceof Date ? row.created_at.toISOString() : new Date(row.created_at).toISOString(), + created_by: row.created_by, + ...(row.correlation_id ? { correlation_id: row.correlation_id } : {}), + }; +} + +function normalizedCount(value: string | number | null | undefined): number | undefined { + return value == null ? undefined : Number(value); +} + +function sameInput(row: VersionRow | DatasetVersion, input: CreateDatasetVersionInput): boolean { + return row.dataset_id === input.dataset_id + && row.tenant_id === input.tenant_id + && row.project_id === input.project_id + && row.schema_version === input.schema_version + && normalizedCount(row.record_count) === normalizedCount(input.record_count) + && (row.description ?? undefined) === (input.description ?? undefined) + && row.created_by === input.created_by + && JSON.stringify(row.artifact_refs) === JSON.stringify(input.artifact_refs); +} + +export class InMemoryDatasetVersionRepository implements DatasetVersionRepository { + private readonly versions = new Map(); + private readonly idempotency = new Map(); + + async create(input: CreateDatasetVersionInput): Promise { + if (input.idempotency_key) { + const existing = this.idempotency.get(`${input.tenant_id}:${input.idempotency_key}`); + if (existing) { + if (!sameInput(existing, input)) { + throw new SiaxError("CONFLICT", "0pt.dataset", "idempotency key reused with different dataset-version input"); + } + return { ...existing, artifact_refs: [...existing.artifact_refs] }; + } + } + const list = this.versions.get(input.dataset_id) ?? []; + const version: DatasetVersion = { + dataset_id: input.dataset_id, + version: (list[list.length - 1]?.version ?? 0) + 1, + tenant_id: input.tenant_id, + project_id: input.project_id, + artifact_refs: [...input.artifact_refs], + schema_version: input.schema_version, + ...(input.record_count === undefined ? {} : { record_count: input.record_count }), + ...(input.description ? { description: input.description } : {}), + created_at: new Date().toISOString(), + created_by: input.created_by, + ...(input.correlation_id ? { correlation_id: input.correlation_id } : {}), + }; + list.push(version); + this.versions.set(input.dataset_id, list); + if (input.idempotency_key) this.idempotency.set(`${input.tenant_id}:${input.idempotency_key}`, version); + return { ...version, artifact_refs: [...version.artifact_refs] }; + } + + async list(datasetId: string, tenantId: string): Promise { + return (this.versions.get(datasetId) ?? []) + .filter((version) => version.tenant_id === tenantId) + .map((version) => ({ ...version, artifact_refs: [...version.artifact_refs] })); + } + + async get(datasetId: string, tenantId: string, version: number): Promise { + const found = (await this.list(datasetId, tenantId)).find((item) => item.version === version); + if (!found) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} has no version ${version}`); + return found; + } +} + +export class PostgresDatasetVersionRepository implements DatasetVersionRepository { + constructor(private readonly sql: RepositorySql) {} + + async create(input: CreateDatasetVersionInput): Promise { + return this.sql.begin(async (tx) => { + const parent = await tx<{ dataset_id: string; tenant_id: string }[]>` + SELECT dataset_id, tenant_id + FROM opt_datasets + WHERE dataset_id = ${input.dataset_id} + FOR UPDATE`; + if (!parent[0] || parent[0].tenant_id !== input.tenant_id) { + throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${input.dataset_id} not found`); + } + + if (input.idempotency_key) { + const existing = await tx` + SELECT * FROM opt_dataset_versions + WHERE tenant_id = ${input.tenant_id} + AND idempotency_key = ${input.idempotency_key} + LIMIT 1`; + if (existing[0]) { + if (!sameInput(existing[0], input)) { + throw new SiaxError("CONFLICT", "0pt.dataset", "idempotency key reused with different dataset-version input"); + } + return toVersion(existing[0]); + } + } + + const next = await tx<{ version: number }[]>` + SELECT COALESCE(max(version), 0)::int + 1 AS version + FROM opt_dataset_versions + WHERE dataset_id = ${input.dataset_id}`; + const version = next[0]?.version ?? 1; + + const rows = await tx` + INSERT INTO opt_dataset_versions + (dataset_id, version, tenant_id, project_id, artifact_refs, + schema_version, record_count, description, created_by, + correlation_id, idempotency_key) + VALUES + (${input.dataset_id}, ${version}, ${input.tenant_id}, ${input.project_id}, + ${tx.array([...input.artifact_refs])}, ${input.schema_version}, ${input.record_count ?? null}, + ${input.description ?? null}, ${input.created_by}, ${input.correlation_id ?? null}, + ${input.idempotency_key ?? null}) + RETURNING *`; + return toVersion(rows[0]); + }); + } + + async list(datasetId: string, tenantId: string): Promise { + const rows = await this.sql` + SELECT * FROM opt_dataset_versions + WHERE dataset_id = ${datasetId} AND tenant_id = ${tenantId} + ORDER BY version ASC`; + return rows.map(toVersion); + } + + async get(datasetId: string, tenantId: string, version: number): Promise { + const rows = await this.sql` + SELECT * FROM opt_dataset_versions + WHERE dataset_id = ${datasetId} AND tenant_id = ${tenantId} AND version = ${version} + LIMIT 1`; + if (!rows[0]) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} has no version ${version}`); + return toVersion(rows[0]); + } +} + +export function createDatasetVersionRepository(databaseUrl?: string): DatasetVersionRepository { + const mode = resolveStoreMode("0pt.dataset_versions", databaseUrl ?? repositoryDatabaseUrl()); + return mode.mode === "postgres" + ? new PostgresDatasetVersionRepository(getRepositorySql(mode.url)) + : new InMemoryDatasetVersionRepository(); +} diff --git a/src/repositories/search-definition-repository.ts b/src/repositories/search-definition-repository.ts new file mode 100644 index 0000000..a5eaf34 --- /dev/null +++ b/src/repositories/search-definition-repository.ts @@ -0,0 +1,186 @@ +import { resolveStoreMode } from "../platform/dependency-mode"; +import { SiaxError } from "../platform/errors"; +import type { SearchIndexDefinition, SearchFieldType } from "../search/search-types"; +import { getRepositorySql, repositoryDatabaseUrl, type RepositorySql } from "./db"; + +export interface SearchDefinitionSaveResult { + definition: SearchIndexDefinition; + created: boolean; + created_at: string; + created_by?: string; +} + +export interface SearchDefinitionRepository { + save(definition: SearchIndexDefinition, actorId?: string): Promise; + latest(indexName: string): Promise; + listLatest(): Promise; +} + +interface DefinitionRow { + index_name: string; + version: number; + fields: Record; + default_search_fields: string[]; + created_at: string | Date; + created_by: string | null; +} + +function iso(value: string | Date): string { + return value instanceof Date ? value.toISOString() : new Date(value).toISOString(); +} + +function toDefinition(row: DefinitionRow): SearchIndexDefinition { + return { + index_name: row.index_name, + version: Number(row.version), + fields: { ...row.fields }, + default_search_fields: [...row.default_search_fields], + }; +} + +function toSaveResult(row: DefinitionRow, created: boolean): SearchDefinitionSaveResult { + return { + definition: toDefinition(row), + created, + created_at: iso(row.created_at), + ...(row.created_by ? { created_by: row.created_by } : {}), + }; +} + +function fingerprint(definition: SearchIndexDefinition): string { + return JSON.stringify({ + index_name: definition.index_name, + version: definition.version, + fields: Object.fromEntries(Object.entries(definition.fields).sort(([a], [b]) => a.localeCompare(b))), + default_search_fields: [...definition.default_search_fields], + }); +} + +interface MemoryRecord { + definition: SearchIndexDefinition; + created_at: string; + created_by?: string; +} + +export class InMemorySearchDefinitionRepository implements SearchDefinitionRepository { + private readonly versions = new Map(); + + async save(definition: SearchIndexDefinition, actorId?: string): Promise { + const list = this.versions.get(definition.index_name) ?? []; + const existing = list.find((item) => item.definition.version === definition.version); + if (existing) { + if (fingerprint(existing.definition) !== fingerprint(definition)) { + throw new SiaxError("CONFLICT", "0pt.search", "search definition version reused with different schema"); + } + return { + definition: { + ...existing.definition, + fields: { ...existing.definition.fields }, + default_search_fields: [...existing.definition.default_search_fields], + }, + created: false, + created_at: existing.created_at, + ...(existing.created_by ? { created_by: existing.created_by } : {}), + }; + } + const expected = (list[list.length - 1]?.definition.version ?? 0) + 1; + if (definition.version !== expected) { + throw new SiaxError("CONFLICT", "0pt.search", `next search definition version must be ${expected}`); + } + const copy = { + ...definition, + fields: { ...definition.fields }, + default_search_fields: [...definition.default_search_fields], + }; + const record: MemoryRecord = { + definition: copy, + created_at: new Date().toISOString(), + ...(actorId ? { created_by: actorId } : {}), + }; + list.push(record); + this.versions.set(definition.index_name, list); + return { definition: copy, created: true, created_at: record.created_at, ...(actorId ? { created_by: actorId } : {}) }; + } + + async latest(indexName: string): Promise { + const list = this.versions.get(indexName) ?? []; + const item = list[list.length - 1]?.definition; + return item ? { ...item, fields: { ...item.fields }, default_search_fields: [...item.default_search_fields] } : undefined; + } + + async listLatest(): Promise { + const result: SearchIndexDefinition[] = []; + for (const key of this.versions.keys()) { + const latest = await this.latest(key); + if (latest) result.push(latest); + } + return result.sort((a, b) => a.index_name.localeCompare(b.index_name)); + } +} + +export class PostgresSearchDefinitionRepository implements SearchDefinitionRepository { + constructor(private readonly sql: RepositorySql) {} + + async save(definition: SearchIndexDefinition, actorId?: string): Promise { + return this.sql.begin(async (tx) => { + await tx`SELECT pg_advisory_xact_lock(hashtext(${`0pt-search:${definition.index_name}`}))`; + const same = await tx` + SELECT index_name, version, fields, default_search_fields, created_at, created_by + FROM opt_search_index_definitions + WHERE index_name = ${definition.index_name} AND version = ${definition.version} + LIMIT 1`; + if (same[0]) { + const existing = toDefinition(same[0]); + if (fingerprint(existing) !== fingerprint(definition)) { + throw new SiaxError("CONFLICT", "0pt.search", "search definition version reused with different schema"); + } + return toSaveResult(same[0], false); + } + + const latest = await tx<{ version: number }[]>` + SELECT COALESCE(max(version), 0)::int AS version + FROM opt_search_index_definitions + WHERE index_name = ${definition.index_name}`; + const expected = (latest[0]?.version ?? 0) + 1; + if (definition.version !== expected) { + throw new SiaxError("CONFLICT", "0pt.search", `next search definition version must be ${expected}`); + } + + const rows = await tx` + INSERT INTO opt_search_index_definitions + (index_name, version, fields, default_search_fields, created_by) + VALUES + (${definition.index_name}, ${definition.version}, + ${tx.json(definition.fields as Parameters[0])}, + ${tx.array(definition.default_search_fields)}, ${actorId ?? null}) + RETURNING index_name, version, fields, default_search_fields, created_at, created_by`; + return toSaveResult(rows[0], true); + }); + } + + async latest(indexName: string): Promise { + const rows = await this.sql` + SELECT index_name, version, fields, default_search_fields, created_at, created_by + FROM opt_search_index_definitions + WHERE index_name = ${indexName} + ORDER BY version DESC + LIMIT 1`; + return rows[0] ? toDefinition(rows[0]) : undefined; + } + + async listLatest(): Promise { + const rows = await this.sql` + SELECT DISTINCT ON (index_name) + index_name, version, fields, default_search_fields, created_at, created_by + FROM opt_search_index_definitions + ORDER BY index_name, version DESC`; + return rows.map(toDefinition); + } +} + +export function createSearchDefinitionRepository(databaseUrl?: string): SearchDefinitionRepository { + const mode = resolveStoreMode("0pt.search_definitions", databaseUrl ?? repositoryDatabaseUrl()); + return mode.mode === "postgres" + ? new PostgresSearchDefinitionRepository(getRepositorySql(mode.url)) + : new InMemorySearchDefinitionRepository(); +} diff --git a/src/search/search-service.ts b/src/search/search-service.ts index b551036..65f5934 100644 --- a/src/search/search-service.ts +++ b/src/search/search-service.ts @@ -1,35 +1,15 @@ /** - * src/search/search-service.ts - * ============================= - * 0pt's Search capability (Master-DoD §10 Search): + * 0pt Search capability. * - * search.index -> registerIndexDefinition + indexDocuments (bulk) - * search.query -> query (tenant + purpose/rights filtering ENFORCED) - * search.rebuild -> rebuild (new generation, bulk load, validate) - * search.alias.switch -> switchGeneration (atomic alias cutover) - * - * Non-negotiables encoded here: - * - OpenSearch is NEVER canonical truth (§42): a rebuild takes a caller- - * supplied `DocumentSource` (an async iterator over canonical data) and - * can rebuild any index from an empty cluster. Deleting every index - * loses nothing canonical. - * - Tenant filter is applied by THIS service on every query — a caller - * cannot opt out; a missing tenant_id is INVALID_REQUEST (§31). - * - Purpose/rights filtering (§10): every document carries purpose_tags + - * rights_tags (mandatory on the envelope); every query names its purpose - * and gets only documents tagged for it. Documents with an EMPTY - * purpose_tags list are never returnable — fail closed. - * - Generation switch: physical indices are `opt-{name}--g{n}`; the alias - * `opt-{name}` points at exactly one generation; cutover is atomic - * (§42 generation isolation + cutover). - * - * Lineage: every completed rebuild emits an OpenLineage COMPLETE run to AUD0 - * (inputs: the document source name; outputs: the physical index). + * OpenSearch remains a disposable projection. Logical index definitions may be + * persisted through SearchDefinitionRepository so a process restart never + * forgets how to query or rebuild an otherwise valid projection. */ -import { SiaxError } from "../platform/errors"; -import type { OpenSearchAdapter } from "./opensearch-adapter"; -import { LineageEmitter } from "../lineage/lineage-emitter"; import type { Aud0Client } from "../clients/aud0-client"; +import { LineageEmitter } from "../lineage/lineage-emitter"; +import { SiaxError } from "../platform/errors"; +import type { SearchDefinitionRepository } from "../repositories/search-definition-repository"; +import type { OpenSearchAdapter } from "./opensearch-adapter"; import { MAX_SEARCH_PAGE_SIZE, type BulkIndexResult, @@ -41,9 +21,7 @@ import { type SearchResult, } from "./search-types"; -/** An async source of canonical documents for (re)builds. */ export interface DocumentSource { - /** Name for lineage, e.g. "postgres:opt_datasets" or "0pt:dataset:ds_x". */ name: string; iterate(): AsyncIterable; } @@ -59,24 +37,34 @@ const ENVELOPE_MAPPINGS: Record = { export class SearchService { private readonly definitions = new Map(); private readonly lineage?: LineageEmitter; + private readonly aud0?: Aud0Client; + private readonly definitionRepository?: SearchDefinitionRepository; constructor( private readonly adapter: OpenSearchAdapter, - opts?: { aud0Client?: Aud0Client } + opts?: { aud0Client?: Aud0Client; definitionRepository?: SearchDefinitionRepository }, ) { + this.aud0 = opts?.aud0Client; this.lineage = opts?.aud0Client ? new LineageEmitter(opts.aud0Client) : undefined; + this.definitionRepository = opts?.definitionRepository; } - // --------------------------------------------------------------------- - // Index definitions - // --------------------------------------------------------------------- - registerIndexDefinition(def: SearchIndexDefinition): SearchIndexDefinition { + private validateDefinition(def: SearchIndexDefinition): void { + if (!def.index_name?.trim() || !/^[a-z0-9][a-z0-9_-]{0,62}$/i.test(def.index_name)) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "index_name must be 1-63 letters, digits, _ or -"); + } + if (!Number.isSafeInteger(def.version) || def.version < 1) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "definition version must be a positive integer"); + } + if (Object.keys(def.fields).length === 0) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "an index definition needs at least one field"); + } for (const f of def.default_search_fields) { if (def.fields[f] !== "text") { throw new SiaxError( "INVALID_REQUEST", "0pt.search", - `default_search_field "${f}" must be a text field in index "${def.index_name}"` + `default_search_field "${f}" must be a text field in index "${def.index_name}"`, ); } } @@ -85,22 +73,85 @@ export class SearchService { throw new SiaxError( "INVALID_REQUEST", "0pt.search", - `field "${reserved}" is reserved by the 0pt search envelope` + `field "${reserved}" is reserved by the 0pt search envelope`, ); } } - this.definitions.set(def.index_name, def); - return def; + } + + /** In-memory registration retained for hermetic unit tests and local callers. */ + registerIndexDefinition(def: SearchIndexDefinition): SearchIndexDefinition { + this.validateDefinition(def); + const copy = { + ...def, + fields: { ...def.fields }, + default_search_fields: [...def.default_search_fields], + }; + this.definitions.set(def.index_name, copy); + return copy; + } + + /** Production registration: durable versioned write, one stable audit event, then cache refresh. */ + async registerIndexDefinitionDurable(def: SearchIndexDefinition, actorId?: string): Promise { + this.validateDefinition(def); + if (!this.definitionRepository) return this.registerIndexDefinition(def); + const stored = await this.definitionRepository.save(def, actorId); + const registered = this.registerIndexDefinition(stored.definition); + if (stored.created && this.aud0) { + const id = `search-definition:${registered.index_name}:v${registered.version}`; + await this.aud0.ingestEvent({ + event_id: `evt_0pt_${id}`, + event_type: "0pt.search_definition_registered", + tenant_id: "t_platform", + project_id: "p_platform", + actor_id: stored.created_by ?? actorId ?? "system", + app_id: "0pt", + capability_id: "0pt.search.definition.register.v1", + resource_type: "search_index_definition", + resource_id: `${registered.index_name}@${registered.version}`, + risk_level: "L2", + result: "success", + payload: { + index_name: registered.index_name, + version: registered.version, + fields: Object.keys(registered.fields).sort(), + default_search_fields: registered.default_search_fields, + }, + created_at: stored.created_at, + occurred_at: stored.created_at, + idempotency_key: id, + }); + } + return registered; } getIndexDefinition(indexName: string): SearchIndexDefinition { const def = this.definitions.get(indexName); if (!def) throw new SiaxError("NOT_FOUND", "0pt.search", `no index definition "${indexName}"`); - return def; + return { ...def, fields: { ...def.fields }, default_search_fields: [...def.default_search_fields] }; + } + + async resolveIndexDefinition(indexName: string): Promise { + const cached = this.definitions.get(indexName); + if (cached) return this.getIndexDefinition(indexName); + const stored = await this.definitionRepository?.latest(indexName); + if (!stored) throw new SiaxError("NOT_FOUND", "0pt.search", `no index definition "${indexName}"`); + return this.registerIndexDefinition(stored); } listIndexDefinitions(): SearchIndexDefinition[] { - return [...this.definitions.values()]; + return [...this.definitions.values()].map((def) => ({ + ...def, + fields: { ...def.fields }, + default_search_fields: [...def.default_search_fields], + })); + } + + async listIndexDefinitionsDurable(): Promise { + if (!this.definitionRepository) return this.listIndexDefinitions(); + const definitions = await this.definitionRepository.listLatest(); + for (const def of definitions) this.registerIndexDefinition(def); + return definitions; } aliasFor(indexName: string): string { @@ -113,51 +164,49 @@ export class SearchService { private mappingsFor(def: SearchIndexDefinition): Record { const properties: Record = { ...ENVELOPE_MAPPINGS }; - for (const [field, type] of Object.entries(def.fields)) { - properties[field] = { type }; - } + for (const [field, type] of Object.entries(def.fields)) properties[field] = { type }; return { properties, _meta: { opt_definition_version: def.version } }; } - // --------------------------------------------------------------------- - // search.index — bulk write into the ACTIVE generation - // --------------------------------------------------------------------- async indexDocuments(indexName: string, docs: SearchDocument[]): Promise { - const def = this.getIndexDefinition(indexName); + const def = await this.resolveIndexDefinition(indexName); const state = await this.generationState(indexName); if (state.active_generation === undefined) { throw new SiaxError( "CONFLICT", "0pt.search", - `index "${indexName}" has no active generation — run a rebuild first` + `index "${indexName}" has no active generation — run a rebuild first`, ); } this.validateDocuments(def, docs); const physical = this.physicalIndexFor(indexName, state.active_generation); const result = await this.adapter.bulkIndex( physical, - docs.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) })) + docs.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) })), ); return { indexed: docs.length - result.errors.length, errors: result.errors }; } - // --------------------------------------------------------------------- - // search.query — tenant + purpose/rights filtering ENFORCED here - // --------------------------------------------------------------------- async query(indexName: string, q: SearchQuery): Promise { - const def = this.getIndexDefinition(indexName); + const def = await this.resolveIndexDefinition(indexName); if (!q.tenant_id?.trim()) { throw new SiaxError("INVALID_REQUEST", "0pt.search", "tenant_id is required on every search query"); } if (!q.purpose?.trim()) { throw new SiaxError("INVALID_REQUEST", "0pt.search", "purpose is required on every search query (§10)"); } - const size = Math.min(q.size ?? 25, MAX_SEARCH_PAGE_SIZE); + const from = q.from ?? 0; + const requestedSize = q.size ?? 25; + if (!Number.isSafeInteger(from) || from < 0) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "from must be a non-negative integer"); + } + if (!Number.isSafeInteger(requestedSize) || requestedSize < 1) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "size must be a positive integer"); + } + const size = Math.min(requestedSize, MAX_SEARCH_PAGE_SIZE); const filter: unknown[] = [ { term: { tenant_id: q.tenant_id } }, - // Purpose fit: the document must be tagged for THIS query's purpose. - // Docs with empty purpose_tags match nothing — fail closed by shape. { term: { purpose_tags: q.purpose } }, { term: { rights_tags: q.rights_tag ?? "internal" } }, ]; @@ -171,10 +220,9 @@ export class SearchService { const must = q.text?.trim() ? [{ multi_match: { query: q.text, fields: def.default_search_fields } }] : [{ match_all: {} }]; - const res = await this.adapter.search(this.aliasFor(indexName), { query: { bool: { must, filter } }, - from: q.from ?? 0, + from, size, }); @@ -196,30 +244,27 @@ export class SearchService { }; } - // --------------------------------------------------------------------- - // search.rebuild — new generation from canonical truth (§42) - // --------------------------------------------------------------------- async rebuild( indexName: string, source: DocumentSource, opts?: { - /** Switch the alias to the new generation when done (default true). */ switch?: boolean; - /** Delete superseded generations after a successful switch (default false — keep for rollback). */ prune?: boolean; batchSize?: number; tenant_id?: string; correlation_id?: string; - } + }, ): Promise { - const def = this.getIndexDefinition(indexName); + const def = await this.resolveIndexDefinition(indexName); const state = await this.generationState(indexName); const newGeneration = (state.active_generation ?? 0) + 1; const physical = this.physicalIndexFor(indexName, newGeneration); + const batchSize = opts?.batchSize ?? 500; + if (!Number.isSafeInteger(batchSize) || batchSize < 1 || batchSize > 5_000) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "batchSize must be between 1 and 5000"); + } await this.adapter.createIndex(physical, this.mappingsFor(def)); - - const batchSize = opts?.batchSize ?? 500; let batch: SearchDocument[] = []; let docsIndexed = 0; const flush = async () => { @@ -227,28 +272,31 @@ export class SearchService { this.validateDocuments(def, batch); const res = await this.adapter.bulkIndex( physical, - batch.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) })) + batch.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) })), ); if (res.errors.length > 0) { - // A rebuild must be all-or-nothing: destroy the failed generation and - // leave the active alias untouched (generation isolation, §42). await this.adapter.deleteIndex(physical); throw new SiaxError( "RETRYABLE_FAILURE", "0pt.search", - `rebuild of "${indexName}" g${newGeneration} failed on ${res.errors.length} document(s); generation discarded` + `rebuild of "${indexName}" g${newGeneration} failed on ${res.errors.length} document(s); generation discarded`, ); } docsIndexed += batch.length; batch = []; }; - for await (const doc of source.iterate()) { - batch.push(doc); - if (batch.length >= batchSize) await flush(); + try { + for await (const doc of source.iterate()) { + batch.push(doc); + if (batch.length >= batchSize) await flush(); + } + await flush(); + await this.adapter.refresh(physical); + } catch (error) { + await this.adapter.deleteIndex(physical).catch(() => undefined); + throw error; } - await flush(); - await this.adapter.refresh(physical); const shouldSwitch = opts?.switch !== false; if (shouldSwitch) { @@ -280,10 +328,11 @@ export class SearchService { }; } - // --------------------------------------------------------------------- - // search.alias.switch — explicit generation cutover / rollback - // --------------------------------------------------------------------- async switchGeneration(indexName: string, generation: number): Promise { + if (!Number.isSafeInteger(generation) || generation < 1) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "generation must be a positive integer"); + } + await this.resolveIndexDefinition(indexName); const physical = this.physicalIndexFor(indexName, generation); if (!(await this.adapter.indexExists(physical))) { throw new SiaxError("NOT_FOUND", "0pt.search", `generation g${generation} of "${indexName}" does not exist`); @@ -308,12 +357,7 @@ export class SearchService { }; } - // --------------------------------------------------------------------- - // internals - // --------------------------------------------------------------------- private docKey(d: SearchDocument): string { - // Physical _id is tenant-prefixed so two tenants can use the same doc_id - // without collision — defense in depth under the tenant term filter. return `${d.tenant_id}:${d.doc_id}`; } @@ -330,10 +374,13 @@ export class SearchService { private validateDocuments(def: SearchIndexDefinition, docs: SearchDocument[]): void { for (const d of docs) { + if (!d.doc_id?.trim()) { + throw new SiaxError("INVALID_REQUEST", "0pt.search", "every document requires doc_id"); + } if (!d.tenant_id?.trim()) { throw new SiaxError("INVALID_REQUEST", "0pt.search", `document ${d.doc_id} is missing tenant_id`); } - if (!Array.isArray(d.purpose_tags)) { + if (!Array.isArray(d.purpose_tags) || d.purpose_tags.length === 0) { throw new SiaxError("INVALID_REQUEST", "0pt.search", `document ${d.doc_id} is missing purpose_tags`); } if (!Array.isArray(d.rights_tags) || d.rights_tags.length === 0) { @@ -344,7 +391,7 @@ export class SearchService { throw new SiaxError( "INVALID_REQUEST", "0pt.search", - `document ${d.doc_id} has field "${field}" not in index definition "${def.index_name}" v${def.version}` + `document ${d.doc_id} has field "${field}" not in index definition "${def.index_name}" v${def.version}`, ); } } diff --git a/src/services/dataset-service.ts b/src/services/dataset-service.ts index d373ca6..b706463 100644 --- a/src/services/dataset-service.ts +++ b/src/services/dataset-service.ts @@ -1,116 +1,72 @@ /** - * src/services/dataset-service.ts - * ================================ - * Owns 0pt.dataset.schema.v1 — the registry of 0pt datasets (raw / structured - * / enriched / curated / feature / derived outputs). This is the thing - * `DatasetIcebergMappingService` (src/services/dataset-iceberg-mapping-service.ts) - * and `IcebergWriteService` (src/services/iceberg-write-service.ts) look a - * `dataset_id` up against before pointing an Iceberg table mapping or a write - * at it. + * Owns 0pt.dataset.schema.v1 — the canonical registry of 0pt datasets. * - * Deliberately small: it is a reference/registry entry, NOT dataset content, - * schema, or quality state — see docs/ICEBERG_WRITE_BOUNDARY.md for that - * boundary. In-memory reference implementation (same convention as the other - * 0pt services in this repo; a real deployment backs this with `opt_datasets`). - * - * In-memory scope: each `new DatasetService()` owns its own registry, the same - * pre-existing limitation source-service.ts and every other P0 registry - * service documents (docs/P0_QUERY_SOURCE_FLOW.md's "Known gaps"). Callers - * that need a shared view across API routes pass an existing instance in via - * the `datasetService` constructor option — see - * src/services/lakehouse-mapping-registry.ts, which does exactly that for the - * whole app/api/0pt/lakehouse/mappings route group. - * - * Registering a dataset emits a best-effort AUD0 `0pt.dataset_registered` - * event; with no AUD0_URL configured that emission is a no-op (see - * aud0-client.ts), so this stays hermetic in tests and local dev. + * Storage is Postgres-backed through DatasetRepository whenever DATABASE_URL + * is configured. Non-production tests/local development may use the in-memory + * repository; production without a persistent store fails at construction via + * resolveStoreMode. Dataset content remains outside this registry. */ import { createAud0Client, type Aud0Client } from "../clients/aud0-client"; +import { + createDatasetRepository, + type DatasetListFilter, + type DatasetRepository, +} from "../repositories/dataset-repository"; import type { Dataset, DatasetStatus, DatasetType, CreateDatasetInput } from "../types/dataset"; -// Re-exported so the historical `import { Dataset, DatasetType, ... } from -// "./dataset-service"` call shape keeps working alongside the canonical -// `../types/dataset` module. export type { Dataset, DatasetStatus, DatasetType, CreateDatasetInput }; -/** Alias so both historical input-type names resolve to the same shape. */ export type RegisterDatasetInput = CreateDatasetInput; -/** - * Storage shape: nullable columns, exactly as the eventual `opt_datasets` - * table would hold them. `toDataset()` converts back to the `Dataset` DTO. - */ -interface DatasetRecord { - dataset_id: string; - tenant_id: string; - project_id: string; - name: string; - dataset_type: string; - status: string; - description: string | null; - correlation_id: string | null; - created_at: string; - updated_at: string | null; - created_by: string | null; +export interface DatasetServiceOptions { + aud0BaseUrl?: string; + databaseUrl?: string; + repository?: DatasetRepository; +} + +function normalizeOptions(input?: string | DatasetServiceOptions): DatasetServiceOptions { + return typeof input === "string" ? { aud0BaseUrl: input } : input ?? {}; } export class DatasetService { - private datasets: Map = new Map(); - private aud0: Aud0Client; + private readonly repo: DatasetRepository; + private readonly aud0: Aud0Client; - constructor(aud0BaseUrl?: string) { - this.aud0 = createAud0Client(aud0BaseUrl); + /** + * A string argument remains supported as the historical aud0BaseUrl form; + * new callers should use the options object so repository/database injection + * is explicit and testable. + */ + constructor(input?: string | DatasetServiceOptions) { + const opts = normalizeOptions(input); + this.aud0 = createAud0Client(opts.aud0BaseUrl); + this.repo = opts.repository ?? createDatasetRepository(opts.databaseUrl); } async registerDataset(input: CreateDatasetInput): Promise { - if (this.datasets.has(input.dataset_id)) { - throw new Error(`Dataset ${input.dataset_id} already exists`); - } - if (!input.name) { - throw new Error("name is required"); - } + const dataset = await this.repo.create(input); - const now = new Date().toISOString(); - const record: DatasetRecord = { - dataset_id: input.dataset_id, - tenant_id: input.tenant_id, - project_id: input.project_id, - name: input.name, - dataset_type: input.dataset_type, - status: input.status ?? "active", - description: input.description ?? null, - correlation_id: input.correlation_id ?? null, - created_at: now, - updated_at: null, - created_by: input.created_by ?? null, - }; - this.datasets.set(record.dataset_id, record); - - // 0pt.dataset_registered -- same "no mutation without an evented trail" - // doctrine every other P0 registry service in this repo follows - // (SourceService.createSource, SourceRatePolicyService.createSourceRatePolicy). await this.aud0.ingestEvent({ - event_id: `evt_0pt_dataset_registered_${record.dataset_id}`, + event_id: `evt_0pt_dataset_registered_${dataset.dataset_id}`, event_type: "0pt.dataset_registered", - tenant_id: record.tenant_id, - project_id: record.project_id, - actor_id: record.created_by ?? "system", + tenant_id: dataset.tenant_id, + project_id: dataset.project_id, + actor_id: dataset.created_by ?? "system", app_id: "0pt", capability_id: "0pt.register_dataset", resource_type: "dataset", - resource_id: record.dataset_id, + resource_id: dataset.dataset_id, risk_level: "L2", - payload: { name: record.name, dataset_type: record.dataset_type }, - created_at: now, - correlation_id: record.correlation_id ?? undefined, + payload: { name: dataset.name, dataset_type: dataset.dataset_type }, + created_at: dataset.created_at, + correlation_id: dataset.correlation_id, + idempotency_key: `dataset-register:${dataset.tenant_id}:${dataset.dataset_id}`, }); - return this.toDataset(record); + return dataset; } async getDataset(datasetId: string): Promise { - const record = this.datasets.get(datasetId); - if (!record) throw new Error(`Dataset ${datasetId} not found`); - return this.toDataset(record); + return this.repo.get(datasetId); } async listDatasets(filter?: { @@ -119,27 +75,6 @@ export class DatasetService { dataset_type?: DatasetType; status?: DatasetStatus; }): Promise { - let results = Array.from(this.datasets.values()); - if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id); - if (filter?.project_id) results = results.filter((r) => r.project_id === filter.project_id); - if (filter?.dataset_type) results = results.filter((r) => r.dataset_type === filter.dataset_type); - if (filter?.status) results = results.filter((r) => r.status === filter.status); - return results.map((r) => this.toDataset(r)); - } - - private toDataset(r: DatasetRecord): Dataset { - return { - dataset_id: r.dataset_id, - tenant_id: r.tenant_id, - project_id: r.project_id, - name: r.name, - dataset_type: r.dataset_type as DatasetType, - status: r.status as DatasetStatus, - description: r.description ?? undefined, - correlation_id: r.correlation_id ?? undefined, - created_at: r.created_at, - updated_at: r.updated_at ?? undefined, - created_by: r.created_by ?? undefined, - }; + return this.repo.list(filter as DatasetListFilter | undefined); } } diff --git a/src/services/dataset-version-service.ts b/src/services/dataset-version-service.ts index 2c1afbf..9852710 100644 --- a/src/services/dataset-version-service.ts +++ b/src/services/dataset-version-service.ts @@ -1,96 +1,62 @@ /** - * src/services/dataset-version-service.ts - * ======================================== - * P0 Dataset capability extensions (Master-DoD §10 Dataset: - * dataset.create/version/query/export) layered ON TOP of DatasetService (the - * owner of 0pt.dataset.schema.v1 registry entries — deliberately untouched: - * riv inte fungerande kod). + * Dataset version/query/export capability. * - * dataset.version -> createVersion: an immutable, monotonically numbered - * version whose artifacts are CANONICAL ST0RE REFS - * (`st0re:artifact:art_x` — Master-DoD §5); 0pt never - * stores bytes here, only references. - * dataset.query -> listVersions/getVersion (tenant-gated). - * dataset.export -> exportDataset: registers the export artifact in ST0RE - * (real client), records AUD0 evidence + OpenLineage, - * returns the canonical `st0re:artifact:...` ref. - * - * Idempotency (§53): createVersion accepts an idempotency_key; the same key - * returns the SAME version instead of minting a duplicate. + * Dataset metadata is canonical in DatasetService; immutable version metadata + * is persisted through DatasetVersionRepository; bytes remain canonical ST0RE + * artifacts. AUD0/OpenLineage record the evidence chain. */ import crypto from "crypto"; -import { SiaxError } from "../platform/errors"; -import { formatResourceRef, isResourceRef } from "../platform/resource-ref"; import { createAud0Client, type Aud0Client } from "../clients/aud0-client"; import { createSt0reClient, type St0reClient } from "../clients/st0re-client"; import { LineageEmitter } from "../lineage/lineage-emitter"; +import { + createDatasetVersionRepository, + type DatasetVersionRepository, +} from "../repositories/dataset-version-repository"; +import { SiaxError } from "../platform/errors"; +import { formatResourceRef, isResourceRef } from "../platform/resource-ref"; +import type { + CreateDatasetVersionInput, + DatasetExport, + DatasetVersion, +} from "../types/dataset-version"; import { DatasetService } from "./dataset-service"; -export interface DatasetVersion { - dataset_id: string; - version: number; - tenant_id: string; - project_id: string; - /** Canonical ST0RE artifact refs (st0re:artifact:...) making up this version. */ - artifact_refs: string[]; - /** Schema version of the dataset content at this version. */ - schema_version: string; - /** Row/record count when known. */ - record_count?: number; - description?: string; - created_at: string; - created_by: string; - correlation_id?: string; -} +export type { CreateDatasetVersionInput, DatasetExport, DatasetVersion } from "../types/dataset-version"; -export interface CreateDatasetVersionInput { - dataset_id: string; - tenant_id: string; - project_id: string; - artifact_refs: string[]; - schema_version: string; - record_count?: number; - description?: string; - created_by: string; - correlation_id?: string; - idempotency_key?: string; -} - -export interface DatasetExport { - export_id: string; - dataset_id: string; - version: number; - tenant_id: string; - /** Canonical ref of the registered export artifact. */ - artifact_ref: string; - format: "parquet" | "csv" | "json"; - created_at: string; +export interface DatasetVersionServiceOptions { + aud0BaseUrl?: string; + st0reBaseUrl?: string; + databaseUrl?: string; + datasetService?: DatasetService; + repository?: DatasetVersionRepository; } export class DatasetVersionService { - private readonly versions = new Map(); - private readonly idempotency = new Map(); + private readonly repo: DatasetVersionRepository; private readonly aud0: Aud0Client; private readonly st0re: St0reClient; private readonly lineage: LineageEmitter; readonly datasets: DatasetService; - constructor(opts?: { aud0BaseUrl?: string; st0reBaseUrl?: string; datasetService?: DatasetService }) { + constructor(opts?: DatasetVersionServiceOptions) { this.aud0 = createAud0Client(opts?.aud0BaseUrl); this.st0re = createSt0reClient(opts?.st0reBaseUrl); this.lineage = new LineageEmitter(this.aud0); - this.datasets = opts?.datasetService ?? new DatasetService(opts?.aud0BaseUrl); + this.datasets = opts?.datasetService ?? new DatasetService({ + aud0BaseUrl: opts?.aud0BaseUrl, + databaseUrl: opts?.databaseUrl, + }); + this.repo = opts?.repository ?? createDatasetVersionRepository(opts?.databaseUrl); } - // ----------------------------------------------------------------------- - // dataset.version - // ----------------------------------------------------------------------- async createVersion(input: CreateDatasetVersionInput): Promise { await this.assertDatasetTenant(input.dataset_id, input.tenant_id); - - if (input.idempotency_key) { - const existing = this.idempotency.get(`${input.tenant_id}:${input.idempotency_key}`); - if (existing) return existing; + if (!input.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "project_id is required"); + if (!input.schema_version?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "schema_version is required"); + if (!input.created_by?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "created_by is required"); + if (input.record_count !== undefined && (!Number.isSafeInteger(input.record_count) || input.record_count < 0)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "record_count must be a non-negative safe integer"); } if (input.artifact_refs.length === 0) { throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "a dataset version needs at least one artifact ref"); @@ -100,37 +66,19 @@ export class DatasetVersionService { throw new SiaxError( "INVALID_REQUEST", "0pt.dataset", - `artifact ref "${ref}" is not a canonical st0re:artifact:... reference (Master-DoD §5)` + `artifact ref "${ref}" is not a canonical st0re:artifact:... reference (Master-DoD §5)`, ); } } - const list = this.versions.get(input.dataset_id) ?? []; - const version: DatasetVersion = { - dataset_id: input.dataset_id, - version: (list[list.length - 1]?.version ?? 0) + 1, - tenant_id: input.tenant_id, - project_id: input.project_id, - artifact_refs: [...input.artifact_refs], - schema_version: input.schema_version, - record_count: input.record_count, - description: input.description, - created_at: new Date().toISOString(), - created_by: input.created_by, - correlation_id: input.correlation_id, - }; - list.push(version); - this.versions.set(input.dataset_id, list); - if (input.idempotency_key) { - this.idempotency.set(`${input.tenant_id}:${input.idempotency_key}`, version); - } - + const version = await this.repo.create(input); const versionRef = formatResourceRef({ product: "0pt", resourceType: "dataset", resourceId: input.dataset_id, version: String(version.version), }); + await this.aud0.ingestEvent({ event_id: `evt_0pt_dataset_version_${input.dataset_id}_v${version.version}`, event_type: "0pt.dataset_version_created", @@ -145,10 +93,11 @@ export class DatasetVersionService { result: "success", input_refs: version.artifact_refs, output_refs: [versionRef], - payload: { schema_version: input.schema_version, record_count: input.record_count }, + payload: { schema_version: version.schema_version, record_count: version.record_count }, created_at: version.created_at, - correlation_id: input.correlation_id, - idempotency_key: input.idempotency_key, + occurred_at: version.created_at, + correlation_id: version.correlation_id, + idempotency_key: `dataset-version:${input.tenant_id}:${input.dataset_id}:v${version.version}`, }); await this.lineage.emit({ tenant_id: input.tenant_id, @@ -156,41 +105,34 @@ export class DatasetVersionService { capability_id: "0pt.dataset.version.v1", run_id: `dataset-version:${input.dataset_id}:v${version.version}`, event_type: "COMPLETE", + event_time: version.created_at, inputs: version.artifact_refs.map((ref) => ({ name: ref, ref })), outputs: [{ name: `dataset/${input.dataset_id}`, ref: versionRef }], - correlation_id: input.correlation_id, + correlation_id: version.correlation_id, }); return version; } - // ----------------------------------------------------------------------- - // dataset.query - // ----------------------------------------------------------------------- async listVersions(datasetId: string, tenantId: string): Promise { await this.assertDatasetTenant(datasetId, tenantId); - return [...(this.versions.get(datasetId) ?? [])]; + return this.repo.list(datasetId, tenantId); } async getVersion(datasetId: string, tenantId: string, version: number): Promise { - const list = await this.listVersions(datasetId, tenantId); - const found = list.find((v) => v.version === version); - if (!found) { - throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} has no version ${version}`); + await this.assertDatasetTenant(datasetId, tenantId); + if (!Number.isSafeInteger(version) || version < 1) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "version must be a positive integer"); } - return found; + return this.repo.get(datasetId, tenantId, version); } - // ----------------------------------------------------------------------- - // dataset.export - // ----------------------------------------------------------------------- async exportDataset(input: { dataset_id: string; tenant_id: string; project_id: string; version: number; format: DatasetExport["format"]; - /** Where the export bytes were written (by the exporting job). */ object_location: { provider: string; bucket: string; key: string; region?: string }; size_bytes: number; checksum_sha256: string; @@ -198,7 +140,18 @@ export class DatasetVersionService { correlation_id?: string; }): Promise { const version = await this.getVersion(input.dataset_id, input.tenant_id, input.version); - const exportId = `dsexp_${crypto.randomUUID()}`; + if (!(["parquet", "csv", "json"] as const).includes(input.format)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported export format"); + } + if (!Number.isSafeInteger(input.size_bytes) || input.size_bytes < 0) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "size_bytes must be a non-negative safe integer"); + } + if (!/^[a-fA-F0-9]{64}$/.test(input.checksum_sha256)) { + throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "checksum_sha256 must be a 64-character hex digest"); + } + + const exportKey = `dsexp:${input.tenant_id}:${input.dataset_id}:v${input.version}:${input.format}:${input.checksum_sha256.toLowerCase()}`; + const exportId = `dsexp_${crypto.createHash("sha256").update(exportKey).digest("hex").slice(0, 32)}`; const artifact = await this.st0re.createArtifact({ tenant_id: input.tenant_id, @@ -214,14 +167,14 @@ export class DatasetVersionService { ? "text/csv" : "application/json", size_bytes: input.size_bytes, - checksum_sha256: input.checksum_sha256, + checksum_sha256: input.checksum_sha256.toLowerCase(), pii_level: "none", retention_policy: "standard", data_residency: "eu", risk_level: "L2", correlation_id: input.correlation_id, created_by: input.actor_id, - idempotency_key: `dsexp:${input.dataset_id}:v${input.version}:${input.format}:${input.checksum_sha256}`, + idempotency_key: exportKey, }); const artifactRef = formatResourceRef({ @@ -229,6 +182,7 @@ export class DatasetVersionService { resourceType: "artifact", resourceId: artifact.artifact_id, }); + const createdAt = artifact.created_at ?? new Date().toISOString(); const record: DatasetExport = { export_id: exportId, dataset_id: input.dataset_id, @@ -236,7 +190,7 @@ export class DatasetVersionService { tenant_id: input.tenant_id, artifact_ref: artifactRef, format: input.format, - created_at: new Date().toISOString(), + created_at: createdAt, }; const versionRef = formatResourceRef({ @@ -260,8 +214,10 @@ export class DatasetVersionService { input_refs: [versionRef], output_refs: [artifactRef], payload: { format: input.format, size_bytes: input.size_bytes }, - created_at: record.created_at, + created_at: createdAt, + occurred_at: createdAt, correlation_id: input.correlation_id, + idempotency_key: exportKey, }); await this.lineage.emit({ tenant_id: input.tenant_id, @@ -269,6 +225,7 @@ export class DatasetVersionService { capability_id: "0pt.dataset.export.v1", run_id: `dataset-export:${exportId}`, event_type: "COMPLETE", + event_time: createdAt, inputs: [{ name: `dataset/${input.dataset_id}`, ref: versionRef }], outputs: [{ name: artifactRef, ref: artifactRef }], correlation_id: input.correlation_id, @@ -277,9 +234,8 @@ export class DatasetVersionService { return record; } - // ----------------------------------------------------------------------- private async assertDatasetTenant(datasetId: string, tenantId: string): Promise { - if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required"); + if (!tenantId?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required"); let dataset; try { dataset = await this.datasets.getDataset(datasetId); diff --git a/src/services/datasets.ts b/src/services/datasets.ts new file mode 100644 index 0000000..c52467a --- /dev/null +++ b/src/services/datasets.ts @@ -0,0 +1,6 @@ +/** Shared composition root for the dataset HTTP surface. */ +import { DatasetService } from "./dataset-service"; +import { DatasetVersionService } from "./dataset-version-service"; + +export const datasetService = new DatasetService(); +export const datasetVersionService = new DatasetVersionService({ datasetService }); diff --git a/src/services/entity-resolution.ts b/src/services/entity-resolution.ts new file mode 100644 index 0000000..4d16b90 --- /dev/null +++ b/src/services/entity-resolution.ts @@ -0,0 +1,10 @@ +import { createN0dClient } from "../clients/n0d-client"; +import { EntityResolutionService } from "../er/entity-resolution-service"; + +export const entityResolutionService = new EntityResolutionService(); + +let batchService: EntityResolutionService | undefined; +export function getBatchEntityResolutionService(): EntityResolutionService { + if (!batchService) batchService = new EntityResolutionService({ n0dClient: createN0dClient() }); + return batchService; +} diff --git a/src/services/search.ts b/src/services/search.ts new file mode 100644 index 0000000..2bbbbcf --- /dev/null +++ b/src/services/search.ts @@ -0,0 +1,33 @@ +import { createAud0Client } from "../clients/aud0-client"; +import { resolveDependencyBaseUrl } from "../platform/dependency-mode"; +import { SiaxError } from "../platform/errors"; +import { createSearchDefinitionRepository } from "../repositories/search-definition-repository"; +import { HttpOpenSearchAdapter } from "../search/opensearch-adapter"; +import { SearchService } from "../search/search-service"; + +let singleton: SearchService | undefined; + +export function getSearchService(): SearchService { + if (singleton) return singleton; + const baseUrl = resolveDependencyBaseUrl("opensearch", process.env.OPENSEARCH_URL); + if (!baseUrl) { + throw new SiaxError("DEPENDENCY_UNAVAILABLE", "opensearch", "OPENSEARCH_URL is required for the search HTTP surface"); + } + singleton = new SearchService( + new HttpOpenSearchAdapter({ + baseUrl, + ...(process.env.OPENSEARCH_BASIC_AUTH + ? { basicAuth: process.env.OPENSEARCH_BASIC_AUTH } + : {}), + }), + { + aud0Client: createAud0Client(), + definitionRepository: createSearchDefinitionRepository(), + }, + ); + return singleton; +} + +export function resetSearchServiceForTests(): void { + singleton = undefined; +} diff --git a/src/types/dataset-version.ts b/src/types/dataset-version.ts new file mode 100644 index 0000000..6639f9e --- /dev/null +++ b/src/types/dataset-version.ts @@ -0,0 +1,38 @@ +export interface DatasetVersion { + dataset_id: string; + version: number; + tenant_id: string; + project_id: string; + /** Canonical ST0RE artifact refs that make up this immutable version. */ + artifact_refs: string[]; + schema_version: string; + record_count?: number; + description?: string; + created_at: string; + created_by: string; + correlation_id?: string; +} + +export interface CreateDatasetVersionInput { + dataset_id: string; + tenant_id: string; + project_id: string; + /** Immutable input — callers may hand us readonly arrays (as const); we never mutate. */ + readonly artifact_refs: readonly string[]; + schema_version: string; + record_count?: number; + description?: string; + created_by: string; + correlation_id?: string; + idempotency_key?: string; +} + +export interface DatasetExport { + export_id: string; + dataset_id: string; + version: number; + tenant_id: string; + artifact_ref: string; + format: "parquet" | "csv" | "json"; + created_at: string; +} diff --git a/tests/integration/search-durable-definition.integration.test.ts b/tests/integration/search-durable-definition.integration.test.ts new file mode 100644 index 0000000..e0f9277 --- /dev/null +++ b/tests/integration/search-durable-definition.integration.test.ts @@ -0,0 +1,117 @@ +import fs from "node:fs"; +import path from "node:path"; +import postgres from "postgres"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; + +import { closeRepositorySqlForTests, getRepositorySql } from "@/src/repositories/db"; +import { PostgresSearchDefinitionRepository } from "@/src/repositories/search-definition-repository"; +import { HttpOpenSearchAdapter } from "@/src/search/opensearch-adapter"; +import { SearchService, type DocumentSource } from "@/src/search/search-service"; +import type { SearchDocument } from "@/src/search/search-types"; + +const OPENSEARCH_URL = process.env.OPENSEARCH_URL; +const DATABASE_URL = process.env.SEARCH_DEFINITION_TEST_DATABASE_URL; +const run = OPENSEARCH_URL && DATABASE_URL ? describe : describe.skip; +let admin: ReturnType | undefined; +let reachable = false; +let reason = ""; + +beforeAll(async () => { + if (!OPENSEARCH_URL || !DATABASE_URL) return; + try { + admin = postgres(DATABASE_URL, { max: 1, prepare: false, onnotice: () => undefined }); + await admin`SELECT 1`; + await admin.unsafe(fs.readFileSync(path.join(process.cwd(), "sql", "013_search_index_definitions.sql"), "utf8")); + const health = await new HttpOpenSearchAdapter({ baseUrl: OPENSEARCH_URL }).health(); + if (!(["green", "yellow"] as string[]).includes(health.status)) throw new Error(`OpenSearch ${health.status}`); + reachable = true; + } catch (error) { + reason = error instanceof Error ? error.message : String(error); + } +}, 30_000); + +afterAll(async () => { + await admin?.end({ timeout: 5 }); + await closeRepositorySqlForTests(); +}); + +run("durable SearchDefinitionRepository + live OpenSearch", () => { + it("reloads a definition after a fresh SearchService and rebuilds/querys the projection", async (ctx) => { + if (!reachable) { + console.warn(`[realinfra] search durable definition dependencies unavailable (${reason}) — skipping honestly`); + ctx.skip(); + return; + } + const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`.replace(/[^a-z0-9_]/gi, "_"); + const indexName = `companies_${stamp}`.slice(0, 60); + const tenant = `t_${stamp}`; + const repo = new PostgresSearchDefinitionRepository(getRepositorySql(DATABASE_URL!)); + const adapter = new HttpOpenSearchAdapter({ baseUrl: OPENSEARCH_URL! }); + + const writer = new SearchService(adapter, { definitionRepository: repo }); + await writer.registerIndexDefinitionDurable({ + index_name: indexName, + version: 1, + fields: { name: "text", org_nr: "keyword" }, + default_search_fields: ["name"], + }, "realinfra-test"); + + // New instance, empty in-memory cache — this is the restart condition. + const reader = new SearchService(adapter, { definitionRepository: repo }); + const loaded = await reader.resolveIndexDefinition(indexName); + expect(loaded.version).toBe(1); + expect(loaded.fields.name).toBe("text"); + + const docs: SearchDocument[] = [{ + doc_id: "company-1", + tenant_id: tenant, + purpose_tags: ["company_enrichment"], + rights_tags: ["internal"], + source_refs: ["0pt:dataset:companies@1"], + payload: { name: "Acme Aktiebolag", org_nr: "5560000000" }, + }]; + const source: DocumentSource = { + name: "0pt:dataset:companies@1", + async *iterate() { for (const doc of docs) yield doc; }, + }; + + const rebuilt = await reader.rebuild(indexName, source, { tenant_id: tenant }); + expect(rebuilt.docs_indexed).toBe(1); + expect(rebuilt.switched).toBe(true); + + const result = await reader.query(indexName, { + tenant_id: tenant, + purpose: "company_enrichment", + rights_tag: "internal", + text: "Acme", + }); + expect(result.total).toBe(1); + expect(result.hits[0].doc_id).toBe("company-1"); + + // A foreign tenant sees no hit even though it queries the same physical alias. + const foreign = await reader.query(indexName, { + tenant_id: `foreign_${tenant}`, + purpose: "company_enrichment", + rights_tag: "internal", + text: "Acme", + }); + expect(foreign.total).toBe(0); + }); + + it("enforces sequential schema versions and rejects same-version drift", async (ctx) => { + if (!reachable) return ctx.skip(); + const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`.replace(/[^a-z0-9_]/gi, "_"); + const indexName = `schema_${stamp}`.slice(0, 60); + const repo = new PostgresSearchDefinitionRepository(getRepositorySql(DATABASE_URL!)); + const v1 = { index_name: indexName, version: 1, fields: { name: "text" as const }, default_search_fields: ["name"] }; + const first = await repo.save(v1, "test"); + const replay = await repo.save(v1, "test"); + expect(first.created).toBe(true); + expect(replay.created).toBe(false); + expect(replay.created_at).toBe(first.created_at); + + await expect(repo.save({ ...v1, fields: { name: "keyword" } }, "test")).rejects.toMatchObject({ code: "CONFLICT" }); + await expect(repo.save({ ...v1, version: 3 }, "test")).rejects.toMatchObject({ code: "CONFLICT" }); + expect((await repo.save({ ...v1, version: 2 }, "test")).definition.version).toBe(2); + }); +}); diff --git a/tests/p0/aud0-outbox-postgres.realinfra.test.ts b/tests/p0/aud0-outbox-postgres.realinfra.test.ts new file mode 100644 index 0000000..b3f0855 --- /dev/null +++ b/tests/p0/aud0-outbox-postgres.realinfra.test.ts @@ -0,0 +1,94 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import postgres from "postgres"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; + +import { PostgresAud0OutboxRepository } from "@/src/evidence/aud0-outbox"; + +const TEST_DATABASE_URL = process.env.TEST_DATABASE_URL; +const describeReal = TEST_DATABASE_URL ? describe : describe.skip; +const schema = `opt_outbox_test_${process.pid}_${Date.now()}`.replace(/[^a-zA-Z0-9_]/g, "_"); +let sql: ReturnType; +let repo: PostgresAud0OutboxRepository; + +const baseEvent = { + event_id: "evt-real-1", + event_type: "0pt.test", + tenant_id: "tenant-real-a", + project_id: "project-a", + actor_id: "tester", + app_id: "0pt", + capability_id: "0pt.test.v1", + resource_type: "test", + resource_id: "r1", + risk_level: "L1", + created_at: "2026-09-07T12:00:00.000Z", + idempotency_key: "idem-real-1", +}; + +describeReal("AUD0 outbox Postgres integration", () => { + beforeAll(async () => { + const admin = postgres(TEST_DATABASE_URL!, { max: 1, onnotice: () => undefined }); + await admin.unsafe(`CREATE SCHEMA ${schema}`); + await admin.end(); + sql = postgres(TEST_DATABASE_URL!, { + max: 3, + prepare: false, + onnotice: () => undefined, + connection: { search_path: schema }, + }); + const migration = await fs.readFile(path.join(process.cwd(), "sql/011_aud0_evidence_outbox.sql"), "utf8"); + await sql.unsafe(migration); + repo = new PostgresAud0OutboxRepository(sql); + }); + + afterAll(async () => { + if (sql) await sql.end({ timeout: 5 }); + if (TEST_DATABASE_URL) { + const admin = postgres(TEST_DATABASE_URL, { max: 1, onnotice: () => undefined }); + await admin.unsafe(`DROP SCHEMA IF EXISTS ${schema} CASCADE`); + await admin.end(); + } + }); + + it("deduplicates identical producer replays and rejects same-key payload drift", async () => { + const first = await repo.enqueueEvent(baseEvent); + const replay = await repo.enqueueEvent({ ...baseEvent }); + expect(replay.outbox_id).toBe(first.outbox_id); + + await expect( + repo.enqueueEvent({ ...baseEvent, resource_id: "different" }), + ).rejects.toMatchObject({ code: "CONFLICT" }); + }); + + it("leases rows to only one worker and recovers retry state", async () => { + await repo.enqueueEvent({ ...baseEvent, event_id: "evt-real-2", idempotency_key: "idem-real-2" }); + const a = await repo.claimPending({ workerId: "worker-a", limit: 100, leaseMs: 60_000 }); + expect(a.length).toBeGreaterThanOrEqual(2); + + const b = await repo.claimPending({ workerId: "worker-b", limit: 100, leaseMs: 60_000 }); + expect(b).toHaveLength(0); + + const target = a.find((r) => r.idempotency_key === "idem-real-2")!; + await repo.markRetry(target.outbox_id, "worker-a", "DEPENDENCY_UNAVAILABLE:aud0", 0); + const reclaimed = await repo.claimPending({ workerId: "worker-b", limit: 100, leaseMs: 60_000 }); + expect(reclaimed.some((r) => r.outbox_id === target.outbox_id)).toBe(true); + }); + + it("keeps delivered/dead rows as visible terminal evidence-delivery state", async () => { + const delivered = await repo.enqueueEvent({ ...baseEvent, event_id: "evt-real-3", idempotency_key: "idem-real-3" }); + const dead = await repo.enqueueEvent({ ...baseEvent, event_id: "evt-real-4", idempotency_key: "idem-real-4" }); + + const claimed = await repo.claimPending({ workerId: "terminal-worker", limit: 100, leaseMs: 60_000 }); + if (claimed.some((r) => r.outbox_id === delivered.outbox_id)) { + await repo.markDelivered(delivered.outbox_id, "terminal-worker"); + } + if (claimed.some((r) => r.outbox_id === dead.outbox_id)) { + await repo.markDead(dead.outbox_id, "terminal-worker", "INVALID_REQUEST:aud0"); + } + + const stats = await repo.stats(); + expect(stats.delivered).toBeGreaterThanOrEqual(1); + expect(stats.dead).toBeGreaterThanOrEqual(1); + }); +}); diff --git a/tests/p0/aud0-outbox.test.ts b/tests/p0/aud0-outbox.test.ts new file mode 100644 index 0000000..ca7f5e3 --- /dev/null +++ b/tests/p0/aud0-outbox.test.ts @@ -0,0 +1,195 @@ +import { describe, expect, it } from "vitest"; + +import type { Aud0Client, Aud0Event } from "@/src/clients/aud0-client"; +import { + Aud0OutboxDispatcher, + openLineageIdempotencyKey, + outboxId, + type Aud0OutboxRecord, + type Aud0OutboxRepository, + type Aud0OutboxStats, +} from "@/src/evidence/aud0-outbox"; +import { LineageEmitter } from "@/src/lineage/lineage-emitter"; +import type { OpenLineageRunEvent } from "@/src/lineage/openlineage"; +import { SiaxError } from "@/src/platform/errors"; + +const event: Aud0Event = { + event_id: "evt-1", + event_type: "0pt.test", + tenant_id: "tenant-a", + project_id: "project-a", + actor_id: "tester", + app_id: "0pt", + capability_id: "0pt.test.v1", + resource_type: "test", + resource_id: "r1", + risk_level: "L1", + created_at: "2026-09-07T12:00:00.000Z", + idempotency_key: "idem-1", +}; + +const ol: OpenLineageRunEvent = { + eventType: "COMPLETE", + eventTime: "2026-09-07T12:00:00.000Z", + producer: "https://git.siax.io/sax3l/0pt", + run: { runId: "run-1" }, + job: { namespace: "0pt", name: "0pt.test.v1" }, + inputs: [], + outputs: [], +}; + +class FakeRepo implements Aud0OutboxRepository { + claimed: Aud0OutboxRecord[] = []; + delivered: string[] = []; + dead: Array<{ id: string; error: string }> = []; + retried: Array<{ id: string; error: string; delayMs: number }> = []; + + async enqueueEvent(): Promise { throw new Error("unused"); } + async enqueueOpenLineage(): Promise { throw new Error("unused"); } + async claimPending(): Promise { return this.claimed; } + async markDelivered(id: string): Promise { this.delivered.push(id); } + async markRetry(id: string, _worker: string, error: string, delayMs: number): Promise { + this.retried.push({ id, error, delayMs }); + } + async markDead(id: string, _worker: string, error: string): Promise { + this.dead.push({ id, error }); + } + async stats(): Promise { return { pending: this.claimed.length, delivered: 0, dead: 0 }; } +} + +function row( + kind: "event" | "openlineage", + payload: Aud0Event | OpenLineageRunEvent, + overrides: Partial = {}, +): Aud0OutboxRecord { + return { + outbox_id: "obx-1", + tenant_id: "tenant-a", + app_id: "0pt", + evidence_kind: kind, + idempotency_key: "idem-1", + payload, + status: "pending", + attempts: 1, + next_attempt_at: "2026-09-07T12:00:00.000Z", + created_at: "2026-09-07T12:00:00.000Z", + updated_at: "2026-09-07T12:00:00.000Z", + ...overrides, + }; +} + +function client(impl?: Partial): Aud0Client { + return { + async ingestEvent(input) { return input; }, + async ingestOpenLineageEvent(tenantId, _appId, input) { + return { tenant_id: tenantId, run_id: input.run.runId, job: input.job.name, event_id: "aud0-1" }; + }, + async ingestLineageEvent(input) { return input; }, + ...impl, + }; +} + +describe("AUD0 durable outbox", () => { + it("derives stable tenant-bound outbox ids", () => { + expect(outboxId("tenant-a", "event", "k1")).toBe(outboxId("tenant-a", "event", "k1")); + expect(outboxId("tenant-a", "event", "k1")).not.toBe(outboxId("tenant-b", "event", "k1")); + expect(outboxId("tenant-a", "event", "k1")).not.toBe(outboxId("tenant-a", "openlineage", "k1")); + }); + + it("uses logical OpenLineage run+eventType identity rather than delivery time", () => { + expect(openLineageIdempotencyKey(ol)).toBe("ol:run-1:COMPLETE"); + expect(openLineageIdempotencyKey({ ...ol, eventTime: "2030-01-01T00:00:00Z" })).toBe("ol:run-1:COMPLETE"); + }); + + it("marks a successfully delivered generic event delivered", async () => { + const repo = new FakeRepo(); + repo.claimed = [row("event", event)]; + let calls = 0; + const dispatcher = new Aud0OutboxDispatcher(repo, client({ + async ingestEvent(input) { calls += 1; return input; }, + }), "worker-a"); + + expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 1, dead: 0, retried: 0 }); + expect(calls).toBe(1); + expect(repo.delivered).toEqual(["obx-1"]); + }); + + it("retries network/dependency failures with bounded exponential backoff", async () => { + const repo = new FakeRepo(); + repo.claimed = [row("openlineage", ol, { attempts: 4 })]; + const dispatcher = new Aud0OutboxDispatcher(repo, client({ + async ingestOpenLineageEvent() { + throw new SiaxError("DEPENDENCY_UNAVAILABLE", "aud0", "offline"); + }, + }), "worker-a"); + + expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 0, dead: 0, retried: 1 }); + expect(repo.retried[0].delayMs).toBe(8_000); + expect(repo.retried[0].error).toBe("DEPENDENCY_UNAVAILABLE:aud0"); + }); + + it("parks permanent 4xx-style failures dead instead of poisoning the queue", async () => { + const repo = new FakeRepo(); + repo.claimed = [row("event", event)]; + const dispatcher = new Aud0OutboxDispatcher(repo, client({ + async ingestEvent() { + throw new SiaxError("INVALID_REQUEST", "aud0", "bad event"); + }, + }), "worker-a"); + + expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 0, dead: 1, retried: 0 }); + expect(repo.dead).toEqual([{ id: "obx-1", error: "INVALID_REQUEST:aud0" }]); + }); + + it("fails closed on an immutable tenant/app envelope mismatch before calling AUD0", async () => { + const repo = new FakeRepo(); + repo.claimed = [row("event", { ...event, tenant_id: "tenant-b" })]; + let calls = 0; + const dispatcher = new Aud0OutboxDispatcher(repo, client({ + async ingestEvent(input) { calls += 1; return input; }, + }), "worker-a"); + + expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 0, dead: 1, retried: 0 }); + expect(calls).toBe(0); + expect(repo.dead[0].error).toBe("PERMANENT_FAILURE:0pt.aud0_outbox"); + }); + + it("reports a durable client acknowledgement as queued, not emitted", async () => { + const emitter = new LineageEmitter(client({ + async ingestOpenLineageEvent(tenantId, _appId, input) { + return { + tenant_id: tenantId, + run_id: input.run.runId, + job: input.job.name, + event_id: "outbox:obx-queued-1", + }; + }, + })); + + const result = await emitter.emit({ + tenant_id: "tenant-a", + capability_id: "0pt.test.v1", + run_id: "run-queued", + event_type: "COMPLETE", + event_time: "2026-09-07T12:00:00.000Z", + }); + expect(result).toEqual({ + emitted: false, + queued: true, + run_id: "run-queued", + outbox_id: "obx-queued-1", + }); + }); + + it("reports a raw AUD0 acknowledgement as emitted, not queued", async () => { + const emitter = new LineageEmitter(client()); + const result = await emitter.emit({ + tenant_id: "tenant-a", + capability_id: "0pt.test.v1", + run_id: "run-live", + event_type: "COMPLETE", + event_time: "2026-09-07T12:00:00.000Z", + }); + expect(result).toEqual({ emitted: true, queued: false, run_id: "run-live" }); + }); +}); diff --git a/tests/p0/dataset-postgres.realinfra.test.ts b/tests/p0/dataset-postgres.realinfra.test.ts new file mode 100644 index 0000000..cf5447c --- /dev/null +++ b/tests/p0/dataset-postgres.realinfra.test.ts @@ -0,0 +1,222 @@ +/** + * Real-Postgres proof for the dataset registry + immutable version metadata. + * + * Set DATASET_TEST_DATABASE_URL to a disposable Postgres 16 instance. When it + * is unset the suite skips honestly; Gitea CI should provide it for the + * blocking real-infrastructure job before this capability is promoted. + */ +import fs from "node:fs"; +import path from "node:path"; +import postgres from "postgres"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; + +import { closeRepositorySqlForTests } from "@/src/repositories/db"; +import { DatasetService } from "@/src/services/dataset-service"; +import { DatasetVersionService } from "@/src/services/dataset-version-service"; + +const TEST_DATABASE_URL = process.env.DATASET_TEST_DATABASE_URL; +const run = TEST_DATABASE_URL ? describe : describe.skip; +let admin: ReturnType | undefined; +let reachable = false; +let unreachableReason = ""; + +beforeAll(async () => { + if (!TEST_DATABASE_URL) { + console.warn("[realinfra] DATASET_TEST_DATABASE_URL unset — skipping honestly."); + return; + } + try { + admin = postgres(TEST_DATABASE_URL, { prepare: false, ssl: false, max: 3 }); + await Promise.race([ + admin`SELECT 1`, + new Promise((_, reject) => setTimeout(() => reject(new Error("connect timeout")), 5_000)), + ]); + for (const migration of ["003_datasets.sql", "011_aud0_evidence_outbox.sql", "012_dataset_versions.sql"]) { + await admin.unsafe(fs.readFileSync(path.join(process.cwd(), "sql", migration), "utf8")); + } + // Aktivera den durabla AUD0-outboxen: createAud0OutboxRepositoryFromEnv() + // läser DATABASE_URL (se src/repositories/db.ts) — utan den blir klienten + // Noop och inga producer-rader skrivs (testet skulle ljugit om durabilitet). + process.env.DATABASE_URL = TEST_DATABASE_URL; + reachable = true; + } catch (error) { + unreachableReason = error instanceof Error ? error.message : String(error); + } +}, 30_000); + +afterAll(async () => { + delete process.env.DATABASE_URL; + await admin?.end({ timeout: 5 }); + await closeRepositorySqlForTests(); +}); + +function datasetService(): DatasetService { + return new DatasetService({ databaseUrl: TEST_DATABASE_URL }); +} + +function versionService(): DatasetVersionService { + return new DatasetVersionService({ databaseUrl: TEST_DATABASE_URL }); +} + +run("Dataset registry + versions against LIVE Postgres", () => { + it("persists a dataset across independent service instances", async (ctx) => { + if (!reachable) { + console.warn(`[realinfra] Postgres unavailable (${unreachableReason}) — skipping honestly.`); + ctx.skip(); + return; + } + const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`; + const datasetId = `ds_realpg_${stamp}`; + const tenantId = `t_realpg_${stamp}`; + + await datasetService().registerDataset({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + name: "Real Postgres dataset", + dataset_type: "curated", + created_by: "realinfra-test", + }); + + const reloaded = await datasetService().getDataset(datasetId); + expect(reloaded.dataset_id).toBe(datasetId); + expect(reloaded.tenant_id).toBe(tenantId); + + const rows = await admin!<{ tenant_id: string; name: string }[]>` + SELECT tenant_id, name FROM opt_datasets WHERE dataset_id = ${datasetId}`; + expect(rows).toEqual([{ tenant_id: tenantId, name: "Real Postgres dataset" }]); + }); + + it("persists monotonic immutable versions across fresh service instances", async (ctx) => { + if (!reachable) return ctx.skip(); + const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`; + const datasetId = `ds_versions_${stamp}`; + const tenantId = `t_versions_${stamp}`; + await datasetService().registerDataset({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + name: "Versioned dataset", + dataset_type: "curated", + created_by: "realinfra-test", + }); + + const v1 = await versionService().createVersion({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + artifact_refs: ["st0re:artifact:art_realpg_1"], + schema_version: "1", + record_count: 10, + created_by: "realinfra-test", + idempotency_key: `idem_${stamp}_1`, + }); + const v2 = await versionService().createVersion({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + artifact_refs: ["st0re:artifact:art_realpg_2"], + schema_version: "1", + record_count: 11, + created_by: "realinfra-test", + idempotency_key: `idem_${stamp}_2`, + }); + + expect([v1.version, v2.version]).toEqual([1, 2]); + const fresh = versionService(); + expect((await fresh.listVersions(datasetId, tenantId)).map((item) => item.version)).toEqual([1, 2]); + expect((await fresh.getVersion(datasetId, tenantId, 2)).artifact_refs).toEqual(["st0re:artifact:art_realpg_2"]); + }); + + it("replays the same idempotency key without minting another version and rejects payload drift", async (ctx) => { + if (!reachable) return ctx.skip(); + const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`; + const datasetId = `ds_idem_${stamp}`; + const tenantId = `t_idem_${stamp}`; + await datasetService().registerDataset({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + name: "Idempotent dataset", + dataset_type: "derived", + created_by: "realinfra-test", + }); + const input = { + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + artifact_refs: ["st0re:artifact:art_idem_1"], + schema_version: "1", + created_by: "realinfra-test", + idempotency_key: `idem_${stamp}`, + } as const; + + const first = await versionService().createVersion(input); + const replay = await versionService().createVersion(input); + expect(replay.version).toBe(first.version); + expect(await versionService().listVersions(datasetId, tenantId)).toHaveLength(1); + + await expect(versionService().createVersion({ + ...input, + artifact_refs: ["st0re:artifact:art_DIFFERENT"], + })).rejects.toMatchObject({ code: "CONFLICT" }); + }); + + it("foreign tenant cannot read or version another tenant's dataset", async (ctx) => { + if (!reachable) return ctx.skip(); + const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`; + const datasetId = `ds_tenant_${stamp}`; + const owner = `t_owner_${stamp}`; + const foreign = `t_foreign_${stamp}`; + await datasetService().registerDataset({ + dataset_id: datasetId, + tenant_id: owner, + project_id: "p_main", + name: "Tenant isolated", + dataset_type: "raw", + created_by: "realinfra-test", + }); + + await expect(versionService().listVersions(datasetId, foreign)).rejects.toMatchObject({ code: "NOT_FOUND" }); + await expect(versionService().createVersion({ + dataset_id: datasetId, + tenant_id: foreign, + project_id: "p_main", + artifact_refs: ["st0re:artifact:art_foreign"], + schema_version: "1", + created_by: "intruder", + })).rejects.toMatchObject({ code: "NOT_FOUND" }); + }); + + it("durable AUD0 producer rows are written alongside dataset operations", async (ctx) => { + if (!reachable) return ctx.skip(); + const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`; + const datasetId = `ds_evidence_${stamp}`; + const tenantId = `t_evidence_${stamp}`; + await datasetService().registerDataset({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + name: "Evidence dataset", + dataset_type: "curated", + created_by: "realinfra-test", + }); + await versionService().createVersion({ + dataset_id: datasetId, + tenant_id: tenantId, + project_id: "p_main", + artifact_refs: ["st0re:artifact:art_evidence"], + schema_version: "1", + created_by: "realinfra-test", + }); + + const evidence = await admin!<{ evidence_kind: string; status: string }[]>` + SELECT evidence_kind, status + FROM opt_aud0_evidence_outbox + WHERE tenant_id = ${tenantId} + ORDER BY created_at`; + expect(evidence.length).toBeGreaterThanOrEqual(3); // register + version event + lineage + expect(evidence.every((row) => row.status === "pending")).toBe(true); + expect(new Set(evidence.map((row) => row.evidence_kind))).toEqual(new Set(["event", "openlineage"])); + }); +}); diff --git a/tests/p0/entity-resolution-http.test.ts b/tests/p0/entity-resolution-http.test.ts new file mode 100644 index 0000000..be79949 --- /dev/null +++ b/tests/p0/entity-resolution-http.test.ts @@ -0,0 +1,197 @@ +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { NextRequest } from "next/server"; + +import { POST as candidatesPOST } from "@/app/api/0pt/internal/er/candidates/route"; + +const TOKEN = "test-er-internal-token"; +let previousToken: string | undefined; + +function request(body: unknown, token: string | undefined = TOKEN): NextRequest { + const headers: Record = { "content-type": "application/json" }; + if (token !== undefined) headers["x-internal-token"] = token; + return new NextRequest("http://localhost/api/0pt/internal/er/candidates", { + method: "POST", + headers, + body: JSON.stringify(body), + }); +} + +function baseBody(records: Array<{ record_id: string; fields: Record }>) { + return { + tenant_id: "tenant_a", + project_id: "project_a", + records, + blocking_fields: ["city"], + compare_fields: [{ field: "name", weight: 1 }], + auto_match_threshold: 0.9, + review_threshold: 0.6, + }; +} + +beforeEach(() => { + previousToken = process.env.INTERNAL_TOKEN; + process.env.INTERNAL_TOKEN = TOKEN; +}); + +afterEach(() => { + if (previousToken === undefined) delete process.env.INTERNAL_TOKEN; + else process.env.INTERNAL_TOKEN = previousToken; +}); + +describe("POST /api/0pt/internal/er/candidates", () => { + it("fails closed without the internal service token", async () => { + // OBS: request(body, undefined) triggar default-parametern (skickar GILTIGT + // token) — skicka tom string för "utan token" (fail-closed → 401). + const res = await candidatesPOST( + request( + baseBody([ + { record_id: "a", fields: { name: "Acme AB", city: "Stockholm" } }, + { record_id: "b", fields: { name: "Acme AB", city: "Stockholm" } }, + ]), + "", + ), + ); + expect(res.status).toBe(401); + }); + + it("never emits a score-only fuzzy company match even above auto_match_threshold", async () => { + const res = await candidatesPOST( + request( + baseBody([ + { record_id: "a", fields: { name: "Acme Consulting AB", city: "Stockholm" } }, + { record_id: "b", fields: { name: "Acme Consulting A B", city: "Stockholm" } }, + ]), + ), + ); + expect(res.status).toBe(200); + const body = await res.json(); + expect(body.pairs).toHaveLength(1); + expect(body.pairs[0].score).toBeGreaterThanOrEqual(0.9); + expect(body.pairs[0].decided_by).toBe("score"); + expect(body.pairs[0].status).toBe("pending_review"); + expect(body.review_queue.map((pair: { candidate_id: string }) => pair.candidate_id)).toEqual([ + body.pairs[0].candidate_id, + ]); + }); + + it("allows a same-country registration number to hard-link despite punctuation differences", async () => { + const res = await candidatesPOST( + request( + baseBody([ + { + record_id: "a", + fields: { + name: "Acme Sweden AB", + city: "Stockholm", + registration_country: "SE", + org_nr: "556000-0001", + }, + }, + { + record_id: "b", + fields: { + name: "Acme Sverige", + city: "Stockholm", + registration_country: "se", + org_nr: "5560000001", + }, + }, + ]), + ), + ); + expect(res.status).toBe(200); + const body = await res.json(); + expect(body.pairs).toHaveLength(1); + expect(body.pairs[0].status).toBe("match"); + expect(body.pairs[0].decided_by).toBe("0pt.company.stable-identifiers.v3"); + expect(body.review_queue).toEqual([]); + }); + + it("does not treat the same national registration number in different countries as global identity", async () => { + const res = await candidatesPOST( + request( + baseBody([ + { + record_id: "a", + fields: { + name: "Acme Consulting AB", + city: "Stockholm", + registration_country: "SE", + org_nr: "123456789", + }, + }, + { + record_id: "b", + fields: { + name: "Acme Consulting A B", + city: "Stockholm", + registration_country: "NO", + org_nr: "123456789", + }, + }, + ]), + ), + ); + expect(res.status).toBe(200); + const body = await res.json(); + expect(body.pairs).toHaveLength(1); + expect(body.pairs[0].decided_by).toBe("score"); + expect(body.pairs[0].status).toBe("pending_review"); + }); + + it("lets conflicting stable identity evidence dominate a matching global LEI", async () => { + const res = await candidatesPOST( + request( + baseBody([ + { + record_id: "a", + fields: { + name: "Acme Sweden AB", + city: "Stockholm", + registration_country: "SE", + lei: "549300EXAMPLE123", + org_nr: "556000-0001", + }, + }, + { + record_id: "b", + fields: { + name: "Acme Sweden AB", + city: "Stockholm", + registration_country: "SE", + lei: "549300EXAMPLE123", + org_nr: "556000-9999", + }, + }, + ]), + ), + ); + expect(res.status).toBe(200); + const body = await res.json(); + expect(body.pairs).toHaveLength(1); + expect(body.pairs[0].status).toBe("non_match"); + expect(body.pairs[0].decided_by).toBe("0pt.company.stable-identifiers.v3"); + }); + + it("can hard-link a globally matching LEI when no scoped identifier contradicts it", async () => { + const res = await candidatesPOST( + request( + baseBody([ + { + record_id: "a", + fields: { name: "Acme Holding", city: "Stockholm", lei: "549300-EXAMPLE-123" }, + }, + { + record_id: "b", + fields: { name: "ACME Holdings Group", city: "Stockholm", lei: "549300EXAMPLE123" }, + }, + ]), + ), + ); + expect(res.status).toBe(200); + const body = await res.json(); + expect(body.pairs).toHaveLength(1); + expect(body.pairs[0].status).toBe("match"); + expect(body.pairs[0].decided_by).toBe("0pt.company.stable-identifiers.v3"); + }); +}); diff --git a/tests/p0/entity-resolution.test.ts b/tests/p0/entity-resolution.test.ts index 44a613d..07938dd 100644 --- a/tests/p0/entity-resolution.test.ts +++ b/tests/p0/entity-resolution.test.ts @@ -97,7 +97,7 @@ describe("EntityResolutionService — candidate API", () => { expect(() => er.generateCandidates(request({ auto_match_threshold: 0.5, review_threshold: 0.6 })) ).toThrowError(/review_threshold/); - expect(() => er.generateCandidates(request({ records: [RECORDS[0]] }))).toThrowError(/two records/); + expect(() => er.generateCandidates(request({ records: [RECORDS[0]] }))).toThrowError(/between 2 and 5000/); expect(() => er.generateCandidates(request({ tenant_id: "" }))).toThrowError(/tenant_id/); }); diff --git a/tests/p0/search-service.test.ts b/tests/p0/search-service.test.ts index 8bf9b29..56f781e 100644 --- a/tests/p0/search-service.test.ts +++ b/tests/p0/search-service.test.ts @@ -160,12 +160,26 @@ describe("SearchService — tenant + purpose/rights enforcement (§10/§31)", () const { search } = await build([ doc("1", "t_a", "Tagged", { purpose_tags: ["company_enrichment"] }), doc("2", "t_a", "OtherPurpose", { purpose_tags: ["compliance"] }), - doc("3", "t_a", "Untagged", { purpose_tags: [] }), // empty ⇒ never returnable ]); const res = await search.query("companies", { tenant_id: "t_a", purpose: "company_enrichment" }); expect(res.hits.map((h) => h.doc_id)).toEqual(["1"]); }); + it("fails closed: rebuild rejects documents without purpose_tags (§10)", async () => { + const fake = new FakeOpenSearch(); + const search = new SearchService(fake); + search.registerIndexDefinition({ ...DEF, fields: { ...DEF.fields } }); + await expect( + search.rebuild( + "companies", + docsSource([ + doc("1", "t_a", "Ok"), + doc("3", "t_a", "Untagged", { purpose_tags: [] }), // empty ⇒ INVALID_REQUEST, generation discarded + ]), + ), + ).rejects.toThrowError(/missing purpose_tags/); + }); + it("rights filtering: default rights tag is 'internal'; deliverable docs need the tag", async () => { const { search } = await build([ doc("1", "t_a", "Internal", { rights_tags: ["internal"] }),