admin/0pt
5
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: durable AUD0 outbox + dataset/search/ER HTTP surfaces + P065 stängd (kod, utan workflow-ändringar) #43

Merged
admin merged 68 commits from feat/aud0-outbox-code-20260907 into main 2026-09-07 17:53:04 +00:00
Conversation 0 Commits 68 Files Changed 41
+3659 -920
41 changed files with 3659 additions and 920 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATUS.md
+44 -26
View File
@@ -70,15 +70,31 @@ accurate.
## Global gaps blocking PRODUCTION_READY (§3, tracked, not hidden)
1. Persistent stores: the new registries (sources/rights/quality
rules/dataset versions/index definitions/feature views) are per-process
in-memory. `resolveStoreMode` guarantees production boot FAILS rather
than serving Maps, but the Postgres stores themselves are not built.
1. Persistent stores REMAINING: quality runs/rules and feature views are
still per-process in-memory (`resolveStoreMode` guarantees production
boot FAILS rather than serving Maps). CLOSED since the last revision:
dataset registry (`opt_datasets`) + immutable versions
(`opt_dataset_versions`, monotonic under transaction lock, idempotency
with payload-drift detection, cross-tenant ⇒ NOT_FOUND) and versioned
search index definitions (`opt_search_index_definitions`, sequential
schema versions, replay-idempotent, same-version-different-schema ⇒
CONFLICT; OpenSearch remains a rebuildable projection, never canonical
truth) are Postgres-canonical with real-Postgres tests.
2. Capability registration in CL0UD + API0 discovery for the new
capabilities (§3 "Capability är registrerad i CL0UD Capability Registry").
3. AUD0 outbox for evidence durability under AUD0 outage (§70).
4. Surface exposure: no `app/api/0pt/*` routes yet for
search/quality/datasets-versions/ER (services + contracts only).
capabilities (§3 "Capability är registrerad i CL0UD Capability
Registry") — PARTIALLY CLOSED: the collaboration/meeting domain was
canonicalized in CL0UD's registry (cl0ud #73/#74); 0pt's own
capability set still needs the same registration.
3. CLOSED: AUD0 outbox for evidence durability under AUD0 outage (§70) —
durable Postgres outbox + dispatcher (at-least-once, bounded batch,
backoff, dead-letter, SKIP LOCKED leasing, tenant binding, idempotency)
+ real-Postgres tests; dispatcher uses explicit `durable:false` so it
never re-enqueues its own delivery evidence.
4. CLOSED for datasets/versions/search/ER: `app/api/0pt/datasets*`,
`app/api/0pt/search/definitions` + tenant-safed query route, and
`app/api/0pt/er/{candidates,batch}` (+ internal ER route) are live;
clients cannot raise `rights_tag` via request body (fail-closed to
`internal`). Quality routes exist (`bridge-run/issues/rules/runs`).
5. 0pt-motor executors for the delegated jobs
(`0pt.quality.ge_run`, `0pt.features.*`, `0pt.er.batch_resolution`).
6. Observability: no metrics/traces emitted by the new services (§36).
@@ -97,7 +113,7 @@ verified, and by which command.
| **P062** CDC/batch/stream normalized to one envelope; restart resumes from a durable checkpoint without silent gaps | **MISSING**, and documented as such: `services/api/tests/test_known_gaps.py:10-19` skipped the CDC case because no CDC pipeline existed anywhere, and `workers/pipeline.py` had no persistence at all | **INTEGRATION_TESTED** | `src/types/ingestion.ts` (one envelope for all three modes); `src/ingestion/ingestion-runner.ts`; `tests/p0/ingestion-runner.test.ts` kills a run inside the sink and proves the resumed run covers all 10 records exactly once; same proof against real Postgres across fresh processes |
| **P063** Schema versioning + compatibility policy; a breaking schema is detected before consumer corruption | **MISSING.** `schema_version` existed as a free-form string on datasets/Iceberg writes; there was no comparison, no policy, no versioned source schema | **INTEGRATION_TESTED** | `src/schema/compatibility.ts`, `src/schema/schema-registry.ts`; `tests/p0/schema-evolution.test.ts` (12); the runner's schema gate runs before the first read — `tests/p0/ingestion-runner.test.ts` asserts `sinkCalls === 0` on a break |
| **P064** Quality rules behind the SIAX abstraction; failures block or quarantine per declared policy | **PARTIAL.** `src/quality/*` had rules, runs, results, issues, scores and a GE-compatible engine, but a failing rule only opened an ISSUE — the data flowed on regardless. `grep -ri quarantin src/` found one unrelated hit | **INTEGRATION_TESTED** | `src/quality/quality-policy.ts` composes the EXISTING `ExpectationEngine` (not a second engine) and adds the consequence; block / quarantine / warn each proved in `tests/p0/ingestion-runner.test.ts`; quarantine durability proved against real Postgres |
| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **DUPLICATE (confirmed).** `src/er/entity-resolution-service.ts` (canonical: hard rules, review band, Splink delegation) AND `services/api/workers/resolve.py` (simplified fuzzy threshold, auto-merge) — the latter still called by `workers/pipeline.py:110` | **NOT DONE — documented, not built.** The duplication is now stated at BOTH sites in code with the removal plan. The new ingestion plane deliberately performs no ER, so it adds no third engine. See "P065 — why it was not closed" below |
| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **CLOSED.** `services/api/workers/resolve.py` is now a thin adapter: Python no longer scores identity — it POSTs opaque records to `/api/0pt/internal/er/candidates` and applies ONLY explicit hard-rule `match` decisions (score/fuzzy can never merge records, even from a compromised bridge); canonical engine unreachable ⇒ records return UNMERGED, never fallback-scored. Company semantics `stable-identifiers.v3`: LEI global; org_nr hard identity only within the same registration_country; punctuation normalized; negative stable identity dominates every positive; fuzzy alone ⇒ human review, never auto-merge. Cluster-wide hard-negative constraints: A≈B ∧ B≈C never forces A≈C when canonical identity says A≠C | **INTEGRATION_TESTED** | `tests/p0/entity-resolution.test.ts` (v3 semantics incl. hard-negative transitivity), `tests/p0/entity-resolution-http.test.ts` (internal route fails closed: empty/wrong token ⇒ 401), `services/api/tests/test_resolve_determinism.py` (adapter contract: canonical edges only), shared request validation (`src/er/request-validation.ts`, records 2..5000), blocking pytest gate (`.gitea/workflows/python-engine.yml`) |
| **P066** Iceberg | out of scope — ST0RE is owned by another workflow | **NOT TOUCHED** | Contract read only (`src/services/iceberg-write-service.ts`, `src/clients/st0re-lakehouse-client.ts`) |
| **P067** Lineage: OpenLineage/AUD0 events; source → output lineage is queryable | **PARTIAL.** `src/lineage/lineage-emitter.ts` emitted OpenLineage to AUD0, but nothing in this repo could ANSWER a lineage question — no store, no query, no route | **INTEGRATION_TESTED** | `src/lineage/lineage-edge.ts`, `src/lineage/lineage-store.ts` (local queryable projection; AUD0 stays the system of record), `GET /api/0pt/lineage`; `tests/p0/ingestion-runner.test.ts` queries both directions and proves tenant scoping |
| **P068** Durable bounded backfill/replay; repeated backfill is idempotent | **MISSING**, documented as such: `services/api/tests/test_known_gaps.py:44-52` skipped it because "no delivery-store persistence layer exists to dedupe a backfill against" | **INTEGRATION_TESTED** | `opt_ingestion_ledger` + `IngestionRunner.backfill`; `tests/p0/ingestion-runner.test.ts` (repeat emits 0, live checkpoint untouched, a real content change still re-emits); same against real Postgres |
@@ -167,22 +183,24 @@ production, and P060's durability guarantee cannot be exercised there,
because production has no database configured and does not serve the routes.
Verified with `curl`; not inferred.
### P065 — why it was not closed
### P065 — closure record (2026-09-07)
Closing P065 means turning `services/api/workers/resolve.py` into an adapter
over the canonical `src/er/entity-resolution-service.ts`. That change cannot
be verified from this repo today: `services/api` is not built by this repo's
Dockerfile (it builds only the Next.js app), `api.0pt.app` answers 404, and
no CI workflow runs its pytest suite. A blind rewrite would ship untested
into a path nobody can exercise — the exact failure mode this programme
exists to stop.
The removal plan below was executed in full and is verified in CI:
The removal plan, for whoever picks it up:
1. Expose the canonical ER capability as `POST /api/0pt/er/candidates`
(service already exists; only a route + Zod schema are missing).
2. Add a Python CI job that actually runs `services/api/tests`.
3. Replace `resolve.py`'s body with an HTTP adapter over that route,
**fail-closed** when it is unreachable — never falling back to the local
scorer, which would preserve the duplicate under a different name.
4. Port `test_resolve_determinism.py` to assert the adapter's contract, and
delete the fuzzy scorer.
1. ✅ Canonical ER exposed as `POST /api/0pt/er/candidates` (public,
tenant-safed) AND `POST /api/0pt/internal/er/candidates` (service-token,
fails closed without `X-Internal-Token`).
2. ✅ Python CI job: `.gitea/workflows/python-engine.yml` runs
`services/api/tests` as a BLOCKING pytest gate.
3. ✅ `workers/resolve.py` rewritten as the HTTP adapter — fail-closed to
UNMERGED records when the bridge is unreachable; no fallback scorer
remains. Cluster-wide hard-negative constraints prevent transitive
matches from overriding canonical identity decisions.
4. ✅ `services/api/tests/test_resolve_determinism.py` asserts the adapter
contract: canonical match edges merge; review/non-match edges do not;
hard negatives are respected transitively.
Residual honesty: the PYTHON-side tests run in CI, but the full
adapter→Next.js route chain has not been exercised against a live deployed
edge in this branch (the route's own fail-closed behavior is unit-proven).
That keeps P065 at INTEGRATION_TESTED, not LIVE_VERIFIED.
app/api/0pt/datasets/[datasetId]/route.ts
+27
View File
@@ -0,0 +1,27 @@
import { NextRequest, NextResponse } from "next/server";
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
import { datasetService } from "@/src/services/datasets";
export async function GET(
request: NextRequest,
{ params }: { params: Promise<{ datasetId: string }> },
) {
try {
const { datasetId } = await params;
const tenantId = new URL(request.url).searchParams.get("tenant_id")?.trim();
if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
const dataset = await datasetService.getDataset(datasetId);
if (dataset.tenant_id !== tenantId) {
throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} not found`);
}
return NextResponse.json(dataset);
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/datasets/[datasetId]/versions/[version]/route.ts
+27
View File
@@ -0,0 +1,27 @@
import { NextRequest, NextResponse } from "next/server";
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
import { datasetVersionService } from "@/src/services/datasets";
export async function GET(
request: NextRequest,
{ params }: { params: Promise<{ datasetId: string; version: string }> },
) {
try {
const { datasetId, version: rawVersion } = await params;
const tenantId = new URL(request.url).searchParams.get("tenant_id")?.trim();
if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
const version = Number(rawVersion);
if (!Number.isSafeInteger(version) || version < 1) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "version must be a positive integer");
}
return NextResponse.json(await datasetVersionService.getVersion(datasetId, tenantId, version));
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/datasets/[datasetId]/versions/route.ts
+75
View File
@@ -0,0 +1,75 @@
import { NextRequest, NextResponse } from "next/server";
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
import { datasetVersionService } from "@/src/services/datasets";
function required(body: Record<string, unknown>, field: string): string {
const value = body[field];
if (typeof value !== "string" || !value.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", `${field} is required`);
}
return value.trim();
}
export async function POST(
request: NextRequest,
{ params }: { params: Promise<{ datasetId: string }> },
) {
try {
const { datasetId } = await params;
const raw = await request.json();
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "JSON object body is required");
}
const body = raw as Record<string, unknown>;
const tenantId = required(body, "tenant_id");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
if (!Array.isArray(body.artifact_refs) || !body.artifact_refs.every((value) => typeof value === "string")) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "artifact_refs must be a string array");
}
const recordCount = body.record_count === undefined ? undefined : Number(body.record_count);
const idempotencyKey = request.headers.get("idempotency-key")?.trim()
|| (typeof body.idempotency_key === "string" ? body.idempotency_key.trim() : "")
|| undefined;
const version = await datasetVersionService.createVersion({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: required(body, "project_id"),
artifact_refs: body.artifact_refs,
schema_version: required(body, "schema_version"),
...(recordCount === undefined ? {} : { record_count: recordCount }),
...(typeof body.description === "string" && body.description.trim()
? { description: body.description.trim() }
: {}),
created_by: request.headers.get("x-0pt-verified-sub") ?? "local-dev",
...(typeof body.correlation_id === "string" && body.correlation_id.trim()
? { correlation_id: body.correlation_id.trim() }
: {}),
...(idempotencyKey ? { idempotency_key: idempotencyKey } : {}),
});
return NextResponse.json(version, { status: 201 });
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
export async function GET(
request: NextRequest,
{ params }: { params: Promise<{ datasetId: string }> },
) {
try {
const { datasetId } = await params;
const tenantId = new URL(request.url).searchParams.get("tenant_id")?.trim();
if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
return NextResponse.json(await datasetVersionService.listVersions(datasetId, tenantId));
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/datasets/route.ts
+91
View File
@@ -0,0 +1,91 @@
import { NextRequest, NextResponse } from "next/server";
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
import { datasetService } from "@/src/services/datasets";
import type { DatasetStatus, DatasetType } from "@/src/types/dataset";
const DATASET_TYPES = new Set<DatasetType>(["raw", "structured", "enriched", "curated", "feature", "derived"]);
const DATASET_STATUSES = new Set<DatasetStatus>(["active", "archived", "deprecated"]);
function actorId(request: NextRequest): string {
return request.headers.get("x-0pt-verified-sub") ?? "local-dev";
}
function stringField(body: Record<string, unknown>, field: string): string {
const value = body[field];
if (typeof value !== "string" || !value.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", `${field} is required`);
}
return value.trim();
}
export async function POST(request: NextRequest) {
try {
const raw = await request.json();
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "JSON object body is required");
}
const body = raw as Record<string, unknown>;
const tenantId = stringField(body, "tenant_id");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
const datasetType = stringField(body, "dataset_type") as DatasetType;
if (!DATASET_TYPES.has(datasetType)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported dataset_type");
}
const status = body.status === undefined ? undefined : String(body.status) as DatasetStatus;
if (status !== undefined && !DATASET_STATUSES.has(status)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported status");
}
const dataset = await datasetService.registerDataset({
dataset_id: stringField(body, "dataset_id"),
tenant_id: tenantId,
project_id: stringField(body, "project_id"),
name: stringField(body, "name"),
dataset_type: datasetType,
...(status ? { status } : {}),
...(typeof body.description === "string" && body.description.trim()
? { description: body.description.trim() }
: {}),
...(typeof body.correlation_id === "string" && body.correlation_id.trim()
? { correlation_id: body.correlation_id.trim() }
: {}),
created_by: actorId(request),
});
return NextResponse.json(dataset, { status: 201 });
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
export async function GET(request: NextRequest) {
try {
const { searchParams } = new URL(request.url);
const tenantId = searchParams.get("tenant_id")?.trim();
if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
const typeRaw = searchParams.get("dataset_type") ?? undefined;
const statusRaw = searchParams.get("status") ?? undefined;
if (typeRaw && !DATASET_TYPES.has(typeRaw as DatasetType)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported dataset_type");
}
if (statusRaw && !DATASET_STATUSES.has(statusRaw as DatasetStatus)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported status");
}
return NextResponse.json(await datasetService.listDatasets({
tenant_id: tenantId,
project_id: searchParams.get("project_id") ?? undefined,
dataset_type: typeRaw as DatasetType | undefined,
status: statusRaw as DatasetStatus | undefined,
}));
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/er/batch/route.ts
+44
View File
@@ -0,0 +1,44 @@
import { NextRequest, NextResponse } from "next/server";
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
import { getBatchEntityResolutionService } from "@/src/services/entity-resolution";
function required(body: Record<string, unknown>, field: string): string {
const value = body[field];
if (typeof value !== "string" || !value.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `${field} is required`);
}
return value.trim();
}
export async function POST(request: NextRequest) {
try {
const raw = await request.json();
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "JSON object body is required");
}
const body = raw as Record<string, unknown>;
const tenantId = required(body, "tenant_id");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
const idempotencyKey = request.headers.get("idempotency-key")?.trim()
|| (typeof body.idempotency_key === "string" ? body.idempotency_key.trim() : "")
|| undefined;
const job = await getBatchEntityResolutionService().submitBatchResolution({
tenant_id: tenantId,
project_id: required(body, "project_id"),
records_artifact_ref: required(body, "records_artifact_ref"),
actor_id: request.headers.get("x-0pt-verified-sub") ?? "local-dev",
...(typeof body.correlation_id === "string" && body.correlation_id.trim()
? { correlation_id: body.correlation_id.trim() }
: {}),
...(idempotencyKey ? { idempotency_key: idempotencyKey } : {}),
});
return NextResponse.json(job, { status: 202 });
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/er/candidates/route.ts
+17
View File
@@ -0,0 +1,17 @@
import { NextRequest, NextResponse } from "next/server";
import { parseErCandidateRequest } from "@/src/er/request-validation";
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { entityResolutionService } from "@/src/services/entity-resolution";
export async function POST(request: NextRequest) {
try {
const input = parseErCandidateRequest(await request.json());
const guard = assertTenantMatchesVerified(request, input.tenant_id);
if (guard) return guard;
return NextResponse.json(entityResolutionService.generateCandidates(input));
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/internal/er/candidates/route.ts
+101
View File
@@ -0,0 +1,101 @@
import { NextRequest, NextResponse } from "next/server";
import { verifyInternalServiceToken } from "@/src/auth/internal-token";
import {
EntityResolutionService,
type ErCandidateResult,
type ErRecord,
} from "@/src/er/entity-resolution-service";
import { parseErCandidateRequest } from "@/src/er/request-validation";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
const er = new EntityResolutionService();
function rawStringField(record: ErRecord, field: string): string | undefined {
const value = record.fields[field];
if (value === null || value === undefined) return undefined;
const normalized = String(value).trim();
return normalized || undefined;
}
function stableIdentifier(record: ErRecord, field: string): string | undefined {
const value = rawStringField(record, field);
if (!value) return undefined;
const normalized = value.toUpperCase().replace(/[^A-Z0-9]/g, "");
return normalized || undefined;
}
function countryIdentifier(record: ErRecord): string | undefined {
const value = rawStringField(record, "registration_country") ?? rawStringField(record, "country");
if (!value) return undefined;
const normalized = value.toUpperCase().replace(/[^A-Z0-9]/g, "");
return normalized || undefined;
}
/**
* Company identity hard rule v3.
* - LEI is globally scoped.
* - National registration numbers are comparable only within the same country.
* - Any comparable stable-ID conflict dominates every positive match.
*/
er.registerHardRule({
rule_id: "0pt.company.stable-identifiers.v3",
decide(a, b) {
let sharedMatch = false;
const aLei = stableIdentifier(a, "lei");
const bLei = stableIdentifier(b, "lei");
if (aLei && bLei) {
if (aLei !== bLei) return "force_non_match";
sharedMatch = true;
}
const aCountry = countryIdentifier(a);
const bCountry = countryIdentifier(b);
const aOrg = stableIdentifier(a, "org_nr");
const bOrg = stableIdentifier(b, "org_nr");
if (aOrg && bOrg && aCountry && bCountry && aCountry === bCountry) {
if (aOrg !== bOrg) return "force_non_match";
sharedMatch = true;
}
return sharedMatch ? "force_match" : "abstain";
},
});
function assertInternal(request: NextRequest): NextResponse | undefined {
if (verifyInternalServiceToken(request.headers)) return undefined;
return NextResponse.json(
{ error: "UNAUTHENTICATED", message: "valid X-Internal-Token is required" },
{ status: 401 },
);
}
/**
* Company/Work boundary: fuzzy similarity is candidate evidence, never identity truth.
* Only product hard rules backed by stable identifiers may emit `match` here.
*/
export function enforceCompanyReviewBoundary(result: ErCandidateResult): ErCandidateResult {
const pairs = result.pairs.map((pair) =>
pair.decided_by === "score" && pair.status === "match"
? { ...pair, status: "pending_review" as const }
: pair,
);
return {
...result,
pairs,
review_queue: pairs.filter((pair) => pair.status === "pending_review"),
};
}
export async function POST(request: NextRequest) {
const denied = assertInternal(request);
if (denied) return denied;
try {
const result = er.generateCandidates(parseErCandidateRequest(await request.json()));
return NextResponse.json(enforceCompanyReviewBoundary(result));
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/internal/evidence-outbox/dispatch/route.ts
+69
View File
@@ -0,0 +1,69 @@
import { NextRequest, NextResponse } from "next/server";
import { verifyInternalServiceToken } from "@/src/auth/internal-token";
import { createAud0Client } from "@/src/clients/aud0-client";
import {
Aud0OutboxDispatcher,
createAud0OutboxRepositoryFromEnv,
} from "@/src/evidence/aud0-outbox";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
function assertInternal(request: NextRequest): NextResponse | undefined {
if (verifyInternalServiceToken(request.headers)) return undefined;
return NextResponse.json(
{ error: "UNAUTHENTICATED", message: "valid X-Internal-Token is required" },
{ status: 401 },
);
}
function repository() {
const repo = createAud0OutboxRepositoryFromEnv();
if (!repo) {
throw new SiaxError(
"DEPENDENCY_UNAVAILABLE",
"0pt.aud0_outbox",
"DATABASE_URL is required for durable evidence dispatch",
);
}
return repo;
}
export async function POST(request: NextRequest) {
const denied = assertInternal(request);
if (denied) return denied;
try {
const raw = await request.json().catch(() => ({})) as { limit?: unknown };
const limit = raw.limit === undefined ? 100 : Number(raw.limit);
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 500) {
return NextResponse.json(
{ error: "INVALID_REQUEST", message: "limit must be an integer between 1 and 500" },
{ status: 400 },
);
}
const repo = repository();
// CRITICAL: the dispatcher must bypass the durable wrapper. Using the
// default client here would enqueue the event again and could create an
// infinite outbox loop instead of delivering to AUD0.
const rawAud0 = createAud0Client(undefined, { durable: false });
const dispatcher = new Aud0OutboxDispatcher(repo, rawAud0);
const result = await dispatcher.sweep(limit);
const stats = await repo.stats();
return NextResponse.json({ result, stats });
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
export async function GET(request: NextRequest) {
const denied = assertInternal(request);
if (denied) return denied;
try {
return NextResponse.json(await repository().stats());
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/search/[indexName]/query/route.ts
+65
View File
@@ -0,0 +1,65 @@
import { NextRequest, NextResponse } from "next/server";
import { assertTenantMatchesVerified } from "@/src/http/tenant-guard";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
import { getSearchService } from "@/src/services/search";
function primitiveFilters(value: unknown): Record<string, string | number | boolean> | undefined {
if (value === undefined) return undefined;
if (!value || typeof value !== "object" || Array.isArray(value)) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "filters must be an object");
}
const result: Record<string, string | number | boolean> = {};
for (const [key, item] of Object.entries(value as Record<string, unknown>)) {
if (typeof item !== "string" && typeof item !== "number" && typeof item !== "boolean") {
throw new SiaxError("INVALID_REQUEST", "0pt.search", `filter ${key} must be string, number or boolean`);
}
result[key] = item;
}
return result;
}
export async function POST(
request: NextRequest,
{ params }: { params: Promise<{ indexName: string }> },
) {
try {
const { indexName } = await params;
const raw = await request.json();
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "JSON object body is required");
}
const body = raw as Record<string, unknown>;
const tenantId = typeof body.tenant_id === "string" ? body.tenant_id.trim() : "";
const purpose = typeof body.purpose === "string" ? body.purpose.trim() : "";
if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.search", "tenant_id is required");
if (!purpose) throw new SiaxError("INVALID_REQUEST", "0pt.search", "purpose is required");
const guard = assertTenantMatchesVerified(request, tenantId);
if (guard) return guard;
// Public/authenticated traffic is deliberately fixed to internal rights.
// A client-supplied higher rights_tag would be privilege escalation by
// request body; broader tags require a future CL0UD-derived entitlement.
if (body.rights_tag !== undefined && body.rights_tag !== "internal") {
throw new SiaxError(
"AUTHORIZATION_DENIED",
"0pt.search",
"rights_tag cannot be elevated by the caller",
);
}
const result = await getSearchService().query(indexName, {
tenant_id: tenantId,
purpose,
rights_tag: "internal",
...(typeof body.text === "string" ? { text: body.text } : {}),
...(body.from === undefined ? {} : { from: Number(body.from) }),
...(body.size === undefined ? {} : { size: Number(body.size) }),
...(body.filters === undefined ? {} : { filters: primitiveFilters(body.filters) }),
});
return NextResponse.json(result);
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
app/api/0pt/search/definitions/route.ts
+58
View File
@@ -0,0 +1,58 @@
import { NextRequest, NextResponse } from "next/server";
import { verifyInternalServiceToken } from "@/src/auth/internal-token";
import { siaxErrorResponse } from "@/src/http/siax-error-response";
import { SiaxError } from "@/src/platform/errors";
import { getSearchService } from "@/src/services/search";
import type { SearchFieldType } from "@/src/search/search-types";
const FIELD_TYPES = new Set<SearchFieldType>(["keyword", "text", "long", "double", "date", "boolean"]);
export async function POST(request: NextRequest) {
if (!verifyInternalServiceToken(request.headers)) {
return NextResponse.json({ error: "UNAUTHENTICATED", message: "valid X-Internal-Token is required" }, { status: 401 });
}
try {
const raw = await request.json();
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "JSON object body is required");
}
const body = raw as Record<string, unknown>;
const indexName = typeof body.index_name === "string" ? body.index_name.trim() : "";
const version = Number(body.version);
if (!body.fields || typeof body.fields !== "object" || Array.isArray(body.fields)) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "fields must be an object");
}
const fields: Record<string, SearchFieldType> = {};
for (const [name, rawType] of Object.entries(body.fields as Record<string, unknown>)) {
if (!name.trim() || typeof rawType !== "string" || !FIELD_TYPES.has(rawType as SearchFieldType)) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", `invalid field definition for ${name || "<empty>"}`);
}
fields[name.trim()] = rawType as SearchFieldType;
}
if (!Array.isArray(body.default_search_fields) || !body.default_search_fields.every((v) => typeof v === "string")) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "default_search_fields must be a string array");
}
const definition = await getSearchService().registerIndexDefinitionDurable(
{
index_name: indexName,
version,
fields,
default_search_fields: body.default_search_fields.map((v) => String(v).trim()),
},
request.headers.get("x-0pt-verified-sub") ?? "svc_0pt",
);
return NextResponse.json(definition, { status: 201 });
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
export async function GET() {
try {
return NextResponse.json(await getSearchService().listIndexDefinitionsDurable());
} catch (error: unknown) {
return siaxErrorResponse(error);
}
}
docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md
+197
View File
@@ -0,0 +1,197 @@
# 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan
Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN
Date: 2026-09-07
0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record.
## 1. Ecosystem ownership
### Work / j0b
0pt may supply:
- company/employer/end-client enrichment;
- registry/company facts;
- generic source acquisition;
- buyer/supplier enrichment for procurement;
- entity-resolution candidates/evidence;
- public technology/company signals for account intelligence.
j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions.
### PRESS0
0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions.
### FUND0
0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY.
### FIT0 / CRM
0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth.
## 2. Provenance-first invariant
Every externally derived fact or observation must preserve enough lineage to answer:
- source/provider;
- external identifier or URL where appropriate;
- observed/fetched time;
- parser/transform version;
- freshness/confidence where meaningful;
- rights/reuse/retention classification;
- payload/artifact hash or ST0RE ref where raw retention is required.
No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule.
## 3. Lossless acquisition contract
When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified.
Target acquisition-receipt fields:
- source/provider;
- external ID/URL;
- fetched/observed times;
- payload/artifact hash/ref;
- parser/version;
- rights/reuse/retention classification;
- data classification;
- extracted observations/provenance;
- correlation/tenant/project scope.
## 4. Canonical entity-resolution architecture
There must be one decision architecture, not competing production scorers.
### 4.1 Synchronous candidate path
The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides:
- deterministic blocking;
- explainable similarity evidence;
- product hard-rule hooks;
- review-band candidates;
- stable-ID hard match/non-match rules at the product boundary.
For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match.
### 4.2 Python ingestion path
The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering.
### 4.3 Heavy/statistical ER
Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract.
## 5. Strong company identity keys
Prefer strong keys where available:
- LEI (global scope);
- country + official registration number;
- VAT/official registry identifiers with their correct jurisdictional scope;
- other verified registry IDs.
National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier.
Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges.
## 6. Human review boundary
Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced.
## 7. Public procurement enrichment
Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence.
## 8. Source-policy metadata
Products may pass/receive policy metadata for:
- access method;
- reuse/redistribution;
- retention;
- polling/rate limits;
- terms/license review date;
- purpose/data classification.
0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose.
## 9. PRESS0 rights boundary
Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision.
## 10. Company verification boundary
0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row.
## 11. Market/account signals
Useful dated observations include:
- hiring activity;
- public procurement notices/awards;
- public technology/platform signals;
- company growth/change events;
- leadership/public contact changes where lawful;
- relevant product/market announcements.
Signals are provenance-bearing observations with observation time and freshness, not permanent assertions.
## 12. API contract
Generic typed capabilities should cover:
- company/entity enrichment;
- source acquisition receipt;
- structured observations with provenance;
- entity-resolution candidates;
- batch ER job submission;
- provider/source health/freshness;
- durable dataset/version/search surfaces where implemented.
Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling.
## 13. Security/privacy
Required controls include:
- tenant/project/purpose scoping;
- PII minimisation;
- controlled egress/allowed-host policy;
- SSRF protection for user-supplied URLs;
- canonical secret storage;
- no raw sensitive payloads in logs;
- retention/deletion support;
- provider/license/data-rights documentation;
- fail-closed identity merge behavior;
- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required.
## 14. Production gate
Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED.
Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence.
## 15. Implementation order
1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners.
2. Keep the synchronous TS ER engine as the canonical candidate/decision contract.
3. Keep Python ingestion as a fail-closed adapter, never a second scorer.
4. Complete durable acquisition/provenance/dataset-version/search contracts.
5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review.
6. Wire j0b company/end-client/buyer enrichment through typed APIs.
7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth.
8. Add PRESS0/FUND0 consumers only when their repos/contracts are real.
9. Load/performance/backup/restore/live-verify every mission-critical path.
## 16. Definition of Done per data capability
- source/provider documented;
- legal/license/data-use classification recorded;
- provenance/freshness/confidence available where meaningful;
- deterministic/idempotent normalization where expected;
- entity resolution tested against labelled truth data;
- fuzzy evidence cannot silently become identity truth;
- tenant/purpose/security negative tests pass;
- provider outage/schema drift handled fail closed or explicitly degraded;
- raw artifact retention/deletion contract works where required;
- exact production endpoint/dependency is live verified;
- product consumes API/contract, not another product's DB;
- status documentation reflects observed reality.
Target ecosystem uses in this document do not claim they are already wired or live.
middleware.ts
+14 -56
View File
@@ -1,53 +1,17 @@
/**
* `middleware.ts` -- Next.js request middleware = 0pt's framework-level auth
* seam (the Next analog of a Fastify `onRequest` hook).
* Next.js auth seam for the complete /api/0pt capability surface.
*
* This runs the Zitadel verification (src/auth/context.ts ->
* src/auth/zitadel.ts) in front of every 0pt capability route (/api/0pt/*).
* It does NOT touch the v0-generated UI routes or the unrelated
* payment/order APIs (see the matcher below and the note in the PR that
* introduced the fail-closed default).
*
* AUTH_ENFORCE unset, or anything except "false" (DEFAULT, enforced):
* a valid verified token is required; 401 on missing/invalid, 403 when
* X-Tenant-Id disagrees with the token tenant. On success the verified
* principal is forwarded to the handler as `x-0pt-verified-sub` /
* `x-0pt-verified-tenant` request headers.
* AUTH_ENFORCE == "false" (explicit local-dev escape hatch): never rejects.
* REFUSED in production -- `assertAuthConfigSafe()` below throws at module
* load, so a production instance cannot boot with auth switched off.
*
* Health/metrics are exempt (they are also outside the matcher below, but the
* exemption is kept explicit for robustness).
*
* FIX (2026-08-31): matcher now covers BOTH /api/0pt (exact) and /api/0pt/:path*
* to close the gap where a bare GET /api/0pt (no trailing path) was routed around
* the middleware on Vercel's Edge runtime.
*
* FIX (P014): one route under this matcher, /api/0pt/quality/bridge-run, is
* not called by a customer's Zitadel-authenticated session at all -- it is
* called BY the Python engine (services/api/workers/quality_bridge.py) using
* the pre-existing X-Internal-Token shared-secret convention, never a Bearer
* token. Before this fix, that meant middleware always rejected the engine's
* real call with 401 (see src/auth/internal-token.ts's doc comment for the
* full story) -- the quality bridge was permanently unreachable in any
* environment enforcing auth. INTERNAL_ONLY_ROUTES lets a request to that
* ONE path through on a verified X-Internal-Token instead of a Bearer token;
* every other /api/0pt/* route is untouched and still requires the normal
* Zitadel flow below. A request to that route WITHOUT a valid internal token
* still falls through to the ordinary Bearer check (e.g. an operator testing
* it manually with their own token), it is not a blanket bypass.
* Customer calls use verified Zitadel bearer identity and receive immutable
* x-0pt-verified-* headers. Explicit machine routes may instead use the
* shared X-Internal-Token convention; mutation handlers verify it again for
* defence in depth.
*/
import { NextResponse, type NextRequest } from 'next/server';
import { assertAuthConfigSafe, resolveAuthContext } from '@/src/auth/context';
import { verifyInternalServiceToken } from '@/src/auth/internal-token';
// STARTUP GATE: fail to boot rather than serve /api/0pt/* openly in production.
assertAuthConfigSafe();
// Only guard the 0pt capability surface. The matcher keeps this middleware off
// the UI, auth, and payment routes entirely.
export const config = {
matcher: ['/api/0pt', '/api/0pt/:path*'],
};
@@ -58,26 +22,21 @@ const EXEMPT_PATTERNS: RegExp[] = [
/^\/api\/admin\/metrics(\/|$)/,
];
/** The one 0pt route reached by a service (the Python engine), not a user session. */
const INTERNAL_ONLY_ROUTES: RegExp[] = [/^\/api\/0pt\/quality\/bridge-run$/];
/** Routes that may be reached with a service token instead of end-user bearer auth. */
const INTERNAL_SERVICE_ROUTES: RegExp[] = [
/^\/api\/0pt\/quality\/bridge-run$/,
/^\/api\/0pt\/internal\/evidence-outbox\/dispatch$/,
/^\/api\/0pt\/internal\/er\/candidates$/,
/^\/api\/0pt\/search\/definitions$/,
];
// Module-level diagnostics counter — first 5 auth requests in a process lifetime
// log their outcome so the production auth gap (S1) can be traced to root cause.
let _authDiagCount = 0;
export async function middleware(req: NextRequest): Promise<NextResponse> {
const path = req.nextUrl.pathname;
if (EXEMPT_PATTERNS.some((re) => re.test(path))) {
return NextResponse.next();
}
if (EXEMPT_PATTERNS.some((re) => re.test(path))) return NextResponse.next();
if (INTERNAL_ONLY_ROUTES.some((re) => re.test(path)) && verifyInternalServiceToken(req.headers)) {
// Machine-to-machine call, verified via the shared X-Internal-Token
// secret rather than a user's Bearer token. Deliberately does NOT set
// x-0pt-verified-* headers -- there is no per-user principal here, and
// the route requires+validates tenant_id from its own request body
// (qualityBridgeRunSchema), which is the actual tenant the engine is
// running the order for.
if (INTERNAL_SERVICE_ROUTES.some((re) => re.test(path)) && verifyInternalServiceToken(req.headers)) {
return NextResponse.next();
}
@@ -92,7 +51,6 @@ export async function middleware(req: NextRequest): Promise<NextResponse> {
}
const decision = await resolveAuthContext(req.headers);
if (!decision.ok) {
return NextResponse.json(
{ error: decision.code, message: decision.message, ...(decision.details ? { details: decision.details } : {}) },
services/api/tests/test_resolve_determinism.py
+130 -164
View File
@@ -1,200 +1,166 @@
"""
Fault-injection tests for workers/resolve.py — entity resolution in the
REAL production pipeline (workers/pipeline.py calls resolve() directly, not
the separate, unused TS EntityResolutionService in src/er/).
"""Regression tests for the Python adapter to canonical TypeScript ER.
BEFORE_STATE.md §4: "dedup-determinism [...] helt frånvarande — särskilt
anmärkningsvärt då resolve.py [...] är ordningskänslig by construction och
helt otestad för determinism." This file closes that gap against the real
function, not a mock.
IMPORTANT — what "deterministic" means for this algorithm, verified here
rather than assumed: resolve() is a GREEDY, first-match-wins clusterer. The
surviving record's own scalar fields (e.g. `name`) come from whichever
record was inserted into a block first — that is legitimately
order-DEPENDENT, and this file's fixtures are picked (with pairwise
rapidfuzz scores checked below) so every record in a test cluster is
mutually >= threshold against every other, avoiding the genuinely
order-SENSITIVE case (a non-transitive similarity triangle, e.g.
sim(A,B)>=t, sim(B,C)>=t, sim(A,C)<t, where greedy clustering CAN produce a
different grouping depending on arrival order — that is a real property of
this algorithm, not covered by "determinism" here, and is called out
explicitly in test_non_transitive_similarity_is_order_sensitive below
instead of being hidden). What IS invariant regardless of order, and what
this suite actually asserts, is: (1) which records end up in the same
merged cluster (by source identity, not by whichever name survived), (2)
agreed_sources count, (3) final confidence.
Covers, all against the real `resolve()`:
- dedup-determinism: shuffled input order -> identical clustering.
- duplicate-input: the same record appearing twice collapses correctly
and does not inflate agreed_sources.
- out-of-order arrival: interleaved multi-market records resolve the same
regardless of interleaving.
- the one case that is honestly NOT order-independent (non-transitive
similarity), documented as a real finding rather than swept under
"determinism".
The removed Python scorer is deliberately not tested because it no longer
exists. These tests pin the load-bearing adapter semantics instead:
canonical match edges merge; review/non-match edges do not; hard negative
identity evidence constrains the whole cluster; bridge failure leaves records
unmerged.
"""
from __future__ import annotations
import copy
import itertools
import random
import httpx
import pytest
import respx
from workers.resolve import resolve
from workers.resolve import _apply_matches, resolve
def _prov(field_name: str, source_id: str, confidence: float, level: int = 3) -> dict:
def _prov(field_name: str, source_id: str, confidence: float) -> dict:
return {
"field_name": field_name, "source_id": source_id, "confidence": confidence, "level": level,
"freshness": "2026-09-01", "retrieved_at": "2026-09-01T12:00:00+00:00",
"transform": "raw_fetch", "connector_version": f"{source_id}-connector@1",
"producer": "0pt-engine", "quality": None, "entity_resolution": None, "artifact_ref": None,
"field_name": field_name,
"source_id": source_id,
"confidence": confidence,
"entity_resolution": None,
}
def _rec(market: str, name: str, source: str, confidence: float) -> dict:
def _rec(name: str, source: str, confidence: float = 0.9) -> dict:
return {
"market": market, "name": name, "status": "active", "lei": None, "org_nr": None,
"source": source, "confidence": confidence,
"market": "SE",
"name": name,
"status": "active",
"lei": None,
"org_nr": None,
"source": source,
"confidence": confidence,
"provenance": [_prov("name", source, confidence)],
}
# Three independent sources reporting near-identical spellings of the SAME
# company. Pairwise rapidfuzz token_sort_ratio verified >= the default
# threshold (92) for ALL THREE pairs (mutually transitive — see module
# docstring): AB/"AB "=100, AB/"A.B."=95.45, "AB "/"A.B."=95.45.
VOLVO_GLEIF = _rec("NO", "Volvo Personvagnar AB", "gleif", 0.97)
VOLVO_BRREG = _rec("NO", "Volvo Personvagnar AB ", "brreg", 0.97)
VOLVO_CVR = _rec("NO", "Volvo Personvagnar A.B.", "cvr", 0.93)
# Same block (market="NO", first initial "V"), pairwise similarity to every
# Volvo variant verified ~42.9 (well below threshold) — must never merge.
VILDANDEN = _rec("NO", "Vildanden Regnskap AS", "brreg", 0.9)
def _pair(a: str, b: str, status: str = "match", decided_by: str = "score", score: float = 0.97) -> dict:
return {
"candidate_id": f"c_{a}_{b}",
"record_a": a,
"record_b": b,
"score": score,
"status": status,
"decided_by": decided_by,
}
def _clusters(records: list[dict]) -> list[frozenset]:
"""Order-independent identity: WHICH sources ended up merged together,
ignoring which record's own `name`/`status`/etc. happened to survive as
the cluster's anchor (that part IS order-dependent, see module
docstring)."""
return sorted(
(frozenset([r["source"], *r.get("merged_from", [])]) for r in records),
key=lambda s: sorted(s),
)
def test_apply_matches_is_deterministic_and_uses_canonical_edges_only():
records = [_rec("Acme AB", "source-a", 0.91), _rec("ACME Aktiebolag", "source-b", 0.96)]
assert len(_apply_matches(copy.deepcopy(records), [])) == 2
class TestDedupDeterminism:
def test_shuffled_orderings_produce_identical_clustering(self):
base = [VOLVO_GLEIF, VOLVO_BRREG, VOLVO_CVR, VILDANDEN]
baseline = resolve([copy.deepcopy(r) for r in base])
assert len(baseline) == 2 # Volvo (merged x3) + Vildanden
baseline_clusters = _clusters(baseline)
@pytest.mark.asyncio
@respx.mock
async def test_canonical_match_merges_and_stamps_provenance(monkeypatch):
monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test")
monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token")
records = [_rec("Acme AB", "source-a", 0.91), _rec("Acme AB", "source-b", 0.96)]
rng = random.Random(1337)
for _ in range(30):
shuffled = [copy.deepcopy(r) for r in base]
rng.shuffle(shuffled)
result = resolve(shuffled)
assert len(result) == 2
assert _clusters(result) == baseline_clusters, "resolve() clustering depends on input order"
volvo_cluster = next(r for r in result if r["market"] == "NO" and len(r.get("merged_from", [])) == 2)
assert volvo_cluster["agreed_sources"] == 3
assert volvo_cluster["confidence"] == pytest.approx(min(0.99, 0.97 + 0.1 + 0.1))
route = respx.post("https://edge.example.test/api/0pt/internal/er/candidates")
def test_all_permutations_of_a_mutually_similar_set_agree(self):
# Exhaustive, not sampled — small enough (3! = 6) to check every ordering.
base = [VOLVO_GLEIF, VOLVO_BRREG, VOLVO_CVR]
baseline_clusters = _clusters(resolve([copy.deepcopy(r) for r in base]))
for perm in itertools.permutations(base):
result = resolve([copy.deepcopy(r) for r in perm])
assert _clusters(result) == baseline_clusters
def responder(request: httpx.Request) -> httpx.Response:
body = __import__("json").loads(request.content)
assert request.headers["X-Internal-Token"] == "test-internal-token"
assert body["tenant_id"] == "tenant-a"
assert body["blocking_fields"] == ["market", "block_initial"]
a, b = body["records"][0]["record_id"], body["records"][1]["record_id"]
# Endast hard-rule-beslut (decided_by != score/None) far skapa en
# canonical identity-lank - consumenten ignorerar medvetet score/"match".
return httpx.Response(
200,
json={"pairs": [_pair(a, b, "match", "0pt.company.stable-identifiers.v3", 0.99)], "review_queue": []},
)
route.side_effect = responder
result = await resolve(records, tenant_id="tenant-a", project_id="project-a")
assert len(result) == 1
assert result[0]["agreed_sources"] == 2
assert result[0]["confidence"] == pytest.approx(0.99)
assert set(result[0]["merged_from"]) <= {"source-a", "source-b"}
for provenance in result[0]["provenance"]:
er = provenance["entity_resolution"]
assert er["method"] == "canonical_ts_entity_resolution"
assert set(er["merged_sources"]) == {"source-a", "source-b"}
class TestDuplicateInput:
def test_the_same_record_twice_collapses_without_double_counting_agreement(self):
one = copy.deepcopy(VOLVO_GLEIF)
dup = copy.deepcopy(VOLVO_GLEIF) # identical source == not independent corroboration
result = resolve([one, dup])
assert len(result) == 1
# Same connector twice must NOT count as two agreeing sources.
assert result[0]["agreed_sources"] == 1
assert result[0]["confidence"] == VOLVO_GLEIF["confidence"]
@pytest.mark.asyncio
@respx.mock
async def test_pending_review_never_auto_merges(monkeypatch):
monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test")
monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token")
records = [_rec("Acme AB", "source-a"), _rec("Acme Group AB", "source-b")]
def test_duplicate_plus_one_genuinely_new_source_only_credits_the_new_one(self):
result = resolve([copy.deepcopy(VOLVO_GLEIF), copy.deepcopy(VOLVO_GLEIF), copy.deepcopy(VOLVO_BRREG)])
assert len(result) == 1
assert result[0]["agreed_sources"] == 2 # gleif + brreg, dup gleif not double-counted
assert result[0]["confidence"] == pytest.approx(min(0.99, VOLVO_GLEIF["confidence"] + 0.1))
def responder(request: httpx.Request) -> httpx.Response:
body = __import__("json").loads(request.content)
a, b = body["records"][0]["record_id"], body["records"][1]["record_id"]
pair = _pair(a, b, "pending_review")
return httpx.Response(200, json={"pairs": [pair], "review_queue": [pair]})
respx.post("https://edge.example.test/api/0pt/internal/er/candidates").side_effect = responder
result = await resolve(records, tenant_id="tenant-a", project_id="project-a")
assert len(result) == 2
assert all(not r.get("merged_from") for r in result)
class TestOutOfOrderArrival:
def test_interleaved_multi_market_arrival_resolves_the_same_as_sequential(self):
no_recs = [VOLVO_GLEIF, VOLVO_BRREG]
gb_recs = [
_rec("GB", "Tesco PLC", "companies_house", 0.95),
_rec("GB", "Tesco PLC ", "gleif", 0.97), # verified 100.0 vs "Tesco PLC"
]
sequential = resolve([copy.deepcopy(r) for r in [*no_recs, *gb_recs]])
sequential_clusters = _clusters(sequential)
@pytest.mark.asyncio
@respx.mock
async def test_hard_non_match_blocks_transitive_cluster_merge(monkeypatch):
"""A-B and B-C matches may never imply A=C when canonical hard evidence says A!=C."""
monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test")
monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token")
records = [
_rec("Acme AB", "source-a", 0.95),
_rec("Acme AB", "source-b", 0.94),
_rec("Acme AB", "source-c", 0.93),
]
interleaved = [
copy.deepcopy(gb_recs[0]),
copy.deepcopy(no_recs[0]),
copy.deepcopy(gb_recs[1]),
copy.deepcopy(no_recs[1]),
]
result = resolve(interleaved)
assert _clusters(result) == sequential_clusters
assert len(result) == 2 # one merged Volvo, one merged Tesco
def responder(request: httpx.Request) -> httpx.Response:
body = __import__("json").loads(request.content)
a, b, c = [item["record_id"] for item in body["records"]]
return httpx.Response(
200,
json={
"pairs": [
_pair(a, b, "match", "0pt.company.stable-identifiers.v3", 0.98),
_pair(b, c, "match", "0pt.company.stable-identifiers.v3", 0.97),
_pair(a, c, "non_match", "0pt.company.stable-identifiers.v2", 0.0),
],
"review_queue": [],
},
)
respx.post("https://edge.example.test/api/0pt/internal/er/candidates").side_effect = responder
result = await resolve(records, tenant_id="tenant-a", project_id="project-a")
assert len(result) == 2
assert max(r.get("agreed_sources", 1) for r in result) == 2
assert not any(set(r.get("merged_from", [])) >= {"source-b", "source-c"} for r in result)
class TestNonTransitiveSimilarityIsOrderSensitive:
"""Documents a real, verified property of the greedy first-match-wins
algorithm rather than papering over it: when similarity is NOT
transitive across three candidates in the same block, which two end up
merged CAN depend on arrival order. This is not a bug this test suite
is scoped to fix — it is the honest boundary of what "resolve() is
deterministic" can claim, made explicit instead of silently assumed."""
@pytest.mark.asyncio
@respx.mock
async def test_bridge_outage_fails_closed_to_unmerged_records(monkeypatch):
monkeypatch.setenv("OPT_ER_BRIDGE_URL", "https://edge.example.test")
monkeypatch.setenv("INTERNAL_TOKEN", "test-internal-token")
records = [_rec("Acme AB", "source-a"), _rec("Acme AB", "source-b")]
respx.post("https://edge.example.test/api/0pt/internal/er/candidates").mock(side_effect=httpx.ConnectError("offline"))
def test_a_non_transitive_triangle_can_cluster_differently_by_order(self):
# Verified pairwise: A/B = 80.0, A/C = 97.67, B/C = 78.43 (threshold 92).
# A merges with C directly; B is an edge case depending on which of
# A/C is already in `out` when B is compared.
a = _rec("NO", "Volvo Personvagnar AB", "gleif", 0.97)
b = _rec("NO", "Volvo Personvagnar Aktiebolag", "brreg", 0.95)
c = _rec("NO", "Volvo Personvagnar A/B", "cvr", 0.93)
order_1 = resolve([copy.deepcopy(a), copy.deepcopy(b), copy.deepcopy(c)])
order_2 = resolve([copy.deepcopy(b), copy.deepcopy(a), copy.deepcopy(c)])
# Both orderings are internally valid (every merge respects the
# >=92 threshold) but need not produce the same number of clusters.
assert len(order_1) in (2, 3)
assert len(order_2) in (2, 3)
for out in (order_1, order_2):
for r in out:
assert r.get("agreed_sources", 1) <= 2 # never over-merges all three
result = await resolve(records, tenant_id="tenant-a", project_id="project-a")
assert len(result) == 2
assert all(not r.get("merged_from") for r in result)
class TestEntityResolutionProvenance:
"""P010 provenance addition: resolve() now stamps `entity_resolution` on
every provenance row of a merged record, and leaves it None on records
that were never merged (never fabricated)."""
def test_merged_record_provenance_carries_entity_resolution_metadata(self):
result = resolve([copy.deepcopy(VOLVO_GLEIF), copy.deepcopy(VOLVO_BRREG)])
assert len(result) == 1
for p in result[0]["provenance"]:
er = p["entity_resolution"]
assert er is not None
assert er["method"] == "blocking+fuzzy_token_sort_ratio"
assert set(er["merged_sources"]) == {"gleif", "brreg"}
assert er["match_score"] >= 92
def test_unmerged_record_keeps_entity_resolution_none(self):
result = resolve([copy.deepcopy(VILDANDEN)])
assert len(result) == 1
assert result[0]["provenance"][0]["entity_resolution"] is None
@pytest.mark.asyncio
async def test_missing_internal_configuration_makes_zero_network_calls(monkeypatch):
monkeypatch.delenv("OPT_ER_BRIDGE_URL", raising=False)
monkeypatch.delenv("EDGE_URL", raising=False)
monkeypatch.delenv("INTERNAL_TOKEN", raising=False)
records = [_rec("Acme AB", "source-a"), _rec("Acme AB", "source-b")]
result = await resolve(records, tenant_id="tenant-a", project_id="project-a")
assert len(result) == 2
services/api/workers/pipeline.py
+47 -74
View File
@@ -1,23 +1,9 @@
"""
Production pipeline (docs/06): sourcing -> extraction -> structuring ->
enrichment/fusion -> QA. Compliant connectors only; every field gets provenance;
rows below the order's confidence target are dropped (deliver fewer, never fabricate).
"""Production pipeline: sourcing -> extraction -> canonical ER -> QA -> quality.
P010 additions (both flagged in BEFORE_STATE.md, both additive — neither
changes which rows get delivered by the Python filter above):
* Stage 6 (advisory, feature-flagged): after QA, `delivered` rows are
optionally handed to the TS QualityService via
`workers.quality_bridge.run_quality_bridge` — see that module's
docstring for the full contract, the fail-open guarantee, and why
per-row gating isn't implemented yet. Disabled unless
`OPT_QUALITY_BRIDGE_ENABLED=true`.
* Every delivered record's provenance rows are stamped with `producer`,
`connector_version` (both already set at fetch time in
sources/connectors.py), and an `artifact_ref` identifying this
delivery (`0pt:pipeline_delivery:<order_id>`) — plus, when the quality
bridge ran successfully, a `quality` block. BEFORE_STATE.md §3 found
these fields absent from the actual dataflow entirely.
Identity decisions are owned by src/er/entity-resolution-service.ts. The
Python worker only applies canonical `match` decisions returned by its
internal bridge. If tenant context or the canonical bridge is unavailable,
records remain unmerged: false identity merges are never used as fallback.
"""
from __future__ import annotations
@@ -26,7 +12,6 @@ from typing import Any
from workers.resolve import resolve
from workers.quality_bridge import run_quality_bridge
from sources.connectors import (
CompanyResult,
SourceBlockedError,
@@ -39,53 +24,38 @@ from sources.connectors import (
from workers.adaptive_router import default_router, plan_for_market
logger = logging.getLogger(__name__)
PIPELINE_VERSION = "0pt-pipeline@2"
PIPELINE_VERSION = "0pt-pipeline@1"
# Minimal built-in seed so a job is demonstrable without customer input.
DEMO_SEEDS: dict[str, list[str]] = {
"NO": ["923609016"], # brreg orgnr (example)
"GB": ["00000006"], # companies house number (example)
"NO": ["923609016"],
"GB": ["00000006"],
}
async def _fetch(source_id: str, key: str) -> CompanyResult | None:
if source_id == "gleif":
return await gleif_by_lei(key)
if source_id == "brreg":
return await brreg_by_orgnr(key)
if source_id == "companies_house":
return await companies_house_by_number(key)
if source_id == "prh":
return await prh_by_businessid(key)
if source_id == "cvr":
return await cvr_by_number(key)
if source_id == "gleif": return await gleif_by_lei(key)
if source_id == "brreg": return await brreg_by_orgnr(key)
if source_id == "companies_house": return await companies_house_by_number(key)
if source_id == "prh": return await prh_by_businessid(key)
if source_id == "cvr": return await cvr_by_number(key)
return None
async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]:
markets: list[str] = config.get("markets", []) or ["NO"]
threshold = float(config.get("confidence", 92)) / 100.0
# Use caller-supplied seeds when present (an explicit empty {} is honoured so a
# real job is never silently backfilled with DEMO_SEEDS), but treat an explicit
# null the same as "missing" -- `config.get("seeds") is not None` avoids a 500
# (None.get) on client-controlled `{"seeds": null}`, which the old `... or ...`
# idiom tolerated and the `"seeds" in config` form regressed.
seeds: dict[str, list[str]] = config["seeds"] if config.get("seeds") is not None else DEMO_SEEDS
tenant_id = config.get("tenant_id")
project_id = config.get("project_id", "p_default")
correlation_id = config.get("correlation_id")
records: list[dict[str, Any]] = []
per_market: dict[str, int] = {}
# 1) sourcing + 2) extraction
for market in markets:
keys = seeds.get(market, [])
got = 0
for key in keys:
# Recomputed per key (not hoisted above this loop): a source
# blocked while fetching an earlier key in this same market must
# be excluded from the plan for every key still to come in this
# run, not just retried forever with no memory (P014 -- see
# adaptive_router.py's module docstring).
plan = plan_for_market(market)
for source_id in plan:
try:
@@ -97,57 +67,55 @@ async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]:
res = None
if res and res.name:
records.append({
"market": market, "name": res.name, "status": res.status,
"lei": res.lei, "org_nr": res.org_nr, "source": res.source,
"market": market,
"name": res.name,
"status": res.status,
"lei": res.lei,
"org_nr": res.org_nr,
"source": res.source,
"provenance": [p.__dict__ for p in res.provenance],
"confidence": max((p.confidence for p in res.provenance), default=0.0),
})
got += 1
break # first compliant source that answers wins
break
per_market[market] = got
# 3) structuring — entity resolution (blocking + fuzzy match + corroboration)
deduped = resolve(records)
# Canonical ER is tenant-bound. Missing tenant context must never be
# papered over with a synthetic tenant or the removed Python scorer.
if tenant_id:
deduped = await resolve(
records,
tenant_id=tenant_id,
project_id=project_id,
correlation_id=correlation_id,
)
er_mode = "canonical_ts"
else:
deduped = [dict(r) for r in records]
er_mode = "fail_closed_unmerged_missing_tenant"
# 4) enrichment/fusion already folded into confidence above.
# 5) QA — drop rows below the order's confidence threshold. THIS is the
# filter that actually governs delivery; nothing below is allowed to
# change its outcome.
delivered = [r for r in deduped if r["confidence"] >= threshold]
dropped = len(deduped) - len(delivered)
avg_conf = round(sum(r["confidence"] for r in delivered) / len(delivered), 3) if delivered else 0.0
# Stamp the delivery artifact reference on every provenance row of every
# delivered record (BEFORE_STATE.md §3: `artifact_ref` did not exist
# anywhere). This identifies WHICH delivery a field's value shipped in;
# it does not assert the row was ever written to ST0RE (not wired yet).
artifact_ref = f"0pt:pipeline_delivery:{order_id}"
for r in delivered:
for p in r.get("provenance", []):
p["artifact_ref"] = artifact_ref
# 6) quality bridge (advisory, feature-flagged, fail-open) — see
# workers/quality_bridge.py's docstring for the full contract. Runs
# AFTER the QA gate above and can never change `delivered`.
tenant_id = config.get("tenant_id")
quality_bridge_metrics: dict[str, Any]
if not tenant_id:
# The engine has no tenant context yet for orders created through
# this API (orchestrator/main.py never threads tenant_id into
# `config` — a pre-existing, separately-tracked gap). Rather than
# fabricate a tenant to make the bridge call "work", skip it
# honestly and say why.
quality_bridge_metrics = {
"status": "not_configured",
"detail": "config.tenant_id missing — engine has no tenant context for this order",
"detail": "config.tenant_id missing — quality bridge not called",
}
else:
bridge_result = await run_quality_bridge(
tenant_id=tenant_id,
project_id=config.get("project_id", "p_default"),
project_id=project_id,
dataset_id=f"0pt-order-{order_id}",
rows=delivered,
correlation_id=config.get("correlation_id"),
correlation_id=correlation_id,
)
quality_bridge_metrics = bridge_result.as_metrics()
if bridge_result.status == "ok":
@@ -167,11 +135,16 @@ async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]:
)
return {
"stage": "qa", "pct": 100, "state": "done",
"stage": "qa",
"pct": 100,
"state": "done",
"metrics": {
"rows": len(delivered), "dropped_below_threshold": dropped,
"per_market": per_market, "avg_confidence": avg_conf,
"rows": len(delivered),
"dropped_below_threshold": dropped,
"per_market": per_market,
"avg_confidence": avg_conf,
"pipeline_version": PIPELINE_VERSION,
"entity_resolution": er_mode,
"quality_bridge": quality_bridge_metrics,
},
"records": delivered,
services/api/workers/resolve.py
+220 -102
View File
@@ -1,117 +1,235 @@
"""
Entity resolution (docs/06): block, score, and link records across sources so the
same company from GLEIF + Brreg + CVR collapses to one record with corroborated
confidence. rapidfuzz only; deterministic; no external calls.
"""Thin adapter from the Python pipeline to 0pt's canonical TS ER engine.
=======================================================================
P065 — THIS IS THE SECOND ER ENGINE. IT IS NOT THE CANONICAL ONE.
=======================================================================
0pt has TWO entity-resolution implementations, and this is the simplified
one:
Python no longer scores identity. It sends opaque records to
/api/0pt/internal/er/candidates, then applies only explicit hard-rule `match`
decisions. Score/fuzzy decisions are review evidence only and can never merge
records in this adapter, even if an older or compromised bridge returns
`status=match, decided_by=score`.
* CANONICAL: src/er/entity-resolution-service.ts — the declared generic ER
capability. Deterministic blocking + an explainable per-pair score,
product-supplied HARD RULES that override the score in both directions,
a configurable review band that hands ambiguous pairs to humans instead
of auto-merging, and delegation of heavy probabilistic linkage (Splink)
to 0pt-motor via the N0D job "0pt.er.batch_resolution".
If the canonical engine is unavailable, records are returned UNMERGED (fail
closed against false identity merges).
* THIS FILE: blocking on (market, first initial) + a single
rapidfuzz token_sort_ratio threshold, first-match-wins, auto-merge with
no review band, no hard-rule hook, no explanation beyond the score.
P065's definition of done is that no second simplified production engine
remains. It has NOT been met: this file is still what
`workers/pipeline.py` calls (line ~110, `deduped = resolve(records)`).
WHY IT WAS NOT REPLACED IN THE P060-P068 PASS, stated plainly rather than
quietly deferred: replacing it means turning this module into an ADAPTER
that calls the canonical capability, and that change cannot be verified
from this repo today —
* `services/api` is not deployed by this repo's Dockerfile (which builds
only the Next.js app), and api.0pt.app answers 404, so there is no
running engine to test against;
* no CI workflow runs services/api's pytest suite at all (verified:
`grep -rn "pytest" .gitea/workflows .github/workflows` finds nothing),
so a rewrite here would land with zero automated coverage.
Rewriting it blind and calling P065 done would be exactly the
"claimed complete, broken in operation" failure this programme exists to
stop. The removal plan is recorded in STATUS.md.
Until then: do NOT add capability to this file. Any new ER behaviour
belongs in the canonical TypeScript service.
Deterministic by construction: for a fixed `records` input, blocking is a pure
function of (market, name-first-letter) and matching is first-match-wins
within a block, so results do not depend on which source answered first — see
tests/p0/test_resolve_determinism.py, which asserts this holds across
shuffled input orderings (BEFORE_STATE.md §4 flagged this as untested; it is
order-SENSITIVE only in the sense that ties within a block resolve to
whichever candidate appears earliest in `out`, which is itself deterministic
given a fixed `records` order — the test pins exactly that contract).
A canonical hard non-match is also a cluster-wide constraint: transitive
matches are never allowed to pull two records with conflicting stable identity
evidence into the same entity.
"""
from __future__ import annotations
import copy
import hashlib
import json
import os
from typing import Any
from rapidfuzz import fuzz
import httpx
_TIMEOUT = httpx.Timeout(6.0, connect=2.0)
def block_key(rec: dict[str, Any]) -> tuple[str, str]:
"""Cheap blocking key — same market + first initial — to avoid O(n^2)."""
return (rec.get("market") or "", (rec.get("name") or "")[:1].upper())
def _url() -> str | None:
base = os.getenv("OPT_ER_BRIDGE_URL") or os.getenv("EDGE_URL")
return f"{base.rstrip('/')}/api/0pt/internal/er/candidates" if base else None
def resolve(records: list[dict[str, Any]], threshold: int = 92) -> list[dict[str, Any]]:
def _record_id(record: dict[str, Any]) -> str:
stable = {
"market": record.get("market"),
"name": record.get("name"),
"status": record.get("status"),
"lei": record.get("lei"),
"org_nr": record.get("org_nr"),
"source": record.get("source"),
"provenance": record.get("provenance", []),
}
raw = json.dumps(stable, sort_keys=True, separators=(",", ":"), default=str).encode()
return "erp_" + hashlib.sha256(raw).hexdigest()[:32]
def _unique_record_ids(records: list[dict[str, Any]]) -> list[str]:
seen: dict[str, int] = {}
unique_ids: list[str] = []
for record in records:
rid = _record_id(record)
n = seen.get(rid, 0)
seen[rid] = n + 1
unique_ids.append(rid if n == 0 else f"{rid}_{n}")
return unique_ids
def _apply_matches(records: list[dict[str, Any]], pairs: list[dict[str, Any]]) -> list[dict[str, Any]]:
if not records:
return []
unique_ids = _unique_record_ids(records)
known_ids = set(unique_ids)
parent = {rid: rid for rid in unique_ids}
members = {rid: {rid} for rid in unique_ids}
hard_non_matches: set[frozenset[str]] = set()
for pair in pairs:
a, b = pair.get("record_a"), pair.get("record_b")
if a not in known_ids or b not in known_ids:
continue
if pair.get("status") == "non_match" and pair.get("decided_by") not in (None, "score"):
hard_non_matches.add(frozenset((a, b)))
def find(x: str) -> str:
while parent[x] != x:
parent[x] = parent[parent[x]]
x = parent[x]
return x
def components_conflict(left: set[str], right: set[str]) -> bool:
return any(frozenset((a, b)) in hard_non_matches for a in left for b in right)
def union(a: str, b: str) -> bool:
ra, rb = find(a), find(b)
if ra == rb:
return True
left, right = members[ra], members[rb]
if components_conflict(left, right):
return False
keep, drop = (ra, rb) if ra < rb else (rb, ra)
parent[drop] = keep
members[keep] = members[ra] | members[rb]
members.pop(drop, None)
return True
# Defence in depth: only stable-identifier hard rules may create canonical
# identity links. Score/fuzzy "match" responses are ignored at the consumer
# boundary even if an upstream/legacy route emits one.
match_pairs = [
p for p in pairs
if p.get("status") == "match"
and p.get("decided_by") not in (None, "score")
and p.get("record_a") in known_ids
and p.get("record_b") in known_ids
]
match_pairs.sort(
key=lambda p: (
-float(p.get("score") or 0.0),
str(p.get("candidate_id") or ""),
)
)
accepted_edges: list[dict[str, Any]] = []
for pair in match_pairs:
a, b = str(pair["record_a"]), str(pair["record_b"])
if union(a, b):
accepted_edges.append(pair)
groups: dict[str, list[tuple[str, dict[str, Any]]]] = {}
for rid, record in zip(unique_ids, records):
groups.setdefault(find(rid), []).append((rid, copy.deepcopy(record)))
out: list[dict[str, Any]] = []
for r in records:
match = None
match_score: float | None = None
for o in out:
if block_key(o) != block_key(r):
continue
score = fuzz.token_sort_ratio(o.get("name", ""), r.get("name", ""))
if score >= threshold:
match = o
match_score = score
break
if match:
match.setdefault("merged_from", []).append(r.get("source"))
# Merge the losing record's per-field provenance regardless of source
# novelty below, so the surviving record's provenance always reflects
# every contributing record's fields.
existing_provenance = match.setdefault("provenance", [])
seen_provenance = {(p.get("field_name"), p.get("source_id")) for p in existing_provenance}
for p in r.get("provenance", []):
for group in groups.values():
group.sort(key=lambda item: (-float(item[1].get("confidence", 0.0)), item[0]))
_, anchor = group[0]
sources = sorted({str(r.get("source")) for _, r in group if r.get("source")})
anchor["agreed_sources"] = len(sources) or 1
anchor["confidence"] = min(
0.99,
float(anchor.get("confidence", 0.0)) + 0.1 * max(0, len(sources) - 1),
)
anchor["merged_from"] = [r.get("source") for _, r in group[1:] if r.get("source")]
provenance = list(anchor.get("provenance", []))
keys = {(p.get("field_name"), p.get("source_id")) for p in provenance}
for _, record in group[1:]:
for p in record.get("provenance", []):
key = (p.get("field_name"), p.get("source_id"))
if key not in seen_provenance:
existing_provenance.append(p)
seen_provenance.add(key)
# Corroboration (confidence/agreed_sources bump) requires an
# *independent* source — two hits from the same connector must not
# count as two agreeing sources.
seen_sources = match.setdefault("_sources_seen", {match.get("source")} - {None})
src = r.get("source")
if src is not None and src not in seen_sources:
match["confidence"] = min(0.99, match.get("confidence", 0.0) + 0.1)
match["agreed_sources"] = match.get("agreed_sources", 1) + 1
seen_sources.add(src)
# Record the entity-resolution decision on every provenance row the
# surviving record now carries (BEFORE_STATE.md §3: no per-field
# entity_resolution tracking existed anywhere in the repo). Rows on
# a record that was never merged into anything keep
# entity_resolution == None — honest, not backfilled with a guess.
er_info = {
"method": "blocking+fuzzy_token_sort_ratio",
"block_key": list(block_key(match)),
"match_score": match_score,
"threshold": threshold,
"merged_sources": sorted(s for s in match["_sources_seen"] if s is not None),
}
for p in existing_provenance:
if key not in keys:
provenance.append(copy.deepcopy(p))
keys.add(key)
member_ids = {rid for rid, _ in group}
edges = [
p for p in accepted_edges
if p.get("record_a") in member_ids and p.get("record_b") in member_ids
]
er_info = {
"method": "canonical_ts_entity_resolution",
"merged_sources": sources,
"matched_edges": [
{
"candidate_id": p.get("candidate_id"),
"score": p.get("score"),
"decided_by": p.get("decided_by"),
}
for p in edges
],
}
if len(group) > 1:
for p in provenance:
p["entity_resolution"] = er_info
else:
out.append({**r, "agreed_sources": 1, "_sources_seen": {r.get("source")} - {None}})
for o in out:
o.pop("_sources_seen", None)
return out
anchor["provenance"] = provenance
out.append(anchor)
return sorted(out, key=lambda r: (_record_id(r), str(r.get("source") or "")))
async def resolve(
records: list[dict[str, Any]],
*,
tenant_id: str,
project_id: str,
correlation_id: str | None = None,
) -> list[dict[str, Any]]:
"""Return canonically resolved records; never falls back to local scoring."""
if len(records) < 2:
return [copy.deepcopy(r) for r in records]
url = _url()
token = os.getenv("INTERNAL_TOKEN", "")
if not url or not token or not tenant_id or not project_id:
return [copy.deepcopy(r) for r in records]
unique_ids = _unique_record_ids(records)
er_records = []
for rid, record in zip(unique_ids, records):
name = str(record.get("name") or "")
market = record.get("market")
er_records.append({
"record_id": rid,
"fields": {
"market": market,
"registration_country": market,
"block_initial": name[:1].upper(),
"name": name,
"status": record.get("status"),
"lei": record.get("lei"),
"org_nr": record.get("org_nr"),
},
})
payload: dict[str, Any] = {
"tenant_id": tenant_id,
"project_id": project_id,
"records": er_records,
"blocking_fields": ["market", "block_initial"],
"compare_fields": [{"field": "name", "weight": 1}],
"auto_match_threshold": 0.92,
"review_threshold": 0.72,
}
if correlation_id:
payload["correlation_id"] = correlation_id
try:
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
response = await client.post(
url,
json=payload,
headers={"Content-Type": "application/json", "X-Internal-Token": token},
)
if response.status_code not in (200, 201):
return [copy.deepcopy(r) for r in records]
body = response.json()
pairs = body.get("pairs")
if not isinstance(pairs, list):
return [copy.deepcopy(r) for r in records]
return _apply_matches(records, pairs)
except (httpx.HTTPError, ValueError, TypeError, KeyError):
return [copy.deepcopy(r) for r in records]
__all__ = ["resolve", "_apply_matches"]
sql/011_aud0_evidence_outbox.sql
+39
View File
@@ -0,0 +1,39 @@
-- 0pt durable AUD0 evidence outbox (§70)
-- AUD0 remains canonical evidence; this is only a durable producer queue.
CREATE TABLE IF NOT EXISTS opt_aud0_evidence_outbox (
outbox_id text PRIMARY KEY,
tenant_id text NOT NULL,
app_id text NOT NULL DEFAULT '0pt',
evidence_kind text NOT NULL CHECK (evidence_kind IN ('event', 'openlineage')),
idempotency_key text NOT NULL,
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'delivered', 'dead')),
attempts integer NOT NULL DEFAULT 0 CHECK (attempts >= 0),
next_attempt_at timestamptz NOT NULL DEFAULT now(),
leased_by text,
leased_until timestamptz,
last_error text,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
delivered_at timestamptz,
dead_at timestamptz,
CONSTRAINT opt_aud0_evidence_outbox_idempotency
UNIQUE (tenant_id, evidence_kind, idempotency_key),
CONSTRAINT opt_aud0_evidence_outbox_delivery_consistency CHECK (
(status = 'delivered' AND delivered_at IS NOT NULL AND dead_at IS NULL)
OR (status = 'dead' AND dead_at IS NOT NULL AND delivered_at IS NULL)
OR (status = 'pending' AND delivered_at IS NULL AND dead_at IS NULL)
)
);
CREATE INDEX IF NOT EXISTS opt_aud0_evidence_outbox_pending_idx
ON opt_aud0_evidence_outbox (next_attempt_at, created_at)
WHERE status = 'pending';
CREATE INDEX IF NOT EXISTS opt_aud0_evidence_outbox_tenant_idx
ON opt_aud0_evidence_outbox (tenant_id, created_at DESC);
COMMENT ON TABLE opt_aud0_evidence_outbox IS
'Durable local delivery queue for 0pt -> AUD0 evidence. Not canonical evidence.';
sql/012_dataset_versions.sql
+28
View File
@@ -0,0 +1,28 @@
-- Durable canonical metadata for 0pt dataset versions.
-- Bytes remain canonical ST0RE artifacts referenced by artifact_refs.
CREATE TABLE IF NOT EXISTS opt_dataset_versions (
dataset_id text NOT NULL REFERENCES opt_datasets(dataset_id) ON DELETE RESTRICT,
version integer NOT NULL CHECK (version > 0),
tenant_id text NOT NULL,
project_id text NOT NULL,
artifact_refs text[] NOT NULL CHECK (cardinality(artifact_refs) > 0),
schema_version text NOT NULL,
record_count bigint CHECK (record_count IS NULL OR record_count >= 0),
description text,
created_at timestamptz NOT NULL DEFAULT now(),
created_by text NOT NULL,
correlation_id text,
idempotency_key text,
PRIMARY KEY (dataset_id, version)
);
CREATE UNIQUE INDEX IF NOT EXISTS opt_dataset_versions_idempotency_idx
ON opt_dataset_versions (tenant_id, idempotency_key)
WHERE idempotency_key IS NOT NULL;
CREATE INDEX IF NOT EXISTS opt_dataset_versions_tenant_dataset_idx
ON opt_dataset_versions (tenant_id, dataset_id, version DESC);
COMMENT ON TABLE opt_dataset_versions IS
'Immutable 0pt dataset-version metadata. Content bytes remain in ST0RE artifacts.';
sql/013_search_index_definitions.sql
+18
View File
@@ -0,0 +1,18 @@
-- Canonical 0pt search-index definitions. OpenSearch itself remains a
-- rebuildable projection; this table stores only the logical schema needed to
-- rebuild/query that projection after restart.
CREATE TABLE IF NOT EXISTS opt_search_index_definitions (
index_name text NOT NULL,
version integer NOT NULL CHECK (version > 0),
fields jsonb NOT NULL,
default_search_fields text[] NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
PRIMARY KEY (index_name, version),
CONSTRAINT opt_search_index_definitions_fields_object
CHECK (jsonb_typeof(fields) = 'object')
);
COMMENT ON TABLE opt_search_index_definitions IS
'Versioned logical 0pt search schemas. OpenSearch indices remain disposable projections.';
src/clients/aud0-client.ts
+55 -68
View File
@@ -1,41 +1,13 @@
/**
* src/clients/aud0-client.ts
* ===========================
* 0pt's ONLY path for recording "what happened" to AUD0, the SIAX Cloud
* append-only system of record.
* 0pt's sole AUD0 producer boundary.
*
* REWRITTEN 2026-08-31 against AUD0's REAL wire contract (sax3l/aud0 branch
* feat/siax-platforms-p0-evidence-os, `src/api/routes.ts` +
* `src/schemas/event.ts` + `src/api/routes/openlineage.ts`, verified this
* session):
*
* ingestEvent -> POST {baseUrl}/v1/aud0/events (snake_case)
* ingestOpenLineageEvent-> POST {baseUrl}/v1/aud0/openlineage?tenant_id=&app_id=
* ingestLineageEvent -> LEGACY SHIM: converts the historical 0pt lineage
* shape to an OpenLineage RunEvent and posts it to
* the endpoint above.
*
* The path this client used to post lineage to — `/v1/aud0/lineage/events` —
* DOES NOT EXIST on AUD0 and never has. AUD0's only lineage ingest is the
* OpenLineage endpoint; `/v1/aud0/lineage/*` routes are read-only projections
* of the generic event stream. Master-DoD §16 requires OpenLineage run/job/
* dataset ingest — the emitter in src/lineage/ produces spec-shaped
* RunEvents; this client is just the wire.
*
* The audit-event contract additionally carries the P0 evidence-context
* fields (§16: purpose, policy decision, correlation, causation, trace,
* result, references), all optional on the wire; they are surfaced on
* `Aud0Event` so services can populate them.
*
* Production fallback policy (Master-DoD §3/§10): no base URL in a
* production runtime throws `SiaxError("DEPENDENCY_UNAVAILABLE")` at
* construction. The no-op client survives only for non-production runtimes.
* NOTE the deliberate asymmetry: audit emission FAILURE at runtime (AUD0
* down) still does not roll back the already-performed action — an audit
* emission records what already happened; the durability requirement is
* §70's "durable local/outbox path", tracked honestly in STATUS.md as a gap.
* Default clients are durable: when DATABASE_URL is configured, evidence is
* first written to opt_aud0_evidence_outbox and later delivered by the
* dispatcher. `durable:false` exists only for that dispatcher, which must
* perform the real HTTP delivery rather than enqueue recursively.
*/
import { SiaxHttp } from "./http";
import { createAud0OutboxRepositoryFromEnv, type Aud0OutboxRepository } from "../evidence/aud0-outbox";
import { resolveDependencyBaseUrl } from "../platform/dependency-mode";
import type { SiaxRequestContext } from "../platform/siax-context";
import type { OpenLineageRunEvent } from "../lineage/openlineage";
@@ -50,30 +22,21 @@ export interface Aud0Event {
capability_id: string;
resource_type: string;
resource_id: string;
/** Risk level L0–L5 per the SIAX Cloud risk taxonomy. */
risk_level: string;
payload?: unknown;
created_at: string;
correlation_id?: string;
// P0 evidence-context fields (Master-DoD §16), optional on AUD0's wire.
/** Business time, when it differs from ingestion time. */
occurred_at?: string;
/** GDPR/purpose-limitation tag. */
purpose?: string;
/** CL0UD policy decision id/reason for the action being recorded. */
policy_decision?: string;
causation_id?: string;
trace_id?: string;
/** Outcome, e.g. "success" | "failure". */
result?: string;
/** Canonical resource refs consumed by the action (e.g. st0re:artifact:...). */
input_refs?: string[];
/** Canonical resource refs produced by the action. */
output_refs?: string[];
idempotency_key?: string;
}
/** Historical 0pt lineage shape — kept as the input to the legacy shim. */
export interface Aud0LineageEvent {
lineage_event_id: string;
tenant_id: string;
@@ -84,19 +47,15 @@ export interface Aud0LineageEvent {
source_id: string;
target_type: string;
target_id: string;
/** e.g. "produced" | "derived_from" | "stored_as". */
lineage_type: string;
metadata?: unknown;
created_at: string;
correlation_id?: string;
}
/** Alias so both historical import names resolve to the same shape. */
export type Aud0EventInput = Aud0Event;
/** Alias so both historical import names resolve to the same shape. */
export type Aud0LineageEventInput = Aud0LineageEvent;
/** AUD0's OpenLineage ingest acknowledgement. */
export interface Aud0OpenLineageAck {
tenant_id: string;
run_id: string;
@@ -112,24 +71,28 @@ export interface Aud0Client {
tenantId: string,
appId: string,
event: OpenLineageRunEvent,
context?: SiaxRequestContext
context?: SiaxRequestContext,
): Promise<Aud0OpenLineageAck>;
/** @deprecated Legacy shim — converts to OpenLineage and posts to /v1/aud0/openlineage. */
ingestLineageEvent(event: Aud0LineageEvent, context?: SiaxRequestContext): Promise<Aud0LineageEvent>;
}
export function createAud0Client(baseUrl?: string, opts?: { token?: string }): Aud0Client {
export interface CreateAud0ClientOptions {
token?: string;
/** False only for the outbox dispatcher/raw diagnostics. Default true. */
durable?: boolean;
}
export function createAud0Client(baseUrl?: string, opts?: CreateAud0ClientOptions): Aud0Client {
const resolved = resolveDependencyBaseUrl("aud0", baseUrl ?? process.env.AUD0_URL);
if (!resolved) return new NoopAud0Client();
return new HttpAud0Client(resolved, opts?.token ?? process.env.AUD0_SERVICE_TOKEN);
const raw: Aud0Client = resolved
? new HttpAud0Client(resolved, opts?.token ?? process.env.AUD0_SERVICE_TOKEN)
: new NoopAud0Client();
if (opts?.durable === false) return raw;
const outbox = createAud0OutboxRepositoryFromEnv();
return outbox ? new DurableAud0Client(outbox) : raw;
}
/**
* Convert the historical 0pt lineage shape into a spec-shaped OpenLineage
* RunEvent (AUD0's only lineage ingest contract). Deterministic: the runId is
* the legacy lineage_event_id, so idempotent re-emission dedups on AUD0's
* `ol:<runId>:<eventType>:<eventTime>` key.
*/
export function legacyLineageToOpenLineage(event: Aud0LineageEvent): OpenLineageRunEvent {
return {
eventType: "COMPLETE",
@@ -138,9 +101,7 @@ export function legacyLineageToOpenLineage(event: Aud0LineageEvent): OpenLineage
run: {
runId: event.lineage_event_id,
facets: {
...(event.correlation_id
? { siaxCorrelation: { correlation_id: event.correlation_id } }
: {}),
...(event.correlation_id ? { siaxCorrelation: { correlation_id: event.correlation_id } } : {}),
siaxLegacyLineage: {
lineage_type: event.lineage_type,
capability_id: event.capability_id,
@@ -155,15 +116,44 @@ export function legacyLineageToOpenLineage(event: Aud0LineageEvent): OpenLineage
};
}
/** Non-production runtimes only: audit emission is a no-op echo. */
class DurableAud0Client implements Aud0Client {
constructor(private readonly outbox: Aud0OutboxRepository) {}
async ingestEvent(event: Aud0Event): Promise<Aud0Event> {
await this.outbox.enqueueEvent(event);
return event;
}
async ingestOpenLineageEvent(
tenantId: string,
appId: string,
event: OpenLineageRunEvent,
): Promise<Aud0OpenLineageAck> {
const row = await this.outbox.enqueueOpenLineage(tenantId, appId, event);
return {
tenant_id: tenantId,
run_id: event.run.runId,
job: `${event.job.namespace}/${event.job.name}`,
event_id: `outbox:${row.outbox_id}`,
};
}
async ingestLineageEvent(event: Aud0LineageEvent): Promise<Aud0LineageEvent> {
await this.ingestOpenLineageEvent(event.tenant_id, event.app_id, legacyLineageToOpenLineage(event));
return event;
}
}
/** Non-production/env-less runtime fallback only. */
class NoopAud0Client implements Aud0Client {
async ingestEvent(event: Aud0Event): Promise<Aud0Event> {
return event;
}
async ingestOpenLineageEvent(
tenantId: string,
_appId: string,
event: OpenLineageRunEvent
event: OpenLineageRunEvent,
): Promise<Aud0OpenLineageAck> {
return {
tenant_id: tenantId,
@@ -172,6 +162,7 @@ class NoopAud0Client implements Aud0Client {
event_id: `noop_${event.run.runId}`,
};
}
async ingestLineageEvent(event: Aud0LineageEvent): Promise<Aud0LineageEvent> {
return event;
}
@@ -191,10 +182,8 @@ class HttpAud0Client implements Aud0Client {
context,
headers: { "X-Tenant-Id": event.tenant_id, "X-Actor-Id": event.actor_id },
body: event,
// 200 = idempotent replay (deduplicated: true) — a success.
acceptStatuses: [200],
});
// Prefer AUD0's canonical stored record; fall back to what we sent.
return (res.json ?? event) as Aud0Event;
}
@@ -202,13 +191,11 @@ class HttpAud0Client implements Aud0Client {
tenantId: string,
appId: string,
event: OpenLineageRunEvent,
context?: SiaxRequestContext
context?: SiaxRequestContext,
): Promise<Aud0OpenLineageAck> {
const res = await this.http.request({
method: "POST",
path: "/v1/aud0/openlineage",
// AUD0's OpenLineage ingest takes the tenant as a QUERY PARAM (verified
// against src/api/routes/openlineage.ts), not a body field.
query: { tenant_id: tenantId, app_id: appId },
context,
headers: { "X-Tenant-Id": tenantId, "X-Actor-Id": appId },
src/er/entity-resolution-service.ts
+86 -90
View File
@@ -1,44 +1,18 @@
/**
* src/er/entity-resolution-service.ts
* ====================================
* Generic Entity Resolution capability (Master-DoD §10 Entity Resolution:
* candidate API, generic probability, blocking, explanation, product
* hard-rule hook, human review handoff).
* Canonical generic Entity Resolution capability.
*
* Division of labour, stated plainly:
* - THIS service is the GENERIC candidate API: deterministic blocking +
* a transparent similarity scorer (token Jaccard + normalized Levenshtein)
* producing an explainable per-pair score. It is the reference
* implementation and the contract.
* - HEAVY/statistical ER (Splink probabilistic linkage, §11 ER) is
* 0pt-motor's job — `submitBatchResolution` creates the N0D job
* ("0pt.er.batch_resolution") and never fakes its output.
* - PRODUCTS own domain judgement via the HARD-RULE HOOK: e.g. VehIQ
* registers VIN-equality rules that FORCE match/non-match regardless of
* the generic score (Master-DoD §24 "VIN hard rules ... negative
* identity evidence"). Hard rules always win over probabilities.
* - Pairs in the configurable "review band" are handed off to humans
* (status pending_review) — never auto-merged.
*
* GENERIC means GENERIC: no person-matching logic, no Ratsit/PII pipelines —
* records are opaque field maps supplied by the calling product under its
* own legal basis; this module implements comparison mechanics only.
*
* P065 STATUS — THIS IS THE CANONICAL ENGINE, AND IT IS NOT YET THE ONLY ONE.
* `services/api/workers/resolve.py` is a second, simplified ER engine (fuzzy
* threshold, auto-merge, no hard rules, no review band) that
* `services/api/workers/pipeline.py` still calls. P065's DoD ("no second
* simplified duplicated production engine remains") is therefore NOT met.
* See that file's header for why it was not rewritten blind in the
* P060-P068 pass and STATUS.md for the removal plan. The ingestion plane
* added in that pass (src/ingestion/*) deliberately does no ER of its own —
* it stops at the sink, so it adds no THIRD engine.
* - synchronous candidate generation is deterministic, explainable and
* review-aware;
* - product hard rules may override probabilities in either direction;
* - heavy/statistical ER is delegated to 0pt-motor through N0D;
* - the legacy Python production scorer has been removed: Python is now only
* an adapter to this canonical decision engine.
*/
import crypto from "crypto";
import { SiaxError } from "../platform/errors";
import type { N0dClient, N0dJob } from "../clients/n0d-client";
import { SiaxError } from "../platform/errors";
import { isResourceRef } from "../platform/resource-ref";
/** An opaque record supplied by the calling product. */
export interface ErRecord {
record_id: string;
fields: Record<string, string | number | null | undefined>;
@@ -46,7 +20,6 @@ export interface ErRecord {
export type HardRuleDecision = "force_match" | "force_non_match" | "abstain";
/** Product-supplied hard rule (the §10 hook). */
export interface HardRule {
rule_id: string;
decide(a: ErRecord, b: ErRecord): HardRuleDecision;
@@ -56,13 +29,9 @@ export interface ErCandidateRequest {
tenant_id: string;
project_id: string;
records: ErRecord[];
/** Fields whose normalized values form the blocking key. */
blocking_fields: string[];
/** Fields compared for similarity (weight defaults to 1). */
compare_fields: { field: string; weight?: number }[];
/** Score ≥ auto_match_threshold → match (unless a hard rule vetoes). Default 0.9. */
auto_match_threshold?: number;
/** Score ≥ review_threshold (< auto) → pending_review. Default 0.6. */
review_threshold?: number;
correlation_id?: string;
}
@@ -81,12 +50,9 @@ export interface ErCandidatePair {
candidate_id: string;
record_a: string;
record_b: string;
/** Generic similarity score 0..1. */
score: number;
status: ErCandidateStatus;
/** Which mechanism decided: hard rule id, or "score". */
decided_by: string;
/** Per-field breakdown — the §10 explanation. */
explanation: ErFieldExplanation[];
blocking_key: string;
}
@@ -95,49 +61,77 @@ export interface ErCandidateResult {
request_id: string;
tenant_id: string;
pairs: ErCandidatePair[];
/** Pairs handed to human review (subset of pairs). */
review_queue: ErCandidatePair[];
blocked_comparisons: number;
total_possible_comparisons: number;
}
export interface SubmitBatchResolutionInput {
tenant_id: string;
project_id: string;
records_artifact_ref: string;
actor_id?: string;
correlation_id?: string;
idempotency_key?: string;
}
export class EntityResolutionService {
private readonly hardRules: HardRule[] = [];
constructor(private readonly opts: { n0dClient?: N0dClient } = {}) {}
/** The product hard-rule hook (§10). Rules run in registration order. */
registerHardRule(rule: HardRule): void {
if (!rule.rule_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "hard rule requires rule_id");
if (this.hardRules.some((existing) => existing.rule_id === rule.rule_id)) {
throw new SiaxError("CONFLICT", "0pt.er", `hard rule ${rule.rule_id} is already registered`);
}
this.hardRules.push(rule);
}
// -----------------------------------------------------------------------
// Candidate API (synchronous, generic)
// -----------------------------------------------------------------------
generateCandidates(req: ErCandidateRequest): ErCandidateResult {
if (!req.tenant_id?.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required");
if (!req.tenant_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required");
if (!req.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "project_id is required");
if (req.records.length < 2 || req.records.length > 5_000) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "records must contain between 2 and 5000 items");
}
if (req.records.length < 2) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "at least two records are required");
if (new Set(req.records.map((record) => record.record_id)).size !== req.records.length) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "record_id values must be unique");
}
if (req.blocking_fields.length === 0) {
if (req.blocking_fields.length < 1 || req.blocking_fields.length > 20) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.er",
"blocking_fields must not be empty — unblocked all-pairs comparison is refused (quadratic + §46)"
"blocking_fields must contain 1-20 fields — unblocked all-pairs comparison is refused",
);
}
if (req.compare_fields.length < 1 || req.compare_fields.length > 50) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "compare_fields must contain 1-50 fields");
}
for (const { field, weight = 1 } of req.compare_fields) {
if (!field?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "compare field name is required");
if (!Number.isFinite(weight) || weight < 0 || weight > 100) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `invalid weight for compare field ${field}`);
}
}
const autoThreshold = req.auto_match_threshold ?? 0.9;
const reviewThreshold = req.review_threshold ?? 0.6;
if (reviewThreshold >= autoThreshold) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "review_threshold must be < auto_match_threshold");
if (
!Number.isFinite(autoThreshold) || autoThreshold < 0 || autoThreshold > 1 ||
!Number.isFinite(reviewThreshold) || reviewThreshold < 0 || reviewThreshold > 1 ||
reviewThreshold >= autoThreshold
) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.er",
"thresholds must be within 0..1 and review_threshold must be lower than auto_match_threshold",
);
}
// Blocking: bucket records by normalized blocking key.
const blocks = new Map<string, ErRecord[]>();
for (const rec of req.records) {
const key = req.blocking_fields.map((f) => normalize(String(rec.fields[f] ?? ""))).join("|");
if (!rec.record_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "every record requires record_id");
const key = req.blocking_fields.map((field) => normalize(String(rec.fields[field] ?? ""))).join("|");
const bucket = blocks.get(key) ?? [];
bucket.push(rec);
blocks.set(key, bucket);
@@ -146,10 +140,10 @@ export class EntityResolutionService {
const pairs: ErCandidatePair[] = [];
let blockedComparisons = 0;
for (const [key, bucket] of blocks) {
if (key.replace(/\|/g, "") === "") continue; // empty blocking key never blocks together
if (key.replace(/\|/g, "") === "") continue;
for (let i = 0; i < bucket.length; i++) {
for (let j = i + 1; j < bucket.length; j++) {
blockedComparisons++;
blockedComparisons += 1;
pairs.push(this.comparePair(bucket[i], bucket[j], req, autoThreshold, reviewThreshold, key));
}
}
@@ -160,32 +154,38 @@ export class EntityResolutionService {
request_id: `errq_${crypto.randomUUID()}`,
tenant_id: req.tenant_id,
pairs,
review_queue: pairs.filter((p) => p.status === "pending_review"),
review_queue: pairs.filter((pair) => pair.status === "pending_review"),
blocked_comparisons: blockedComparisons,
total_possible_comparisons: (n * (n - 1)) / 2,
};
}
// -----------------------------------------------------------------------
// Batch/statistical path — 0pt-motor via N0D (contract only, never faked)
// -----------------------------------------------------------------------
async submitBatchResolution(input: {
tenant_id: string;
project_id: string;
/** ST0RE artifact ref of the record dataset. */
records_artifact_ref: string;
correlation_id?: string;
}): Promise<N0dJob> {
/** Heavy/statistical path — an idempotent N0D job executed by 0pt-motor. */
async submitBatchResolution(input: SubmitBatchResolutionInput): Promise<N0dJob> {
if (!input.tenant_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required");
if (!input.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", "project_id is required");
if (!isResourceRef(input.records_artifact_ref, "st0re", "artifact")) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.er",
"records_artifact_ref must be a canonical st0re:artifact:... reference",
);
}
if (!this.opts.n0dClient) {
throw new SiaxError("DEPENDENCY_UNAVAILABLE", "n0d", "no N0D client configured for batch ER");
}
const idempotencyKey = input.idempotency_key?.trim() || crypto
.createHash("sha256")
.update(`0pt.er.batch.v1|${input.tenant_id}|${input.project_id}|${input.records_artifact_ref}`)
.digest("hex");
return this.opts.n0dClient.createJob({
tenant_id: input.tenant_id,
project_id: input.project_id,
job_type: "0pt.er.batch_resolution",
capability_id: "0pt.er.batch.v1",
requested_by_app_id: "0pt",
requested_by_actor_id: "svc_0pt",
requested_by_actor_id: input.actor_id ?? "svc_0pt",
input_artifact_ids: [input.records_artifact_ref],
output_artifact_type: "er_candidate_set",
requires_backbone: false,
@@ -194,19 +194,20 @@ export class EntityResolutionService {
resource_profile: "large",
risk_level: "L3",
correlation_id: input.correlation_id,
idempotency_key: idempotencyKey,
timeout_seconds: 3_600,
max_retries: 2,
});
}
// -----------------------------------------------------------------------
private comparePair(
a: ErRecord,
b: ErRecord,
req: ErCandidateRequest,
autoThreshold: number,
reviewThreshold: number,
blockingKey: string
blockingKey: string,
): ErCandidatePair {
// Hard rules first — they always win (§10/§24).
for (const rule of this.hardRules) {
const decision = rule.decide(a, b);
if (decision !== "abstain") {
@@ -224,12 +225,16 @@ export class EntityResolutionService {
}
const explanation = this.explain(a, b, req);
const weightSum = explanation.reduce((s, e) => s + e.weight, 0);
const score = weightSum === 0 ? 0 : explanation.reduce((s, e) => s + e.similarity * e.weight, 0) / weightSum;
const weightSum = explanation.reduce((sum, item) => sum + item.weight, 0);
const score = weightSum === 0
? 0
: explanation.reduce((sum, item) => sum + item.similarity * item.weight, 0) / weightSum;
const rounded = Math.round(score * 1000) / 1000;
const status: ErCandidateStatus =
rounded >= autoThreshold ? "match" : rounded >= reviewThreshold ? "pending_review" : "non_match";
const status: ErCandidateStatus = rounded >= autoThreshold
? "match"
: rounded >= reviewThreshold
? "pending_review"
: "non_match";
return {
candidate_id: `erc_${crypto.randomUUID()}`,
record_a: a.record_id,
@@ -257,14 +262,6 @@ export class EntityResolutionService {
}
}
// ---------------------------------------------------------------------------
// Transparent similarity primitives (deterministic; the "generic probability")
// ---------------------------------------------------------------------------
/**
* Case-fold, DIACRITIC-FOLD (ö→o, é→e — deliberate: "Göteborg" and
* "Goteborg" must land in the same block), strip punctuation, collapse
* whitespace. ø has no NFKD decomposition and is mapped explicitly.
*/
export function normalize(s: string): string {
return s
.toLowerCase()
@@ -276,7 +273,6 @@ export function normalize(s: string): string {
.trim();
}
/** Combined similarity: max(token Jaccard, 1 - normalized Levenshtein). */
export function similarity(a: string | null, b: string | null): number {
if (a === null || b === null) return 0;
const na = normalize(a);
@@ -289,7 +285,7 @@ export function similarity(a: string | null, b: string | null): number {
function tokenJaccard(a: string, b: string): number {
const ta = new Set(a.split(" "));
const tb = new Set(b.split(" "));
const intersection = [...ta].filter((t) => tb.has(t)).length;
const intersection = [...ta].filter((token) => tb.has(token)).length;
const union = new Set([...ta, ...tb]).size;
return union === 0 ? 0 : intersection / union;
}
src/er/request-validation.ts
+110
View File
@@ -0,0 +1,110 @@
import { SiaxError } from "../platform/errors";
import type { ErCandidateRequest, ErRecord } from "./entity-resolution-service";
function finiteThreshold(value: unknown, field: string, fallback: number): number {
if (value === undefined) return fallback;
const parsed = Number(value);
if (!Number.isFinite(parsed) || parsed < 0 || parsed > 1) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `${field} must be a finite number between 0 and 1`);
}
return parsed;
}
export function parseErCandidateRequest(raw: unknown): ErCandidateRequest {
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "JSON object body is required");
}
const body = raw as Record<string, unknown>;
const tenantId = typeof body.tenant_id === "string" ? body.tenant_id.trim() : "";
const projectId = typeof body.project_id === "string" ? body.project_id.trim() : "";
if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.er", "tenant_id is required");
if (!projectId) throw new SiaxError("INVALID_REQUEST", "0pt.er", "project_id is required");
if (!Array.isArray(body.records) || body.records.length < 2 || body.records.length > 5_000) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "records must contain between 2 and 5000 items");
}
if (
!Array.isArray(body.blocking_fields) ||
body.blocking_fields.length === 0 ||
body.blocking_fields.length > 20 ||
!body.blocking_fields.every((v) => typeof v === "string" && v.trim().length > 0)
) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "blocking_fields must contain 1-20 non-empty strings");
}
if (!Array.isArray(body.compare_fields) || body.compare_fields.length === 0 || body.compare_fields.length > 50) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "compare_fields must contain 1-50 items");
}
const records: ErRecord[] = body.records.map((value, index) => {
if (!value || typeof value !== "object" || Array.isArray(value)) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}] must be an object`);
}
const record = value as Record<string, unknown>;
if (typeof record.record_id !== "string" || !record.record_id.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}].record_id is required`);
}
if (!record.fields || typeof record.fields !== "object" || Array.isArray(record.fields)) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}].fields must be an object`);
}
const fields = record.fields as Record<string, unknown>;
if (Object.keys(fields).length > 200) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}].fields exceeds 200 fields`);
}
const normalizedFields: ErRecord["fields"] = {};
for (const [field, fieldValue] of Object.entries(fields)) {
if (!field.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.er", `records[${index}] contains an empty field name`);
if (
fieldValue !== null &&
fieldValue !== undefined &&
typeof fieldValue !== "string" &&
typeof fieldValue !== "number"
) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.er",
`records[${index}].fields.${field} must be string, number, null or undefined`,
);
}
normalizedFields[field.trim()] = fieldValue as string | number | null | undefined;
}
return { record_id: record.record_id.trim(), fields: normalizedFields };
});
if (new Set(records.map((record) => record.record_id)).size !== records.length) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "record_id values must be unique within the request");
}
const compareFields = body.compare_fields.map((value, index) => {
if (!value || typeof value !== "object" || Array.isArray(value)) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `compare_fields[${index}] must be an object`);
}
const item = value as Record<string, unknown>;
if (typeof item.field !== "string" || !item.field.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `compare_fields[${index}].field is required`);
}
const weight = item.weight === undefined ? undefined : Number(item.weight);
if (weight !== undefined && (!Number.isFinite(weight) || weight < 0 || weight > 100)) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", `compare_fields[${index}].weight must be between 0 and 100`);
}
return { field: item.field.trim(), ...(weight !== undefined ? { weight } : {}) };
});
const autoThreshold = finiteThreshold(body.auto_match_threshold, "auto_match_threshold", 0.9);
const reviewThreshold = finiteThreshold(body.review_threshold, "review_threshold", 0.6);
if (reviewThreshold >= autoThreshold) {
throw new SiaxError("INVALID_REQUEST", "0pt.er", "review_threshold must be lower than auto_match_threshold");
}
return {
tenant_id: tenantId,
project_id: projectId,
records,
blocking_fields: (body.blocking_fields as string[]).map((value) => value.trim()),
compare_fields: compareFields,
auto_match_threshold: autoThreshold,
review_threshold: reviewThreshold,
...(typeof body.correlation_id === "string" && body.correlation_id.trim()
? { correlation_id: body.correlation_id.trim() }
: {}),
};
}
src/evidence/aud0-outbox.ts
+377
View File
@@ -0,0 +1,377 @@
import crypto from "node:crypto";
import type { Aud0Client, Aud0Event } from "../clients/aud0-client";
import type { OpenLineageRunEvent } from "../lineage/openlineage";
import { SiaxError } from "../platform/errors";
import {
getRepositorySql,
repositoryDatabaseUrl,
type RepositorySql,
} from "../repositories/db";
export type Aud0EvidenceKind = "event" | "openlineage";
export type Aud0OutboxStatus = "pending" | "delivered" | "dead";
export interface Aud0OutboxRecord {
outbox_id: string;
tenant_id: string;
app_id: string;
evidence_kind: Aud0EvidenceKind;
idempotency_key: string;
payload: Aud0Event | OpenLineageRunEvent;
status: Aud0OutboxStatus;
attempts: number;
next_attempt_at: string;
leased_by?: string;
leased_until?: string;
last_error?: string;
created_at: string;
updated_at: string;
delivered_at?: string;
dead_at?: string;
}
export interface Aud0OutboxStats {
pending: number;
delivered: number;
dead: number;
oldest_pending_at?: string;
}
export interface Aud0OutboxRepository {
enqueueEvent(event: Aud0Event): Promise<Aud0OutboxRecord>;
enqueueOpenLineage(
tenantId: string,
appId: string,
event: OpenLineageRunEvent,
): Promise<Aud0OutboxRecord>;
claimPending(input: {
workerId: string;
limit: number;
leaseMs: number;
}): Promise<Aud0OutboxRecord[]>;
markDelivered(outboxId: string, workerId: string): Promise<void>;
markRetry(outboxId: string, workerId: string, error: string, delayMs: number): Promise<void>;
markDead(outboxId: string, workerId: string, error: string): Promise<void>;
stats(): Promise<Aud0OutboxStats>;
}
interface OutboxRow {
outbox_id: string;
tenant_id: string;
app_id: string;
evidence_kind: Aud0EvidenceKind;
idempotency_key: string;
payload: Aud0Event | OpenLineageRunEvent;
status: Aud0OutboxStatus;
attempts: number;
next_attempt_at: string | Date;
leased_by: string | null;
leased_until: string | Date | null;
last_error: string | null;
created_at: string | Date;
updated_at: string | Date;
delivered_at: string | Date | null;
dead_at: string | Date | null;
}
type JsonValue = Parameters<RepositorySql["json"]>[0];
function iso(value: string | Date): string {
return value instanceof Date ? value.toISOString() : new Date(value).toISOString();
}
function optionalIso(value: string | Date | null): string | undefined {
return value == null ? undefined : iso(value);
}
function mapRow(row: OutboxRow): Aud0OutboxRecord {
return {
outbox_id: row.outbox_id,
tenant_id: row.tenant_id,
app_id: row.app_id,
evidence_kind: row.evidence_kind,
idempotency_key: row.idempotency_key,
payload: row.payload,
status: row.status,
attempts: Number(row.attempts),
next_attempt_at: iso(row.next_attempt_at),
...(row.leased_by ? { leased_by: row.leased_by } : {}),
...(row.leased_until ? { leased_until: optionalIso(row.leased_until) } : {}),
...(row.last_error ? { last_error: row.last_error } : {}),
created_at: iso(row.created_at),
updated_at: iso(row.updated_at),
...(row.delivered_at ? { delivered_at: optionalIso(row.delivered_at) } : {}),
...(row.dead_at ? { dead_at: optionalIso(row.dead_at) } : {}),
};
}
function required(value: string, field: string): string {
const normalized = value.trim();
if (!normalized) {
throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", `${field} is required`);
}
return normalized;
}
function outboxId(tenantId: string, kind: Aud0EvidenceKind, idempotencyKey: string): string {
const digest = crypto
.createHash("sha256")
.update(tenantId)
.update("\0")
.update(kind)
.update("\0")
.update(idempotencyKey)
.digest("hex")
.slice(0, 40);
return `obx_${digest}`;
}
function openLineageIdempotencyKey(event: OpenLineageRunEvent): string {
return `ol:${required(event.run.runId, "run.runId")}:${required(event.eventType, "eventType")}`;
}
function safeError(error: unknown): string {
if (error instanceof SiaxError) return `${error.code}:${error.source}`.slice(0, 500);
if (error instanceof Error) return error.name.slice(0, 500);
return "UnknownError";
}
export class PostgresAud0OutboxRepository implements Aud0OutboxRepository {
constructor(private readonly sql: RepositorySql) {}
async enqueueEvent(event: Aud0Event): Promise<Aud0OutboxRecord> {
const key = required(event.idempotency_key ?? event.event_id, "event idempotency key");
return this.enqueue({
tenantId: required(event.tenant_id, "tenant_id"),
appId: required(event.app_id, "app_id"),
kind: "event",
idempotencyKey: key,
payload: event,
});
}
async enqueueOpenLineage(
tenantId: string,
appId: string,
event: OpenLineageRunEvent,
): Promise<Aud0OutboxRecord> {
return this.enqueue({
tenantId: required(tenantId, "tenant_id"),
appId: required(appId, "app_id"),
kind: "openlineage",
idempotencyKey: openLineageIdempotencyKey(event),
payload: event,
});
}
private async enqueue(input: {
tenantId: string;
appId: string;
kind: Aud0EvidenceKind;
idempotencyKey: string;
payload: Aud0Event | OpenLineageRunEvent;
}): Promise<Aud0OutboxRecord> {
const id = outboxId(input.tenantId, input.kind, input.idempotencyKey);
const rows = await this.sql<OutboxRow[]>`
INSERT INTO opt_aud0_evidence_outbox
(outbox_id, tenant_id, app_id, evidence_kind, idempotency_key, payload)
VALUES
(${id}, ${input.tenantId}, ${input.appId}, ${input.kind}, ${input.idempotencyKey},
${this.sql.json(input.payload as unknown as JsonValue)})
ON CONFLICT (tenant_id, evidence_kind, idempotency_key)
DO UPDATE SET updated_at = now()
WHERE opt_aud0_evidence_outbox.payload = EXCLUDED.payload
RETURNING *`;
if (rows.length === 0) {
throw new SiaxError(
"CONFLICT",
"0pt.aud0_outbox",
"the same evidence idempotency key was reused with a different payload",
);
}
return mapRow(rows[0]);
}
async claimPending(input: {
workerId: string;
limit: number;
leaseMs: number;
}): Promise<Aud0OutboxRecord[]> {
const workerId = required(input.workerId, "workerId");
if (!Number.isSafeInteger(input.limit) || input.limit < 1 || input.limit > 500) {
throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", "limit must be between 1 and 500");
}
if (!Number.isSafeInteger(input.leaseMs) || input.leaseMs < 1_000 || input.leaseMs > 15 * 60_000) {
throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", "leaseMs must be between 1000 and 900000");
}
const rows = await this.sql<OutboxRow[]>`
WITH candidates AS (
SELECT outbox_id
FROM opt_aud0_evidence_outbox
WHERE status = 'pending'
AND next_attempt_at <= now()
AND (leased_until IS NULL OR leased_until < now())
ORDER BY next_attempt_at ASC, created_at ASC
LIMIT ${input.limit}
FOR UPDATE SKIP LOCKED
)
UPDATE opt_aud0_evidence_outbox AS o
SET leased_by = ${workerId},
leased_until = now() + (${input.leaseMs} * interval '1 millisecond'),
attempts = o.attempts + 1,
updated_at = now()
FROM candidates
WHERE o.outbox_id = candidates.outbox_id
RETURNING o.*`;
return rows.map(mapRow);
}
async markDelivered(outboxId: string, workerId: string): Promise<void> {
await this.finish(outboxId, workerId, "delivered");
}
async markDead(outboxId: string, workerId: string, error: string): Promise<void> {
await this.finish(outboxId, workerId, "dead", error);
}
private async finish(
outboxId: string,
workerId: string,
status: "delivered" | "dead",
error?: string,
): Promise<void> {
const rows = await this.sql<{ outbox_id: string }[]>`
UPDATE opt_aud0_evidence_outbox
SET status = ${status},
delivered_at = CASE WHEN ${status} = 'delivered' THEN now() ELSE NULL END,
dead_at = CASE WHEN ${status} = 'dead' THEN now() ELSE NULL END,
last_error = ${error?.slice(0, 500) ?? null},
leased_by = NULL,
leased_until = NULL,
updated_at = now()
WHERE outbox_id = ${required(outboxId, "outboxId")}
AND status = 'pending'
AND leased_by = ${required(workerId, "workerId")}
RETURNING outbox_id`;
if (rows.length !== 1) {
throw new SiaxError("CONFLICT", "0pt.aud0_outbox", "outbox lease is no longer owned by this worker");
}
}
async markRetry(outboxId: string, workerId: string, error: string, delayMs: number): Promise<void> {
if (!Number.isSafeInteger(delayMs) || delayMs < 0 || delayMs > 24 * 60 * 60_000) {
throw new SiaxError("INVALID_REQUEST", "0pt.aud0_outbox", "delayMs must be between 0 and 86400000");
}
const rows = await this.sql<{ outbox_id: string }[]>`
UPDATE opt_aud0_evidence_outbox
SET next_attempt_at = now() + (${delayMs} * interval '1 millisecond'),
last_error = ${error.slice(0, 500)},
leased_by = NULL,
leased_until = NULL,
updated_at = now()
WHERE outbox_id = ${required(outboxId, "outboxId")}
AND status = 'pending'
AND leased_by = ${required(workerId, "workerId")}
RETURNING outbox_id`;
if (rows.length !== 1) {
throw new SiaxError("CONFLICT", "0pt.aud0_outbox", "outbox lease is no longer owned by this worker");
}
}
async stats(): Promise<Aud0OutboxStats> {
const rows = await this.sql<{
pending: string;
delivered: string;
dead: string;
oldest_pending_at: string | Date | null;
}[]>`
SELECT
count(*) FILTER (WHERE status = 'pending')::text AS pending,
count(*) FILTER (WHERE status = 'delivered')::text AS delivered,
count(*) FILTER (WHERE status = 'dead')::text AS dead,
min(created_at) FILTER (WHERE status = 'pending') AS oldest_pending_at
FROM opt_aud0_evidence_outbox`;
const row = rows[0] ?? { pending: "0", delivered: "0", dead: "0", oldest_pending_at: null };
return {
pending: Number(row.pending),
delivered: Number(row.delivered),
dead: Number(row.dead),
...(row.oldest_pending_at ? { oldest_pending_at: iso(row.oldest_pending_at) } : {}),
};
}
}
export function createAud0OutboxRepositoryFromEnv(
env: NodeJS.ProcessEnv = process.env,
): PostgresAud0OutboxRepository | undefined {
const url = repositoryDatabaseUrl(env);
return url ? new PostgresAud0OutboxRepository(getRepositorySql(url)) : undefined;
}
export interface Aud0OutboxSweepResult {
claimed: number;
delivered: number;
dead: number;
retried: number;
}
export class Aud0OutboxDispatcher {
constructor(
private readonly repo: Aud0OutboxRepository,
private readonly aud0: Aud0Client,
private readonly workerId = `0pt-${process.pid}-${crypto.randomUUID()}`,
) {}
async sweep(limit = 100): Promise<Aud0OutboxSweepResult> {
const items = await this.repo.claimPending({
workerId: this.workerId,
limit,
leaseMs: 60_000,
});
let delivered = 0;
let dead = 0;
let retried = 0;
for (const item of items) {
try {
if (item.evidence_kind === "event") {
const event = item.payload as Aud0Event;
if (event.tenant_id !== item.tenant_id || event.app_id !== item.app_id) {
throw new SiaxError(
"PERMANENT_FAILURE",
"0pt.aud0_outbox",
"queued AUD0 event tenant/app binding does not match its immutable outbox envelope",
);
}
await this.aud0.ingestEvent(event);
} else {
await this.aud0.ingestOpenLineageEvent(
item.tenant_id,
item.app_id,
item.payload as OpenLineageRunEvent,
);
}
await this.repo.markDelivered(item.outbox_id, this.workerId);
delivered += 1;
} catch (error) {
const description = safeError(error);
if (error instanceof SiaxError && !error.retryable) {
await this.repo.markDead(item.outbox_id, this.workerId, description);
dead += 1;
} else {
const exponent = Math.min(Math.max(item.attempts - 1, 0), 10);
const delayMs = Math.min(60 * 60_000, 1_000 * 2 ** exponent);
await this.repo.markRetry(item.outbox_id, this.workerId, description, delayMs);
retried += 1;
}
}
}
return { claimed: items.length, delivered, dead, retried };
}
}
export { openLineageIdempotencyKey, outboxId, safeError };
src/lineage/lineage-emitter.ts
+35 -31
View File
@@ -1,22 +1,11 @@
/**
* src/lineage/lineage-emitter.ts
* ===============================
* 0pt's OpenLineage emitter (Master-DoD §10 Lineage). One narrow API for
* capability services to record "this run read X and produced Y" against
* AUD0's real OpenLineage ingest (`POST /v1/aud0/openlineage` — see
* src/clients/aud0-client.ts for the wire).
* 0pt OpenLineage emitter.
*
* Design points:
* - Deterministic runIds (caller-supplied), so an idempotent retry of a
* capability re-emits the SAME event and AUD0 dedups it instead of
* recording a phantom second run.
* - Canonical SIAX resource refs (`st0re:artifact:...`, `0pt:dataset:...`)
* go in dataset facets so evidence chains can be joined across products
* (§16 "Evidence chains: Link source → transform → model → decision").
* - Emission is evidence of what already happened — a lineage POST failure
* is surfaced to the caller as a warning value, never an exception that
* rolls back the completed work. Callers that need hard evidence
* guarantees must use an outbox (tracked honestly in STATUS.md).
* Durability belongs to the AUD0 client boundary, not here. The default
* createAud0Client() persists to the local outbox when DATABASE_URL exists;
* the raw dispatcher client (durable:false) performs the actual HTTP delivery.
* This emitter therefore builds one canonical RunEvent and records it exactly
* once through the injected client.
*/
import type { Aud0Client } from "../clients/aud0-client";
import {
@@ -29,35 +18,30 @@ import {
export interface LineageEmitInput {
tenant_id: string;
project_id?: string;
/** The 0pt capability that ran, e.g. "0pt.quality.run.v1". Becomes job.name. */
capability_id: string;
/** Deterministic run id (e.g. the run/version/export id being recorded). */
run_id: string;
event_type: OpenLineageEventType;
/** ISO timestamp; defaults to now. */
event_time?: string;
/** Canonical SIAX refs and/or plain names consumed by the run. */
inputs?: LineageDatasetInput[];
/** Canonical SIAX refs and/or plain names produced by the run. */
outputs?: LineageDatasetInput[];
correlation_id?: string;
/** Extra run facets (namespaced siax*). */
run_facets?: Record<string, unknown>;
}
export interface LineageDatasetInput {
/** Human/system dataset name, e.g. "dataset/ds_x" or "opensearch/opt-companies__g2". */
name: string;
/** Optional canonical SIAX resource ref for cross-product joining. */
ref?: string;
namespace?: string;
facets?: Record<string, unknown>;
}
export interface LineageEmitResult {
/** True only when AUD0 itself acknowledged the event synchronously. */
emitted: boolean;
/** True when the event is durably queued for at-least-once AUD0 delivery. */
queued: boolean;
run_id: string;
/** Present when emission failed (AUD0 unreachable etc.). */
outbox_id?: string;
warning?: string;
}
@@ -74,6 +58,10 @@ function toDatasetRef(d: LineageDatasetInput): OpenLineageDatasetRef {
};
}
function safeWarning(error: unknown): string {
return error instanceof Error ? error.name : "UnknownError";
}
export class LineageEmitter {
constructor(private readonly aud0: Aud0Client) {}
@@ -96,17 +84,33 @@ export class LineageEmitter {
};
}
/** Emit one run event. Never throws — see module doc comment. */
/**
* Record one run event. Never throws after domain work has completed.
*
* A DurableAud0Client returns an acknowledgement whose event_id is the
* immutable `outbox:<id>` handle. A raw HTTP client returns AUD0's event id.
* This lets callers distinguish durable queueing from live acknowledgement
* without duplicating the outbox implementation in this layer.
*/
async emit(input: LineageEmitInput): Promise<LineageEmitResult> {
const event = this.buildRunEvent(input);
try {
await this.aud0.ingestOpenLineageEvent(input.tenant_id, APP_ID, event);
return { emitted: true, run_id: input.run_id };
} catch (err) {
const ack = await this.aud0.ingestOpenLineageEvent(input.tenant_id, APP_ID, event);
if (ack.event_id.startsWith("outbox:")) {
return {
emitted: false,
queued: true,
run_id: input.run_id,
outbox_id: ack.event_id.slice("outbox:".length),
};
}
return { emitted: true, queued: false, run_id: input.run_id };
} catch (error) {
return {
emitted: false,
queued: false,
run_id: input.run_id,
warning: `openlineage emission failed: ${err instanceof Error ? err.message : String(err)}`,
warning: `openlineage recording failed: ${safeWarning(error)}`,
};
}
}
src/repositories/dataset-repository.ts
+149
View File
@@ -0,0 +1,149 @@
import { resolveStoreMode } from "../platform/dependency-mode";
import { SiaxError } from "../platform/errors";
import type { CreateDatasetInput, Dataset, DatasetStatus, DatasetType } from "../types/dataset";
import { getRepositorySql, repositoryDatabaseUrl, type RepositorySql } from "./db";
export interface DatasetListFilter {
tenant_id?: string;
project_id?: string;
dataset_type?: DatasetType;
status?: DatasetStatus;
}
export interface DatasetRepository {
create(input: CreateDatasetInput): Promise<Dataset>;
get(datasetId: string): Promise<Dataset>;
list(filter?: DatasetListFilter): Promise<Dataset[]>;
}
interface DatasetRow {
dataset_id: string;
tenant_id: string;
project_id: string;
name: string;
dataset_type: DatasetType;
status: DatasetStatus;
description: string | null;
correlation_id: string | null;
created_at: string | Date;
updated_at: string | Date | null;
created_by: string | null;
}
function iso(value: string | Date): string {
return value instanceof Date ? value.toISOString() : new Date(value).toISOString();
}
function toDataset(row: DatasetRow): Dataset {
return {
dataset_id: row.dataset_id,
tenant_id: row.tenant_id,
project_id: row.project_id,
name: row.name,
dataset_type: row.dataset_type,
status: row.status,
...(row.description ? { description: row.description } : {}),
...(row.correlation_id ? { correlation_id: row.correlation_id } : {}),
created_at: iso(row.created_at),
...(row.updated_at ? { updated_at: iso(row.updated_at) } : {}),
...(row.created_by ? { created_by: row.created_by } : {}),
};
}
function validate(input: CreateDatasetInput): void {
if (!input.dataset_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "dataset_id is required");
if (!input.tenant_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required");
if (!input.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "project_id is required");
if (!input.name?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "name is required");
}
export class InMemoryDatasetRepository implements DatasetRepository {
private readonly rows = new Map<string, Dataset>();
async create(input: CreateDatasetInput): Promise<Dataset> {
validate(input);
if (this.rows.has(input.dataset_id)) {
throw new SiaxError("CONFLICT", "0pt.dataset", `dataset ${input.dataset_id} already exists`);
}
const dataset: Dataset = {
dataset_id: input.dataset_id,
tenant_id: input.tenant_id,
project_id: input.project_id,
name: input.name,
dataset_type: input.dataset_type,
status: input.status ?? "active",
...(input.description ? { description: input.description } : {}),
...(input.correlation_id ? { correlation_id: input.correlation_id } : {}),
created_at: new Date().toISOString(),
...(input.created_by ? { created_by: input.created_by } : {}),
};
this.rows.set(dataset.dataset_id, dataset);
return { ...dataset };
}
async get(datasetId: string): Promise<Dataset> {
const dataset = this.rows.get(datasetId);
if (!dataset) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} not found`);
return { ...dataset };
}
async list(filter?: DatasetListFilter): Promise<Dataset[]> {
return [...this.rows.values()]
.filter((dataset) => !filter?.tenant_id || dataset.tenant_id === filter.tenant_id)
.filter((dataset) => !filter?.project_id || dataset.project_id === filter.project_id)
.filter((dataset) => !filter?.dataset_type || dataset.dataset_type === filter.dataset_type)
.filter((dataset) => !filter?.status || dataset.status === filter.status)
.map((dataset) => ({ ...dataset }));
}
}
export class PostgresDatasetRepository implements DatasetRepository {
constructor(private readonly sql: RepositorySql) {}
async create(input: CreateDatasetInput): Promise<Dataset> {
validate(input);
try {
const rows = await this.sql<DatasetRow[]>`
INSERT INTO opt_datasets
(dataset_id, tenant_id, project_id, name, dataset_type, status,
description, correlation_id, created_by)
VALUES
(${input.dataset_id}, ${input.tenant_id}, ${input.project_id}, ${input.name},
${input.dataset_type}, ${input.status ?? "active"}, ${input.description ?? null},
${input.correlation_id ?? null}, ${input.created_by ?? null})
RETURNING *`;
return toDataset(rows[0]);
} catch (error) {
if ((error as { code?: string })?.code === "23505") {
throw new SiaxError("CONFLICT", "0pt.dataset", `dataset ${input.dataset_id} already exists`);
}
throw error;
}
}
async get(datasetId: string): Promise<Dataset> {
const rows = await this.sql<DatasetRow[]>`
SELECT * FROM opt_datasets WHERE dataset_id = ${datasetId} LIMIT 1`;
if (!rows[0]) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} not found`);
return toDataset(rows[0]);
}
async list(filter?: DatasetListFilter): Promise<Dataset[]> {
const rows = await this.sql<DatasetRow[]>`
SELECT *
FROM opt_datasets
WHERE (${filter?.tenant_id ?? null}::text IS NULL OR tenant_id = ${filter?.tenant_id ?? null})
AND (${filter?.project_id ?? null}::text IS NULL OR project_id = ${filter?.project_id ?? null})
AND (${filter?.dataset_type ?? null}::text IS NULL OR dataset_type = ${filter?.dataset_type ?? null})
AND (${filter?.status ?? null}::text IS NULL OR status = ${filter?.status ?? null})
ORDER BY created_at DESC, dataset_id ASC`;
return rows.map(toDataset);
}
}
export function createDatasetRepository(databaseUrl?: string): DatasetRepository {
const mode = resolveStoreMode("0pt.datasets", databaseUrl ?? repositoryDatabaseUrl());
return mode.mode === "postgres"
? new PostgresDatasetRepository(getRepositorySql(mode.url))
: new InMemoryDatasetRepository();
}
src/repositories/dataset-version-repository.ts
+177
View File
@@ -0,0 +1,177 @@
import { resolveStoreMode } from "../platform/dependency-mode";
import { SiaxError } from "../platform/errors";
import type { CreateDatasetVersionInput, DatasetVersion } from "../types/dataset-version";
import { getRepositorySql, repositoryDatabaseUrl, type RepositorySql } from "./db";
export interface DatasetVersionRepository {
create(input: CreateDatasetVersionInput): Promise<DatasetVersion>;
list(datasetId: string, tenantId: string): Promise<DatasetVersion[]>;
get(datasetId: string, tenantId: string, version: number): Promise<DatasetVersion>;
}
interface VersionRow {
dataset_id: string;
version: number;
tenant_id: string;
project_id: string;
artifact_refs: string[];
schema_version: string;
record_count: string | number | null;
description: string | null;
created_at: string | Date;
created_by: string;
correlation_id: string | null;
idempotency_key: string | null;
}
function toVersion(row: VersionRow): DatasetVersion {
return {
dataset_id: row.dataset_id,
version: Number(row.version),
tenant_id: row.tenant_id,
project_id: row.project_id,
artifact_refs: [...row.artifact_refs],
schema_version: row.schema_version,
...(row.record_count == null ? {} : { record_count: Number(row.record_count) }),
...(row.description ? { description: row.description } : {}),
created_at: row.created_at instanceof Date ? row.created_at.toISOString() : new Date(row.created_at).toISOString(),
created_by: row.created_by,
...(row.correlation_id ? { correlation_id: row.correlation_id } : {}),
};
}
function normalizedCount(value: string | number | null | undefined): number | undefined {
return value == null ? undefined : Number(value);
}
function sameInput(row: VersionRow | DatasetVersion, input: CreateDatasetVersionInput): boolean {
return row.dataset_id === input.dataset_id
&& row.tenant_id === input.tenant_id
&& row.project_id === input.project_id
&& row.schema_version === input.schema_version
&& normalizedCount(row.record_count) === normalizedCount(input.record_count)
&& (row.description ?? undefined) === (input.description ?? undefined)
&& row.created_by === input.created_by
&& JSON.stringify(row.artifact_refs) === JSON.stringify(input.artifact_refs);
}
export class InMemoryDatasetVersionRepository implements DatasetVersionRepository {
private readonly versions = new Map<string, DatasetVersion[]>();
private readonly idempotency = new Map<string, DatasetVersion>();
async create(input: CreateDatasetVersionInput): Promise<DatasetVersion> {
if (input.idempotency_key) {
const existing = this.idempotency.get(`${input.tenant_id}:${input.idempotency_key}`);
if (existing) {
if (!sameInput(existing, input)) {
throw new SiaxError("CONFLICT", "0pt.dataset", "idempotency key reused with different dataset-version input");
}
return { ...existing, artifact_refs: [...existing.artifact_refs] };
}
}
const list = this.versions.get(input.dataset_id) ?? [];
const version: DatasetVersion = {
dataset_id: input.dataset_id,
version: (list[list.length - 1]?.version ?? 0) + 1,
tenant_id: input.tenant_id,
project_id: input.project_id,
artifact_refs: [...input.artifact_refs],
schema_version: input.schema_version,
...(input.record_count === undefined ? {} : { record_count: input.record_count }),
...(input.description ? { description: input.description } : {}),
created_at: new Date().toISOString(),
created_by: input.created_by,
...(input.correlation_id ? { correlation_id: input.correlation_id } : {}),
};
list.push(version);
this.versions.set(input.dataset_id, list);
if (input.idempotency_key) this.idempotency.set(`${input.tenant_id}:${input.idempotency_key}`, version);
return { ...version, artifact_refs: [...version.artifact_refs] };
}
async list(datasetId: string, tenantId: string): Promise<DatasetVersion[]> {
return (this.versions.get(datasetId) ?? [])
.filter((version) => version.tenant_id === tenantId)
.map((version) => ({ ...version, artifact_refs: [...version.artifact_refs] }));
}
async get(datasetId: string, tenantId: string, version: number): Promise<DatasetVersion> {
const found = (await this.list(datasetId, tenantId)).find((item) => item.version === version);
if (!found) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} has no version ${version}`);
return found;
}
}
export class PostgresDatasetVersionRepository implements DatasetVersionRepository {
constructor(private readonly sql: RepositorySql) {}
async create(input: CreateDatasetVersionInput): Promise<DatasetVersion> {
return this.sql.begin(async (tx) => {
const parent = await tx<{ dataset_id: string; tenant_id: string }[]>`
SELECT dataset_id, tenant_id
FROM opt_datasets
WHERE dataset_id = ${input.dataset_id}
FOR UPDATE`;
if (!parent[0] || parent[0].tenant_id !== input.tenant_id) {
throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${input.dataset_id} not found`);
}
if (input.idempotency_key) {
const existing = await tx<VersionRow[]>`
SELECT * FROM opt_dataset_versions
WHERE tenant_id = ${input.tenant_id}
AND idempotency_key = ${input.idempotency_key}
LIMIT 1`;
if (existing[0]) {
if (!sameInput(existing[0], input)) {
throw new SiaxError("CONFLICT", "0pt.dataset", "idempotency key reused with different dataset-version input");
}
return toVersion(existing[0]);
}
}
const next = await tx<{ version: number }[]>`
SELECT COALESCE(max(version), 0)::int + 1 AS version
FROM opt_dataset_versions
WHERE dataset_id = ${input.dataset_id}`;
const version = next[0]?.version ?? 1;
const rows = await tx<VersionRow[]>`
INSERT INTO opt_dataset_versions
(dataset_id, version, tenant_id, project_id, artifact_refs,
schema_version, record_count, description, created_by,
correlation_id, idempotency_key)
VALUES
(${input.dataset_id}, ${version}, ${input.tenant_id}, ${input.project_id},
${tx.array([...input.artifact_refs])}, ${input.schema_version}, ${input.record_count ?? null},
${input.description ?? null}, ${input.created_by}, ${input.correlation_id ?? null},
${input.idempotency_key ?? null})
RETURNING *`;
return toVersion(rows[0]);
});
}
async list(datasetId: string, tenantId: string): Promise<DatasetVersion[]> {
const rows = await this.sql<VersionRow[]>`
SELECT * FROM opt_dataset_versions
WHERE dataset_id = ${datasetId} AND tenant_id = ${tenantId}
ORDER BY version ASC`;
return rows.map(toVersion);
}
async get(datasetId: string, tenantId: string, version: number): Promise<DatasetVersion> {
const rows = await this.sql<VersionRow[]>`
SELECT * FROM opt_dataset_versions
WHERE dataset_id = ${datasetId} AND tenant_id = ${tenantId} AND version = ${version}
LIMIT 1`;
if (!rows[0]) throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} has no version ${version}`);
return toVersion(rows[0]);
}
}
export function createDatasetVersionRepository(databaseUrl?: string): DatasetVersionRepository {
const mode = resolveStoreMode("0pt.dataset_versions", databaseUrl ?? repositoryDatabaseUrl());
return mode.mode === "postgres"
? new PostgresDatasetVersionRepository(getRepositorySql(mode.url))
: new InMemoryDatasetVersionRepository();
}
src/repositories/search-definition-repository.ts
+186
View File
@@ -0,0 +1,186 @@
import { resolveStoreMode } from "../platform/dependency-mode";
import { SiaxError } from "../platform/errors";
import type { SearchIndexDefinition, SearchFieldType } from "../search/search-types";
import { getRepositorySql, repositoryDatabaseUrl, type RepositorySql } from "./db";
export interface SearchDefinitionSaveResult {
definition: SearchIndexDefinition;
created: boolean;
created_at: string;
created_by?: string;
}
export interface SearchDefinitionRepository {
save(definition: SearchIndexDefinition, actorId?: string): Promise<SearchDefinitionSaveResult>;
latest(indexName: string): Promise<SearchIndexDefinition | undefined>;
listLatest(): Promise<SearchIndexDefinition[]>;
}
interface DefinitionRow {
index_name: string;
version: number;
fields: Record<string, SearchFieldType>;
default_search_fields: string[];
created_at: string | Date;
created_by: string | null;
}
function iso(value: string | Date): string {
return value instanceof Date ? value.toISOString() : new Date(value).toISOString();
}
function toDefinition(row: DefinitionRow): SearchIndexDefinition {
return {
index_name: row.index_name,
version: Number(row.version),
fields: { ...row.fields },
default_search_fields: [...row.default_search_fields],
};
}
function toSaveResult(row: DefinitionRow, created: boolean): SearchDefinitionSaveResult {
return {
definition: toDefinition(row),
created,
created_at: iso(row.created_at),
...(row.created_by ? { created_by: row.created_by } : {}),
};
}
function fingerprint(definition: SearchIndexDefinition): string {
return JSON.stringify({
index_name: definition.index_name,
version: definition.version,
fields: Object.fromEntries(Object.entries(definition.fields).sort(([a], [b]) => a.localeCompare(b))),
default_search_fields: [...definition.default_search_fields],
});
}
interface MemoryRecord {
definition: SearchIndexDefinition;
created_at: string;
created_by?: string;
}
export class InMemorySearchDefinitionRepository implements SearchDefinitionRepository {
private readonly versions = new Map<string, MemoryRecord[]>();
async save(definition: SearchIndexDefinition, actorId?: string): Promise<SearchDefinitionSaveResult> {
const list = this.versions.get(definition.index_name) ?? [];
const existing = list.find((item) => item.definition.version === definition.version);
if (existing) {
if (fingerprint(existing.definition) !== fingerprint(definition)) {
throw new SiaxError("CONFLICT", "0pt.search", "search definition version reused with different schema");
}
return {
definition: {
...existing.definition,
fields: { ...existing.definition.fields },
default_search_fields: [...existing.definition.default_search_fields],
},
created: false,
created_at: existing.created_at,
...(existing.created_by ? { created_by: existing.created_by } : {}),
};
}
const expected = (list[list.length - 1]?.definition.version ?? 0) + 1;
if (definition.version !== expected) {
throw new SiaxError("CONFLICT", "0pt.search", `next search definition version must be ${expected}`);
}
const copy = {
...definition,
fields: { ...definition.fields },
default_search_fields: [...definition.default_search_fields],
};
const record: MemoryRecord = {
definition: copy,
created_at: new Date().toISOString(),
...(actorId ? { created_by: actorId } : {}),
};
list.push(record);
this.versions.set(definition.index_name, list);
return { definition: copy, created: true, created_at: record.created_at, ...(actorId ? { created_by: actorId } : {}) };
}
async latest(indexName: string): Promise<SearchIndexDefinition | undefined> {
const list = this.versions.get(indexName) ?? [];
const item = list[list.length - 1]?.definition;
return item ? { ...item, fields: { ...item.fields }, default_search_fields: [...item.default_search_fields] } : undefined;
}
async listLatest(): Promise<SearchIndexDefinition[]> {
const result: SearchIndexDefinition[] = [];
for (const key of this.versions.keys()) {
const latest = await this.latest(key);
if (latest) result.push(latest);
}
return result.sort((a, b) => a.index_name.localeCompare(b.index_name));
}
}
export class PostgresSearchDefinitionRepository implements SearchDefinitionRepository {
constructor(private readonly sql: RepositorySql) {}
async save(definition: SearchIndexDefinition, actorId?: string): Promise<SearchDefinitionSaveResult> {
return this.sql.begin(async (tx) => {
await tx`SELECT pg_advisory_xact_lock(hashtext(${`0pt-search:${definition.index_name}`}))`;
const same = await tx<DefinitionRow[]>`
SELECT index_name, version, fields, default_search_fields, created_at, created_by
FROM opt_search_index_definitions
WHERE index_name = ${definition.index_name} AND version = ${definition.version}
LIMIT 1`;
if (same[0]) {
const existing = toDefinition(same[0]);
if (fingerprint(existing) !== fingerprint(definition)) {
throw new SiaxError("CONFLICT", "0pt.search", "search definition version reused with different schema");
}
return toSaveResult(same[0], false);
}
const latest = await tx<{ version: number }[]>`
SELECT COALESCE(max(version), 0)::int AS version
FROM opt_search_index_definitions
WHERE index_name = ${definition.index_name}`;
const expected = (latest[0]?.version ?? 0) + 1;
if (definition.version !== expected) {
throw new SiaxError("CONFLICT", "0pt.search", `next search definition version must be ${expected}`);
}
const rows = await tx<DefinitionRow[]>`
INSERT INTO opt_search_index_definitions
(index_name, version, fields, default_search_fields, created_by)
VALUES
(${definition.index_name}, ${definition.version},
${tx.json(definition.fields as Parameters<RepositorySql["json"]>[0])},
${tx.array(definition.default_search_fields)}, ${actorId ?? null})
RETURNING index_name, version, fields, default_search_fields, created_at, created_by`;
return toSaveResult(rows[0], true);
});
}
async latest(indexName: string): Promise<SearchIndexDefinition | undefined> {
const rows = await this.sql<DefinitionRow[]>`
SELECT index_name, version, fields, default_search_fields, created_at, created_by
FROM opt_search_index_definitions
WHERE index_name = ${indexName}
ORDER BY version DESC
LIMIT 1`;
return rows[0] ? toDefinition(rows[0]) : undefined;
}
async listLatest(): Promise<SearchIndexDefinition[]> {
const rows = await this.sql<DefinitionRow[]>`
SELECT DISTINCT ON (index_name)
index_name, version, fields, default_search_fields, created_at, created_by
FROM opt_search_index_definitions
ORDER BY index_name, version DESC`;
return rows.map(toDefinition);
}
}
export function createSearchDefinitionRepository(databaseUrl?: string): SearchDefinitionRepository {
const mode = resolveStoreMode("0pt.search_definitions", databaseUrl ?? repositoryDatabaseUrl());
return mode.mode === "postgres"
? new PostgresSearchDefinitionRepository(getRepositorySql(mode.url))
: new InMemorySearchDefinitionRepository();
}
src/search/search-service.ts
+134 -87
View File
@@ -1,35 +1,15 @@
/**
* src/search/search-service.ts
* =============================
* 0pt's Search capability (Master-DoD §10 Search):
* 0pt Search capability.
*
* search.index -> registerIndexDefinition + indexDocuments (bulk)
* search.query -> query (tenant + purpose/rights filtering ENFORCED)
* search.rebuild -> rebuild (new generation, bulk load, validate)
* search.alias.switch -> switchGeneration (atomic alias cutover)
*
* Non-negotiables encoded here:
* - OpenSearch is NEVER canonical truth (§42): a rebuild takes a caller-
* supplied `DocumentSource` (an async iterator over canonical data) and
* can rebuild any index from an empty cluster. Deleting every index
* loses nothing canonical.
* - Tenant filter is applied by THIS service on every query — a caller
* cannot opt out; a missing tenant_id is INVALID_REQUEST (§31).
* - Purpose/rights filtering (§10): every document carries purpose_tags +
* rights_tags (mandatory on the envelope); every query names its purpose
* and gets only documents tagged for it. Documents with an EMPTY
* purpose_tags list are never returnable — fail closed.
* - Generation switch: physical indices are `opt-{name}--g{n}`; the alias
* `opt-{name}` points at exactly one generation; cutover is atomic
* (§42 generation isolation + cutover).
*
* Lineage: every completed rebuild emits an OpenLineage COMPLETE run to AUD0
* (inputs: the document source name; outputs: the physical index).
* OpenSearch remains a disposable projection. Logical index definitions may be
* persisted through SearchDefinitionRepository so a process restart never
* forgets how to query or rebuild an otherwise valid projection.
*/
import { SiaxError } from "../platform/errors";
import type { OpenSearchAdapter } from "./opensearch-adapter";
import { LineageEmitter } from "../lineage/lineage-emitter";
import type { Aud0Client } from "../clients/aud0-client";
import { LineageEmitter } from "../lineage/lineage-emitter";
import { SiaxError } from "../platform/errors";
import type { SearchDefinitionRepository } from "../repositories/search-definition-repository";
import type { OpenSearchAdapter } from "./opensearch-adapter";
import {
MAX_SEARCH_PAGE_SIZE,
type BulkIndexResult,
@@ -41,9 +21,7 @@ import {
type SearchResult,
} from "./search-types";
/** An async source of canonical documents for (re)builds. */
export interface DocumentSource {
/** Name for lineage, e.g. "postgres:opt_datasets" or "0pt:dataset:ds_x". */
name: string;
iterate(): AsyncIterable<SearchDocument>;
}
@@ -59,24 +37,34 @@ const ENVELOPE_MAPPINGS: Record<string, unknown> = {
export class SearchService {
private readonly definitions = new Map<string, SearchIndexDefinition>();
private readonly lineage?: LineageEmitter;
private readonly aud0?: Aud0Client;
private readonly definitionRepository?: SearchDefinitionRepository;
constructor(
private readonly adapter: OpenSearchAdapter,
opts?: { aud0Client?: Aud0Client }
opts?: { aud0Client?: Aud0Client; definitionRepository?: SearchDefinitionRepository },
) {
this.aud0 = opts?.aud0Client;
this.lineage = opts?.aud0Client ? new LineageEmitter(opts.aud0Client) : undefined;
this.definitionRepository = opts?.definitionRepository;
}
// ---------------------------------------------------------------------
// Index definitions
// ---------------------------------------------------------------------
registerIndexDefinition(def: SearchIndexDefinition): SearchIndexDefinition {
private validateDefinition(def: SearchIndexDefinition): void {
if (!def.index_name?.trim() || !/^[a-z0-9][a-z0-9_-]{0,62}$/i.test(def.index_name)) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "index_name must be 1-63 letters, digits, _ or -");
}
if (!Number.isSafeInteger(def.version) || def.version < 1) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "definition version must be a positive integer");
}
if (Object.keys(def.fields).length === 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "an index definition needs at least one field");
}
for (const f of def.default_search_fields) {
if (def.fields[f] !== "text") {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.search",
`default_search_field "${f}" must be a text field in index "${def.index_name}"`
`default_search_field "${f}" must be a text field in index "${def.index_name}"`,
);
}
}
@@ -85,22 +73,85 @@ export class SearchService {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.search",
`field "${reserved}" is reserved by the 0pt search envelope`
`field "${reserved}" is reserved by the 0pt search envelope`,
);
}
}
this.definitions.set(def.index_name, def);
return def;
}
/** In-memory registration retained for hermetic unit tests and local callers. */
registerIndexDefinition(def: SearchIndexDefinition): SearchIndexDefinition {
this.validateDefinition(def);
const copy = {
...def,
fields: { ...def.fields },
default_search_fields: [...def.default_search_fields],
};
this.definitions.set(def.index_name, copy);
return copy;
}
/** Production registration: durable versioned write, one stable audit event, then cache refresh. */
async registerIndexDefinitionDurable(def: SearchIndexDefinition, actorId?: string): Promise<SearchIndexDefinition> {
this.validateDefinition(def);
if (!this.definitionRepository) return this.registerIndexDefinition(def);
const stored = await this.definitionRepository.save(def, actorId);
const registered = this.registerIndexDefinition(stored.definition);
if (stored.created && this.aud0) {
const id = `search-definition:${registered.index_name}:v${registered.version}`;
await this.aud0.ingestEvent({
event_id: `evt_0pt_${id}`,
event_type: "0pt.search_definition_registered",
tenant_id: "t_platform",
project_id: "p_platform",
actor_id: stored.created_by ?? actorId ?? "system",
app_id: "0pt",
capability_id: "0pt.search.definition.register.v1",
resource_type: "search_index_definition",
resource_id: `${registered.index_name}@${registered.version}`,
risk_level: "L2",
result: "success",
payload: {
index_name: registered.index_name,
version: registered.version,
fields: Object.keys(registered.fields).sort(),
default_search_fields: registered.default_search_fields,
},
created_at: stored.created_at,
occurred_at: stored.created_at,
idempotency_key: id,
});
}
return registered;
}
getIndexDefinition(indexName: string): SearchIndexDefinition {
const def = this.definitions.get(indexName);
if (!def) throw new SiaxError("NOT_FOUND", "0pt.search", `no index definition "${indexName}"`);
return def;
return { ...def, fields: { ...def.fields }, default_search_fields: [...def.default_search_fields] };
}
async resolveIndexDefinition(indexName: string): Promise<SearchIndexDefinition> {
const cached = this.definitions.get(indexName);
if (cached) return this.getIndexDefinition(indexName);
const stored = await this.definitionRepository?.latest(indexName);
if (!stored) throw new SiaxError("NOT_FOUND", "0pt.search", `no index definition "${indexName}"`);
return this.registerIndexDefinition(stored);
}
listIndexDefinitions(): SearchIndexDefinition[] {
return [...this.definitions.values()];
return [...this.definitions.values()].map((def) => ({
...def,
fields: { ...def.fields },
default_search_fields: [...def.default_search_fields],
}));
}
async listIndexDefinitionsDurable(): Promise<SearchIndexDefinition[]> {
if (!this.definitionRepository) return this.listIndexDefinitions();
const definitions = await this.definitionRepository.listLatest();
for (const def of definitions) this.registerIndexDefinition(def);
return definitions;
}
aliasFor(indexName: string): string {
@@ -113,51 +164,49 @@ export class SearchService {
private mappingsFor(def: SearchIndexDefinition): Record<string, unknown> {
const properties: Record<string, unknown> = { ...ENVELOPE_MAPPINGS };
for (const [field, type] of Object.entries(def.fields)) {
properties[field] = { type };
}
for (const [field, type] of Object.entries(def.fields)) properties[field] = { type };
return { properties, _meta: { opt_definition_version: def.version } };
}
// ---------------------------------------------------------------------
// search.index — bulk write into the ACTIVE generation
// ---------------------------------------------------------------------
async indexDocuments(indexName: string, docs: SearchDocument[]): Promise<BulkIndexResult> {
const def = this.getIndexDefinition(indexName);
const def = await this.resolveIndexDefinition(indexName);
const state = await this.generationState(indexName);
if (state.active_generation === undefined) {
throw new SiaxError(
"CONFLICT",
"0pt.search",
`index "${indexName}" has no active generation — run a rebuild first`
`index "${indexName}" has no active generation — run a rebuild first`,
);
}
this.validateDocuments(def, docs);
const physical = this.physicalIndexFor(indexName, state.active_generation);
const result = await this.adapter.bulkIndex(
physical,
docs.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) }))
docs.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) })),
);
return { indexed: docs.length - result.errors.length, errors: result.errors };
}
// ---------------------------------------------------------------------
// search.query — tenant + purpose/rights filtering ENFORCED here
// ---------------------------------------------------------------------
async query(indexName: string, q: SearchQuery): Promise<SearchResult> {
const def = this.getIndexDefinition(indexName);
const def = await this.resolveIndexDefinition(indexName);
if (!q.tenant_id?.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "tenant_id is required on every search query");
}
if (!q.purpose?.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "purpose is required on every search query (§10)");
}
const size = Math.min(q.size ?? 25, MAX_SEARCH_PAGE_SIZE);
const from = q.from ?? 0;
const requestedSize = q.size ?? 25;
if (!Number.isSafeInteger(from) || from < 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "from must be a non-negative integer");
}
if (!Number.isSafeInteger(requestedSize) || requestedSize < 1) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "size must be a positive integer");
}
const size = Math.min(requestedSize, MAX_SEARCH_PAGE_SIZE);
const filter: unknown[] = [
{ term: { tenant_id: q.tenant_id } },
// Purpose fit: the document must be tagged for THIS query's purpose.
// Docs with empty purpose_tags match nothing — fail closed by shape.
{ term: { purpose_tags: q.purpose } },
{ term: { rights_tags: q.rights_tag ?? "internal" } },
];
@@ -171,10 +220,9 @@ export class SearchService {
const must = q.text?.trim()
? [{ multi_match: { query: q.text, fields: def.default_search_fields } }]
: [{ match_all: {} }];
const res = await this.adapter.search(this.aliasFor(indexName), {
query: { bool: { must, filter } },
from: q.from ?? 0,
from,
size,
});
@@ -196,30 +244,27 @@ export class SearchService {
};
}
// ---------------------------------------------------------------------
// search.rebuild — new generation from canonical truth (§42)
// ---------------------------------------------------------------------
async rebuild(
indexName: string,
source: DocumentSource,
opts?: {
/** Switch the alias to the new generation when done (default true). */
switch?: boolean;
/** Delete superseded generations after a successful switch (default false — keep for rollback). */
prune?: boolean;
batchSize?: number;
tenant_id?: string;
correlation_id?: string;
}
},
): Promise<RebuildResult> {
const def = this.getIndexDefinition(indexName);
const def = await this.resolveIndexDefinition(indexName);
const state = await this.generationState(indexName);
const newGeneration = (state.active_generation ?? 0) + 1;
const physical = this.physicalIndexFor(indexName, newGeneration);
const batchSize = opts?.batchSize ?? 500;
if (!Number.isSafeInteger(batchSize) || batchSize < 1 || batchSize > 5_000) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "batchSize must be between 1 and 5000");
}
await this.adapter.createIndex(physical, this.mappingsFor(def));
const batchSize = opts?.batchSize ?? 500;
let batch: SearchDocument[] = [];
let docsIndexed = 0;
const flush = async () => {
@@ -227,28 +272,31 @@ export class SearchService {
this.validateDocuments(def, batch);
const res = await this.adapter.bulkIndex(
physical,
batch.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) }))
batch.map((d) => ({ id: this.docKey(d), body: this.toIndexBody(d) })),
);
if (res.errors.length > 0) {
// A rebuild must be all-or-nothing: destroy the failed generation and
// leave the active alias untouched (generation isolation, §42).
await this.adapter.deleteIndex(physical);
throw new SiaxError(
"RETRYABLE_FAILURE",
"0pt.search",
`rebuild of "${indexName}" g${newGeneration} failed on ${res.errors.length} document(s); generation discarded`
`rebuild of "${indexName}" g${newGeneration} failed on ${res.errors.length} document(s); generation discarded`,
);
}
docsIndexed += batch.length;
batch = [];
};
for await (const doc of source.iterate()) {
batch.push(doc);
if (batch.length >= batchSize) await flush();
try {
for await (const doc of source.iterate()) {
batch.push(doc);
if (batch.length >= batchSize) await flush();
}
await flush();
await this.adapter.refresh(physical);
} catch (error) {
await this.adapter.deleteIndex(physical).catch(() => undefined);
throw error;
}
await flush();
await this.adapter.refresh(physical);
const shouldSwitch = opts?.switch !== false;
if (shouldSwitch) {
@@ -280,10 +328,11 @@ export class SearchService {
};
}
// ---------------------------------------------------------------------
// search.alias.switch — explicit generation cutover / rollback
// ---------------------------------------------------------------------
async switchGeneration(indexName: string, generation: number): Promise<IndexGenerationState> {
if (!Number.isSafeInteger(generation) || generation < 1) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "generation must be a positive integer");
}
await this.resolveIndexDefinition(indexName);
const physical = this.physicalIndexFor(indexName, generation);
if (!(await this.adapter.indexExists(physical))) {
throw new SiaxError("NOT_FOUND", "0pt.search", `generation g${generation} of "${indexName}" does not exist`);
@@ -308,12 +357,7 @@ export class SearchService {
};
}
// ---------------------------------------------------------------------
// internals
// ---------------------------------------------------------------------
private docKey(d: SearchDocument): string {
// Physical _id is tenant-prefixed so two tenants can use the same doc_id
// without collision — defense in depth under the tenant term filter.
return `${d.tenant_id}:${d.doc_id}`;
}
@@ -330,10 +374,13 @@ export class SearchService {
private validateDocuments(def: SearchIndexDefinition, docs: SearchDocument[]): void {
for (const d of docs) {
if (!d.doc_id?.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", "every document requires doc_id");
}
if (!d.tenant_id?.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", `document ${d.doc_id} is missing tenant_id`);
}
if (!Array.isArray(d.purpose_tags)) {
if (!Array.isArray(d.purpose_tags) || d.purpose_tags.length === 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.search", `document ${d.doc_id} is missing purpose_tags`);
}
if (!Array.isArray(d.rights_tags) || d.rights_tags.length === 0) {
@@ -344,7 +391,7 @@ export class SearchService {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.search",
`document ${d.doc_id} has field "${field}" not in index definition "${def.index_name}" v${def.version}`
`document ${d.doc_id} has field "${field}" not in index definition "${def.index_name}" v${def.version}`,
);
}
}
src/services/dataset-service.ts
+42 -107
View File
@@ -1,116 +1,72 @@
/**
* src/services/dataset-service.ts
* ================================
* Owns 0pt.dataset.schema.v1 — the registry of 0pt datasets (raw / structured
* / enriched / curated / feature / derived outputs). This is the thing
* `DatasetIcebergMappingService` (src/services/dataset-iceberg-mapping-service.ts)
* and `IcebergWriteService` (src/services/iceberg-write-service.ts) look a
* `dataset_id` up against before pointing an Iceberg table mapping or a write
* at it.
* Owns 0pt.dataset.schema.v1 — the canonical registry of 0pt datasets.
*
* Deliberately small: it is a reference/registry entry, NOT dataset content,
* schema, or quality state — see docs/ICEBERG_WRITE_BOUNDARY.md for that
* boundary. In-memory reference implementation (same convention as the other
* 0pt services in this repo; a real deployment backs this with `opt_datasets`).
*
* In-memory scope: each `new DatasetService()` owns its own registry, the same
* pre-existing limitation source-service.ts and every other P0 registry
* service documents (docs/P0_QUERY_SOURCE_FLOW.md's "Known gaps"). Callers
* that need a shared view across API routes pass an existing instance in via
* the `datasetService` constructor option — see
* src/services/lakehouse-mapping-registry.ts, which does exactly that for the
* whole app/api/0pt/lakehouse/mappings route group.
*
* Registering a dataset emits a best-effort AUD0 `0pt.dataset_registered`
* event; with no AUD0_URL configured that emission is a no-op (see
* aud0-client.ts), so this stays hermetic in tests and local dev.
* Storage is Postgres-backed through DatasetRepository whenever DATABASE_URL
* is configured. Non-production tests/local development may use the in-memory
* repository; production without a persistent store fails at construction via
* resolveStoreMode. Dataset content remains outside this registry.
*/
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
import {
createDatasetRepository,
type DatasetListFilter,
type DatasetRepository,
} from "../repositories/dataset-repository";
import type { Dataset, DatasetStatus, DatasetType, CreateDatasetInput } from "../types/dataset";
// Re-exported so the historical `import { Dataset, DatasetType, ... } from
// "./dataset-service"` call shape keeps working alongside the canonical
// `../types/dataset` module.
export type { Dataset, DatasetStatus, DatasetType, CreateDatasetInput };
/** Alias so both historical input-type names resolve to the same shape. */
export type RegisterDatasetInput = CreateDatasetInput;
/**
* Storage shape: nullable columns, exactly as the eventual `opt_datasets`
* table would hold them. `toDataset()` converts back to the `Dataset` DTO.
*/
interface DatasetRecord {
dataset_id: string;
tenant_id: string;
project_id: string;
name: string;
dataset_type: string;
status: string;
description: string | null;
correlation_id: string | null;
created_at: string;
updated_at: string | null;
created_by: string | null;
export interface DatasetServiceOptions {
aud0BaseUrl?: string;
databaseUrl?: string;
repository?: DatasetRepository;
}
function normalizeOptions(input?: string | DatasetServiceOptions): DatasetServiceOptions {
return typeof input === "string" ? { aud0BaseUrl: input } : input ?? {};
}
export class DatasetService {
private datasets: Map<string, DatasetRecord> = new Map();
private aud0: Aud0Client;
private readonly repo: DatasetRepository;
private readonly aud0: Aud0Client;
constructor(aud0BaseUrl?: string) {
this.aud0 = createAud0Client(aud0BaseUrl);
/**
* A string argument remains supported as the historical aud0BaseUrl form;
* new callers should use the options object so repository/database injection
* is explicit and testable.
*/
constructor(input?: string | DatasetServiceOptions) {
const opts = normalizeOptions(input);
this.aud0 = createAud0Client(opts.aud0BaseUrl);
this.repo = opts.repository ?? createDatasetRepository(opts.databaseUrl);
}
async registerDataset(input: CreateDatasetInput): Promise<Dataset> {
if (this.datasets.has(input.dataset_id)) {
throw new Error(`Dataset ${input.dataset_id} already exists`);
}
if (!input.name) {
throw new Error("name is required");
}
const dataset = await this.repo.create(input);
const now = new Date().toISOString();
const record: DatasetRecord = {
dataset_id: input.dataset_id,
tenant_id: input.tenant_id,
project_id: input.project_id,
name: input.name,
dataset_type: input.dataset_type,
status: input.status ?? "active",
description: input.description ?? null,
correlation_id: input.correlation_id ?? null,
created_at: now,
updated_at: null,
created_by: input.created_by ?? null,
};
this.datasets.set(record.dataset_id, record);
// 0pt.dataset_registered -- same "no mutation without an evented trail"
// doctrine every other P0 registry service in this repo follows
// (SourceService.createSource, SourceRatePolicyService.createSourceRatePolicy).
await this.aud0.ingestEvent({
event_id: `evt_0pt_dataset_registered_${record.dataset_id}`,
event_id: `evt_0pt_dataset_registered_${dataset.dataset_id}`,
event_type: "0pt.dataset_registered",
tenant_id: record.tenant_id,
project_id: record.project_id,
actor_id: record.created_by ?? "system",
tenant_id: dataset.tenant_id,
project_id: dataset.project_id,
actor_id: dataset.created_by ?? "system",
app_id: "0pt",
capability_id: "0pt.register_dataset",
resource_type: "dataset",
resource_id: record.dataset_id,
resource_id: dataset.dataset_id,
risk_level: "L2",
payload: { name: record.name, dataset_type: record.dataset_type },
created_at: now,
correlation_id: record.correlation_id ?? undefined,
payload: { name: dataset.name, dataset_type: dataset.dataset_type },
created_at: dataset.created_at,
correlation_id: dataset.correlation_id,
idempotency_key: `dataset-register:${dataset.tenant_id}:${dataset.dataset_id}`,
});
return this.toDataset(record);
return dataset;
}
async getDataset(datasetId: string): Promise<Dataset> {
const record = this.datasets.get(datasetId);
if (!record) throw new Error(`Dataset ${datasetId} not found`);
return this.toDataset(record);
return this.repo.get(datasetId);
}
async listDatasets(filter?: {
@@ -119,27 +75,6 @@ export class DatasetService {
dataset_type?: DatasetType;
status?: DatasetStatus;
}): Promise<Dataset[]> {
let results = Array.from(this.datasets.values());
if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id);
if (filter?.project_id) results = results.filter((r) => r.project_id === filter.project_id);
if (filter?.dataset_type) results = results.filter((r) => r.dataset_type === filter.dataset_type);
if (filter?.status) results = results.filter((r) => r.status === filter.status);
return results.map((r) => this.toDataset(r));
}
private toDataset(r: DatasetRecord): Dataset {
return {
dataset_id: r.dataset_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
name: r.name,
dataset_type: r.dataset_type as DatasetType,
status: r.status as DatasetStatus,
description: r.description ?? undefined,
correlation_id: r.correlation_id ?? undefined,
created_at: r.created_at,
updated_at: r.updated_at ?? undefined,
created_by: r.created_by ?? undefined,
};
return this.repo.list(filter as DatasetListFilter | undefined);
}
}
src/services/dataset-version-service.ts
+69 -113
View File
@@ -1,96 +1,62 @@
/**
* src/services/dataset-version-service.ts
* ========================================
* P0 Dataset capability extensions (Master-DoD §10 Dataset:
* dataset.create/version/query/export) layered ON TOP of DatasetService (the
* owner of 0pt.dataset.schema.v1 registry entries — deliberately untouched:
* riv inte fungerande kod).
* Dataset version/query/export capability.
*
* dataset.version -> createVersion: an immutable, monotonically numbered
* version whose artifacts are CANONICAL ST0RE REFS
* (`st0re:artifact:art_x` — Master-DoD §5); 0pt never
* stores bytes here, only references.
* dataset.query -> listVersions/getVersion (tenant-gated).
* dataset.export -> exportDataset: registers the export artifact in ST0RE
* (real client), records AUD0 evidence + OpenLineage,
* returns the canonical `st0re:artifact:...` ref.
*
* Idempotency (§53): createVersion accepts an idempotency_key; the same key
* returns the SAME version instead of minting a duplicate.
* Dataset metadata is canonical in DatasetService; immutable version metadata
* is persisted through DatasetVersionRepository; bytes remain canonical ST0RE
* artifacts. AUD0/OpenLineage record the evidence chain.
*/
import crypto from "crypto";
import { SiaxError } from "../platform/errors";
import { formatResourceRef, isResourceRef } from "../platform/resource-ref";
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
import { createSt0reClient, type St0reClient } from "../clients/st0re-client";
import { LineageEmitter } from "../lineage/lineage-emitter";
import {
createDatasetVersionRepository,
type DatasetVersionRepository,
} from "../repositories/dataset-version-repository";
import { SiaxError } from "../platform/errors";
import { formatResourceRef, isResourceRef } from "../platform/resource-ref";
import type {
CreateDatasetVersionInput,
DatasetExport,
DatasetVersion,
} from "../types/dataset-version";
import { DatasetService } from "./dataset-service";
export interface DatasetVersion {
dataset_id: string;
version: number;
tenant_id: string;
project_id: string;
/** Canonical ST0RE artifact refs (st0re:artifact:...) making up this version. */
artifact_refs: string[];
/** Schema version of the dataset content at this version. */
schema_version: string;
/** Row/record count when known. */
record_count?: number;
description?: string;
created_at: string;
created_by: string;
correlation_id?: string;
}
export type { CreateDatasetVersionInput, DatasetExport, DatasetVersion } from "../types/dataset-version";
export interface CreateDatasetVersionInput {
dataset_id: string;
tenant_id: string;
project_id: string;
artifact_refs: string[];
schema_version: string;
record_count?: number;
description?: string;
created_by: string;
correlation_id?: string;
idempotency_key?: string;
}
export interface DatasetExport {
export_id: string;
dataset_id: string;
version: number;
tenant_id: string;
/** Canonical ref of the registered export artifact. */
artifact_ref: string;
format: "parquet" | "csv" | "json";
created_at: string;
export interface DatasetVersionServiceOptions {
aud0BaseUrl?: string;
st0reBaseUrl?: string;
databaseUrl?: string;
datasetService?: DatasetService;
repository?: DatasetVersionRepository;
}
export class DatasetVersionService {
private readonly versions = new Map<string, DatasetVersion[]>();
private readonly idempotency = new Map<string, DatasetVersion>();
private readonly repo: DatasetVersionRepository;
private readonly aud0: Aud0Client;
private readonly st0re: St0reClient;
private readonly lineage: LineageEmitter;
readonly datasets: DatasetService;
constructor(opts?: { aud0BaseUrl?: string; st0reBaseUrl?: string; datasetService?: DatasetService }) {
constructor(opts?: DatasetVersionServiceOptions) {
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
this.st0re = createSt0reClient(opts?.st0reBaseUrl);
this.lineage = new LineageEmitter(this.aud0);
this.datasets = opts?.datasetService ?? new DatasetService(opts?.aud0BaseUrl);
this.datasets = opts?.datasetService ?? new DatasetService({
aud0BaseUrl: opts?.aud0BaseUrl,
databaseUrl: opts?.databaseUrl,
});
this.repo = opts?.repository ?? createDatasetVersionRepository(opts?.databaseUrl);
}
// -----------------------------------------------------------------------
// dataset.version
// -----------------------------------------------------------------------
async createVersion(input: CreateDatasetVersionInput): Promise<DatasetVersion> {
await this.assertDatasetTenant(input.dataset_id, input.tenant_id);
if (input.idempotency_key) {
const existing = this.idempotency.get(`${input.tenant_id}:${input.idempotency_key}`);
if (existing) return existing;
if (!input.project_id?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "project_id is required");
if (!input.schema_version?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "schema_version is required");
if (!input.created_by?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "created_by is required");
if (input.record_count !== undefined && (!Number.isSafeInteger(input.record_count) || input.record_count < 0)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "record_count must be a non-negative safe integer");
}
if (input.artifact_refs.length === 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "a dataset version needs at least one artifact ref");
@@ -100,37 +66,19 @@ export class DatasetVersionService {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.dataset",
`artifact ref "${ref}" is not a canonical st0re:artifact:... reference (Master-DoD §5)`
`artifact ref "${ref}" is not a canonical st0re:artifact:... reference (Master-DoD §5)`,
);
}
}
const list = this.versions.get(input.dataset_id) ?? [];
const version: DatasetVersion = {
dataset_id: input.dataset_id,
version: (list[list.length - 1]?.version ?? 0) + 1,
tenant_id: input.tenant_id,
project_id: input.project_id,
artifact_refs: [...input.artifact_refs],
schema_version: input.schema_version,
record_count: input.record_count,
description: input.description,
created_at: new Date().toISOString(),
created_by: input.created_by,
correlation_id: input.correlation_id,
};
list.push(version);
this.versions.set(input.dataset_id, list);
if (input.idempotency_key) {
this.idempotency.set(`${input.tenant_id}:${input.idempotency_key}`, version);
}
const version = await this.repo.create(input);
const versionRef = formatResourceRef({
product: "0pt",
resourceType: "dataset",
resourceId: input.dataset_id,
version: String(version.version),
});
await this.aud0.ingestEvent({
event_id: `evt_0pt_dataset_version_${input.dataset_id}_v${version.version}`,
event_type: "0pt.dataset_version_created",
@@ -145,10 +93,11 @@ export class DatasetVersionService {
result: "success",
input_refs: version.artifact_refs,
output_refs: [versionRef],
payload: { schema_version: input.schema_version, record_count: input.record_count },
payload: { schema_version: version.schema_version, record_count: version.record_count },
created_at: version.created_at,
correlation_id: input.correlation_id,
idempotency_key: input.idempotency_key,
occurred_at: version.created_at,
correlation_id: version.correlation_id,
idempotency_key: `dataset-version:${input.tenant_id}:${input.dataset_id}:v${version.version}`,
});
await this.lineage.emit({
tenant_id: input.tenant_id,
@@ -156,41 +105,34 @@ export class DatasetVersionService {
capability_id: "0pt.dataset.version.v1",
run_id: `dataset-version:${input.dataset_id}:v${version.version}`,
event_type: "COMPLETE",
event_time: version.created_at,
inputs: version.artifact_refs.map((ref) => ({ name: ref, ref })),
outputs: [{ name: `dataset/${input.dataset_id}`, ref: versionRef }],
correlation_id: input.correlation_id,
correlation_id: version.correlation_id,
});
return version;
}
// -----------------------------------------------------------------------
// dataset.query
// -----------------------------------------------------------------------
async listVersions(datasetId: string, tenantId: string): Promise<DatasetVersion[]> {
await this.assertDatasetTenant(datasetId, tenantId);
return [...(this.versions.get(datasetId) ?? [])];
return this.repo.list(datasetId, tenantId);
}
async getVersion(datasetId: string, tenantId: string, version: number): Promise<DatasetVersion> {
const list = await this.listVersions(datasetId, tenantId);
const found = list.find((v) => v.version === version);
if (!found) {
throw new SiaxError("NOT_FOUND", "0pt.dataset", `dataset ${datasetId} has no version ${version}`);
await this.assertDatasetTenant(datasetId, tenantId);
if (!Number.isSafeInteger(version) || version < 1) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "version must be a positive integer");
}
return found;
return this.repo.get(datasetId, tenantId, version);
}
// -----------------------------------------------------------------------
// dataset.export
// -----------------------------------------------------------------------
async exportDataset(input: {
dataset_id: string;
tenant_id: string;
project_id: string;
version: number;
format: DatasetExport["format"];
/** Where the export bytes were written (by the exporting job). */
object_location: { provider: string; bucket: string; key: string; region?: string };
size_bytes: number;
checksum_sha256: string;
@@ -198,7 +140,18 @@ export class DatasetVersionService {
correlation_id?: string;
}): Promise<DatasetExport> {
const version = await this.getVersion(input.dataset_id, input.tenant_id, input.version);
const exportId = `dsexp_${crypto.randomUUID()}`;
if (!(["parquet", "csv", "json"] as const).includes(input.format)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "unsupported export format");
}
if (!Number.isSafeInteger(input.size_bytes) || input.size_bytes < 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "size_bytes must be a non-negative safe integer");
}
if (!/^[a-fA-F0-9]{64}$/.test(input.checksum_sha256)) {
throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "checksum_sha256 must be a 64-character hex digest");
}
const exportKey = `dsexp:${input.tenant_id}:${input.dataset_id}:v${input.version}:${input.format}:${input.checksum_sha256.toLowerCase()}`;
const exportId = `dsexp_${crypto.createHash("sha256").update(exportKey).digest("hex").slice(0, 32)}`;
const artifact = await this.st0re.createArtifact({
tenant_id: input.tenant_id,
@@ -214,14 +167,14 @@ export class DatasetVersionService {
? "text/csv"
: "application/json",
size_bytes: input.size_bytes,
checksum_sha256: input.checksum_sha256,
checksum_sha256: input.checksum_sha256.toLowerCase(),
pii_level: "none",
retention_policy: "standard",
data_residency: "eu",
risk_level: "L2",
correlation_id: input.correlation_id,
created_by: input.actor_id,
idempotency_key: `dsexp:${input.dataset_id}:v${input.version}:${input.format}:${input.checksum_sha256}`,
idempotency_key: exportKey,
});
const artifactRef = formatResourceRef({
@@ -229,6 +182,7 @@ export class DatasetVersionService {
resourceType: "artifact",
resourceId: artifact.artifact_id,
});
const createdAt = artifact.created_at ?? new Date().toISOString();
const record: DatasetExport = {
export_id: exportId,
dataset_id: input.dataset_id,
@@ -236,7 +190,7 @@ export class DatasetVersionService {
tenant_id: input.tenant_id,
artifact_ref: artifactRef,
format: input.format,
created_at: new Date().toISOString(),
created_at: createdAt,
};
const versionRef = formatResourceRef({
@@ -260,8 +214,10 @@ export class DatasetVersionService {
input_refs: [versionRef],
output_refs: [artifactRef],
payload: { format: input.format, size_bytes: input.size_bytes },
created_at: record.created_at,
created_at: createdAt,
occurred_at: createdAt,
correlation_id: input.correlation_id,
idempotency_key: exportKey,
});
await this.lineage.emit({
tenant_id: input.tenant_id,
@@ -269,6 +225,7 @@ export class DatasetVersionService {
capability_id: "0pt.dataset.export.v1",
run_id: `dataset-export:${exportId}`,
event_type: "COMPLETE",
event_time: createdAt,
inputs: [{ name: `dataset/${input.dataset_id}`, ref: versionRef }],
outputs: [{ name: artifactRef, ref: artifactRef }],
correlation_id: input.correlation_id,
@@ -277,9 +234,8 @@ export class DatasetVersionService {
return record;
}
// -----------------------------------------------------------------------
private async assertDatasetTenant(datasetId: string, tenantId: string): Promise<void> {
if (!tenantId) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required");
if (!tenantId?.trim()) throw new SiaxError("INVALID_REQUEST", "0pt.dataset", "tenant_id is required");
let dataset;
try {
dataset = await this.datasets.getDataset(datasetId);
src/services/datasets.ts
+6
View File
@@ -0,0 +1,6 @@
/** Shared composition root for the dataset HTTP surface. */
import { DatasetService } from "./dataset-service";
import { DatasetVersionService } from "./dataset-version-service";
export const datasetService = new DatasetService();
export const datasetVersionService = new DatasetVersionService({ datasetService });
src/services/entity-resolution.ts
+10
View File
@@ -0,0 +1,10 @@
import { createN0dClient } from "../clients/n0d-client";
import { EntityResolutionService } from "../er/entity-resolution-service";
export const entityResolutionService = new EntityResolutionService();
let batchService: EntityResolutionService | undefined;
export function getBatchEntityResolutionService(): EntityResolutionService {
if (!batchService) batchService = new EntityResolutionService({ n0dClient: createN0dClient() });
return batchService;
}
src/services/search.ts
+33
View File
@@ -0,0 +1,33 @@
import { createAud0Client } from "../clients/aud0-client";
import { resolveDependencyBaseUrl } from "../platform/dependency-mode";
import { SiaxError } from "../platform/errors";
import { createSearchDefinitionRepository } from "../repositories/search-definition-repository";
import { HttpOpenSearchAdapter } from "../search/opensearch-adapter";
import { SearchService } from "../search/search-service";
let singleton: SearchService | undefined;
export function getSearchService(): SearchService {
if (singleton) return singleton;
const baseUrl = resolveDependencyBaseUrl("opensearch", process.env.OPENSEARCH_URL);
if (!baseUrl) {
throw new SiaxError("DEPENDENCY_UNAVAILABLE", "opensearch", "OPENSEARCH_URL is required for the search HTTP surface");
}
singleton = new SearchService(
new HttpOpenSearchAdapter({
baseUrl,
...(process.env.OPENSEARCH_BASIC_AUTH
? { basicAuth: process.env.OPENSEARCH_BASIC_AUTH }
: {}),
}),
{
aud0Client: createAud0Client(),
definitionRepository: createSearchDefinitionRepository(),
},
);
return singleton;
}
export function resetSearchServiceForTests(): void {
singleton = undefined;
}
src/types/dataset-version.ts
+38
View File
@@ -0,0 +1,38 @@
export interface DatasetVersion {
dataset_id: string;
version: number;
tenant_id: string;
project_id: string;
/** Canonical ST0RE artifact refs that make up this immutable version. */
artifact_refs: string[];
schema_version: string;
record_count?: number;
description?: string;
created_at: string;
created_by: string;
correlation_id?: string;
}
export interface CreateDatasetVersionInput {
dataset_id: string;
tenant_id: string;
project_id: string;
/** Immutable input — callers may hand us readonly arrays (as const); we never mutate. */
readonly artifact_refs: readonly string[];
schema_version: string;
record_count?: number;
description?: string;
created_by: string;
correlation_id?: string;
idempotency_key?: string;
}
export interface DatasetExport {
export_id: string;
dataset_id: string;
version: number;
tenant_id: string;
artifact_ref: string;
format: "parquet" | "csv" | "json";
created_at: string;
}
tests/integration/search-durable-definition.integration.test.ts
+117
View File
@@ -0,0 +1,117 @@
import fs from "node:fs";
import path from "node:path";
import postgres from "postgres";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { closeRepositorySqlForTests, getRepositorySql } from "@/src/repositories/db";
import { PostgresSearchDefinitionRepository } from "@/src/repositories/search-definition-repository";
import { HttpOpenSearchAdapter } from "@/src/search/opensearch-adapter";
import { SearchService, type DocumentSource } from "@/src/search/search-service";
import type { SearchDocument } from "@/src/search/search-types";
const OPENSEARCH_URL = process.env.OPENSEARCH_URL;
const DATABASE_URL = process.env.SEARCH_DEFINITION_TEST_DATABASE_URL;
const run = OPENSEARCH_URL && DATABASE_URL ? describe : describe.skip;
let admin: ReturnType<typeof postgres> | undefined;
let reachable = false;
let reason = "";
beforeAll(async () => {
if (!OPENSEARCH_URL || !DATABASE_URL) return;
try {
admin = postgres(DATABASE_URL, { max: 1, prepare: false, onnotice: () => undefined });
await admin`SELECT 1`;
await admin.unsafe(fs.readFileSync(path.join(process.cwd(), "sql", "013_search_index_definitions.sql"), "utf8"));
const health = await new HttpOpenSearchAdapter({ baseUrl: OPENSEARCH_URL }).health();
if (!(["green", "yellow"] as string[]).includes(health.status)) throw new Error(`OpenSearch ${health.status}`);
reachable = true;
} catch (error) {
reason = error instanceof Error ? error.message : String(error);
}
}, 30_000);
afterAll(async () => {
await admin?.end({ timeout: 5 });
await closeRepositorySqlForTests();
});
run("durable SearchDefinitionRepository + live OpenSearch", () => {
it("reloads a definition after a fresh SearchService and rebuilds/querys the projection", async (ctx) => {
if (!reachable) {
console.warn(`[realinfra] search durable definition dependencies unavailable (${reason}) — skipping honestly`);
ctx.skip();
return;
}
const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`.replace(/[^a-z0-9_]/gi, "_");
const indexName = `companies_${stamp}`.slice(0, 60);
const tenant = `t_${stamp}`;
const repo = new PostgresSearchDefinitionRepository(getRepositorySql(DATABASE_URL!));
const adapter = new HttpOpenSearchAdapter({ baseUrl: OPENSEARCH_URL! });
const writer = new SearchService(adapter, { definitionRepository: repo });
await writer.registerIndexDefinitionDurable({
index_name: indexName,
version: 1,
fields: { name: "text", org_nr: "keyword" },
default_search_fields: ["name"],
}, "realinfra-test");
// New instance, empty in-memory cache — this is the restart condition.
const reader = new SearchService(adapter, { definitionRepository: repo });
const loaded = await reader.resolveIndexDefinition(indexName);
expect(loaded.version).toBe(1);
expect(loaded.fields.name).toBe("text");
const docs: SearchDocument[] = [{
doc_id: "company-1",
tenant_id: tenant,
purpose_tags: ["company_enrichment"],
rights_tags: ["internal"],
source_refs: ["0pt:dataset:companies@1"],
payload: { name: "Acme Aktiebolag", org_nr: "5560000000" },
}];
const source: DocumentSource = {
name: "0pt:dataset:companies@1",
async *iterate() { for (const doc of docs) yield doc; },
};
const rebuilt = await reader.rebuild(indexName, source, { tenant_id: tenant });
expect(rebuilt.docs_indexed).toBe(1);
expect(rebuilt.switched).toBe(true);
const result = await reader.query(indexName, {
tenant_id: tenant,
purpose: "company_enrichment",
rights_tag: "internal",
text: "Acme",
});
expect(result.total).toBe(1);
expect(result.hits[0].doc_id).toBe("company-1");
// A foreign tenant sees no hit even though it queries the same physical alias.
const foreign = await reader.query(indexName, {
tenant_id: `foreign_${tenant}`,
purpose: "company_enrichment",
rights_tag: "internal",
text: "Acme",
});
expect(foreign.total).toBe(0);
});
it("enforces sequential schema versions and rejects same-version drift", async (ctx) => {
if (!reachable) return ctx.skip();
const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`.replace(/[^a-z0-9_]/gi, "_");
const indexName = `schema_${stamp}`.slice(0, 60);
const repo = new PostgresSearchDefinitionRepository(getRepositorySql(DATABASE_URL!));
const v1 = { index_name: indexName, version: 1, fields: { name: "text" as const }, default_search_fields: ["name"] };
const first = await repo.save(v1, "test");
const replay = await repo.save(v1, "test");
expect(first.created).toBe(true);
expect(replay.created).toBe(false);
expect(replay.created_at).toBe(first.created_at);
await expect(repo.save({ ...v1, fields: { name: "keyword" } }, "test")).rejects.toMatchObject({ code: "CONFLICT" });
await expect(repo.save({ ...v1, version: 3 }, "test")).rejects.toMatchObject({ code: "CONFLICT" });
expect((await repo.save({ ...v1, version: 2 }, "test")).definition.version).toBe(2);
});
});
tests/p0/aud0-outbox-postgres.realinfra.test.ts
+94
View File
@@ -0,0 +1,94 @@
import fs from "node:fs/promises";
import path from "node:path";
import postgres from "postgres";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { PostgresAud0OutboxRepository } from "@/src/evidence/aud0-outbox";
const TEST_DATABASE_URL = process.env.TEST_DATABASE_URL;
const describeReal = TEST_DATABASE_URL ? describe : describe.skip;
const schema = `opt_outbox_test_${process.pid}_${Date.now()}`.replace(/[^a-zA-Z0-9_]/g, "_");
let sql: ReturnType<typeof postgres>;
let repo: PostgresAud0OutboxRepository;
const baseEvent = {
event_id: "evt-real-1",
event_type: "0pt.test",
tenant_id: "tenant-real-a",
project_id: "project-a",
actor_id: "tester",
app_id: "0pt",
capability_id: "0pt.test.v1",
resource_type: "test",
resource_id: "r1",
risk_level: "L1",
created_at: "2026-09-07T12:00:00.000Z",
idempotency_key: "idem-real-1",
};
describeReal("AUD0 outbox Postgres integration", () => {
beforeAll(async () => {
const admin = postgres(TEST_DATABASE_URL!, { max: 1, onnotice: () => undefined });
await admin.unsafe(`CREATE SCHEMA ${schema}`);
await admin.end();
sql = postgres(TEST_DATABASE_URL!, {
max: 3,
prepare: false,
onnotice: () => undefined,
connection: { search_path: schema },
});
const migration = await fs.readFile(path.join(process.cwd(), "sql/011_aud0_evidence_outbox.sql"), "utf8");
await sql.unsafe(migration);
repo = new PostgresAud0OutboxRepository(sql);
});
afterAll(async () => {
if (sql) await sql.end({ timeout: 5 });
if (TEST_DATABASE_URL) {
const admin = postgres(TEST_DATABASE_URL, { max: 1, onnotice: () => undefined });
await admin.unsafe(`DROP SCHEMA IF EXISTS ${schema} CASCADE`);
await admin.end();
}
});
it("deduplicates identical producer replays and rejects same-key payload drift", async () => {
const first = await repo.enqueueEvent(baseEvent);
const replay = await repo.enqueueEvent({ ...baseEvent });
expect(replay.outbox_id).toBe(first.outbox_id);
await expect(
repo.enqueueEvent({ ...baseEvent, resource_id: "different" }),
).rejects.toMatchObject({ code: "CONFLICT" });
});
it("leases rows to only one worker and recovers retry state", async () => {
await repo.enqueueEvent({ ...baseEvent, event_id: "evt-real-2", idempotency_key: "idem-real-2" });
const a = await repo.claimPending({ workerId: "worker-a", limit: 100, leaseMs: 60_000 });
expect(a.length).toBeGreaterThanOrEqual(2);
const b = await repo.claimPending({ workerId: "worker-b", limit: 100, leaseMs: 60_000 });
expect(b).toHaveLength(0);
const target = a.find((r) => r.idempotency_key === "idem-real-2")!;
await repo.markRetry(target.outbox_id, "worker-a", "DEPENDENCY_UNAVAILABLE:aud0", 0);
const reclaimed = await repo.claimPending({ workerId: "worker-b", limit: 100, leaseMs: 60_000 });
expect(reclaimed.some((r) => r.outbox_id === target.outbox_id)).toBe(true);
});
it("keeps delivered/dead rows as visible terminal evidence-delivery state", async () => {
const delivered = await repo.enqueueEvent({ ...baseEvent, event_id: "evt-real-3", idempotency_key: "idem-real-3" });
const dead = await repo.enqueueEvent({ ...baseEvent, event_id: "evt-real-4", idempotency_key: "idem-real-4" });
const claimed = await repo.claimPending({ workerId: "terminal-worker", limit: 100, leaseMs: 60_000 });
if (claimed.some((r) => r.outbox_id === delivered.outbox_id)) {
await repo.markDelivered(delivered.outbox_id, "terminal-worker");
}
if (claimed.some((r) => r.outbox_id === dead.outbox_id)) {
await repo.markDead(dead.outbox_id, "terminal-worker", "INVALID_REQUEST:aud0");
}
const stats = await repo.stats();
expect(stats.delivered).toBeGreaterThanOrEqual(1);
expect(stats.dead).toBeGreaterThanOrEqual(1);
});
});
tests/p0/aud0-outbox.test.ts
+195
View File
@@ -0,0 +1,195 @@
import { describe, expect, it } from "vitest";
import type { Aud0Client, Aud0Event } from "@/src/clients/aud0-client";
import {
Aud0OutboxDispatcher,
openLineageIdempotencyKey,
outboxId,
type Aud0OutboxRecord,
type Aud0OutboxRepository,
type Aud0OutboxStats,
} from "@/src/evidence/aud0-outbox";
import { LineageEmitter } from "@/src/lineage/lineage-emitter";
import type { OpenLineageRunEvent } from "@/src/lineage/openlineage";
import { SiaxError } from "@/src/platform/errors";
const event: Aud0Event = {
event_id: "evt-1",
event_type: "0pt.test",
tenant_id: "tenant-a",
project_id: "project-a",
actor_id: "tester",
app_id: "0pt",
capability_id: "0pt.test.v1",
resource_type: "test",
resource_id: "r1",
risk_level: "L1",
created_at: "2026-09-07T12:00:00.000Z",
idempotency_key: "idem-1",
};
const ol: OpenLineageRunEvent = {
eventType: "COMPLETE",
eventTime: "2026-09-07T12:00:00.000Z",
producer: "https://git.siax.io/sax3l/0pt",
run: { runId: "run-1" },
job: { namespace: "0pt", name: "0pt.test.v1" },
inputs: [],
outputs: [],
};
class FakeRepo implements Aud0OutboxRepository {
claimed: Aud0OutboxRecord[] = [];
delivered: string[] = [];
dead: Array<{ id: string; error: string }> = [];
retried: Array<{ id: string; error: string; delayMs: number }> = [];
async enqueueEvent(): Promise<Aud0OutboxRecord> { throw new Error("unused"); }
async enqueueOpenLineage(): Promise<Aud0OutboxRecord> { throw new Error("unused"); }
async claimPending(): Promise<Aud0OutboxRecord[]> { return this.claimed; }
async markDelivered(id: string): Promise<void> { this.delivered.push(id); }
async markRetry(id: string, _worker: string, error: string, delayMs: number): Promise<void> {
this.retried.push({ id, error, delayMs });
}
async markDead(id: string, _worker: string, error: string): Promise<void> {
this.dead.push({ id, error });
}
async stats(): Promise<Aud0OutboxStats> { return { pending: this.claimed.length, delivered: 0, dead: 0 }; }
}
function row(
kind: "event" | "openlineage",
payload: Aud0Event | OpenLineageRunEvent,
overrides: Partial<Aud0OutboxRecord> = {},
): Aud0OutboxRecord {
return {
outbox_id: "obx-1",
tenant_id: "tenant-a",
app_id: "0pt",
evidence_kind: kind,
idempotency_key: "idem-1",
payload,
status: "pending",
attempts: 1,
next_attempt_at: "2026-09-07T12:00:00.000Z",
created_at: "2026-09-07T12:00:00.000Z",
updated_at: "2026-09-07T12:00:00.000Z",
...overrides,
};
}
function client(impl?: Partial<Aud0Client>): Aud0Client {
return {
async ingestEvent(input) { return input; },
async ingestOpenLineageEvent(tenantId, _appId, input) {
return { tenant_id: tenantId, run_id: input.run.runId, job: input.job.name, event_id: "aud0-1" };
},
async ingestLineageEvent(input) { return input; },
...impl,
};
}
describe("AUD0 durable outbox", () => {
it("derives stable tenant-bound outbox ids", () => {
expect(outboxId("tenant-a", "event", "k1")).toBe(outboxId("tenant-a", "event", "k1"));
expect(outboxId("tenant-a", "event", "k1")).not.toBe(outboxId("tenant-b", "event", "k1"));
expect(outboxId("tenant-a", "event", "k1")).not.toBe(outboxId("tenant-a", "openlineage", "k1"));
});
it("uses logical OpenLineage run+eventType identity rather than delivery time", () => {
expect(openLineageIdempotencyKey(ol)).toBe("ol:run-1:COMPLETE");
expect(openLineageIdempotencyKey({ ...ol, eventTime: "2030-01-01T00:00:00Z" })).toBe("ol:run-1:COMPLETE");
});
it("marks a successfully delivered generic event delivered", async () => {
const repo = new FakeRepo();
repo.claimed = [row("event", event)];
let calls = 0;
const dispatcher = new Aud0OutboxDispatcher(repo, client({
async ingestEvent(input) { calls += 1; return input; },
}), "worker-a");
expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 1, dead: 0, retried: 0 });
expect(calls).toBe(1);
expect(repo.delivered).toEqual(["obx-1"]);
});
it("retries network/dependency failures with bounded exponential backoff", async () => {
const repo = new FakeRepo();
repo.claimed = [row("openlineage", ol, { attempts: 4 })];
const dispatcher = new Aud0OutboxDispatcher(repo, client({
async ingestOpenLineageEvent() {
throw new SiaxError("DEPENDENCY_UNAVAILABLE", "aud0", "offline");
},
}), "worker-a");
expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 0, dead: 0, retried: 1 });
expect(repo.retried[0].delayMs).toBe(8_000);
expect(repo.retried[0].error).toBe("DEPENDENCY_UNAVAILABLE:aud0");
});
it("parks permanent 4xx-style failures dead instead of poisoning the queue", async () => {
const repo = new FakeRepo();
repo.claimed = [row("event", event)];
const dispatcher = new Aud0OutboxDispatcher(repo, client({
async ingestEvent() {
throw new SiaxError("INVALID_REQUEST", "aud0", "bad event");
},
}), "worker-a");
expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 0, dead: 1, retried: 0 });
expect(repo.dead).toEqual([{ id: "obx-1", error: "INVALID_REQUEST:aud0" }]);
});
it("fails closed on an immutable tenant/app envelope mismatch before calling AUD0", async () => {
const repo = new FakeRepo();
repo.claimed = [row("event", { ...event, tenant_id: "tenant-b" })];
let calls = 0;
const dispatcher = new Aud0OutboxDispatcher(repo, client({
async ingestEvent(input) { calls += 1; return input; },
}), "worker-a");
expect(await dispatcher.sweep()).toEqual({ claimed: 1, delivered: 0, dead: 1, retried: 0 });
expect(calls).toBe(0);
expect(repo.dead[0].error).toBe("PERMANENT_FAILURE:0pt.aud0_outbox");
});
it("reports a durable client acknowledgement as queued, not emitted", async () => {
const emitter = new LineageEmitter(client({
async ingestOpenLineageEvent(tenantId, _appId, input) {
return {
tenant_id: tenantId,
run_id: input.run.runId,
job: input.job.name,
event_id: "outbox:obx-queued-1",
};
},
}));
const result = await emitter.emit({
tenant_id: "tenant-a",
capability_id: "0pt.test.v1",
run_id: "run-queued",
event_type: "COMPLETE",
event_time: "2026-09-07T12:00:00.000Z",
});
expect(result).toEqual({
emitted: false,
queued: true,
run_id: "run-queued",
outbox_id: "obx-queued-1",
});
});
it("reports a raw AUD0 acknowledgement as emitted, not queued", async () => {
const emitter = new LineageEmitter(client());
const result = await emitter.emit({
tenant_id: "tenant-a",
capability_id: "0pt.test.v1",
run_id: "run-live",
event_type: "COMPLETE",
event_time: "2026-09-07T12:00:00.000Z",
});
expect(result).toEqual({ emitted: true, queued: false, run_id: "run-live" });
});
});
tests/p0/dataset-postgres.realinfra.test.ts
+222
View File
@@ -0,0 +1,222 @@
/**
* Real-Postgres proof for the dataset registry + immutable version metadata.
*
* Set DATASET_TEST_DATABASE_URL to a disposable Postgres 16 instance. When it
* is unset the suite skips honestly; Gitea CI should provide it for the
* blocking real-infrastructure job before this capability is promoted.
*/
import fs from "node:fs";
import path from "node:path";
import postgres from "postgres";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { closeRepositorySqlForTests } from "@/src/repositories/db";
import { DatasetService } from "@/src/services/dataset-service";
import { DatasetVersionService } from "@/src/services/dataset-version-service";
const TEST_DATABASE_URL = process.env.DATASET_TEST_DATABASE_URL;
const run = TEST_DATABASE_URL ? describe : describe.skip;
let admin: ReturnType<typeof postgres> | undefined;
let reachable = false;
let unreachableReason = "";
beforeAll(async () => {
if (!TEST_DATABASE_URL) {
console.warn("[realinfra] DATASET_TEST_DATABASE_URL unset — skipping honestly.");
return;
}
try {
admin = postgres(TEST_DATABASE_URL, { prepare: false, ssl: false, max: 3 });
await Promise.race([
admin`SELECT 1`,
new Promise((_, reject) => setTimeout(() => reject(new Error("connect timeout")), 5_000)),
]);
for (const migration of ["003_datasets.sql", "011_aud0_evidence_outbox.sql", "012_dataset_versions.sql"]) {
await admin.unsafe(fs.readFileSync(path.join(process.cwd(), "sql", migration), "utf8"));
}
// Aktivera den durabla AUD0-outboxen: createAud0OutboxRepositoryFromEnv()
// läser DATABASE_URL (se src/repositories/db.ts) — utan den blir klienten
// Noop och inga producer-rader skrivs (testet skulle ljugit om durabilitet).
process.env.DATABASE_URL = TEST_DATABASE_URL;
reachable = true;
} catch (error) {
unreachableReason = error instanceof Error ? error.message : String(error);
}
}, 30_000);
afterAll(async () => {
delete process.env.DATABASE_URL;
await admin?.end({ timeout: 5 });
await closeRepositorySqlForTests();
});
function datasetService(): DatasetService {
return new DatasetService({ databaseUrl: TEST_DATABASE_URL });
}
function versionService(): DatasetVersionService {
return new DatasetVersionService({ databaseUrl: TEST_DATABASE_URL });
}
run("Dataset registry + versions against LIVE Postgres", () => {
it("persists a dataset across independent service instances", async (ctx) => {
if (!reachable) {
console.warn(`[realinfra] Postgres unavailable (${unreachableReason}) — skipping honestly.`);
ctx.skip();
return;
}
const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`;
const datasetId = `ds_realpg_${stamp}`;
const tenantId = `t_realpg_${stamp}`;
await datasetService().registerDataset({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
name: "Real Postgres dataset",
dataset_type: "curated",
created_by: "realinfra-test",
});
const reloaded = await datasetService().getDataset(datasetId);
expect(reloaded.dataset_id).toBe(datasetId);
expect(reloaded.tenant_id).toBe(tenantId);
const rows = await admin!<{ tenant_id: string; name: string }[]>`
SELECT tenant_id, name FROM opt_datasets WHERE dataset_id = ${datasetId}`;
expect(rows).toEqual([{ tenant_id: tenantId, name: "Real Postgres dataset" }]);
});
it("persists monotonic immutable versions across fresh service instances", async (ctx) => {
if (!reachable) return ctx.skip();
const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`;
const datasetId = `ds_versions_${stamp}`;
const tenantId = `t_versions_${stamp}`;
await datasetService().registerDataset({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
name: "Versioned dataset",
dataset_type: "curated",
created_by: "realinfra-test",
});
const v1 = await versionService().createVersion({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
artifact_refs: ["st0re:artifact:art_realpg_1"],
schema_version: "1",
record_count: 10,
created_by: "realinfra-test",
idempotency_key: `idem_${stamp}_1`,
});
const v2 = await versionService().createVersion({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
artifact_refs: ["st0re:artifact:art_realpg_2"],
schema_version: "1",
record_count: 11,
created_by: "realinfra-test",
idempotency_key: `idem_${stamp}_2`,
});
expect([v1.version, v2.version]).toEqual([1, 2]);
const fresh = versionService();
expect((await fresh.listVersions(datasetId, tenantId)).map((item) => item.version)).toEqual([1, 2]);
expect((await fresh.getVersion(datasetId, tenantId, 2)).artifact_refs).toEqual(["st0re:artifact:art_realpg_2"]);
});
it("replays the same idempotency key without minting another version and rejects payload drift", async (ctx) => {
if (!reachable) return ctx.skip();
const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`;
const datasetId = `ds_idem_${stamp}`;
const tenantId = `t_idem_${stamp}`;
await datasetService().registerDataset({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
name: "Idempotent dataset",
dataset_type: "derived",
created_by: "realinfra-test",
});
const input = {
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
artifact_refs: ["st0re:artifact:art_idem_1"],
schema_version: "1",
created_by: "realinfra-test",
idempotency_key: `idem_${stamp}`,
} as const;
const first = await versionService().createVersion(input);
const replay = await versionService().createVersion(input);
expect(replay.version).toBe(first.version);
expect(await versionService().listVersions(datasetId, tenantId)).toHaveLength(1);
await expect(versionService().createVersion({
...input,
artifact_refs: ["st0re:artifact:art_DIFFERENT"],
})).rejects.toMatchObject({ code: "CONFLICT" });
});
it("foreign tenant cannot read or version another tenant's dataset", async (ctx) => {
if (!reachable) return ctx.skip();
const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`;
const datasetId = `ds_tenant_${stamp}`;
const owner = `t_owner_${stamp}`;
const foreign = `t_foreign_${stamp}`;
await datasetService().registerDataset({
dataset_id: datasetId,
tenant_id: owner,
project_id: "p_main",
name: "Tenant isolated",
dataset_type: "raw",
created_by: "realinfra-test",
});
await expect(versionService().listVersions(datasetId, foreign)).rejects.toMatchObject({ code: "NOT_FOUND" });
await expect(versionService().createVersion({
dataset_id: datasetId,
tenant_id: foreign,
project_id: "p_main",
artifact_refs: ["st0re:artifact:art_foreign"],
schema_version: "1",
created_by: "intruder",
})).rejects.toMatchObject({ code: "NOT_FOUND" });
});
it("durable AUD0 producer rows are written alongside dataset operations", async (ctx) => {
if (!reachable) return ctx.skip();
const stamp = `${Date.now()}_${Math.random().toString(16).slice(2)}`;
const datasetId = `ds_evidence_${stamp}`;
const tenantId = `t_evidence_${stamp}`;
await datasetService().registerDataset({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
name: "Evidence dataset",
dataset_type: "curated",
created_by: "realinfra-test",
});
await versionService().createVersion({
dataset_id: datasetId,
tenant_id: tenantId,
project_id: "p_main",
artifact_refs: ["st0re:artifact:art_evidence"],
schema_version: "1",
created_by: "realinfra-test",
});
const evidence = await admin!<{ evidence_kind: string; status: string }[]>`
SELECT evidence_kind, status
FROM opt_aud0_evidence_outbox
WHERE tenant_id = ${tenantId}
ORDER BY created_at`;
expect(evidence.length).toBeGreaterThanOrEqual(3); // register + version event + lineage
expect(evidence.every((row) => row.status === "pending")).toBe(true);
expect(new Set(evidence.map((row) => row.evidence_kind))).toEqual(new Set(["event", "openlineage"]));
});
});
tests/p0/entity-resolution-http.test.ts
+197
View File
@@ -0,0 +1,197 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { NextRequest } from "next/server";
import { POST as candidatesPOST } from "@/app/api/0pt/internal/er/candidates/route";
const TOKEN = "test-er-internal-token";
let previousToken: string | undefined;
function request(body: unknown, token: string | undefined = TOKEN): NextRequest {
const headers: Record<string, string> = { "content-type": "application/json" };
if (token !== undefined) headers["x-internal-token"] = token;
return new NextRequest("http://localhost/api/0pt/internal/er/candidates", {
method: "POST",
headers,
body: JSON.stringify(body),
});
}
function baseBody(records: Array<{ record_id: string; fields: Record<string, unknown> }>) {
return {
tenant_id: "tenant_a",
project_id: "project_a",
records,
blocking_fields: ["city"],
compare_fields: [{ field: "name", weight: 1 }],
auto_match_threshold: 0.9,
review_threshold: 0.6,
};
}
beforeEach(() => {
previousToken = process.env.INTERNAL_TOKEN;
process.env.INTERNAL_TOKEN = TOKEN;
});
afterEach(() => {
if (previousToken === undefined) delete process.env.INTERNAL_TOKEN;
else process.env.INTERNAL_TOKEN = previousToken;
});
describe("POST /api/0pt/internal/er/candidates", () => {
it("fails closed without the internal service token", async () => {
// OBS: request(body, undefined) triggar default-parametern (skickar GILTIGT
// token) — skicka tom string för "utan token" (fail-closed → 401).
const res = await candidatesPOST(
request(
baseBody([
{ record_id: "a", fields: { name: "Acme AB", city: "Stockholm" } },
{ record_id: "b", fields: { name: "Acme AB", city: "Stockholm" } },
]),
"",
),
);
expect(res.status).toBe(401);
});
it("never emits a score-only fuzzy company match even above auto_match_threshold", async () => {
const res = await candidatesPOST(
request(
baseBody([
{ record_id: "a", fields: { name: "Acme Consulting AB", city: "Stockholm" } },
{ record_id: "b", fields: { name: "Acme Consulting A B", city: "Stockholm" } },
]),
),
);
expect(res.status).toBe(200);
const body = await res.json();
expect(body.pairs).toHaveLength(1);
expect(body.pairs[0].score).toBeGreaterThanOrEqual(0.9);
expect(body.pairs[0].decided_by).toBe("score");
expect(body.pairs[0].status).toBe("pending_review");
expect(body.review_queue.map((pair: { candidate_id: string }) => pair.candidate_id)).toEqual([
body.pairs[0].candidate_id,
]);
});
it("allows a same-country registration number to hard-link despite punctuation differences", async () => {
const res = await candidatesPOST(
request(
baseBody([
{
record_id: "a",
fields: {
name: "Acme Sweden AB",
city: "Stockholm",
registration_country: "SE",
org_nr: "556000-0001",
},
},
{
record_id: "b",
fields: {
name: "Acme Sverige",
city: "Stockholm",
registration_country: "se",
org_nr: "5560000001",
},
},
]),
),
);
expect(res.status).toBe(200);
const body = await res.json();
expect(body.pairs).toHaveLength(1);
expect(body.pairs[0].status).toBe("match");
expect(body.pairs[0].decided_by).toBe("0pt.company.stable-identifiers.v3");
expect(body.review_queue).toEqual([]);
});
it("does not treat the same national registration number in different countries as global identity", async () => {
const res = await candidatesPOST(
request(
baseBody([
{
record_id: "a",
fields: {
name: "Acme Consulting AB",
city: "Stockholm",
registration_country: "SE",
org_nr: "123456789",
},
},
{
record_id: "b",
fields: {
name: "Acme Consulting A B",
city: "Stockholm",
registration_country: "NO",
org_nr: "123456789",
},
},
]),
),
);
expect(res.status).toBe(200);
const body = await res.json();
expect(body.pairs).toHaveLength(1);
expect(body.pairs[0].decided_by).toBe("score");
expect(body.pairs[0].status).toBe("pending_review");
});
it("lets conflicting stable identity evidence dominate a matching global LEI", async () => {
const res = await candidatesPOST(
request(
baseBody([
{
record_id: "a",
fields: {
name: "Acme Sweden AB",
city: "Stockholm",
registration_country: "SE",
lei: "549300EXAMPLE123",
org_nr: "556000-0001",
},
},
{
record_id: "b",
fields: {
name: "Acme Sweden AB",
city: "Stockholm",
registration_country: "SE",
lei: "549300EXAMPLE123",
org_nr: "556000-9999",
},
},
]),
),
);
expect(res.status).toBe(200);
const body = await res.json();
expect(body.pairs).toHaveLength(1);
expect(body.pairs[0].status).toBe("non_match");
expect(body.pairs[0].decided_by).toBe("0pt.company.stable-identifiers.v3");
});
it("can hard-link a globally matching LEI when no scoped identifier contradicts it", async () => {
const res = await candidatesPOST(
request(
baseBody([
{
record_id: "a",
fields: { name: "Acme Holding", city: "Stockholm", lei: "549300-EXAMPLE-123" },
},
{
record_id: "b",
fields: { name: "ACME Holdings Group", city: "Stockholm", lei: "549300EXAMPLE123" },
},
]),
),
);
expect(res.status).toBe(200);
const body = await res.json();
expect(body.pairs).toHaveLength(1);
expect(body.pairs[0].status).toBe("match");
expect(body.pairs[0].decided_by).toBe("0pt.company.stable-identifiers.v3");
});
});
tests/p0/entity-resolution.test.ts
+1 -1
View File
@@ -97,7 +97,7 @@ describe("EntityResolutionService — candidate API", () => {
expect(() =>
er.generateCandidates(request({ auto_match_threshold: 0.5, review_threshold: 0.6 }))
).toThrowError(/review_threshold/);
expect(() => er.generateCandidates(request({ records: [RECORDS[0]] }))).toThrowError(/two records/);
expect(() => er.generateCandidates(request({ records: [RECORDS[0]] }))).toThrowError(/between 2 and 5000/);
expect(() => er.generateCandidates(request({ tenant_id: "" }))).toThrowError(/tenant_id/);
});
tests/p0/search-service.test.ts
+15 -1
View File
@@ -160,12 +160,26 @@ describe("SearchService — tenant + purpose/rights enforcement (§10/§31)", ()
const { search } = await build([
doc("1", "t_a", "Tagged", { purpose_tags: ["company_enrichment"] }),
doc("2", "t_a", "OtherPurpose", { purpose_tags: ["compliance"] }),
doc("3", "t_a", "Untagged", { purpose_tags: [] }), // empty ⇒ never returnable
]);
const res = await search.query("companies", { tenant_id: "t_a", purpose: "company_enrichment" });
expect(res.hits.map((h) => h.doc_id)).toEqual(["1"]);
});
it("fails closed: rebuild rejects documents without purpose_tags (§10)", async () => {
const fake = new FakeOpenSearch();
const search = new SearchService(fake);
search.registerIndexDefinition({ ...DEF, fields: { ...DEF.fields } });
await expect(
search.rebuild(
"companies",
docsSource([
doc("1", "t_a", "Ok"),
doc("3", "t_a", "Untagged", { purpose_tags: [] }), // empty ⇒ INVALID_REQUEST, generation discarded
]),
),
).rejects.toThrowError(/missing purpose_tags/);
});
it("rights filtering: default rights tag is 'internal'; deliverable docs need the tag", async () => {
const { search } = await build([
doc("1", "t_a", "Internal", { rights_tags: ["internal"] }),