0pt/STATUS.md

0pt — capability status (Master-DoD §56 status honesty)

Classification scale (Master-DoD §2, exactly one per capability): MISSING → DOCUMENTED_ONLY → SKELETON → MOCK_ONLY → IMPLEMENTED → INTEGRATION_TESTED → LIVE_VERIFIED → PRODUCTION_READY. Only PRODUCTION_READY may be called DONE. Docs ≠ implementation. Last full audit + re-classification: 2026-08-31 (SIAX Platforms P0 wave 2).

How to read this

• INTEGRATION_TESTED here means: verified against a REAL dependency (live OpenSearch/ClickHouse container, or a recorded wire-contract double pinned to the sibling repo's real handler code on its main) in CI.
• LIVE_VERIFIED requires a verification against the actual SIAX fleet deployment. No new capability in this wave claims it.
• The unit/contract/integration suites are in tests/ and run in .gitea/workflows/ci.yml (jobs quality + integration).

Cross-product clients (Master-DoD §10 "Cross-product clients")

Capability	Status	Evidence	Honest gaps
CL0UD client (policy evaluate, POST /api/policy/evaluate)	INTEGRATION_TESTED (contract)	src/clients/cl0ud-client.ts; tests/contracts/cl0ud-client.contract.test.ts pins the real camelCase {data} envelope; fail-closed (missing decision ⇒ deny)	Not yet exercised against a live CL0UD deployment (no reachable CL0UD URL this session ⇒ not LIVE_VERIFIED)
CL0UD capability discovery (GET /api/capabilities)	INTEGRATION_TESTED (contract)	same suite	0pt's own capabilities are not yet registered in CL0UD's registry (§3 gap)
ST0RE client (artifact register, camelCase /v1/st0re/artifacts/register)	INTEGRATION_TESTED (contract)	src/clients/st0re-client.ts; tests/contracts/st0re-client.contract.test.ts	Raw-bytes upload still goes via LocalBlobStore interim (pre-existing, unchanged); presign upload path not consumed yet
N0D client (create/get job, get result)	INTEGRATION_TESTED (contract)	src/clients/n0d-client.ts; tests/contracts/n0d-client.contract.test.ts pins required_role/resource_requirements/body idempotency_key	CloudEvents completion callback receiving endpoint not built in 0pt (poll path works)
AUD0 client (events + OpenLineage ingest)	INTEGRATION_TESTED (contract)	src/clients/aud0-client.ts; tests/contracts/aud0-client.contract.test.ts; legacy lineage shape converts to real OpenLineage	No durable local outbox for AUD0 outages (§70 "durable local/outbox path" — emission failure is surfaced, not queued)
No production mock fallback (§3/§10)	INTEGRATION_TESTED	src/platform/dependency-mode.ts; tests/contracts/dependency-mode.contract.test.ts drives NODE_ENV=production for all four factories ⇒ hard DEPENDENCY_UNAVAILABLE	—

Correction of prior state (2026-08-31): the pre-existing clients' HTTP paths were verified against the sibling repos' actual main branches and THREE of four were wrong: cl0ud (/v1/cl0ud/policies/evaluate — never existed), st0re (snake_case body — would 400), aud0 lineage (/v1/aud0/lineage/events — never existed). All fixed and pinned by contract tests. The earlier MOCK_ONLY classification in the README was accurate.

Data-OS capabilities (Master-DoD §10)

Capability	Status	Evidence	Honest gaps
Source Registry — register/query/describe/health, rights/license/purpose, freshness, provenance	IMPLEMENTED	src/services/source-registry-service.ts, src/types/source-registry.ts; tests/p0/source-registry.test.ts (fail-closed purpose gate, cross-tenant NOT_FOUND)	In-memory persistence (per-process); no Postgres store yet — production boot must therefore configure one before this counts further. No API route surface yet
Source rate-policy	IMPLEMENTED (pre-existing)	src/services/source-rate-policy-service.ts	unchanged this wave
Search — OpenSearch adapter, index defs, bulk, tenant + purpose/rights filter, rebuild, generation switch	INTEGRATION_TESTED	src/search/*; unit tests/p0/search-service.test.ts; real OpenSearch 2.11 tests/integration/search-opensearch.integration.test.ts incl. projection-disaster rebuild (§68-style) — runs in CI service container	Hybrid (vector) query not built; no metrics emission; index definitions in-memory
Quality — versioned rules, runs, results, issues, score, trend, GE interchange	IMPLEMENTED (local engine) / SKELETON (GE delegation)	src/quality/*; tests/p0/quality-service.test.ts, tests/p0/expectation-engine cases	Local engine covers 7 GE core expectation types; everything else delegates via N0D job 0pt.quality.ge_run whose EXECUTOR (0pt-motor GE runner) does not exist yet — delegated runs stay "delegated" until it does. Never fabricates results
Dataset — create (pre-existing) + version/query/export with st0re:artifact: refs	IMPLEMENTED	src/services/dataset-version-service.ts; tests/p0/dataset-version-service.test.ts (idempotency §53, canonical refs §5)	Export assumes bytes already written by a job; no route surface; in-memory persistence
Lineage — OpenLineage emission to AUD0's real ingest	INTEGRATION_TESTED (contract)	src/lineage/*; aud0 contract suite; emitted from dataset version/export, quality runs, search rebuilds	Emission is best-effort (never-throw); no outbox. AUD0-side ingestion verified against contract double, not live AUD0
Metadata — OpenMetadata projection (never canonical)	IMPLEMENTED	src/metadata/openmetadata-adapter.ts; tests/contracts/metadata-features.contract.test.ts; projectAll rebuild path	No live OpenMetadata exercised (no local container this pass); lineage/glossary/ownership projection not built — only dataset entities
Analytics — ClickHouse, tenant-isolated, query limits, retention	INTEGRATION_TESTED	src/analytics/clickhouse-analytics.ts; real ClickHouse 24.8 tests/integration/analytics-clickhouse.integration.test.ts (tenant isolation, closed DSL, TTL schema) — runs in CI service container	Closed metric-DSL only (no ad-hoc queries by design); no quota integration with CL0UD
Features — registry + Feast adapter	IMPLEMENTED (online path, registry) / SKELETON (materialize/get-historical)	src/features/feast-adapter.ts; contract suite pins Feast's get-online-features wire shape	No live Feast server exercised; offline/materialization are N0D delegation contracts whose executor (0pt-motor) does not exist yet; point-in-time correctness therefore NOT yet proven
Entity Resolution — candidate API, blocking, explanation, hard-rule hook, review handoff	IMPLEMENTED	src/er/entity-resolution-service.ts; tests/p0/entity-resolution.test.ts (hard rules override both directions, review band, blocking refusals)	Generic reference scorer (Jaccard+Levenshtein), NOT Splink probabilistic linkage — that is the 0pt.er.batch_resolution N0D contract, executor pending in 0pt-motor. No persistence of review queues
Graph (§10 Graph)	MISSING	—	Not started this wave (explicitly out of the delivered P0 subset; next in line per §10)

Canonical platform plumbing

Capability	Status	Evidence
SiaxRequestContext + propagation headers (mirror of cl0ud @siax/contract)	INTEGRATION_TESTED (contract-pinned)	src/platform/siax-context.ts; tests/contracts/siax-context.contract.test.ts
Canonical 12-code error model + typed SiaxError (§6)	INTEGRATION_TESTED (contract-pinned)	src/platform/errors.ts; same suite
Canonical resource refs (§5)	IMPLEMENTED	src/platform/resource-ref.ts; tests/p0/dataset-version-service.test.ts

Pre-existing capabilities (unchanged this wave — prior classifications stand)

Capability	Status	Note
Iceberg write boundary via ST0RE lakehouse	LIVE_VERIFIED (2026-07-13)	docs/ICEBERG_WRITE_BOUNDARY.md; untouched
Document capture/extraction	IMPLEMENTED	LocalBlobStore interim persists bytes; ST0RE raw upload still pending
Edge order/payment app (Next.js)	see docs/architecture/ARCHITECTURE.md	outside this wave's scope
FastAPI engine (services/api)	IMPLEMENTED	outside this wave's scope
services/scrapers/* (WAF-bypass stack)	quarantined, pending deletion PR	feature/0pt-remove-noncompliant-scrapers — NOT touched by this wave; nothing new wires it

Global gaps blocking PRODUCTION_READY (§3, tracked, not hidden)

1. Persistent stores: the new registries (sources/rights/quality rules/dataset versions/index definitions/feature views) are per-process in-memory. resolveStoreMode guarantees production boot FAILS rather than serving Maps, but the Postgres stores themselves are not built.
2. Capability registration in CL0UD + API0 discovery for the new capabilities (§3 "Capability är registrerad i CL0UD Capability Registry").
3. AUD0 outbox for evidence durability under AUD0 outage (§70).
4. Surface exposure: no app/api/0pt/* routes yet for search/quality/datasets-versions/ER (services + contracts only).
5. 0pt-motor executors for the delegated jobs (0pt.quality.ge_run, 0pt.features.*, 0pt.er.batch_resolution).
6. Observability: no metrics/traces emitted by the new services (§36).

---

P060-P068 ingestion plane (added 2026-09-04)

Classification scale unchanged. Every row below states what was actually verified, and by which command.

Requirement	BEFORE this pass	Status now	Evidence
P060 Source Registry: durable canonical registry; restart preserves sources; an unclassified source cannot be activated	PARTIAL / BROKEN. The registry existed with Postgres persistence (sql/002_source_registry.sql, src/repositories/postgres-source-registry-repository.ts) and carried source_id/tenant/project/type/status/license/purpose/rights. It had NO authorization, NO schedule, NO checkpoint, NO schema, and no owner distinct from created_by. The activation DoD was inverted: postgres-source-repository.ts:70 wrote ${input.status ?? "active"}, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered, and no activation operation existed at all	INTEGRATION_TESTED	sql/007_ingestion_plane.sql (authorization/schedule/schemas/checkpoints + ALTER … SET DEFAULT 'pending' + owner_id); SourceRegistryService.classifySource/activateSource; tests/p0/source-activation-gate.test.ts (11); tests/api/0pt-p060-routes.test.ts (9); tests/p0/ingestion-plane-postgres.realinfra.test.ts against a live Postgres 16
P061 Connector contract: discover/validate/connect/checkpoint/read/resume/backfill/health/disconnect; two heterogeneous connectors pass one conformance suite	MISSING. services/api/sources/connectors.py had per-source fetch_*() functions only — no lifecycle, no cursor, no resume, no health, no disconnect, no conformance suite	INTEGRATION_TESTED	src/connectors/connector.ts (all nine operations); src/connectors/conformance.ts (15 checks); HttpJsonConnector (real node:http server, opaque cursor) and JsonlFileConnector (real file, line-offset cursor, CDC deletes) both pass the identical suite — tests/p0/connector-conformance.test.ts, which also proves the suite FAILS a fail-open connector
P062 CDC/batch/stream normalized to one envelope; restart resumes from a durable checkpoint without silent gaps	MISSING, and documented as such: services/api/tests/test_known_gaps.py:10-19 skipped the CDC case because no CDC pipeline existed anywhere, and workers/pipeline.py had no persistence at all	INTEGRATION_TESTED	src/types/ingestion.ts (one envelope for all three modes); src/ingestion/ingestion-runner.ts; tests/p0/ingestion-runner.test.ts kills a run inside the sink and proves the resumed run covers all 10 records exactly once; same proof against real Postgres across fresh processes
P063 Schema versioning + compatibility policy; a breaking schema is detected before consumer corruption	MISSING. schema_version existed as a free-form string on datasets/Iceberg writes; there was no comparison, no policy, no versioned source schema	INTEGRATION_TESTED	src/schema/compatibility.ts, src/schema/schema-registry.ts; tests/p0/schema-evolution.test.ts (12); the runner's schema gate runs before the first read — tests/p0/ingestion-runner.test.ts asserts sinkCalls === 0 on a break
P064 Quality rules behind the SIAX abstraction; failures block or quarantine per declared policy	PARTIAL. src/quality/* had rules, runs, results, issues, scores and a GE-compatible engine, but a failing rule only opened an ISSUE — the data flowed on regardless. grep -ri quarantin src/ found one unrelated hit	INTEGRATION_TESTED	src/quality/quality-policy.ts composes the EXISTING ExpectationEngine (not a second engine) and adds the consequence; block / quarantine / warn each proved in tests/p0/ingestion-runner.test.ts; quarantine durability proved against real Postgres
P065 Normalization/ER: only the canonical highway; no second simplified production engine	DUPLICATE (confirmed). src/er/entity-resolution-service.ts (canonical: hard rules, review band, Splink delegation) AND services/api/workers/resolve.py (simplified fuzzy threshold, auto-merge) — the latter still called by workers/pipeline.py:110	NOT DONE — documented, not built. The duplication is now stated at BOTH sites in code with the removal plan. The new ingestion plane deliberately performs no ER, so it adds no third engine. See "P065 — why it was not closed" below
P066 Iceberg	out of scope — ST0RE is owned by another workflow	NOT TOUCHED	Contract read only (src/services/iceberg-write-service.ts, src/clients/st0re-lakehouse-client.ts)
P067 Lineage: OpenLineage/AUD0 events; source → output lineage is queryable	PARTIAL. src/lineage/lineage-emitter.ts emitted OpenLineage to AUD0, but nothing in this repo could ANSWER a lineage question — no store, no query, no route	INTEGRATION_TESTED	src/lineage/lineage-edge.ts, src/lineage/lineage-store.ts (local queryable projection; AUD0 stays the system of record), GET /api/0pt/lineage; tests/p0/ingestion-runner.test.ts queries both directions and proves tenant scoping
P068 Durable bounded backfill/replay; repeated backfill is idempotent	MISSING, documented as such: services/api/tests/test_known_gaps.py:44-52 skipped it because "no delivery-store persistence layer exists to dedupe a backfill against"	INTEGRATION_TESTED	opt_ingestion_ledger + IngestionRunner.backfill; tests/p0/ingestion-runner.test.ts (repeat emits 0, live checkpoint untouched, a real content change still re-emits); same against real Postgres

What was actually run (not "should pass")

• pnpm run typecheck — clean.
• pnpm run test — 365 passed, 25 skipped (was 311/18 before this pass).
• pnpm run build — clean; the four new routes appear in the route table.
• pnpm run lint — 12 errors / 5 warnings, all pre-existing (identical count on the base commit; advisory in CI). This pass adds none.
• Real Postgres 16: tests/p0/ingestion-plane-postgres.realinfra.test.ts with INGESTION_TEST_DATABASE_URL — 7 passed.
• Real container restart, run manually against that same database: docker restart the Postgres container, then re-read through the real repositories. Before and after were identical — 2 sources (one active with its owner, one pending), 1 checkpoint (cursor 6, 6 committed), 6 ledger rows, 1 lineage edge, 4 quarantined records, 1 schema version. That is P060's "a restart preserves all sources", measured rather than asserted.

A real bug this pass wrote and then caught

The first draft of IngestionRepository combined the idempotency check and its recording into one atomic call, made BEFORE the sink. The crash/resume test then delivered 7 of 10 records: the page in flight when the sink died had already been marked delivered, so the resumed run skipped it as a duplicate — permanent, silent data loss. The fix is the peekDeliveries / commitDeliveries split, with both the ledger commit and the checkpoint commit after the sink resolves. Recorded here because the failure mode is the interesting artifact, not the fix.

Honest gaps in the new plane (not hidden)

1. Delivery is AT-LEAST-ONCE, not exactly-once. Two concurrent runs on the same (source, partition) can both peek "new" and both emit. envelope_id is deterministic so a dedup-capable sink survives it. There is no per-(source, partition) run lock. Do not describe this plane as exactly-once.
2. No CDC source exists. The envelope, the operations and the runner handle change events, and JsonlFileConnector emits real deletes from an _op field — but this repo still has no Debezium/WAL/replication-slot consumer. P062's contract is met; a CDC connector is still MISSING.
3. 0pt does not schedule itself. opt_source_schedule is a declaration ACT0 would read; there is no cron daemon here.
4. The sink is caller-supplied. The runner stops at the sink; the lakehouse write boundary is ST0RE's (P066, another workflow's repo).
5. Nothing here is deployed. See the deployment note below.
6. QualityService's own rule/run/issue registry is still in-process Maps (src/quality/quality-service.ts:47-50). The new enforcement path is durable; the older rule registry is not. Unchanged by this pass.
7. Python tests still run nowhere. grep -rn "pytest" .gitea/workflows .github/workflows finds nothing — services/api/tests/* (9 files) has never been executed by CI. Flagged, not fixed (it needs a workflow change and a Python toolchain decision).

Deployment reality, measured 2026-09-04

https://0pt.app/api/health answers 200 from Vercel with {"db":"not_configured"}, and https://0pt.app/api/0pt/sources answers 404 — the deployed build predates the entire app/api/0pt/* capability layer. https://api.0pt.app/health answers 404 (the FastAPI engine is not reachable), and .gitea/workflows/cd.yml still guards on COOLIFY_APP_ID = TODO-no-coolify-app-provisioned-yet. So: none of the capability layer — neither this pass's work nor P010/P014's — is running in production, and P060's durability guarantee cannot be exercised there, because production has no database configured and does not serve the routes. Verified with curl; not inferred.

P065 — why it was not closed

Closing P065 means turning services/api/workers/resolve.py into an adapter over the canonical src/er/entity-resolution-service.ts. That change cannot be verified from this repo today: services/api is not built by this repo's Dockerfile (it builds only the Next.js app), api.0pt.app answers 404, and no CI workflow runs its pytest suite. A blind rewrite would ship untested into a path nobody can exercise — the exact failure mode this programme exists to stop.

The removal plan, for whoever picks it up:

1. Expose the canonical ER capability as POST /api/0pt/er/candidates (service already exists; only a route + Zod schema are missing).
2. Add a Python CI job that actually runs services/api/tests.
3. Replace resolve.py's body with an HTTP adapter over that route, fail-closed when it is unreachable — never falling back to the local scorer, which would preserve the duplicate under a different name.
4. Port test_resolve_determinism.py to assert the adapter's contract, and delete the fuzzy scorer.