feat: SIAX Platforms P0 wave 2 — real-contract platform clients + Data-OS capability layer #25

Merged

admin merged 5 commits from feat/siax-platforms-p0-data-os into main 2026-08-31 20:26:42 +00:00

Conversation 0 Commits 5 Files Changed 46

+6765 -208

admin commented 2026-08-31 20:18:52 +00:00

Owner

Copy Link

Copy Source

Summary

SIAX Platforms P0 wave 2 for 0pt (Data OS): replaces the mock-only cross-product clients with real-contract clients, and adds the first slice of the P0 Data-OS capability layer (Master-DoD §10).

Cross-product clients (§10 "Cross-product clients")

Rewrote all four platform clients (src/clients/{cl0ud,st0re,n0d,aud0}-client.ts) against the actual wire contracts verified on the sibling repos' real main branches this session — three of the four pre-existing paths never existed on the target service:

• CL0UD: POST /api/policy/evaluate (camelCase {data} envelope), not the invented /v1/cl0ud/policies/evaluate.
• ST0RE: camelCase ArtifactInput on /v1/st0re/artifacts/register incl. X-Tenant-Id/X-Actor-Id/X-Idempotency-Key, not the invented snake_case object_location{} body (would 400).
• N0D: required_role + resource_requirements + body idempotency_key + CloudEvents callback_url per N0D's real createJobSchema; adds getJob/getJobResult polling.
• AUD0: /v1/aud0/openlineage is AUD0's only lineage ingest — /v1/aud0/lineage/events never existed. Legacy lineage shape converts to a spec-shaped OpenLineage RunEvent.

Production can no longer silently fall back to mocks (§3/§10): src/platform/dependency-mode.ts makes every client factory throw SiaxError(DEPENDENCY_UNAVAILABLE) at construction when NODE_ENV=production and no URL is configured. The Next.js build phase (NEXT_PHASE=phase-production-build) is explicitly exempted so next build's page-data collection can still compile.

Also new: src/platform/siax-context.ts + src/platform/errors.ts — a documented mirror of cl0ud's canonical SiaxRequestContext + 12-code error vocabulary (merged to cl0ud main today), pinned by contract tests so drift is caught. src/platform/resource-ref.ts implements the canonical product:resourceType:resourceId[@version] reference format (§5).

P0 Data-OS capability layer (§10)

• Source Registry: rights/license/purpose metadata, provenance, freshness SLA, health checks, describe/query. Purpose gate fails closed (no rights record ⇒ deny).
• Search: OpenSearch REST adapter (no SDK) + tenant/purpose/rights-filtered query, versioned index definitions, bulk, generation-per-rebuild with atomic alias cutover, rollback, failed-rebuild isolation. OpenSearch is never canonical (§42).
• Quality: versioned rule registry (Great Expectations JSON interchange), local engine for the GE core subset (never silently passes an unsupported expectation type), runs/results/issues/score/trend, whole-run delegation to a GE backend via N0D job 0pt.quality.ge_run when needed.
• Dataset: immutable numbered versions holding canonical st0re:artifact:... refs only, idempotent createVersion, export registers the ST0RE artifact + AUD0 evidence + OpenLineage.
• Metadata: OpenMetadata projection adapter (never canonical) + rebuild path.
• Analytics: ClickHouse HTTP adapter, closed metric-query DSL with a forced tenant predicate, query limits, TTL retention. Zero connection to b0ts/quarantined pipelines (legal boundary respected — nothing here touches services/scrapers/*).
• Features: versioned feature-view registry + real Feast get-online-features adapter; materialize/get-historical are N0D delegation contracts.
• Entity Resolution: generic candidate API (deterministic blocking + transparent Jaccard/Levenshtein scoring + per-field explanations), product hard-rule hook (VIN-style rules override both directions), human-review handoff, Splink batch delegated via N0D contract.

Full 8-level Master-DoD classification per capability is in STATUS.md, including the honest gaps that still block PRODUCTION_READY (in-memory persistence, no CL0UD capability registration yet, no AUD0 outbox, no API routes for the new services, missing 0pt-motor executors for delegated jobs).

CI

New blocking integration job in .gitea/workflows/ci.yml running tests/integration/* against live OpenSearch 2.11 + ClickHouse 24.8 service containers, using the self-identifying-container discipline documented in sax3l/st0re's ci.yml (repo-unique aliases, run-unique identity badges, getent-enumerated selection, no-match = hard fail — guards against the measured alias-collision/DNS-round-robin defects across repos on these runners).

Test plan

• pnpm run typecheck — clean
• pnpm run test — 249 passed / 11 skipped (skips are the honest env-gated integration suites when run without live backends)
• pnpm run build — clean (next build)
• pnpm vitest run tests/integration against local OpenSearch 2.11 + ClickHouse 24.8 containers — 10/10 passed (tenant isolation, purpose/rights filtering, generation rebuild/rollback, projection-disaster rebuild, closed analytics DSL, injection-surface refusals)
• CI green on this PR (blocking quality + new blocking integration job)

Legal boundary (unchanged, respected)

No b0ts/Ratsit/person-PII code touched. No connection to quarantined market='sverige'/'norge' data. No live scraping. services/scrapers/* untouched.

🤖 Generated with Claude Code

## Summary SIAX Platforms P0 wave 2 for 0pt (Data OS): replaces the mock-only cross-product clients with real-contract clients, and adds the first slice of the P0 Data-OS capability layer (Master-DoD §10). ### Cross-product clients (§10 "Cross-product clients") Rewrote all four platform clients (`src/clients/{cl0ud,st0re,n0d,aud0}-client.ts`) against the **actual wire contracts** verified on the sibling repos' real `main` branches this session — three of the four pre-existing paths never existed on the target service: - CL0UD: `POST /api/policy/evaluate` (camelCase `{data}` envelope), not the invented `/v1/cl0ud/policies/evaluate`. - ST0RE: camelCase `ArtifactInput` on `/v1/st0re/artifacts/register` incl. `X-Tenant-Id`/`X-Actor-Id`/`X-Idempotency-Key`, not the invented snake_case `object_location{}` body (would 400). - N0D: `required_role` + `resource_requirements` + body `idempotency_key` + CloudEvents `callback_url` per N0D's real `createJobSchema`; adds `getJob`/`getJobResult` polling. - AUD0: `/v1/aud0/openlineage` is AUD0's only lineage ingest — `/v1/aud0/lineage/events` never existed. Legacy lineage shape converts to a spec-shaped OpenLineage RunEvent. **Production can no longer silently fall back to mocks** (§3/§10): `src/platform/dependency-mode.ts` makes every client factory throw `SiaxError(DEPENDENCY_UNAVAILABLE)` at construction when `NODE_ENV=production` and no URL is configured. The Next.js build phase (`NEXT_PHASE=phase-production-build`) is explicitly exempted so `next build`'s page-data collection can still compile. Also new: `src/platform/siax-context.ts` + `src/platform/errors.ts` — a documented mirror of cl0ud's canonical `SiaxRequestContext` + 12-code error vocabulary (merged to cl0ud main today), pinned by contract tests so drift is caught. `src/platform/resource-ref.ts` implements the canonical `product:resourceType:resourceId[@version]` reference format (§5). ### P0 Data-OS capability layer (§10) - **Source Registry**: rights/license/purpose metadata, provenance, freshness SLA, health checks, describe/query. Purpose gate fails closed (no rights record ⇒ deny). - **Search**: OpenSearch REST adapter (no SDK) + tenant/purpose/rights-filtered query, versioned index definitions, bulk, generation-per-rebuild with atomic alias cutover, rollback, failed-rebuild isolation. OpenSearch is never canonical (§42). - **Quality**: versioned rule registry (Great Expectations JSON interchange), local engine for the GE core subset (never silently passes an unsupported expectation type), runs/results/issues/score/trend, whole-run delegation to a GE backend via N0D job `0pt.quality.ge_run` when needed. - **Dataset**: immutable numbered versions holding canonical `st0re:artifact:...` refs only, idempotent `createVersion`, export registers the ST0RE artifact + AUD0 evidence + OpenLineage. - **Metadata**: OpenMetadata projection adapter (never canonical) + rebuild path. - **Analytics**: ClickHouse HTTP adapter, closed metric-query DSL with a forced tenant predicate, query limits, TTL retention. Zero connection to b0ts/quarantined pipelines (legal boundary respected — nothing here touches `services/scrapers/*`). - **Features**: versioned feature-view registry + real Feast `get-online-features` adapter; materialize/get-historical are N0D delegation contracts. - **Entity Resolution**: generic candidate API (deterministic blocking + transparent Jaccard/Levenshtein scoring + per-field explanations), product hard-rule hook (VIN-style rules override both directions), human-review handoff, Splink batch delegated via N0D contract. Full 8-level Master-DoD classification per capability is in `STATUS.md`, including the honest gaps that still block PRODUCTION_READY (in-memory persistence, no CL0UD capability registration yet, no AUD0 outbox, no API routes for the new services, missing 0pt-motor executors for delegated jobs). ### CI New blocking `integration` job in `.gitea/workflows/ci.yml` running `tests/integration/*` against **live OpenSearch 2.11 + ClickHouse 24.8** service containers, using the self-identifying-container discipline documented in sax3l/st0re's `ci.yml` (repo-unique aliases, run-unique identity badges, `getent`-enumerated selection, no-match = hard fail — guards against the measured alias-collision/DNS-round-robin defects across repos on these runners). ## Test plan - [x] `pnpm run typecheck` — clean - [x] `pnpm run test` — 249 passed / 11 skipped (skips are the honest env-gated integration suites when run without live backends) - [x] `pnpm run build` — clean (`next build`) - [x] `pnpm vitest run tests/integration` against local OpenSearch 2.11 + ClickHouse 24.8 containers — 10/10 passed (tenant isolation, purpose/rights filtering, generation rebuild/rollback, projection-disaster rebuild, closed analytics DSL, injection-surface refusals) - [ ] CI green on this PR (blocking `quality` + new blocking `integration` job) ## Legal boundary (unchanged, respected) No b0ts/Ratsit/person-PII code touched. No connection to quarantined `market='sverige'/'norge'` data. No live scraping. `services/scrapers/*` untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

admin added 4 commits 2026-08-31 20:18:53 +00:00

feat(platform): canonical SIAX context/errors + real-contract clients, no prod mock fallback ... b7038d7675

- src/platform/: SiaxRequestContext + propagation headers + 12 canonical
  error codes (documented mirror of cl0ud packages/contract), SiaxError,
  resource refs (st0re:artifact:...), and the §3/§10 dependency-mode guard:
  production runtime + unconfigured dependency/store => hard
  DEPENDENCY_UNAVAILABLE at construction, never silent mock/in-memory.
- src/clients/: all four platform clients rewritten against the REAL wire
  contracts verified on sibling main branches this session:
  * cl0ud: POST /api/policy/evaluate ({data} envelope, allow+requiresApproval)
    - the old /v1/cl0ud/policies/evaluate path never existed on CL0UD
  * st0re: camelCase ArtifactInput on /v1/st0re/artifacts/register incl.
    X-Tenant-Id/X-Actor-Id + X-Idempotency-Key - the old snake_case body
    would have been a 400 against the live registry
  * n0d: required_role + resource_requirements + body idempotency_key +
    CloudEvents callback_url per n0d createJobSchema; adds getJob/getJobResult
  * aud0: /v1/aud0/openlineage is AUD0's only lineage ingest -
    /v1/aud0/lineage/events never existed; legacy shape converts to a
    spec-shaped OpenLineage RunEvent (deterministic runId dedup)
- src/lineage/: OpenLineage subset types + LineageEmitter (never-throw,
  §16 evidence semantics)
- shared SiaxHttp transport: timeouts, typed SiaxError mapping, context
  propagation, no raw upstream bodies in errors

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test(contracts): per-client wire-contract suites + §3 production-guard tests ... da58176735

38 tests: recording provider double answers with the sibling services' real
response shapes; each suite asserts the exact request wire shape (path,
method, tenant/actor/idempotency headers, camelCase vs snake_case body) so a
regression to the pre-rewrite invented shapes fails loudly. Includes the
fail-closed policy cases (missing decision => deny, CL0UD down => typed
error, never implicit allow) and the production factory guard for all four
clients.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feat(data-os): P0 capability layer — source registry, search, quality, dataset versions, metadata, analytics, features, ER ... b46807da75

- Source Registry (§10): rights/license/purpose metadata, provenance,
  freshness SLA, health checks, describe/query; purpose gate FAILS CLOSED
  (no rights record => deny); cross-tenant probes => NOT_FOUND.
- Search (§10/§42): OpenSearch REST adapter (no SDK) + SearchService with
  MANDATORY tenant + purpose/rights filtering, versioned index definitions,
  bulk, generation-per-rebuild (opt-{name}--g{n}) with atomic alias cutover,
  rollback, failed-rebuild generation discard. OpenSearch never canonical.
- Quality (§10): versioned rule registry (GE-JSON interchange), local
  engine for the GE core subset (never silently passes unsupported types),
  runs/results/issues/score/trend, whole-run delegation to the GE backend
  via N0D (0pt.quality.ge_run) - never fabricated results.
- Dataset (§10/§5/§53): immutable numbered versions holding canonical
  st0re:artifact refs only, idempotent createVersion, export registers the
  ST0RE artifact + AUD0 evidence + OpenLineage.
- Metadata (§10): OpenMetadata PUT /api/v1/tables projection adapter +
  projectAll rebuild path; projection never canonical.
- Analytics (§10): ClickHouse HTTP adapter, closed metric-query DSL with
  FORCED tenant predicate + readonly/max_execution_time/max_result_rows,
  TTL retention. Zero connection to b0ts/quarantined pipelines.
- Features (§10): versioned feature-view registry, real Feast
  get-online-features adapter, materialize/get-historical as N0D delegation.
- ER (§10): generic candidate API with deterministic blocking + transparent
  scoring + per-field explanations, product hard-rule hook (force both
  directions), review-band handoff, Splink batch via N0D contract.

93 new unit/contract tests; suite 248 passed / 1 skipped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test(integration)+ci+docs: live OpenSearch/ClickHouse suites, CI service containers, STATUS.md (§56) ... c47e3b468e

- tests/integration/: 10 tests against REAL OpenSearch 2.11 (tenant
  isolation, purpose/rights filters, generation rebuild/rollback,
  projection-disaster rebuild §68) + REAL ClickHouse 24.8 (tenant-isolated
  metric DSL, TTL schema, injection-surface refusals). Honest skip without
  env - and CI fails hard if the suites skip instead of running.
- .gitea/workflows/ci.yml: new blocking 'integration' job with
  self-identifying service containers (st0re ci.yml's measured
  alias-collision discipline: run-unique cluster.name / credentials,
  getent-enumerated IP selection, no-match = hard fail).
- STATUS.md: full 8-level Master-DoD classification per capability incl.
  the honest gaps blocking PRODUCTION_READY.
- dependency-mode: NEXT_PHASE=phase-production-build is not a serving
  runtime (next build page-data collection must compile; guard re-arms at
  boot) - pinned by contract test.
- docs/02-architecture.md: superseded the accurate-but-now-fixed 'every
  client runs against its mock' paragraph; .env.example: client URLs now
  actually read from env + new P0 backend vars.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

admin referenced this issue from a commit 2026-08-31 20:20:40 +00:00

fix(ci): pin OpenSearch service container workdir — act_runner clobbers WORKDIR-relative ENTRYPOINT

admin added 1 commit 2026-08-31 20:20:40 +00:00

fix(ci): pin OpenSearch service container workdir — act_runner clobbers WORKDIR-relative ENTRYPOINT ... 62f7b0049f

PR #25 CI run 7745 job 'integration' failed before any test ran: 'exec:
"./opensearch-docker-entrypoint.sh": stat ...: no such file or directory'.
Root cause (confirmed against the job log's merged container.Config, and
reproduced locally with plain docker run -w): this runner's act_runner
merges the JOB's WorkingDir (/workspace/sax3l/0pt) into every SERVICE
container, clobbering the OpenSearch image's own WORKDIR
(/usr/share/opensearch) that its ENTRYPOINT ./opensearch-docker-entrypoint.sh
is relative to. ClickHouse's entrypoint is an absolute path so it started
fine under the same defect -- a green ClickHouse service is not evidence
this class of bug is absent elsewhere.

Fix: --workdir /usr/share/opensearch in the opt-ci-opensearch service's
options, restoring the image's own working directory. Verified locally:
'docker run -w /workspace/sax3l/0pt opensearchproject/opensearch:2.11.1'
reproduces the exact failure; '-w /usr/share/opensearch' fixes it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

admin merged commit into main 2026-08-31 20:26:42 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

siax-bot siax-ci siax-healer (siax-healer (CI self-healing agent)) admin (Simon Axelsson)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: admin/0pt#25