BI-P129/P130/P131 — analytics ConnectorProvider: native (verified), dlt + Singer (honest readiness) #37

2026-09-07T04:22:38Z

admin commented

2026-09-07 04:22:38 +00:00

What this is

BI-P129 asks for a 0pt ConnectorProvider covering native, dlt, Meltano/Singer, with Airbyte optional "where legal/license architecture accepts it". BI-P130 and BI-P131 are the two third-party providers.

This PR opens with the seam and the native provider. dlt (BI-P130) and Singer (BI-P131) land on this branch next; the PR body is updated as each arrives.

What is deliberately NOT here

A second ingestion stack. 0pt already has the P061 connector contract, the P062 runner with durable checkpoints and crash/resume tests, the schema registry, the quality gate and the idempotency ledger. A provider hands back a P061 Connector; the highway is the one that exists.

The four gates

What the P061 stack has no opinion about is where a run LANDS, and that is an analytics contract question. All four are fail-closed and each is attacked twice — with the guard (must refuse) and without it (must succeed):

tier — RAW, or NORMALIZED with a declared normalization. Never directly CURATED/SERVING. BI-P003's "no business dashboard reads arbitrary raw ingestion tables" is defeated by relabelling the landing table SERVING, so the tier is checked at WRITE, not only at read.
contract coverage — every field discovery actually reported must be declared in the BI-P004 contract. An undeclared column is an unclassified column. The counterfactual test lands personnummer into a RAW table whose contract says pii_classification: "none", and the runner reports success. That is the harm, shown rather than asserted.
physical home — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE (st0re#45 shapes), never straight into ClickHouse.
provider readiness — a provider whose own providerReadiness() says verified: false cannot run in a production runtime without a recorded, named exception. The structural form of the honesty rule 0pt#35 set with MetricFlow.

Airbyte

A declared kind with no implementation. Airbyte's server is Elastic License 2.0, which restricts offering the software as a managed service to third parties — which is what 0pt does. Registering or planning one is refused at the registry, in one constant, so a future licence review has a single place to change.

BI-P129 DoD

"Two real source types ingest into canonical analytics tiers." Proven with the two real heterogeneous P061 connectors — a JSONL file on actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API driven through a real fetch/Response cycle (cursor = opaque server token). Both go through the real runner into the declared RAW tier, paged, 9 rows landed each, distinct envelope ids.

Verification

24 tests in tests/analytics/analytics-ingestion-providers.test.ts. Mutation-checked: neutering gates 1 and 2 fails 5 of them, so they are load-bearing rather than incidental. tsc -p tsconfig.check.json clean; eslint clean on changed files.

🤖 Generated with Claude Code

## What this is BI-P129 asks for a 0pt `ConnectorProvider` covering `native`, `dlt`, `Meltano/Singer`, with Airbyte optional "where legal/license architecture accepts it". BI-P130 and BI-P131 are the two third-party providers. This PR opens with the seam and the **native** provider. dlt (BI-P130) and Singer (BI-P131) land on this branch next; the PR body is updated as each arrives. ## What is deliberately NOT here A second ingestion stack. 0pt already has the P061 connector contract, the P062 runner with durable checkpoints and crash/resume tests, the schema registry, the quality gate and the idempotency ledger. A provider hands back a P061 `Connector`; the highway is the one that exists. ## The four gates What the P061 stack has no opinion about is where a run LANDS, and that is an analytics contract question. All four are fail-closed and each is attacked twice — with the guard (must refuse) and without it (must succeed): 1. **tier** — RAW, or NORMALIZED with a declared normalization. Never directly CURATED/SERVING. BI-P003's "no business dashboard reads arbitrary raw ingestion tables" is defeated by relabelling the landing table SERVING, so the tier is checked at WRITE, not only at read. 2. **contract coverage** — every field discovery actually reported must be declared in the BI-P004 contract. An undeclared column is an unclassified column. The counterfactual test lands `personnummer` into a RAW table whose contract says `pii_classification: "none"`, and the runner reports success. That is the harm, shown rather than asserted. 3. **physical home** — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE (st0re#45 shapes), never straight into ClickHouse. 4. **provider readiness** — a provider whose own `providerReadiness()` says `verified: false` cannot run in a production runtime without a recorded, named exception. The structural form of the honesty rule `0pt#35` set with MetricFlow. ## Airbyte A declared kind with no implementation. Airbyte's server is Elastic License 2.0, which restricts offering the software as a managed service to third parties — which is what 0pt does. Registering or planning one is refused at the registry, in one constant, so a future licence review has a single place to change. ## BI-P129 DoD "Two real source types ingest into canonical analytics tiers." Proven with the two real heterogeneous P061 connectors — a JSONL file on actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API driven through a real fetch/Response cycle (cursor = opaque server token). Both go through the real runner into the declared RAW tier, paged, 9 rows landed each, distinct envelope ids. ## Verification 24 tests in `tests/analytics/analytics-ingestion-providers.test.ts`. Mutation-checked: neutering gates 1 and 2 fails 5 of them, so they are load-bearing rather than incidental. `tsc -p tsconfig.check.json` clean; eslint clean on changed files. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

admin added 1 commit 2026-09-07 04:22:39 +00:00

feat(analytics): BI-P129 — ConnectorProvider, and four gates on where data lands

CI (SIAX Cloud) / security (push) Successful in 27s

Details

Security Scan (SIAX Cloud) / security (push) Successful in 30s

Details

CI (SIAX Cloud) / quality (push) Successful in 39s

Details

CI (SIAX Cloud) / security (pull_request) Successful in 43s

Details

CI (SIAX Cloud) / integration (push) Successful in 51s

Details

CI (SIAX Cloud) / quality (pull_request) Successful in 54s

Details

CI (SIAX Cloud) / integration (pull_request) Successful in 1m15s

Details

Security Scan (SIAX Cloud) / security (pull_request) Successful in 13s

Details

b5ab885a67

BI-P129 names four provider kinds (native, dlt, Meltano/Singer, Airbyte-if-
licensed). This commit ships the seam and the `native` provider; dlt and
Singer follow on this branch.

What is deliberately NOT here: a second ingestion stack. 0pt already has the
P061 connector contract, the P062 runner with durable checkpoints and
crash/resume tests, the schema registry, the quality gate and the idempotency
ledger. A provider's job is to hand back a P061 `Connector`; the highway is
the one that exists.

What is new is the question the P061 stack has no opinion about: an ingestion
run has to LAND somewhere, and where it lands is an analytics contract
question. Four fail-closed gates, each attacked with the guard (must refuse)
AND without it (must succeed — otherwise the guard is decoration):

  1. tier — source data lands in RAW, or NORMALIZED with a declared
     normalization. Never directly in CURATED/SERVING. BI-P003's "no business
     dashboard reads arbitrary raw ingestion tables" is defeated in one line
     by relabelling the landing table SERVING; the tier is therefore checked
     at WRITE, not only at read.
  2. contract coverage — every field discovery actually reported must be
     declared in the BI-P004 contract. An undeclared column is an
     unclassified column. The counterfactual test lands `personnummer` into a
     RAW table whose contract says pii_classification "none", and the runner
     reports success: that is the harm, shown rather than asserted.
  3. physical home — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE
     (st0re#45 shapes), never straight into ClickHouse.
  4. provider readiness — a provider whose own providerReadiness() says
     verified:false cannot run in a production runtime without a recorded,
     named exception. This is the structural form of the honesty rule 0pt#35
     set with MetricFlow.

Airbyte is a declared kind with NO implementation: its server is ELv2, which
restricts offering the software as a managed service, and that is exactly
what 0pt does. Registering or planning one is refused at the registry, in one
place, so a future licence review has a single constant to change.

BI-P129 DoD — "two real source types ingest into canonical analytics tiers" —
is proven with the two real, heterogeneous P061 connectors: a JSONL file on
actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API
driven through a real fetch/Response cycle (cursor = opaque server token).
Both run through the real runner into the declared RAW tier, paged, with
9 rows landed each and distinct envelope ids.

Verification: 24 tests. Mutation-checked — neutering gates 1 and 2 fails 5 of
them, so they are load-bearing rather than incidental.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CODEOWNERS rules requested review from sax3l 2026-09-07 04:22:39 +00:00

admin merged commit into main

2026-09-07 05:39:47 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: admin/0pt#37