66 Commits

Include renames

Author	SHA1	Message	Date
admin	17bd1b40a0	feat(0pt): record legitimate-interest stance — B2B contacts, company-linked only ... Recorded decision (Simon 2026-08-31): personal data may be stored under legitimate interest ONLY as business contacts bound to a company/org entity. Gate enforces it: legitimate_interest requires purpose=business_contact and entity_linked_only=true; recordStored() requires entity_id (orgnr/rgs/LEI, hashed) and refuses person-only rows. +1 test; suite 167 pass | 1 skip; typecheck clean.	2026-09-15 02:40:52 +02:00
admin	12080fc517	feat(0pt): PII-Gate GDPR-guardrail for personal-data sources (fail-closed) ... PiiGateService makes any future personal-data source (ratsit.se and other directory domains) technically GDPR-OK before the first byte is stored: - recorded legal basis (art. 6) with basis_ref evidence, no policy => fail-closed - data minimisation via field allowlist (art. 5(1)(c)) - storage limitation + enforceRetention purge (art. 5(1)(e)) - sha256-only identity keys + log redaction (art. 25/32) - accessStatement / erase subject-rights surfaces (art. 15/17) - robots/rate confirmation required for web+api sources; no stealth/bypass - audit trail via AUD0 at risk L3 (0pt.pii_*) - gate enabled by default in SourceQueryService 10 new tests; full suite 166 pass | 1 skip; typecheck (blocking gate) clean.	2026-09-15 02:40:52 +02:00
adminandClaude Sonnet 5	4caf122129	fix(security): add SSRF guard to Fabric/OneLake adapter's sqlEndpoint ... connection_config.sqlEndpoint on POST /api/0pt/sources/{sourceId}/query was passed straight from the request body to the mssql TDS driver as the outbound connection target, with only its http(s):// scheme stripped — no hostname allowlist or private/loopback/link-local/cloud-metadata guard. This route already requires a Zitadel bearer token + tenant match (P014 tenant guard), so this was not anonymously exploitable, but it is the exact SSRF shape found and exploited-in-principle elsewhere in this estate tonight (press0), so it's worth closing as defense-in-depth. Adds src/lib/security/ssrf-guard.ts: resolves the endpoint's hostname and rejects any address in a private/loopback/link-local/cloud-metadata/ unspecified range (RFC1918, 127.0.0.0/8, 169.254.0.0/16 incl. the 169.254.169.254 metadata address, ::1, fc00::/7, fe80::/10, IPv4-mapped IPv6), failing closed on DNS resolution errors. The range-check logic is ported 1:1 from press0's src/lib/security/ssrf-guard.ts built the same night for the same bug class against a fetch()-based URL there; this module's surface differs because sqlEndpoint is a bare host[:port], not a fetchable URL. The DNS lookup is injectable (mirrors this adapter's existing fetchImpl pattern) so tests are hermetic. FabricOneLakeAdapter.getPool() now calls the guard before loadDriver()/ getToken()/opening the TDS pool, so a blocked target never reaches a real outbound connection attempt or spends an AAD token exchange. Tests: new tests/p0/fabric-onelake-ssrf-guard.test.ts unit-tests the guard module directly (range boundaries, IPv4-mapped-IPv6 bypass attempt, fail-closed on DNS failure/empty answer). New adapter-level tests in tests/p0/fabric-onelake-adapter.test.ts prove (a) a private-IP-resolving sqlEndpoint is rejected before fetch/connect is ever called, (b) a literal loopback sqlEndpoint is rejected the same way, and (c) a normal external Fabric hostname still passes the guard and completes the query — the guard doesn't false-block legitimate targets. Existing adapter tests are updated to inject a deterministic DNS resolver so they stay hermetic instead of depending on real DNS. Full suite: 997 passed, 0 failed, 184 skipped (self-skipping realinfra tests with no live backend this run). typecheck and lint clean on all touched files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-11 02:18:23 +02:00
adminandClaude Opus 5	db868f4bdf	fix(prod): close legacy storage fallbacks in orders/subscriptions/blobs ... The `resolveStoreMode` kill switch (Master-DoD §3/§10) covered the newer `src/repositories/*` stores but not two older paths, both of which are live production execution paths today. 1. LEGACY NEXT.JS STORES. `lib/orders-store.ts` and `lib/subscriptions-store.ts` predate `src/platform/dependency-mode.ts` and called `getSql()` directly, silently falling back to module-global `Map`s when no database is configured. For orders, payments and subscriptions that is not a degraded mode, it is invisible loss of money-bearing state: the API returns 201 and the row evaporates on the next restart or on any other replica. The live 0pt deployment (Coolify app 54) runs NODE_ENV=production with DATABASE_URL, POSTGRES_URL and POSTGRES_PRISMA_URL all unset — so this is the behavior in production right now, across 7 API routes. These now go through `getSqlForDurableStore()`, which reuses the SAME `resolveStoreMode` decision every `src/repositories/*` store already uses rather than re-implementing a second production check. The reuse is load-bearing: `isProductionRuntime` deliberately exempts `NEXT_PHASE=phase-production-build`, because `next build` runs page-data collection under NODE_ENV=production while serving nothing. A naive `process.env.NODE_ENV === "production"` test here would have failed the BUILD instead of protecting a request path. 2. LOCALBLOBSTORE. It writes document bytes to a container-local disk while `document-capture-service` registers an artifact pointer claiming the bytes are stored — durable object storage silently swapped for ephemeral disk. It is the default blob store for both the capture and extraction services. In production it now raises DEPENDENCY_UNAVAILABLE (capability blocked: no ST0RE bytes path exists yet) instead of fabricating success. Asserted per-operation, not in the constructor, so merely wiring the services is not fatal — only actually persisting or reading bytes is. Evidence: full suite 956 passed / 184 skipped, typecheck clean, dependency-cruiser 0 errors, lint identical to main's pre-existing baseline (17 problems, none in changed files). Both guards asserted in both directions and mutation-tested: bypassing resolveStoreMode -> 2 failed; dropping the write assert -> 2 failed; dropping the read assert -> 1 failed; all restored -> 8 passed. 7 non-test call sites. DEPLOYMENT NOTE: merging changes no runtime (no Gitea webhook exists on any repo; deploys are manual). Deploying this build REQUIRES provisioning DATABASE_URL for 0pt first, or the order/subscription routes will correctly return DEPENDENCY_UNAVAILABLE instead of silently accepting orders into memory. That provisioning is a Simon action. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-09 17:38:51 +02:00
admin	2d34cb1629	feat(P-001, P-002): 0pt→N0D quality delegation + internal OpenAPI	2026-09-08 22:38:47 +02:00
admin	3edc56dcdb	fix: slutgranskning av dataset/search/ER-ytorna + STATUS-sanning ... Slutgate inför merge (per fortsättningsplan): Kontrakt: CreateDatasetVersionInput.artifact_refs → readonly string[] (anropare får skicka 'as const'; servicen muterar aldrig); postgres.js tx.array() får muterbar kopia vid INSERT. Testkorrigeringar (stale mot starkare implementation — aldrig svagare): - entity-resolution-http: request(body, undefined) triggade default-param och skickade GILTIGT token → 200. Tomt token = 'utan token' → 401 fail-closed. - entity-resolution: /two records/ → /between 2 and 5000/ (bounds-aware validering) - search-service: omformat purpose-filter-test + NY negativ kontroll — rebuild avvisar dokument utan purpose_tags (fail closed, generation discarded) - dataset-postgres.realinfra: sätter DATABASE_URL i setup så den durabla AUD0-outboxen aktiveras (testet ljög annars om durabilitet) — 5/5 mot live Postgres STATUS.md: AUD0-klient + lineage-rader (outbox nu real-infra testad), gap-listan (persistence/routes/outbox stängda, exakt vad som återstår), P065 STÄNGD med closure-record (thin adapter, v3-semantik, hard-negative transitivitet, fail-closed 401 — 4/4 planpunkter verifierade). Verifierat: typecheck EXIT 0 · 945/945 unit · 5/5 real-Postgres · next build EXIT 0	2026-09-07 19:31:22 +02:00
admin	35a8ff0f2a	feat: add generic ER composition root	2026-09-07 18:18:02 +02:00
admin	243918a255	feat: harden canonical ER service and idempotent batch submission	2026-09-07 18:17:52 +02:00
admin	00e4636655	refactor: centralize ER request validation	2026-09-07 18:16:43 +02:00
admin	f3a8cf6cc2	feat: add production search composition root	2026-09-07 18:11:31 +02:00
admin	82f7475b91	fix: audit only newly created durable search definitions	2026-09-07 18:11:06 +02:00
admin	8dfd429a61	refactor: return stable creation metadata for search definition audit	2026-09-07 18:10:13 +02:00
admin	9961ac7e03	refactor: make search definitions restart-safe	2026-09-07 18:09:05 +02:00
admin	dd01fe76c9	feat: add durable versioned search definition repository	2026-09-07 18:08:09 +02:00
admin	70fd81dc4d	feat: share durable dataset services across route modules	2026-09-07 18:06:21 +02:00
admin	3c5850fbe5	refactor: persist dataset versions and make lineage retry-stable	2026-09-07 18:03:21 +02:00
admin	3ccdcf0cf6	fix: use postgres arrays and normalized idempotency comparison	2026-09-07 18:02:38 +02:00
admin	906fe9ac37	feat: add durable dataset version repository	2026-09-07 18:01:56 +02:00
admin	a3bad9aa12	refactor: extract canonical dataset version types	2026-09-07 18:01:24 +02:00
admin	ea02f4e9f7	refactor: persist DatasetService through repository boundary	2026-09-07 18:00:47 +02:00
admin	5dbd53045c	feat: add durable dataset registry repository	2026-09-07 18:00:29 +02:00
admin	f0958cf708	fix: centralize lineage durability in AUD0 client boundary	2026-09-07 17:54:55 +02:00
admin	020a4be7e8	feat(evidence): make standard AUD0 client durable by default	2026-09-07 17:23:20 +02:00
admin	9f46e12447	feat(lineage): route production evidence through durable AUD0 outbox	2026-09-07 17:14:54 +02:00
admin	79942a2431	feat(evidence): implement durable AUD0 outbox and dispatcher	2026-09-07 17:13:02 +02:00
siax-agentandClaude Opus 5	0e0e1fec56	fix(BI-P147): flytta MetricFlow-pinnen ovanför licensgränsen (0.208.1 var BUSL) ... Certifieringslanan (BI-P147) hittade det; jag verifierade det oberoende mot LICENSE-filen vid respektive git-tagg 2026-09-07: v0.208.1 Business Source License 1.1 — source-available, INTE öppen källkod v0.209.0 Apache License 2.0 dbt Labs licensierade om MetricFlow vid 0.209.0. Den gamla pinnen låg alltså exakt en patch under gränsen. För en självhostad plattform vars hela premiss är att inte ha ett proprietärt runtime-beroende är det en licensdefekt, inte en smaksak: BUSL begränsar produktionsanvändning fram till sitt change date. Notera metodpunkten: PyPI redovisar INGET licensfält för någondera versionen, så en metadatakontroll hade inte hittat något. Skillnaden syns bara i LICENSE-filen vid taggen. Ny grind: `tests/unit/metricflow-licence.test.ts` asserterar att pinnen ligger på eller över `METRICFLOW_FIRST_OPEN_SOURCE_VERSION`, och — som negativ kontroll — att alla tre kända BUSL-versionerna (0.207.3, 0.208.0, 0.208.1) faktiskt förkastas av samma jämförelse. Utan den andra assertionen hade den första kunnat passera med en jämförelse som inte diskriminerar. Detta gör INTE MetricFlow verifierad: `providerReadiness()` returnerar fortfarande `verified: false`, ingen supply-chain-granskning är gjord och providern har aldrig körts (ingen Python-runtime är provisionerad). Att laga licensen är inte samma sak som att bevisa komponenten. typecheck rent, 2/2 tester gröna. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 09:04:30 +02:00
Claude Opus 5	7441de5429	BI-P103: the cited evidence/bi-p103 did not exist — measure it ... Same shape as the BI-P029 finding in the previous commit. `bi-slo.ts` said "evidence/bi-p103 records the first baselines". There was no evidence/bi-p103. The catalogue and the evaluator were both real and both well covered — against SYNTHETIC events. A synthetic event carries whatever duration the test author typed, so that suite proves the arithmetic and nothing about the system: it cannot tell you whether a dashboard query takes 8 ms or 8 s. BI-P103's DoD is one sentence, "measured baselines exist", and it was the part not done. tests/integration/bi-p103-slo-baseline.integration.test.ts runs 460 real queries against a live ClickHouse over a 400 000-row fact table -- 120 cold dashboard queries (each a different bucket so the BI-P020 cache genuinely misses), the same 120 again through the same shipped cache now hitting, and 220 exploration queries, that last number chosen so explore.p99's min_samples of 200 is MET rather than reported low-confidence. Every duration is wall-clock around the real call, and the telemetry goes through the shipped producer and the shipped evaluator, so the evidence is written by the code under test. Measured: 7 of 11 objectives produced a number, `fully_measured` false, measured_fraction 0.64. The four that did not each say why -- dashboard freshness is a property of the dataset, the exploration timeout is a configured ceiling, report completion has no producer in 0pt at all, and report delivery is N0TIFY's receipt. A baseline table listing only the rows that worked is how an estate comes to believe it measures more than it does. The evidence also records what is NARROWER than production, because these numbers will be quoted: the cached path is the in-process reference cache and not Valkey, every latency is the engine round trip with no HTTP or authorization layer, and there is no concurrency. The suite is adversarial about its own inputs: a "cold" sample that came from the cache fails, a cache "hit" that missed fails, a cached p95 that is not faster than the cold p95 fails, and dropping the exploration sample to 60 fails on explore.p99 being low-confidence rather than quietly recording a p99 over 60 requests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 08:27:37 +02:00
Claude Opus 5	468a87c00c	BI-P029: the cited parity test did not exist, and it caught a real bug ... `semantic-provider.ts` cited tests/integration/semantic-parity.integration.test.ts as the proof of the one BI-P029 DoD surface it claimed was closed, and `providerReadiness().proven` asserted the comparison had been made against live ClickHouse. The file did not exist. The citation was the evidence. Writing it exposed a divergence that had been shipping: `compileRequestToClickHouse` had no notion of a metric's OWN filters (BI-P028 `MetricSpec.filters`). `ResolvedMetric` did not carry them and the compiler never asked, so a metric defined as "invoiced revenue EXCLUDING internal transfers" executed as "invoiced revenue". On the committed fixture that is 11 249 instead of 1 250 for one country -- a plausible-looking number, 9x too large, under the correct canonical metric id. The registry, the governance checks and the MetricFlow manifest all carried the filter; only the thing that actually ran did not. * `ResolvedMetric` gains `definition_filters`, already resolved onto the physical dim_names key so the compilers stay dumb. * The ClickHouse compiler folds them into THAT metric's conditional aggregate, never into the shared WHERE -- two metrics in one request may exclude different rows, and a shared predicate would apply one metric's meaning to the other. `renderFilter`'s uniquifier becomes a string so request filter `0` and metric filter `0_0m` cannot collide in the parameter map. * `compileRequestToSemantic` carries them through, and `MetricFlowProvider.compile` emits them per metric. The test runs both compile paths against one live ClickHouse: 0pt's `sumIf` compiler, and a renderer driven only by the provider's structured query that shares no SQL-generation code and emits `sum(if(...))` with inlined literals. It asserts the two SQL strings differ, so parity is not vacuous. Backing the fix out reproduces the pre-fix numbers, and the suite records them itself rather than quoting a console: evidence/bi-p029/semantic-parity.json. Three of BI-P029's four DoD surfaces stay OPEN and still say so: MetricFlow itself is not executed (no Python runtime is provisioned here), Superset is not implemented, and the AI Analyst path has not been compared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 08:23:43 +02:00
Claude Opus 5	b5ab885a67	feat(analytics): BI-P129 — ConnectorProvider, and four gates on where data lands ... BI-P129 names four provider kinds (native, dlt, Meltano/Singer, Airbyte-if- licensed). This commit ships the seam and the `native` provider; dlt and Singer follow on this branch. What is deliberately NOT here: a second ingestion stack. 0pt already has the P061 connector contract, the P062 runner with durable checkpoints and crash/resume tests, the schema registry, the quality gate and the idempotency ledger. A provider's job is to hand back a P061 `Connector`; the highway is the one that exists. What is new is the question the P061 stack has no opinion about: an ingestion run has to LAND somewhere, and where it lands is an analytics contract question. Four fail-closed gates, each attacked with the guard (must refuse) AND without it (must succeed — otherwise the guard is decoration): 1. tier — source data lands in RAW, or NORMALIZED with a declared normalization. Never directly in CURATED/SERVING. BI-P003's "no business dashboard reads arbitrary raw ingestion tables" is defeated in one line by relabelling the landing table SERVING; the tier is therefore checked at WRITE, not only at read. 2. contract coverage — every field discovery actually reported must be declared in the BI-P004 contract. An undeclared column is an unclassified column. The counterfactual test lands `personnummer` into a RAW table whose contract says pii_classification "none", and the runner reports success: that is the harm, shown rather than asserted. 3. physical home — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE (st0re#45 shapes), never straight into ClickHouse. 4. provider readiness — a provider whose own providerReadiness() says verified:false cannot run in a production runtime without a recorded, named exception. This is the structural form of the honesty rule 0pt#35 set with MetricFlow. Airbyte is a declared kind with NO implementation: its server is ELv2, which restricts offering the software as a managed service, and that is exactly what 0pt does. Registering or planning one is refused at the registry, in one place, so a future licence review has a single constant to change. BI-P129 DoD — "two real source types ingest into canonical analytics tiers" — is proven with the two real, heterogeneous P061 connectors: a JSONL file on actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API driven through a real fetch/Response cycle (cursor = opaque server token). Both run through the real runner into the declared RAW tier, paged, with 9 rows landed each and distinct envelope ids. Verification: 24 tests. Mutation-checked — neutering gates 1 and 2 fails 5 of them, so they are load-bearing rather than incidental. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 06:22:07 +02:00
Claude Opus 5	f33cc8d32f	feat(analytics): BI-P104 — a benchmark harness that refuses to quote one number ... Six dataset profiles (small SaaS, medium business, VehIQ market, large event, wide table, geospatial), one representative question each, run on ClickHouse 26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions. evidence/bi-p104/benchmark.json. The harness's job is to stop the benchmark from lying: - overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three of the six real comparisons came out that way, one of them with a 1.6x median gap — which is exactly the headline a noisy host manufactures. - every sentence carries the range, the sample size and the stability, so it cannot be trimmed to "4.4x faster" without visibly deleting the caveats. - under 5 samples nothing is comparable at all. A three-run benchmark is an anecdote. - every engine BI-P104 names appears in the evidence, and the five nothing measured carry a reason each. The running Trino belongs to another lane, so benchmarking through it would have measured their configuration. Measured shape, warm cache, one idle host, these query shapes only: DuckDB faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on medium_business, wide_table and geospatial. One finding from the harness's own tests: an engine declared available that produced no run came out of buildEvidence() with no reason — the dangerous case, since it reads as "we could have and chose not to say". It now gets a generated reason. A second from the runs: DuckDB's progress bar writes to STDOUT and corrupted the first transcript; it is disabled explicitly and the JSON is read from a sentinel. 10 unit tests + 4 against real engines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 04:06:43 +02:00
Claude Opus 5	ed6e62809d	feat(analytics): BI-P103 — BI SLOs measured from AUD0's usage events, or declared unmeasured ... aud0#34 merged seven BI usage event types and recorded that zero producers emit them. This makes 0pt the producer and derives the SLOs from those events rather than opening a second, private measurement path that would disagree with aud0 during an incident. Three honest verdicts instead of one number: insufficient_data an availability objective over an empty window is SILENCE, not 100%. Every objective in an empty window comes back insufficient_data and fully_measured is false. A started query with no terminal event counts as unresolved — not as a success. not_measurable_here dashboard FRESHNESS is a property of the dataset (BI-P026/BI-P105 watermark) and report DELIVERY is N0TIFY's receipt. Neither is in aud0's seven types, so both are declared unmeasured rather than proxied by completion. low_confidence a p99 over 12 samples is the 12th-slowest request wearing a percentile's clothes. Reported with its sample size and not judged. The producer is an allowlist built field by named field, never a spread. The test feeds it an execution record carrying SQL text, a database error message quoting a personnummer, filter VALUES, a tenant id and a result sample, then greps the emitted events — after first proving the grep finds those strings in the input, so a clean result cannot be a broken detector. 14 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 04:01:16 +02:00
Claude Opus 5	96ff601248	feat(analytics): BI-P105 — a pre-aggregate serves only while it is provably undrifted ... A slow query is a complaint; a drifted pre-aggregate answers instantly, with confidence, and is wrong. So the router is fail-closed on eight named reasons, and the one people leave out — drift_never_verified — is the one that matters on launch day: a brand-new aggregate is exactly as unproven as a drifted one. Proved against ClickHouse 26.3.10.60, 2M rows, a real materialized view (evidence/bi-p105/materialization-drift.json): after backfill match, 48 buckets, 801 571 002 both sides GDPR erasure ALTER TABLE ... DELETE on the source. An MV is an INSERT trigger and never sees a mutation, so the aggregate kept the deleted revenue: it would have reported 801 571 002 against a true 761 580 554 — 39 990 448 overstated, 5.251%, with NOTHING stale. Staleness at the moment of the decision: 1 second. guarded router base_table (state_not_serving) counterfactual a staleness-only router SERVES it — measured, not assumed after rebuild match, and routing returns to the materialization Drift detection treats a MISSING bucket as drift even when every value present is correct (late-arriving rows are the sneakiest failure), and an empty comparison as insufficient_data rather than match — the same false green BI-P025 found in a quality check over an empty table. The measured speed-up is reported as a distribution, not a figure: p50 3.4x, but the run-to-run range spans 0.6x-7.5x on one idle host, so quoting "3.4x" as a property of the system would be misleading (BI-P104 doctrine). 19 unit tests + 4 against live ClickHouse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:56:32 +02:00
Claude Opus 5	1a12e8a882	feat(analytics): BI-P061 — dashboard releases, with the publish race measured both ways ... DRAFT -> PUBLISHED -> SUPERSEDED as a ledger (sql/010) rather than a column on the frozen version: sql/008 already makes DashboardVersion immutable at the database, and a lifecycle column there would have meant weakening that. Rollback mints a NEW version carrying the old document and publishes it, so "we went back to v3" is a timestamped fact (v5 = rollback_of_version 3) rather than a hole where v4 used to be. It is verified by BI-P060 canonical hash, and documentOf() refuses to serve a stored document whose hash no longer matches the release — a dashboard that renders an unapproved definition is worse than one that will not open. The claim worth attacking is "only one release can be live". The application check is read-then-write, so it is NOT the guarantee, and the evidence says so in both directions against Postgres 16.15: guard_absent partial unique index dropped -> BOTH concurrent publishes commit, dashboard ends with TWO live versions guard_present index in place -> loser fails 23505 on opt_analytics_dashboard_release_one_published, one live evidence/bi-p061/dashboard-release-race.json, reproducible with scripts/bi-p061-release-race.mjs. Two findings while writing the tests, both fixed here rather than papered over: the one-draft-per-dashboard index makes the two-draft race impossible to even construct (so the race is run as two create-and-publish transactions, which is what a "save and publish" button does), and an UPDATE matching zero rows raises nothing — which reads as "refused" to a careless test, so the immutability test asserts the row exists first. 23 tests: 8 unit + 15 against live Postgres. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:51:04 +02:00
Claude Opus 5	2af126dfc4	feat(analytics): BI-P060 — engine-neutral dashboard model that plans without Superset ... BI-P002 named DashboardDefinition/DashboardVersion and sql/008 stored them, but `layout_json` was a string, and a string holds anything — including a Superset export only Superset can read back. This gives the document a body and makes the DoD checkable rather than asserted. planDashboard() is the DoD: a document in, runnable AnalyticsQueryRequest objects (BI-P016) plus visualization specs out. The test greps the whole serialized plan for vendor traces after first proving the detector CAN find one in the source document — a clean grep over a broken detector is the usual way this test passes for the wrong reason. The counterfactual is a test too: a widget whose only content is a Superset slice id is REFUSED, because that is exactly the document that cannot be reconstructed. vendor_ref survives for round-tripping but is excluded from the canonical hash, so re-exporting to a new Superset instance does not fabricate a new dashboard version. No SQL surface: query widgets are parsed by parseAnalyticsQueryRequest, so `sql`, `where` and `tenant_id` are refused structurally rather than by review. 18 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:43:06 +02:00
Claude Opus 5	c4862fb7a7	docs(analytics): quote the numbers the COMMITTED evidence actually contains ... The live suites regenerate their evidence on every run, so the BI-P021 overshoot figures moved when they were last re-run — and the doc and module comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with the evidence it cites is worse than prose with no number in it. Both now match the committed file, and both say what the earlier version implied but did not state: the multiple is MACHINE-DEPENDENT and moves between runs on the same box. That is the actual finding. The bound is "roughly one poll interval of engine throughput", not a number, and quoting any specific multiple as if it were a property of the system is the mistake this correction exists to prevent. The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows read before cancellation in the current BI-P019 record, not the 6.5 billion of the earlier run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:18:12 +02:00
Claude Opus 5	c864c8c417	fix(analytics): revert 18 files reformatted by mistake, and actually apply the Iceberg fix a previous message claimed ... Two corrections, both of my own errors. 1. SCOPE CREEP, NOW REVERTED. Running `prettier --write` on whole directories reformatted 18 files this branch had no business touching — the entire merged BI-P002..P012 core (arrow/, domain/, engine/, repositories/, five analytics integration suites) and, absurdly, search-opensearch.integration.test.ts, which has nothing to do with this work. The changes were purely stylistic quote churn, but they added ~1,400 lines of noise to the diff, made the PR harder to review honestly, and invited conflicts with the four other lanes working in this estate. All 18 are now byte-identical to origin/main again (verified with `git diff origin/main --name-only`, which reports nothing for them). Formatting is now applied only to the files this branch actually owns. 2. A COMMIT MESSAGE THAT DESCRIBED A FIX THAT WAS NOT IN THE CODE. Commit 3ee0230 said the "does NOT ship the Iceberg table" assertion had been moved from a filesystem check to `git ls-files`. It had not. The patch anchor used double quotes, prettier had already converted that file to single quotes, so the replacement silently matched nothing and the script printed its success line anyway. The suite then passed — but only because I had just deleted the fixture directory, so the OLD filesystem assertion happened to be true. That is precisely the failure mode this branch keeps writing tests against: a green run that means something different from what it appears to mean. The fix is now genuinely applied and verified the way it should have been the first time — by running the suite WITH the Iceberg fixture present on disk, which is the case that previously failed. It passes, and it now also asserts that the parquet/csv fixtures ARE tracked, so a version shipping no fixture at all would fail rather than pass vacuously. 957 tests pass with ClickHouse, Postgres and DuckDB live. typecheck, eslint and prettier clean on the files this branch owns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:16:30 +02:00
Claude Opus 5	ba5c796b36	docs(analytics): anchor two security claims to the measurements behind them ... Neither changes behaviour; both replace an assertion with a measurement. BI-P014's escaping strategy rests on DuckDB treating a backslash as a literal character, so that doubling `'` is sufficient. Verified on 1.5.5 rather than assumed: `SELECT length('a\\b')` returns 4, and `SELECT 'a\''` is a parser error ("unterminated quoted string") — which is exactly what a NON-escaping backslash produces. In a dialect where `\'` closed the literal, doubling would not be enough, so the claim is worth pinning to a run. BI-P017's FROM/JOIN scan is conservative where it is unsure, and that has a cost worth naming rather than discovering: ClickHouse's `ARRAY JOIN arr` takes a COLUMN, not a table, and this guard refuses it. A false refusal an analyst can see and report is the right side of that trade; a false ADMISSION is not. Supporting it means teaching the scan which JOIN flavours take a non-table target, which is the direction that grows holes — so it stays refused until someone needs it and can add it WITH a corpus entry proving the narrowing did not open anything else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:11:19 +02:00
Claude Opus 5	3ee023017b	fix(analytics): move the DuckDB live suite out of tests/integration, and add the BI docs ... CI's integration job failed, and it failed CORRECTLY. That job provides a live OpenSearch and a live ClickHouse and then asserts that nothing in tests/integration/ skipped, on the stated principle that "a skipped suite must never masquerade as a green integration gate". The DuckDB suite needs a Python interpreter that job does not provide, so it skipped, and the guard fired — the first suite in that directory ever to skip there. Moving the file to tests/duckdb/ respects the guard rather than weakening it. Loosening it to tolerate skips would have traded a real gate for one green job, which is the trade this estate keeps regretting. The suite still runs, still fails rather than skipping when DUCKDB_PYTHON is set and DuckDB is broken, and is still pinned by the committed evidence. Also fixed: "does NOT ship the Iceberg table" asserted against the FILESYSTEM when the claim is about what is COMMITTED. The live suite rebuilds that table locally, so the assertion failed whenever both suites ran in one pass — it was testing the wrong thing rather than catching a real problem. It now asserts against `git ls-files`, and also asserts the parquet/csv fixtures ARE tracked, so a version that stopped shipping any fixture would fail too. docs/analytics/BI_QUERY_METRIC_QUALITY.md is organised around what is PROVEN and what is NOT, because the second list is the one that gets dropped. It records the five findings and the six open items, including that BI-P029's MetricFlow was never executed and that BI-P021's budget is a bound rather than a hard cap (measured 130x overshoot at a 100ms poll). It also reports, without changing it, a latent flaw in .gitea/workflows/ci.yml (a protected file this branch does not touch): the integration guard's second pattern `Tests [0-9]+ passed` never matches, because vitest writes that summary with ANSI colour codes between the words. The guard is therefore STRICTER than it reads — "any skip fails" rather than "skipped and nothing passed" — so nothing is currently unguarded, but the escape hatch does not work and someone will eventually be puzzled by it. 957 tests pass with ClickHouse, Postgres and DuckDB all live. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:08:11 +02:00
Claude Opus 5	a9ae7f25a4	feat(analytics): BI-P014 — DuckDB provider, and the measured finding that its sandbox does not work the way it looks ... BI-P014's security requirement is the whole prompt: DuckDB is a local engine with `read_csv('/etc/passwd')` built in. Two layers. LAYER 1, the mount registry: a caller names ARTIFACT IDS, never paths. The registry resolves an id, checks the artifact belongs to the caller's tenant, and realpath()s the result to confirm it is still inside the mount root once symlinks are followed. A symlink to /etc/passwd, a traversing relative path, an absolute path, and a sibling directory sharing the root's prefix (/mnt/artifacts-evil for root /mnt/artifacts) are each refused. A cross-tenant read is AUTHORIZATION_DENIED, deliberately not NOT_FOUND — an operator reading the audit trail needs to see an attempted cross-tenant read as one. LAYER 2, THE FINDING: `SET allowed_directories=[...]` ALONE IS A NO-OP. Measured on DuckDB 1.5.5. With allowed_directories set to a single mount directory and nothing else, `read_csv('/etc/passwd')` RETURNED THE FILE and an https:// URL was fetched over the network. Only when `enable_external_access = false` is ALSO set does the allow-list bind; the same reads then fail with "Permission Error: Cannot access file". `lock_configuration=true` afterwards stops a query re-opening either. So the obvious spelling of this control — set the allow-list and move on — yields a setting that appears in duckdb_settings(), looks like a sandbox, and permits everything. That is `trivy --exit-code 0` in a different costume. The bootstrap is therefore emitted as ONE unit, `assertSessionSandboxed` refuses a session missing any part of it, and the measurement itself is PINNED by a test that never skips: if someone simplifies the bootstrap, the pinned evidence (`read_etc_passwd_succeeded: true` for the naive spelling) keeps the finding attached to the build rather than to a comment that would rot. Twelve bypass attempts run against a real DuckDB with LAYER 1 REMOVED — sent straight to the engine, registry not in the picture. Zero succeeded. The counterfactual runs in the same session and is asserted: the mounted file IS readable, so "nothing succeeded" is not a sandbox that blocks everything, which would be an outage rather than a control. A locked session is then shown unable to re-open its own sandbox, all three ways. DoD, against real fixtures: Parquet aggregates correctly (SE 394.5, NO 105.5), CSV round-trips a value containing a quote, and an ICEBERG SNAPSHOT reads via iceberg_scan on an exact metadata JSON — which is not a workaround: DuckDB refuses to glob for the latest version because globbing can read uncommitted data, and an exact metadata file is what "snapshot" means. Arrow is reported UNPROVEN by `readiness()` with the reason: DuckDB's arrow extension is a network install the offline environment could not download. THREE DEFECTS IN MY OWN TESTS, found and fixed rather than tidied away: * a bypass-corpus entry was `SELECT 1 FROM (SELECT 1) t` — not an attack at all. It succeeded and was reported as a leak. A corpus entry that cannot fail is as useless as a check that cannot. Replaced with real ATTACH and COPY attempts. * the harness ran the session bootstrap outside its try block, so a statement that correctly REFUSED to re-open a locked sandbox killed the subprocess and surfaced as an unreadable "Command failed" instead of as the meaningful refusal it was. * the Iceberg fallback was `expect(evidence.iceberg ?? "unbuilt").toBeDefined()` — a check that can never fail. It let a run without the fixture ERASE the recorded Iceberg result from the evidence file while both suites stayed green. It now rebuilds the fixture when pyiceberg is available and, when it is not, asserts against the previously recorded result and marks it as carried forward. The Iceberg fixture is deliberately NOT committed: pyiceberg writes absolute paths into its metadata and Avro manifests, so a committed table resolves only on the machine that built it, and a fixture that silently fails everywhere else is worse than none. tests/fixtures/duckdb/README.md says so and gives the rebuild command; the recorded result is the artefact instead. Heavy artifacts are refused with CAPABILITY_UNAVAILABLE and pointed at N0D, per BI-P014 — silently running a 40 GB scan in the API process is how a preview engine takes down a control plane. 35 unit tests, 12 live tests. Evidence: evidence/bi-p014/duckdb-sandbox.json. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:54:53 +02:00
Claude Opus 5	ec26ddc296	feat(analytics): BI-P022 — dbt canonical transformation CONTRACT and reference project ... Scope stated in the module's first paragraph: this defines the contract, it does not run dbt, and nothing in this repo does. Execution is N0D's (n0d#38), which consumes this from the other side. There is deliberately no DbtExecutor implementation here — only the interface — so "0pt can run dbt" cannot become true by accident and the two repos share one versioned shape. `analytics/dbt/` is a real reference project covering raw -> staging -> curated -> serving, with the four layers mapped onto BI-P003's existing tiers rather than inventing a fifth vocabulary. The tenant predicate is applied at the FIRST modelled layer so every downstream model inherits it and no later model has to remember — the same forced-predicate rule as the query compiler's, one layer up. Two singular tests ship with it, both aimed at false greens: * `assert_mart_is_not_empty` — without it every column test passes trivially over zero rows and the whole suite reports green while the pipeline is dead; * `assert_no_cross_tenant_rows` — which should be impossible to fail, and exists precisely for that reason. An isolation guarantee nobody tests is the one that turns out to have four holes rather than the one reported. BI-P022's version rule is a GATE. `DBT_RUNTIME_PIN` records dbt Core 1.10.13 (stable) with dbt-clickhouse 1.9.2 and BOTH review flags FALSE — no licence or supply-chain review has been performed in this repo and none is claimed. `assertDbtRuntimeCertified` refuses a production run until they are true, and refuses an alpha line in production without a separate certification reference. A version constant on its own would be the documented-but-unenforced pattern; a test pins the flags at false so flipping them is a visible deliberate act. The result is treated as UNTRUSTED INPUT, because it arrives from another repo's executor and this estate has been burned by green statuses that meant nothing. Refused: `status: success` with ZERO nodes executed (a dbt selector matching nothing exits 0 — the dbt-shaped false green); an observed runtime different from the pin (the models mean something else and nothing says so); a revision that is not an exact 40-char SHA (a branch name is not reproducible, and reproducibility is the DoD); a missing manifest.json or run_results.json, without which the run cannot be audited or its tests ingested by BI-P025. Layer dependencies are checked against the manifest GRAPH, not asserted as a convention: a mart reading a source directly, or skipping a layer, or a staging model reading a mart, are each refused. Selectors are validated rather than passed through — a free-form argument list is a shell surface, and a shell surface on a job holding warehouse credentials is the whole problem. The job also cannot name an arbitrary target schema: it must be scoped to the job's tenant. 39 tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:45:41 +02:00
Claude Opus 5	6cb00add57	feat(analytics): BI-P025/BI-P026 — quality providers that fail on empty, freshness that refuses to be silent ... Both prompts are about a FALSE GREEN, so both are tested by constructing the false green and requiring a failure. BI-P025. Three provider kinds, and the interface makes the difference visible instead of implying parity: `providerKind` is "executing" or "ingesting". The native provider runs checks; the dbt and Great Expectations adapters INGEST results produced elsewhere (n0d#38 owns dbt execution against BI-P022, and GE is Python — nothing here runs it). Calling all three "providers" while only one executes is how "we have Great Expectations coverage" ends up in a status report. THE FALSE GREEN, proven against a live ClickHouse 24.8.14.39: a `not_null` check over an EMPTY table returns violations=0, evaluated=0. The raw counts are asserted first — the engine really does say zero-over-zero — and then the provider is required to call it a FAILURE, not a pass. Every rule compiles to a VIOLATION expression rather than a "rows that are fine" count, so the distinction cannot be lost at the runner layer. Same treatment for the ingesting providers: a MISSING dbt result is a failure, and dbt's `error`, `warn` and `skipped` are each asserted not to be passes. `skipped` in particular is the exact shape of l0re's 28 silently-skipped Postgres tests. The gate: zero verdicts QUARANTINES. "No rules ran" and "everything passed" produce the same green in most implementations, which is how a quality gate stops gating without anyone noticing — api0's `trivy --exit-code 0` in another costume. A later passing run does not auto-release either; release takes a principal and a reason, or a flapping rule would quarantine and un-quarantine a dataset between two reads of the same dashboard. BI-P025's DoD is the READ side: a quarantined projection is REFUSED, not annotated. An annotation on a dashboard nobody reads is not a quarantine. Live counts, all exact, from evidence/bi-p025/native-quality-live.json: not_null 5/118, range 3, distribution 4, unique 11. The unique figure is 11 rather than the 6 the first version of the test asserted — the five empty-string customers are duplicates of each other, so not_null and unique overlap on the same defect. The engine was right and the fixture arithmetic was wrong; the breakdown is written into the test rather than the number quietly adjusted. Tenant isolation re-proven on this path specifically, because a quality run uses an ADMINISTRATIVE credential: with 618 rows in the base table, t_alpha evaluates 118 and t_beta 500, and t_beta's blocking failure quarantines t_beta alone. Without the forced predicate one tenant's dirty rows would quarantine another tenant's projection. BI-P026. Five states, and the DoD's operative word is "silently". So the gate sits at the boundary every result crosses: a result with NO freshness envelope is unreleasable — not defaulted to FRESH, not defaulted to UNKNOWN — because "we forgot to attach it" and "the data is current" are indistinguishable to a reader, and any default makes forgetting the silent path. A non-current state without an explicit acknowledgement is refused, so a consumer that ignores the envelope cannot render stale numbers as current. FAILED outranks age: a pipeline that just failed may still hold recently-timestamped data from the previous run, and reporting that as FRESH is how a broken pipeline stays invisible for a day. UNKNOWN is never collapsed into FRESH. 54 unit tests, 11 live tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:40:35 +02:00
Claude Opus 5	65cce86144	feat(analytics): BI-P028/BI-P029/BI-P030/BI-P031 — metric model, semantic registry, governance, semantic provider ... BI-P028 extends the BI-P002 MetricDefinition rather than defining a second, parallel metric — a second metric type would be the exact failure the DoD is about. `MetricSpec` is the fifteen-field shape the prompt names; explicit bridge functions map it onto the persisted row plus the six columns it adds. The DoD ("no production metric exists only inside a dashboard tool") is enforced by something that refuses: a dashboard panel carrying an inline expression, inline aggregation or inline SQL is rejected, as is one naming an unregistered or RETIRED metric. So a metric cannot come into existence inside Superset and quietly become production. Validation refuses the metrics that render as a plausible number and mean nothing: no description, no owner, no time dimension, a ratio without a denominator, a currency format with no currency (which is how SEK gets rendered as USD), a filter on a dimension the metric does not declare. BI-P031's DoD is checked on a MEANING HASH rather than a field diff, so a field this code has never heard of still counts as changing the meaning — fail-closed rather than fail-open on future additions. Eight semantic fields are each mutated and each refused; three editorial fields are each mutated and each allowed, because forcing a version bump to fix a typo is how teams learn to route around governance. A new version starts DRAFT and EXPERIMENTAL: it never inherits the certification, or a certified metric's meaning could change while still wearing the badge. BI-P030 stores the semantic model separately from any engine, and that negative is enforced two ways after the first way turned out to be insufficient: * a VOCABULARY scan rejects a model mentioning metricflow / clickhouse / trino / cube / superset anywhere in its serialized form; * a STRUCTURAL check rejects any top-level key outside BI-P030's field list. The second exists because a test expected the vocabulary scan to catch a MetricFlow-shaped manifest and it did not — the manifest never says "metricflow". The structural check holds regardless of what a document calls itself. The manifest is now also stamped with its generator so a leaked copy is detectable. The replaceability DoD is tested with TWO providers whose naming rules differ (MetricFlow-shaped and Cube-shaped, `ds_sales.amount_sek`): the canonical id set must be identical across both AND the provider bindings must differ — otherwise the equality would hold trivially for a registry that only ever had one provider. BI-P029 — WHAT IS NOT PROVEN IS STATED IN THE MODULE AND RETURNED BY THE CODE. MetricFlow is a Python package and was NOT executed: no Python runtime is provisioned here, and BI-P029's "pin a currently verified version" requires a licence/supply-chain review that has not happened. `providerReadiness()` returns `verified: false` and names all four open items, including that the Superset path is unimplemented and the AI Analyst path uncompared. The constant `METRICFLOW_PINNED_VERSION` is labelled a declared intent, not a certification. A test asserts the readiness report says so, so the honesty cannot rot silently. 353 tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:34:52 +02:00
Claude Opus 5	ca0a3292df	feat(analytics): BI-P020 — provider-neutral query cache that cannot serve a stale or foreign result ... All six key components BI-P020 lists are REQUIRED, and the builder refuses to produce a key when one is missing rather than defaulting it. A defaulted component is how a cache serves a result computed under a metric definition that no longer exists — that answer is not stale, it is WRONG, and it looks current. "Never share tenant result cache" is enforced structurally, in three layers, because the obvious one is a naming convention: * the key is prefixed with the tenant id, derived here from the verified identity and never passed in; * the stored ENTRY records its own tenant id, and every read checks the entry against the caller — so a prefix collision or a shared backing store still cannot serve a foreign result; * a cross-tenant hit is REFUSED and the poisoned entry EVICTED, not returned as a miss. Returning undefined would hide a security event. The third point is the one the test corpus exercises directly: an entry stored under t_alpha's key carrying t_beta's id, and a forged key that a SCAN glob would happily include. ST0RE was relying on naming conventions when it turned out to have four cross-tenant paths rather than the one reported. Staleness is tested by mutating each key component ONE AT A TIME and requiring a miss for every one — and a coverage assertion compares the mutation list against the component type, so adding a field to CacheKeyComponents without adding a mutation fails the suite instead of going untested. ValkeyResultCache takes the client as an injected interface, so no dependency and no CVE surface is added — the same reason BI-P005's Arrow writer is hand-written. It sets a store-side TTL as well as recording one (a recorded timestamp alone leaves expired results resident forever), pages with SCAN rather than blocking on KEYS, and removes a corrupt entry instead of letting it become a permanent silent miss. 32 tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:28:27 +02:00
Claude Opus 5	8f6357044f	feat(analytics): BI-P018/BI-P019/BI-P021 — execution model, verified cancellation, a budget that stops queries ... BI-P018 — every field the prompt names, plus the one it names by its absence. The record carries `compiled_query_hash` and never the statement, and that is ENFORCED: `assertNoRawSql` runs before any record is emitted and refuses one whose `error` field looks like SQL. The realistic leak is ClickHouse's own error text, which echoes the query — the test uses exactly that string. The recorder writes at EVERY transition, because a recorder that only writes at the end loses precisely the executions worth investigating. Illegal state transitions throw rather than overwrite a terminal row. The one place raw SQL is retained stays BI-P017's analyst statement history, where a privileged human ran unreviewed SQL and "who ran what" must survive. Different record, different retention posture, stated in both files. BI-P019 — "stops" is VERIFIED, not requested. Aborting the client stops US waiting, not the server; KILL QUERY without SYNC tells you nothing; even SYNC can return while the query unwinds. So cancelAndVerify aborts, kills, then POLLS the process list until the query is GONE or the deadline passes — and if the deadline passes it returns `verified_stopped: false` rather than a cheerful `true`. The return type is an outcome object, not a boolean, so a caller cannot mistake "we asked" for "it stopped". An unreachable engine is read as "still running" (fail closed), so the kill is still attempted at the moment an operator most needs it to be. Proven live against ClickHouse 24.8.14.39, with the counterfactual first: * an identical bomb left alone IS still running after 2s — so "gone after we cancelled" is not just "the query was short"; * the cancelled query had read 6,559,476,156 rows, and was verified gone 57ms after the request; * system.query_log records ExceptionWhileProcessing with code 394, QUERY_WAS_CANCELLED — ClickHouse's verdict, not ours. BI-P021 — the honest version. A budget checked once before execution bounds nothing, and the module says so in its first paragraph. ClickHouse's max_rows_to_read bounds ONE query; it cannot express "this tenant may scan X per hour", which is what a budget is for. So enforcement polls the engine's live read_rows/read_bytes counters and CANCELS the running query, through BI-P019 with verification, when window spend would be exceeded. Measured live: a 1,000,000-row window cancelled the query at 130,883,409 rows, 92ms in, verified stopped, code 394. THE OVERSHOOT IS DOCUMENTED, in the module and in the evidence: enforcement is granular to the poll interval, so this is a bound of the form "within roughly one poll interval of engine throughput", not a hard cap. Quoting it as a hard cap would be wrong. Two more things stated rather than implied: * `enforceInFlight: false` is reported by describeEnforcement() as "NOT ENFORCED — this budget is advisory", and the constructor REFUSES to accept enforceInFlight without a process control; * an unshared window store is reported as non-binding across replicas ("with N replicas a tenant can spend up to N times this budget"). * a query the budget killed is still charged for the work it burned. Not charging is how a tenant learns the cheapest query is a cancelled one. Evidence: evidence/bi-p019-p021/cancellation-and-budget.json 268 unit tests, 7 live tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:26:05 +02:00
Claude Opus 5	adc7a9be31	fix(analytics): replace the control-character class with a code-point scan (no-control-regex) ... The ASCII-only check used a regex character class of control characters. It tripped eslint's no-control-regex, which the repo's quality gate enforces — and the rule has a point: such a class is unreadable and is exactly the kind of expression a reviewer nods at without checking. Same behaviour, written as an explicit scan naming the permitted set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:19:10 +02:00
Claude Opus 5	d9c8f7190b	feat(analytics): BI-P017 — analyst SQL contract, proven against live ClickHouse with the guard removed ... AnalyticsSQLSession is a SEPARATE class, not a method on SiaxAnalyticsEngine. executeMetric has no raw-SQL surface at all; on the same object the closed DSL would be one refactor away from growing one. ClickHouseAnalyticsEngine .executeQuery stays a documented refusal. FOUR LAYERS, and the guard is deliberately called the weakest: 1 role — the CL0UD analyst role is checked when the SESSION IS OPENED, so a caller without it never obtains an object that can run SQL. 2 guard — statement admission. Comments and literals removed with ClickHouse's own lexical rules, ASCII-only outside literals, one statement, SELECT/WITH only, and every table reference POSITIVELY resolved against the tenant's catalog. Deny-lists are a second opinion, never the mechanism. 3 grants — a per-tenant ClickHouse user with SELECT on that tenant's catalog alone, plus a restrictive row policy on the base tables. 4 views — the tenant catalog holds only VIEWS, each baking `WHERE tenant_id = '<tenant>'`, and tenant_id is not projected. 61 bypass attempts, in one shared corpus, run TWICE: against the guard, and against a live ClickHouse 24.8.14.39 with LAYER 2 REMOVED. The live assertion is not "the engine errored" — it is "no canary came back", with BETA_CANARY seeded into the other tenant's rows and SECRET_CANARY into an ungranted table. The counterfactual runs first: as admin the same base table returns 100 rows and BETA_CANARY IS readable, so the isolation numbers mean something. Result: 0 of 61 leaked. Evidence: evidence/bi-p017/analyst-sql-live-bypass.json. FOUR DEFECTS FOUND BY WRITING TESTS THAT CAN FAIL — three of them mine: * `CREATE ROW POLICY IF NOT EXISTS` is a SILENT NO-OP on re-provisioning. Dropping the user empties the policy's apply-to list; the next run skips re-creation, so the policy survives with `apply_to_list = []`, applying to nobody, while `system.row_policies` still shows the correct select_filter. The restricted user read 100 of 100 rows — BOTH TENANTS — from the base table. Fixed with OR REPLACE + `rowPolicyVerificationSql`, which turns "the policy is applied" from an assumption into a query, and a test that asserts applied_to_user = 1 per object. * `CREATE SETTINGS PROFILE IF NOT EXISTS` has the identical shape: a changed budget never lands. A stale 60-row ceiling survived re-provisioning and starved every legitimate statement, with the DDL reporting success throughout. Also OR REPLACE now. * `describe.skipIf(!reachable)` where `reachable` is set in `beforeAll` skips ALWAYS — vitest evaluates skipIf at collection time. Ten tests reported green while skipping against a live, reachable server. Replaced with the repo's existing env-var gate, which FAILS when CLICKHOUSE_URL is set and the server is down rather than degrading to a skip. * MEASURED, and it changes the design: a ClickHouse user whose profile carries `readonly = 1` cannot have ANY setting changed from the request (Code: 164). The BI-P012 ceilings therefore cannot travel with an analyst statement; they are baked into the user's SETTINGS PROFILE at provisioning. Proven both ways: over-budget scan fails with code 158 from the profile with no setting sent, and an attempt to raise it from the request fails with code 164. Honest about what the guard does NOT catch: `SELECT tenant_id FROM t_alpha_sql.sales` is structurally legitimate and IS admitted. It fails at the engine because the view does not project the column. Recorded in its own GUARD_ADMITS_ENGINE_REFUSES list and asserted live, rather than deleted from the corpus. evidence/bi-p017 is pinned by tests/analytics/analyst-sql-evidence.test.ts, which never skips: adding an attack without re-running the live suite breaks the build, and the pin also checks the counterfactual numbers so the evidence cannot be a recording of an empty database. 246 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:17:07 +02:00
Claude Opus 5	b1884a89a5	feat(analytics): BI-P016 — canonical query request contract, two compile targets ... The eight-field AnalyticsQueryRequest BI-P016 names, verbatim, with the rule that makes it safe enforced structurally rather than by convention: there is no field that can carry SQL, and parseAnalyticsQueryRequest REJECTS unknown keys instead of ignoring them. Ignoring an unsupported field is how it becomes a supported one after the next refactor. `tenant`, `tenant_id`, `project_id`, `database`, `catalog` and `table` are rejected by name. INF0's Wave-2 cross-tenant hole was a tenant read out of a request body; the fix that sticks is a request type in which the field does not exist, and a parser that refuses it when a caller sends it anyway. BI-P016's DoD ("same request compiles to ClickHouse/semantic engine without consumer rewrite") is tested as an equality, not narrated: both targets are projected onto a comparable shape and compared. The ClickHouse side of that projection is obtained by PARSING the emitted SQL and its bound parameters in the test — not by reading a field the compiler helpfully returned — so a compiler that silently drops a filter fails the comparison instead of passing it. The parser throws on any condition it does not recognise, which is what would catch unbound caller text appearing in the statement. BI-P011's forced tenant predicate is re-proven against this new path: five request shapes are enumerated and each must have FORCED_TENANT_PREDICATE as condition[0]; a blank tenant refuses to compile rather than emitting no predicate; a hostile filter value appears only in params, never in sql. 45 tests, all failing-capable: every rejection assertion checks the SiaxError code, and the DoD assertions compare reconstructed meaning rather than strings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:00:07 +02:00
Claude Opus 5	84bb9faa4f	feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance ... Implements the eight BI program prompts that make 0pt the canonical SIAX analytics capability. Every file is NEW: not one existing file is modified, and src/analytics/clickhouse-analytics.ts is byte-identical to main. BI-P002 — the nineteen canonical entities, verbatim, with tenant_id + project_id on every tenant-owned record. sql/008 gives each one a table. The four version entities BI-P002 names as immutable are enforced by BEFORE UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL against a live Postgres. BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced: assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a recorded exception exists. All eight required declarations are NOT NULL. BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen breaking-change classes are each tested twice: refused under a minor bump, accepted under a major one. BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so no npm dependency (or CVE surface) is added. Conformance comes from three independent implementations, not from self-agreement: pyarrow 25.0.1 and DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates. BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates. BI-P010 — four purpose-built tables, not one giant one, and the difference is measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000 rows for the same question; 5/660 granules vs 660/1149. BI-P011 — layered isolation, each layer proven with the layer above removed. Without the forced predicate the same statement returns 50,050 (both tenants); with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2 tenants; the restricted user running the IDENTICAL statement sees 1. BI-P012 — every named limit, fired individually and identified by ClickHouse error code, plus the half usually skipped: an unrelated tenant is timed while a real CPU bomb is in flight and must still return the CORRECT answer. Three defects found by writing tests that can fail: * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test that caught it is pinned. * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with code 158 exactly when the estimate would be over budget. Planner calls now use a narrower envelope. * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows and no indication it is short. Every SELECT ships `throw`. Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51 tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this module is prohibited from touching — has no `opt_analytics` database, zero tenant_id columns and zero row policies. Nothing there was modified; every proof ran against disposable containers. Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json, bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md} Docs: docs/analytics/BI_ANALYTICS_CORE.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-06 23:01:27 +02:00
Simon AxelssonandClaude Opus 5	9267576b7c	feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill ... BEFORE_STATE (verified, with evidence): * P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/ freshness to Postgres, but carried no authorization, no schedule, no checkpoint, no schema and no owner distinct from created_by — and the activation DoD was inverted: postgres-source-repository.ts wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered. No activation operation existed at all. * P061 MISSING. services/api/sources/connectors.py had per-source fetch_*() functions only — no lifecycle, no cursor, no resume, no health, no conformance suite. * P062 MISSING, already documented as such in services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere; workers/pipeline.py has no persistence at all). * P063 MISSING. schema_version existed as a free string; no comparison, no policy, no versioned source schema. * P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only opened an issue and the data flowed on. No quarantine anywhere. * P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution- service.ts (canonical) and services/api/workers/resolve.py (simplified, still called by pipeline.py:110). * P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a lineage question — no store, no query, no route. * P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store persistence layer exists to dedupe a backfill against"). BUILT (only what was PARTIAL/BROKEN/MISSING): * P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/ checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot produce an unclassified-but-active source. SourceRegistryService gains classifySource/activateSource/deactivateSource + authorization/schedule accessors; registration NEVER activates (asking for status:"active" goes through the gate, not around it); a service identity cannot authorize a source. Routes: activate (GET classification / POST activate), authorization, schedule. * P061 src/connectors/connector.ts — all nine operations on one interface. src/connectors/conformance.ts — a runnable suite (15 checks, no "skip" verdict). HttpJsonConnector (real HTTP, opaque cursor) and JsonlFileConnector (real file, line offsets, CDC deletes) both pass it. The suite caught a real defect while being written: an API that drops next_cursor at the end left the connector unable to resume, which is why HttpJsonConnectorConfig has cursor_field. * P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts. One envelope for batch/stream/cdc; durable per-(source,partition) checkpoints; resume proven by killing a run inside the sink. * P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two enforcement points, both upstream of any consumer write: a discovery gate before the first read, and per-record conformance before the page reaches the sink. A producer cannot widen its own policy retroactively. * P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine (not a second engine) and adds the missing consequence: block / quarantine / warn, with quarantine durable and recoverable. No policy is reported as "none_declared", never as a pass; an unexecutable expectation counts as a violation, never a silent pass. * P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage. AUD0 stays the system of record; this is 0pt's own queryable projection. * P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent on repeat, leaves the live checkpoint alone, still re-emits real changes. A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft combined the idempotency check and its recording into one atomic call BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the page in flight when the sink died was already marked delivered, so the resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by splitting peekDeliveries/commitDeliveries and committing both the ledger and the checkpoint only after the sink resolves. NOT DONE, stated rather than deferred quietly: * P065 is NOT met. resolve.py is still a second simplified production engine. Closing it means rewriting it as an adapter over the canonical service, which cannot be verified from this repo today: services/api is not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI workflow runs its pytest suite. The duplication is now stated at both sites in code and the removal plan is in STATUS.md. The new ingestion plane performs no ER, so it adds no third engine. * P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract read only. * Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock). * No CDC *source* exists; the contract is met, a CDC connector is not built. * 0pt does not schedule itself — the schedule row is a declaration ACT0 would read. VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was 311/18); pnpm build clean with all four new routes in the route table; lint unchanged at 12 pre-existing errors. Real Postgres 16 (tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real `docker restart` of that container with an identical read-back through the production code path — that is P060's "a restart preserves all sources", measured. DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404; api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify app id. None of the capability layer runs in production. Nothing here is LIVE_VERIFIED and STATUS.md says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 17:03:14 +02:00
Simon AxelssonandClaude Sonnet 5	dc6aaa0e10	fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill ... Tenant isolation (security-critical): every app/api/0pt/** route reading tenant_id from client-controlled query/body never cross-checked it against the token middleware.ts already verified (x-0pt-verified-tenant). Adds a shared assertTenantMatchesVerified guard and wires it into all 20+ affected routes. Also closes two full-leak-by-omission bugs found in this pass (documents/capture and lakehouse/mappings GETs returned every tenant's rows when tenant_id was simply omitted) and two "no tenant check at all" reads (rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone. Duplicate-delivery idempotency (correctness-critical): POST /api/orders now honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres + in-memory); the Python orchestrator's POST /orders/{id}/run — the actual delivery trigger — is now idempotent under an asyncio lock, closing a real concurrent-request race, not just the sequential-retry case. Engine-duplication resolution: adaptive_router.py's docstring wrongly invited a bypass ("see b0ts for scraping") contradicting the product's own compliance stance; corrected, and given real (additive, not escalating) block-signal tracking so a rate-limited/blocked source is excluded from the plan instead of hammered. Also fixes a previously-undiscovered production bug: the Python engine's only call into the TS quality bridge (X-Internal-Token) was unconditionally 401'd by middleware.ts, which never checked that credential — the advisory quality bridge was unreachable in any environment enforcing auth. Restore drill: executed for real against throwaway Docker Postgres containers (not simulated). Found and fixed two real blockers along the way: sql/004 referenced opt_datasets, which no migration ever created (chain was never actually runnable end to end); lib/db.ts hardcoded ssl:"require", making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev container) unreachable. After both fixes: full migration chain applies cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts + content checksums), and real application code (lib/orders-store.ts, SourceRegistryService's Postgres repository) reads the restored data back correctly, including cross-tenant isolation still holding. 82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS + 55 Python) still green; typecheck clean on every changed/added file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-03 12:00:43 +02:00