BI-P060/P061/P103/P104/P105 — dashboard model + versioning, BI SLO, benchmark harness, materialization #36

Merged

admin merged 6 commits from feat/bi-p060-p061-p103-p105-dashboards-slo-materialization into main 2026-09-07 02:53:38 +00:00

Conversation 0 Commits 6 Files Changed 21

+6484

6 Commits

Author	SHA1	Message	Date
Claude Opus 5	02b993faf9	docs(analytics): BI-P060/P061/P103/P104/P105 — what is proven, and what is not ... Includes BI-P105's freshness semantics as its DoD asks: watermark, staleness, the eight refusal reasons, grain rollup limits, and the sentence the evidence exists to support — the drifted aggregate was ONE SECOND old and overstated revenue by 5.251%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 04:08:22 +02:00
Claude Opus 5	f33cc8d32f	feat(analytics): BI-P104 — a benchmark harness that refuses to quote one number ... Six dataset profiles (small SaaS, medium business, VehIQ market, large event, wide table, geospatial), one representative question each, run on ClickHouse 26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions. evidence/bi-p104/benchmark.json. The harness's job is to stop the benchmark from lying: - overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three of the six real comparisons came out that way, one of them with a 1.6x median gap — which is exactly the headline a noisy host manufactures. - every sentence carries the range, the sample size and the stability, so it cannot be trimmed to "4.4x faster" without visibly deleting the caveats. - under 5 samples nothing is comparable at all. A three-run benchmark is an anecdote. - every engine BI-P104 names appears in the evidence, and the five nothing measured carry a reason each. The running Trino belongs to another lane, so benchmarking through it would have measured their configuration. Measured shape, warm cache, one idle host, these query shapes only: DuckDB faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on medium_business, wide_table and geospatial. One finding from the harness's own tests: an engine declared available that produced no run came out of buildEvidence() with no reason — the dangerous case, since it reads as "we could have and chose not to say". It now gets a generated reason. A second from the runs: DuckDB's progress bar writes to STDOUT and corrupted the first transcript; it is disabled explicitly and the JSON is read from a sentinel. 10 unit tests + 4 against real engines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 04:06:43 +02:00
Claude Opus 5	ed6e62809d	feat(analytics): BI-P103 — BI SLOs measured from AUD0's usage events, or declared unmeasured ... aud0#34 merged seven BI usage event types and recorded that zero producers emit them. This makes 0pt the producer and derives the SLOs from those events rather than opening a second, private measurement path that would disagree with aud0 during an incident. Three honest verdicts instead of one number: insufficient_data an availability objective over an empty window is SILENCE, not 100%. Every objective in an empty window comes back insufficient_data and fully_measured is false. A started query with no terminal event counts as unresolved — not as a success. not_measurable_here dashboard FRESHNESS is a property of the dataset (BI-P026/BI-P105 watermark) and report DELIVERY is N0TIFY's receipt. Neither is in aud0's seven types, so both are declared unmeasured rather than proxied by completion. low_confidence a p99 over 12 samples is the 12th-slowest request wearing a percentile's clothes. Reported with its sample size and not judged. The producer is an allowlist built field by named field, never a spread. The test feeds it an execution record carrying SQL text, a database error message quoting a personnummer, filter VALUES, a tenant id and a result sample, then greps the emitted events — after first proving the grep finds those strings in the input, so a clean result cannot be a broken detector. 14 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 04:01:16 +02:00
Claude Opus 5	96ff601248	feat(analytics): BI-P105 — a pre-aggregate serves only while it is provably undrifted ... A slow query is a complaint; a drifted pre-aggregate answers instantly, with confidence, and is wrong. So the router is fail-closed on eight named reasons, and the one people leave out — drift_never_verified — is the one that matters on launch day: a brand-new aggregate is exactly as unproven as a drifted one. Proved against ClickHouse 26.3.10.60, 2M rows, a real materialized view (evidence/bi-p105/materialization-drift.json): after backfill match, 48 buckets, 801 571 002 both sides GDPR erasure ALTER TABLE ... DELETE on the source. An MV is an INSERT trigger and never sees a mutation, so the aggregate kept the deleted revenue: it would have reported 801 571 002 against a true 761 580 554 — 39 990 448 overstated, 5.251%, with NOTHING stale. Staleness at the moment of the decision: 1 second. guarded router base_table (state_not_serving) counterfactual a staleness-only router SERVES it — measured, not assumed after rebuild match, and routing returns to the materialization Drift detection treats a MISSING bucket as drift even when every value present is correct (late-arriving rows are the sneakiest failure), and an empty comparison as insufficient_data rather than match — the same false green BI-P025 found in a quality check over an empty table. The measured speed-up is reported as a distribution, not a figure: p50 3.4x, but the run-to-run range spans 0.6x-7.5x on one idle host, so quoting "3.4x" as a property of the system would be misleading (BI-P104 doctrine). 19 unit tests + 4 against live ClickHouse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:56:32 +02:00
Claude Opus 5	1a12e8a882	feat(analytics): BI-P061 — dashboard releases, with the publish race measured both ways ... DRAFT -> PUBLISHED -> SUPERSEDED as a ledger (sql/010) rather than a column on the frozen version: sql/008 already makes DashboardVersion immutable at the database, and a lifecycle column there would have meant weakening that. Rollback mints a NEW version carrying the old document and publishes it, so "we went back to v3" is a timestamped fact (v5 = rollback_of_version 3) rather than a hole where v4 used to be. It is verified by BI-P060 canonical hash, and documentOf() refuses to serve a stored document whose hash no longer matches the release — a dashboard that renders an unapproved definition is worse than one that will not open. The claim worth attacking is "only one release can be live". The application check is read-then-write, so it is NOT the guarantee, and the evidence says so in both directions against Postgres 16.15: guard_absent partial unique index dropped -> BOTH concurrent publishes commit, dashboard ends with TWO live versions guard_present index in place -> loser fails 23505 on opt_analytics_dashboard_release_one_published, one live evidence/bi-p061/dashboard-release-race.json, reproducible with scripts/bi-p061-release-race.mjs. Two findings while writing the tests, both fixed here rather than papered over: the one-draft-per-dashboard index makes the two-draft race impossible to even construct (so the race is run as two create-and-publish transactions, which is what a "save and publish" button does), and an UPDATE matching zero rows raises nothing — which reads as "refused" to a careless test, so the immutability test asserts the row exists first. 23 tests: 8 unit + 15 against live Postgres. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:51:04 +02:00
Claude Opus 5	2af126dfc4	feat(analytics): BI-P060 — engine-neutral dashboard model that plans without Superset ... BI-P002 named DashboardDefinition/DashboardVersion and sql/008 stored them, but `layout_json` was a string, and a string holds anything — including a Superset export only Superset can read back. This gives the document a body and makes the DoD checkable rather than asserted. planDashboard() is the DoD: a document in, runnable AnalyticsQueryRequest objects (BI-P016) plus visualization specs out. The test greps the whole serialized plan for vendor traces after first proving the detector CAN find one in the source document — a clean grep over a broken detector is the usual way this test passes for the wrong reason. The counterfactual is a test too: a widget whose only content is a Superset slice id is REFUSED, because that is exactly the document that cannot be reconstructed. vendor_ref survives for round-tripping but is excluded from the canonical hash, so re-exporting to a new Superset instance does not fabricate a new dashboard version. No SQL surface: query widgets are parsed by parseAnalyticsQueryRequest, so `sql`, `where` and `tenant_id` are refused structurally rather than by review. 18 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:43:06 +02:00