diff --git a/docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md b/docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md new file mode 100644 index 0000000..0cd7e45 --- /dev/null +++ b/docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md @@ -0,0 +1,257 @@ +# BI-P060, P061, P103, P104, P105 — dashboards, SLOs, benchmarks, pre-aggregation + +Extends the analytics core (`0pt#34`, BI-P002..P012) and the query/metric/quality +layer (`0pt#35`). Nothing there is rebuilt: `DashboardDefinition` and +`DashboardVersion` were already canonical entities with tables in `sql/008`, and +these prompts give them behaviour. + +Organised, like its predecessor, around what is **proven** and what is **not**. + +--- + +## Where things live + +| prompt | module | live proof | +| ---------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| BI-P060 dashboard model | `src/analytics/dashboards/dashboard-document.ts` | unit (pure model + planner) | +| BI-P061 dashboard versioning | `src/analytics/dashboards/dashboard-release.ts`, `sql/010` | `tests/analytics/dashboard-release-postgres.realinfra.test.ts`, Postgres 16.15 | +| BI-P103 BI SLO | `src/analytics/slo/bi-slo.ts`, `slo/usage-events.ts` | unit; measurement path is `aud0#34`'s seven event types | +| BI-P104 benchmark harness | `src/analytics/bench/benchmark-harness.ts` | `tests/integration/bi-p104-benchmark.integration.test.ts`, ClickHouse 26.3.10.60 + DuckDB 1.5.5 | +| BI-P105 materialization | `src/analytics/materialization/` | `tests/integration/materialization-drift-clickhouse.integration.test.ts` | + +Evidence: `evidence/bi-p061/dashboard-release-race.json`, +`evidence/bi-p104/benchmark.json`, `evidence/bi-p105/materialization-drift.json`. + +--- + +## BI-P060 — the dashboard is a document, not a Superset export + +`layout_json` was a string, and a string holds anything. The canonical document +carries `layout / filters / widgets / parameters / refresh / permissions / +version`, and a widget references **a metric, a contract query and a +visualization spec** — never a vendor's internal id. + +The DoD is a function. `planDashboard(document, { parameters })` returns, per +widget, a runnable `AnalyticsQueryRequest` (BI-P016) plus a render spec. If it +returns a plan, the dashboard can be drawn by anything that executes the +contract and draws the twelve canonical visualization types. + +Rules that are enforced rather than documented: + +- a widget whose only content is a vendor id is **refused** — that is exactly + the document that cannot be reconstructed; +- `vendor_ref` may carry a Superset/Metabase id for round-tripping, but it is + excluded from the canonical hash, so re-exporting into a new Superset instance + does not fabricate a new dashboard version; +- vendor-native keys (`viz_type`, `form_data`, `position_json`, `slice_id`, …) + are refused with a message that names the rule; +- query widgets go through `parseAnalyticsQueryRequest`, so a dashboard cannot + become an SQL surface and cannot name a tenant; +- a locked filter cannot be switched off by a viewer — locked filters are how a + saved dashboard pins its scope, and removing one silently widens the result. + +--- + +## BI-P061 — releases are a ledger + +`DRAFT → PUBLISHED → SUPERSEDED`, stored in `opt_analytics_dashboard_release` +(`sql/010`) rather than as a column on the version row: `sql/008` already makes +`DashboardVersion` immutable at the database, and a lifecycle column there would +have meant weakening that. The **document** is frozen; the **release** moves. + +The database enforces the state machine, not the application: + +| attempted | result | +| ------------------------------------------- | -------------------------------------------------- | +| `published → draft` (un-publishing) | refused — it is how a rollback gets faked | +| `superseded → published` | refused | +| repointing a release at another version | refused (every content column is frozen on UPDATE) | +| rewriting `document_hash` or `published_at` | refused | +| `DELETE` on a release | refused — the ledger has no eraser | +| a `published` row with no `published_at` | refused by CHECK | +| a second open draft for one dashboard | refused by a partial unique index | + +**Rollback mints a new version.** It does not resurrect the old release row. +"We went back to v3" becomes a timestamped fact (v5 carries +`rollback_of_version = 3`) instead of a hole in the history where v4 used to be. +The restore is verified by BI-P060 canonical hash, and `documentOf()` refuses to +serve a stored document whose hash no longer matches its release. + +### The concurrency claim, measured both ways + +`DashboardReleaseService.publish()` checks for a live release before writing. +That check is a read-then-write race and is **not** the guarantee. +`evidence/bi-p061/dashboard-release-race.json`, Postgres 16.15: + +| run | partial unique index | transactions committed | published releases after | +| --------------- | -------------------- | ---------------------- | --------------------------- | +| `guard_absent` | dropped | 2 | **2 — two live dashboards** | +| `guard_present` | in place | 1 (loser `23505`) | 1 | + +Reproduce with `node scripts/bi-p061-release-race.mjs`. + +--- + +## BI-P103 — SLOs measured from AUD0, or declared unmeasured + +The measurement path is `aud0#34`'s seven BI usage event types (BI-P076). aud0's +own doc records that zero producers emit them; `src/analytics/slo/usage-events.ts` +makes 0pt the producer. There is deliberately no second telemetry path — two +measurement systems disagree, and the disagreement surfaces during an incident. + +| workload | objectives | +| --------------------- | ---------------------------------------------------------------- | +| interactive dashboard | availability, p95 first query, p95 cached, error rate, freshness | +| exploration | p95, p99, max result, timeout | +| report | completion SLA, delivery SLA | + +Three verdicts that are not "met": + +- **`insufficient_data`** — an availability objective over an empty window is + silence, not 100%. A started query with no terminal event is _unresolved_, not + a success. +- **`not_measurable_here`** — dashboard **freshness** is a property of the + dataset (BI-P026 / BI-P105 watermark) and report **delivery** is N0TIFY's + receipt. Neither exists in the seven event types, so both are declared + unmeasured rather than proxied by completion. +- **`low_confidence`** — a p99 over 12 samples is the 12th-slowest request + wearing a percentile's clothes. Reported with its sample size, not judged. + +The producer is an allowlist built field by named field. `compiled_sql`, +`error_message`, `tenant_id`, filter **values** and result samples exist on the +0pt-side telemetry type and are never mapped; a `query_hash` that is not a +64-hex digest is dropped rather than reshaped. + +### Honest status + +`BI_SLO_CATALOGUE`'s targets are **starting thresholds, not derived baselines**. +No production BI traffic exists yet, so the first real baselines will come from +the first weeks of emitted events. The evaluator is built so that absence shows +up as `insufficient_data` until then, rather than as a green dashboard. + +--- + +## BI-P104 — the harness refuses to quote one number + +Six dataset profiles × two engines, 2 warmups discarded, 9 timed repetitions, +statistics from the shipped module. `evidence/bi-p104/benchmark.json`. + +- **Overlapping ranges are `indistinguishable`**, whatever the medians say. + Three of the six real comparisons came out that way, one with a 1.6x median + gap — the headline a noisy host manufactures for you. +- **Every sentence carries the range, the sample size and the stability**, so it + cannot be trimmed to "4.4x faster" without visibly deleting the caveats. +- **Under 5 samples nothing is comparable.** A three-run benchmark is an anecdote. +- **Every engine BI-P104 names appears**, and the five nothing measured carry a + reason each. + +Measured (warm cache, one idle host, these query shapes only): + +| dataset | verdict, DuckDB 1.5.5 vs ClickHouse 26.3.10.60 | +| --------------- | ------------------------------------------------- | +| small_saas | faster, 1.6x–8.2x (median 4.4x) | +| medium_business | indistinguishable (medians 1.6x apart) | +| vehiq_market | faster, 2.4x–13.3x (median 5.8x) | +| large_event | **slower**, 0.2x–0.9x (`count DISTINCT`, 5M rows) | +| wide_table | indistinguishable | +| geospatial | indistinguishable | + +Not exercised, with reasons in the evidence: **trino** (the running instance +belongs to another lane; benchmarking through it would measure their +configuration), **semantic_query** (MetricFlow has never executed in 0pt), +**superset**, **native_charts** (DES1GN owns rendering), **perspective**. + +--- + +## BI-P105 — freshness semantics, and why freshness is not correctness + +### The vocabulary + +| term | definition | +| ------------------------------ | ------------------------------------------------------------------------------- | +| `data_watermark` | the newest **source** timestamp the aggregate represents. Absent = **UNKNOWN**. | +| staleness | `now − data_watermark`, in seconds. `undefined` when the watermark is unknown. | +| `max_staleness_seconds` | the freshness contract. Older than this and the aggregate stops serving. | +| `drift_check_interval_seconds` | how often the aggregate must be re-proved against its source to keep serving. | +| `last_drift_verdict` | `match` / `drifted` / `insufficient_data` from the last comparison. | + +**UNKNOWN is never treated as current.** This is the same doctrine BI-P026 +applies to result freshness: a missing watermark is a refusal, not a zero. + +### The routing rule + +A materialization serves **only** while it is provably current _and_ provably +undrifted. Eight named refusals, each a real way an aggregate becomes a +confident liar: + +`coverage_miss` (metric/dimension/grain not carried — including a filter on a +dimension the aggregate already summed across) · `watermark_unknown` · +`staleness_exceeded` · `window_beyond_watermark` (the question reaches past the +newest data; answering would silently shorten the period) · +`drift_never_verified` · `drift_verification_stale` · `drift_detected` · +`state_not_serving`. + +`drift_never_verified` is the one people leave out and the one that matters on +launch day: a brand-new aggregate is exactly as unproven as a drifted one. + +Grain rollup is restricted, not assumed: a `day` aggregate answers `day` and +coarser, never `hour`; a `week` aggregate answers only `week`, because weekly +buckets do not compose into calendar months. + +### Drift detection + +- **A missing bucket is drift** even when every value present is correct — late + arrivals and a materialized view created after the backfill both look like + this, and a value-only comparison passes while the total is wrong. +- **An extra bucket is drift** — the source rows are gone and the aggregate is not. +- **An empty comparison is `insufficient_data`, never `match`** (the BI-P025 + false green, restated for pre-aggregates), and it takes the aggregate out of + service rather than annotating it. +- **Both tolerances must hold.** 0.1% of a 40M bucket is 40 000 kr of invisible + error; an absolute-only bound flags every rounding difference on a large bucket. + +### The proof, on a live ClickHouse + +`evidence/bi-p105/materialization-drift.json`, ClickHouse 26.3.10.60, 2M rows, +a real materialized view: + +| step | result | +| ------------------------ | ----------------------------------------------------------------------------------- | +| after backfill | `match`, 48 buckets, 801 571 002 on both sides | +| GDPR erasure | `ALTER TABLE sales DELETE …`. An MV is an INSERT trigger and never sees a mutation. | +| what would be served | 801 571 002 against a true 761 580 554 — **39 990 448 overstated, 5.251%** | +| staleness at that moment | **1 second.** Every freshness check passed. | +| guarded router | `base_table` (`state_not_serving`) | +| counterfactual | a staleness-only router **serves it** — measured, not assumed | +| after rebuild | `match`, and routing returns to the materialization | + +That last row of the counterfactual is the whole point of the prompt: staleness +is not correctness. The aggregate was one second old and wrong by 5%. + +### Measured improvement, honestly + +Same drill, base table vs pre-aggregate, 9 repetitions interleaved: p50 3.4x, +but the run-to-run range spans **0.6x–7.5x** on an otherwise idle host — one +sample had the pre-aggregate slower than the base table's best run. Quoting +"3.4x" as a property of the system would be misleading, which is why BI-P104's +harness exists. + +--- + +## What is NOT proven here + +- **No dashboard has been rendered.** BI-P060 produces a plan; drawing it is + DES1GN (`DES1GN#9`) and `siax-portal#5`. Nothing in this PR proves a pixel. +- **The release service has no Postgres adapter yet.** The lifecycle is proven + in-memory and the _schema_ is proven against live Postgres; wiring + `DashboardReleaseStore` to `postgres.js` is the next step and is the only + place where the two could drift apart. +- **Zero real SLO events have been emitted in production.** The producer and the + evaluator are tested; no BI surface calls the producer yet, so every objective + would today report `insufficient_data`. That is the honest empty state, and it + is deliberately visible rather than hidden behind a default. +- **No materialization is deployed.** The router and detector are proven against + a live ClickHouse in a throwaway database. Nothing schedules a refresh or a + periodic drift check yet — that belongs with ACT0/N0D scheduling. +- **Trino, Superset, MetricFlow and Perspective are unmeasured**, for the + reasons recorded in the benchmark evidence. diff --git a/evidence/bi-p061/dashboard-release-race.json b/evidence/bi-p061/dashboard-release-race.json new file mode 100644 index 0000000..99e2f25 --- /dev/null +++ b/evidence/bi-p061/dashboard-release-race.json @@ -0,0 +1,42 @@ +{ + "recorded_at": "2026-09-07T01:50:09.216Z", + "postgres_version": "16.15", + "guard_present": { + "partial_unique_index_present": true, + "transactions_committed": 1, + "rejections": [ + { + "code": "23505", + "constraint": "opt_analytics_dashboard_release_one_published" + } + ], + "published_releases_after": [ + { + "release_id": "rel_1", + "version": 1 + } + ] + }, + "guard_absent": { + "partial_unique_index_present": false, + "transactions_committed": 2, + "rejections": [], + "published_releases_after": [ + { + "release_id": "rel_1", + "version": 1 + }, + { + "release_id": "rel_2", + "version": 2 + } + ] + }, + "verdict": { + "counterfactual_produced_two_live_dashboards": true, + "guard_left_exactly_one": true, + "loser_error_code": "23505", + "application_check_alone_is_sufficient": false, + "note": "DashboardReleaseService.publish() reads for a live release before writing. Both sessions in guard_absent passed that check and both committed. The partial unique index is what holds." + } +} diff --git a/evidence/bi-p104/benchmark.json b/evidence/bi-p104/benchmark.json new file mode 100644 index 0000000..91c744b --- /dev/null +++ b/evidence/bi-p104/benchmark.json @@ -0,0 +1,638 @@ +{ + "recorded_at": "2026-09-07T02:06:11.079Z", + "host_note": "One idle host, warm caches, both engines local, single-threaded client. These numbers describe these query shapes on this data on this machine. They are not engine rankings.", + "datasets": [ + { + "id": "small_saas", + "label": "small SaaS", + "approx_rows": 100000, + "shape": "narrow fact table, one tenant, high selectivity" + }, + { + "id": "medium_business", + "label": "medium business", + "approx_rows": 2000000, + "shape": "multi-tenant fact table with a date and a few dimensions" + }, + { + "id": "vehiq_market", + "label": "VehIQ market", + "approx_rows": 1000000, + "shape": "vehicle listings: make/model/year/price/mileage, skewed cardinalities" + }, + { + "id": "large_event", + "label": "large event", + "approx_rows": 5000000, + "shape": "append-only event stream, timestamp-ordered, sparse dimensions" + }, + { + "id": "wide_table", + "label": "wide table", + "approx_rows": 200000, + "shape": "many columns, few read per query — the columnar-store best case" + }, + { + "id": "geospatial", + "label": "geospatial", + "approx_rows": 500000, + "shape": "lat/lon points with bounding-box and distance predicates" + } + ], + "engines": [ + { + "engine": "clickhouse", + "available": true, + "version": "26.3.10.60" + }, + { + "engine": "duckdb", + "available": true, + "version": "1.5.5" + }, + { + "engine": "trino", + "available": false, + "reason": "a Trino is running on this host but belongs to another lane (st0re-verify-trino); benchmarking through someone else’s container would measure their configuration" + }, + { + "engine": "semantic_query", + "available": false, + "reason": "MetricFlow has never been executed in 0pt (BI-P029 honest status)" + }, + { + "engine": "superset", + "available": false, + "reason": "no Superset deployment in the estate yet" + }, + { + "engine": "native_charts", + "available": false, + "reason": "render-side; DES1GN owns it (BI-P051..P057)" + }, + { + "engine": "perspective", + "available": false, + "reason": "not deployed" + } + ], + "runs": [ + { + "engine": "clickhouse", + "dataset": "small_saas", + "query_label": "revenue by day", + "warmup_ms": [58.856714000000466, 58.77232499999991], + "samples_ms": [ + 51.25258499999927, 23.924909000000298, 39.89413500000046, 29.40832799999953, 43.058938000000126, + 39.98396799999955, 41.99595799999952, 32.193271000000095, 25.963267999999516 + ], + "distribution": { + "n": 9, + "min": 23.924909000000298, + "p50": 39.89413500000046, + "p95": 51.25258499999927, + "p99": 51.25258499999927, + "max": 51.25258499999927, + "mean": 36.40837333333315, + "stdev": 8.514314573768598, + "cv": 0.23385594560395923, + "iqr": 12.58762999999999, + "spread_ratio": 2.1422269568506462, + "stability": "noisy" + }, + "rows_returned": 365 + }, + { + "engine": "clickhouse", + "dataset": "medium_business", + "query_label": "revenue by month and region for one tenant", + "warmup_ms": [44.60071400000015, 70.7402890000003], + "samples_ms": [ + 70.63034199999947, 69.65554000000066, 58.036339000000225, 46.82895400000052, 44.67711700000018, + 87.97018200000002, 49.249023999999736, 59.75650299999961, 41.953865000000405 + ], + "distribution": { + "n": 9, + "min": 41.953865000000405, + "p50": 58.036339000000225, + "p95": 87.97018200000002, + "p99": 87.97018200000002, + "max": 87.97018200000002, + "mean": 58.750874000000096, + "stdev": 14.251513712332908, + "cv": 0.24257534811027467, + "iqr": 22.826586000000134, + "spread_ratio": 2.096831412314436, + "stability": "noisy" + }, + "rows_returned": 120 + }, + { + "engine": "clickhouse", + "dataset": "vehiq_market", + "query_label": "average asking price by make and model year", + "warmup_ms": [83.03972700000031, 59.77780099999927], + "samples_ms": [ + 74.38707999999951, 60.41443299999992, 59.13954199999989, 98.55565299999944, 98.45707200000015, + 74.05828699999984, 194.5833190000003, 159.76290199999949, 87.88670300000013 + ], + "distribution": { + "n": 9, + "min": 59.13954199999989, + "p50": 87.88670300000013, + "p95": 194.5833190000003, + "p99": 194.5833190000003, + "max": 194.5833190000003, + "mean": 100.80499899999985, + "stdev": 43.75254087132061, + "cv": 0.4340314598021143, + "iqr": 24.4973659999996, + "spread_ratio": 3.290240546671813, + "stability": "noisy" + }, + "rows_returned": 60 + }, + { + "engine": "clickhouse", + "dataset": "large_event", + "query_label": "hourly event volume for one event type", + "warmup_ms": [122.44560499999989, 136.30929500000002], + "samples_ms": [ + 142.78732200000013, 136.18330800000058, 82.08186000000023, 95.91282499999943, 74.52815499999997, + 93.95668399999977, 112.81592300000011, 76.3508299999994, 93.78873100000055 + ], + "distribution": { + "n": 9, + "min": 74.52815499999997, + "p50": 93.95668399999977, + "p95": 142.78732200000013, + "p99": 142.78732200000013, + "max": 142.78732200000013, + "mean": 100.9339597777778, + "stdev": 23.374602687288522, + "cv": 0.23158313355337923, + "iqr": 30.73406299999988, + "spread_ratio": 1.9158842990276652, + "stability": "noisy" + }, + "rows_returned": 720 + }, + { + "engine": "clickhouse", + "dataset": "wide_table", + "query_label": "three columns of fifty, grouped", + "warmup_ms": [28.096958000000086, 14.89937800000007], + "samples_ms": [ + 19.301219999999375, 21.628866000000926, 32.125191999999515, 25.563218999999663, 24.388967999999295, + 16.8507410000002, 19.57230399999935, 20.52187000000049, 18.078355999999985 + ], + "distribution": { + "n": 9, + "min": 16.8507410000002, + "p50": 20.52187000000049, + "p95": 32.125191999999515, + "p99": 32.125191999999515, + "max": 32.125191999999515, + "mean": 22.00341511111098, + "stdev": 4.451179566867264, + "cv": 0.2022949412348072, + "iqr": 5.08774799999992, + "spread_ratio": 1.9064557457739775, + "stability": "noisy" + }, + "rows_returned": 500 + }, + { + "engine": "clickhouse", + "dataset": "geospatial", + "query_label": "bounding box, aggregated to a grid cell", + "warmup_ms": [38.77805500000068, 17.665412000000288], + "samples_ms": [ + 29.736737000001085, 27.89697900000101, 25.586840999998458, 21.336174000000028, 22.492867999999362, + 30.176470000000336, 22.63948900000105, 29.021173000001, 36.31744599999911 + ], + "distribution": { + "n": 9, + "min": 21.336174000000028, + "p50": 27.89697900000101, + "p95": 36.31744599999911, + "p99": 36.31744599999911, + "max": 36.31744599999911, + "mean": 27.244908555555714, + "stdev": 4.498748211227398, + "cv": 0.16512252929951657, + "iqr": 7.097248000000036, + "spread_ratio": 1.7021536288558137, + "stability": "noisy" + }, + "rows_returned": 201 + }, + { + "engine": "duckdb", + "dataset": "small_saas", + "query_label": "revenue by day", + "warmup_ms": [12.830814997869311, 8.479544001602335], + "samples_ms": [ + 7.375242999842158, 9.8867060005432, 9.773318997758906, 9.169243003270822, 7.295589999557706, + 10.63253999745939, 6.214401000761427, 15.184157000476262, 7.2293630000785924 + ], + "distribution": { + "n": 9, + "min": 6.214401000761427, + "p50": 9.169243003270822, + "p95": 15.184157000476262, + "p99": 15.184157000476262, + "max": 15.184157000476262, + "mean": 9.19561799997205, + "stdev": 2.5477019265669423, + "cv": 0.27705608547187216, + "iqr": 2.591116000985494, + "spread_ratio": 2.443382234042477, + "stability": "noisy" + }, + "rows_returned": 365 + }, + { + "engine": "duckdb", + "dataset": "medium_business", + "query_label": "revenue by month and region for one tenant", + "warmup_ms": [60.29315100022359, 47.50260100263404], + "samples_ms": [ + 47.711714996694354, 36.605457000405295, 30.622688002040377, 44.33469899959164, 49.469069999759085, + 33.921502999874065, 33.03168200000073, 40.499283000826836, 33.27470799922594 + ], + "distribution": { + "n": 9, + "min": 30.622688002040377, + "p50": 36.605457000405295, + "p95": 49.469069999759085, + "p99": 49.469069999759085, + "max": 49.469069999759085, + "mean": 38.8300894442687, + "stdev": 6.549355811539825, + "cv": 0.1686670287211122, + "iqr": 11.059991000365699, + "spread_ratio": 1.615438527031428, + "stability": "noisy" + }, + "rows_returned": 120 + }, + { + "engine": "duckdb", + "dataset": "vehiq_market", + "query_label": "average asking price by make and model year", + "warmup_ms": [17.676940999081125, 24.05106900187093], + "samples_ms": [ + 14.901875001669396, 14.926752002793364, 17.798165001295274, 15.059515000757528, 19.75441800095723, + 14.601105001929682, 14.954296002542833, 20.32349000000977, 24.610873999336036 + ], + "distribution": { + "n": 9, + "min": 14.601105001929682, + "p50": 15.059515000757528, + "p95": 24.610873999336036, + "p99": 24.610873999336036, + "max": 24.610873999336036, + "mean": 17.436721112365678, + "stdev": 3.2978767006322265, + "cv": 0.18913399367805778, + "iqr": 4.827665998163866, + "spread_ratio": 1.6855487304613908, + "stability": "noisy" + }, + "rows_returned": 60 + }, + { + "engine": "duckdb", + "dataset": "large_event", + "query_label": "hourly event volume for one event type", + "warmup_ms": [184.43432099957136, 163.9354379985889], + "samples_ms": [ + 160.28642499804846, 161.1321640011738, 177.48030700022355, 213.59406000192394, 337.0364080001309, + 334.6067929996934, 232.15600899857236, 204.8815359994478, 196.73219499964034 + ], + "distribution": { + "n": 9, + "min": 160.28642499804846, + "p50": 204.8815359994478, + "p95": 337.0364080001309, + "p99": 337.0364080001309, + "max": 337.0364080001309, + "mean": 224.21176633320607, + "stdev": 63.65742323059742, + "cv": 0.28391651460429923, + "iqr": 54.67570199834881, + "spread_ratio": 2.1027133645549485, + "stability": "noisy" + }, + "rows_returned": 720 + }, + { + "engine": "duckdb", + "dataset": "wide_table", + "query_label": "three columns of fifty, grouped", + "warmup_ms": [18.729354997049086, 21.23752199986484], + "samples_ms": [ + 14.388244999281596, 15.164796001045033, 18.07164799902239, 22.90704599727178, 19.191682997188764, + 16.838232000736753, 16.024758999265032, 24.479674000758678, 22.105899999587564 + ], + "distribution": { + "n": 9, + "min": 14.388244999281596, + "p50": 18.07164799902239, + "p95": 24.479674000758678, + "p99": 24.479674000758678, + "max": 24.479674000758678, + "mean": 18.796886999350843, + "stdev": 3.4151315245862714, + "cv": 0.1816860166635153, + "iqr": 6.081141000322532, + "spread_ratio": 1.7013662195758374, + "stability": "noisy" + }, + "rows_returned": 500 + }, + { + "engine": "duckdb", + "dataset": "geospatial", + "query_label": "bounding box, aggregated to a grid cell", + "warmup_ms": [27.98272599829943, 22.078424000937957], + "samples_ms": [ + 30.961295000452083, 28.811156000301708, 19.97572500113165, 22.212601001228904, 33.500631998322206, + 34.46119200089015, 24.307676001626533, 20.643488001951482, 24.275364998175064 + ], + "distribution": { + "n": 9, + "min": 19.97572500113165, + "p50": 24.307676001626533, + "p95": 34.46119200089015, + "p99": 34.46119200089015, + "max": 34.46119200089015, + "mean": 26.572125556008864, + "stdev": 5.191758162396437, + "cv": 0.19538362301703047, + "iqr": 8.748693999223178, + "spread_ratio": 1.7251535050136042, + "stability": "noisy" + }, + "rows_returned": 201 + } + ], + "comparisons": [ + { + "dataset": "small_saas", + "query_label": "revenue by day", + "description": "duckdb vs clickhouse (small_saas): FASTER — 1.6x to 8.2x (median 4.4x) over n=9/9, stability noisy/noisy. Warm cache, one host, this query shape.", + "p50_ratio": 4.350864622714171, + "conservative_ratio": 1.5756494745971, + "optimistic_ratio": 8.247389409489264, + "ranges_overlap": false, + "baseline": { + "n": 9, + "min": 23.924909000000298, + "p50": 39.89413500000046, + "p95": 51.25258499999927, + "p99": 51.25258499999927, + "max": 51.25258499999927, + "mean": 36.40837333333315, + "stdev": 8.514314573768598, + "cv": 0.23385594560395923, + "iqr": 12.58762999999999, + "spread_ratio": 2.1422269568506462, + "stability": "noisy" + }, + "candidate": { + "n": 9, + "min": 6.214401000761427, + "p50": 9.169243003270822, + "p95": 15.184157000476262, + "p99": 15.184157000476262, + "max": 15.184157000476262, + "mean": 9.19561799997205, + "stdev": 2.5477019265669423, + "cv": 0.27705608547187216, + "iqr": 2.591116000985494, + "spread_ratio": 2.443382234042477, + "stability": "noisy" + }, + "verdict": "faster" + }, + { + "dataset": "medium_business", + "query_label": "revenue by month and region for one tenant", + "description": "duckdb vs clickhouse (medium_business): INDISTINGUISHABLE at this sample size. Medians differ by 1.6x but the observed ranges overlap (baseline 42.0-88.0ms, candidate 30.6-49.5ms; n=9/9, CV 0.24/0.17).", + "p50_ratio": 1.5854559335062433, + "conservative_ratio": 0.8480827515092707, + "optimistic_ratio": 2.872712610798197, + "ranges_overlap": true, + "baseline": { + "n": 9, + "min": 41.953865000000405, + "p50": 58.036339000000225, + "p95": 87.97018200000002, + "p99": 87.97018200000002, + "max": 87.97018200000002, + "mean": 58.750874000000096, + "stdev": 14.251513712332908, + "cv": 0.24257534811027467, + "iqr": 22.826586000000134, + "spread_ratio": 2.096831412314436, + "stability": "noisy" + }, + "candidate": { + "n": 9, + "min": 30.622688002040377, + "p50": 36.605457000405295, + "p95": 49.469069999759085, + "p99": 49.469069999759085, + "max": 49.469069999759085, + "mean": 38.8300894442687, + "stdev": 6.549355811539825, + "cv": 0.1686670287211122, + "iqr": 11.059991000365699, + "spread_ratio": 1.615438527031428, + "stability": "noisy" + }, + "verdict": "indistinguishable" + }, + { + "dataset": "vehiq_market", + "query_label": "average asking price by make and model year", + "description": "duckdb vs clickhouse (vehiq_market): FASTER — 2.4x to 13.3x (median 5.8x) over n=9/9, stability noisy/noisy. Warm cache, one host, this query shape.", + "p50_ratio": 5.835958395444954, + "conservative_ratio": 2.4029842256555125, + "optimistic_ratio": 13.32661596326334, + "ranges_overlap": false, + "baseline": { + "n": 9, + "min": 59.13954199999989, + "p50": 87.88670300000013, + "p95": 194.5833190000003, + "p99": 194.5833190000003, + "max": 194.5833190000003, + "mean": 100.80499899999985, + "stdev": 43.75254087132061, + "cv": 0.4340314598021143, + "iqr": 24.4973659999996, + "spread_ratio": 3.290240546671813, + "stability": "noisy" + }, + "candidate": { + "n": 9, + "min": 14.601105001929682, + "p50": 15.059515000757528, + "p95": 24.610873999336036, + "p99": 24.610873999336036, + "max": 24.610873999336036, + "mean": 17.436721112365678, + "stdev": 3.2978767006322265, + "cv": 0.18913399367805778, + "iqr": 4.827665998163866, + "spread_ratio": 1.6855487304613908, + "stability": "noisy" + }, + "verdict": "faster" + }, + { + "dataset": "large_event", + "query_label": "hourly event volume for one event type", + "description": "duckdb vs clickhouse (large_event): SLOWER — 0.2x to 0.9x (median 0.5x) over n=9/9, stability noisy/noisy. Warm cache, one host, this query shape.", + "p50_ratio": 0.4585902948338546, + "conservative_ratio": 0.2211279055643479, + "optimistic_ratio": 0.8908260446992852, + "ranges_overlap": false, + "baseline": { + "n": 9, + "min": 74.52815499999997, + "p50": 93.95668399999977, + "p95": 142.78732200000013, + "p99": 142.78732200000013, + "max": 142.78732200000013, + "mean": 100.9339597777778, + "stdev": 23.374602687288522, + "cv": 0.23158313355337923, + "iqr": 30.73406299999988, + "spread_ratio": 1.9158842990276652, + "stability": "noisy" + }, + "candidate": { + "n": 9, + "min": 160.28642499804846, + "p50": 204.8815359994478, + "p95": 337.0364080001309, + "p99": 337.0364080001309, + "max": 337.0364080001309, + "mean": 224.21176633320607, + "stdev": 63.65742323059742, + "cv": 0.28391651460429923, + "iqr": 54.67570199834881, + "spread_ratio": 2.1027133645549485, + "stability": "noisy" + }, + "verdict": "slower" + }, + { + "dataset": "wide_table", + "query_label": "three columns of fifty, grouped", + "description": "duckdb vs clickhouse (wide_table): INDISTINGUISHABLE at this sample size. Medians differ by 1.1x but the observed ranges overlap (baseline 16.9-32.1ms, candidate 14.4-24.5ms; n=9/9, CV 0.20/0.18).", + "p50_ratio": 1.1355837608784018, + "conservative_ratio": 0.688356429888648, + "optimistic_ratio": 2.232738739269697, + "ranges_overlap": true, + "baseline": { + "n": 9, + "min": 16.8507410000002, + "p50": 20.52187000000049, + "p95": 32.125191999999515, + "p99": 32.125191999999515, + "max": 32.125191999999515, + "mean": 22.00341511111098, + "stdev": 4.451179566867264, + "cv": 0.2022949412348072, + "iqr": 5.08774799999992, + "spread_ratio": 1.9064557457739775, + "stability": "noisy" + }, + "candidate": { + "n": 9, + "min": 14.388244999281596, + "p50": 18.07164799902239, + "p95": 24.479674000758678, + "p99": 24.479674000758678, + "max": 24.479674000758678, + "mean": 18.796886999350843, + "stdev": 3.4151315245862714, + "cv": 0.1816860166635153, + "iqr": 6.081141000322532, + "spread_ratio": 1.7013662195758374, + "stability": "noisy" + }, + "verdict": "indistinguishable" + }, + { + "dataset": "geospatial", + "query_label": "bounding box, aggregated to a grid cell", + "description": "duckdb vs clickhouse (geospatial): INDISTINGUISHABLE at this sample size. Medians differ by 1.1x but the observed ranges overlap (baseline 21.3-36.3ms, candidate 20.0-34.5ms; n=9/9, CV 0.17/0.20).", + "p50_ratio": 1.1476612983542442, + "conservative_ratio": 0.619136273621902, + "optimistic_ratio": 1.8180789932751718, + "ranges_overlap": true, + "baseline": { + "n": 9, + "min": 21.336174000000028, + "p50": 27.89697900000101, + "p95": 36.31744599999911, + "p99": 36.31744599999911, + "max": 36.31744599999911, + "mean": 27.244908555555714, + "stdev": 4.498748211227398, + "cv": 0.16512252929951657, + "iqr": 7.097248000000036, + "spread_ratio": 1.7021536288558137, + "stability": "noisy" + }, + "candidate": { + "n": 9, + "min": 19.97572500113165, + "p50": 24.307676001626533, + "p95": 34.46119200089015, + "p99": 34.46119200089015, + "max": 34.46119200089015, + "mean": 26.572125556008864, + "stdev": 5.191758162396437, + "cv": 0.19538362301703047, + "iqr": 8.748693999223178, + "spread_ratio": 1.7251535050136042, + "stability": "noisy" + }, + "verdict": "indistinguishable" + } + ], + "not_exercised": [ + { + "engine": "trino", + "available": false, + "reason": "a Trino is running on this host but belongs to another lane (st0re-verify-trino); benchmarking through someone else’s container would measure their configuration" + }, + { + "engine": "semantic_query", + "available": false, + "reason": "MetricFlow has never been executed in 0pt (BI-P029 honest status)" + }, + { + "engine": "superset", + "available": false, + "reason": "no Superset deployment in the estate yet" + }, + { + "engine": "native_charts", + "available": false, + "reason": "render-side; DES1GN owns it (BI-P051..P057)" + }, + { + "engine": "perspective", + "available": false, + "reason": "not deployed" + } + ] +} diff --git a/evidence/bi-p105/materialization-drift.json b/evidence/bi-p105/materialization-drift.json new file mode 100644 index 0000000..669368a --- /dev/null +++ b/evidence/bi-p105/materialization-drift.json @@ -0,0 +1,75 @@ +{ + "clickhouse_version": "26.3.10.60", + "recorded_at": "2026-09-07T01:55:42.821Z", + "source_rows": 2000000, + "after_backfill": { + "verdict": "match", + "buckets_compared": 48, + "revenue_total": { + "source": 801571002, + "aggregate": 801571002, + "delta": 0 + } + }, + "performance": { + "note": "ClickHouse caches marks and pages between repetitions, so these are WARM numbers on one host with nothing else running. They describe this shape of query on this data, not a system property.", + "repetitions": 9, + "base_table_ms": { + "n": 9, + "min": 48.70366200000012, + "p50": 76.67811500000016, + "p95": 115.04416399999991, + "max": 115.04416399999991, + "spread_ratio": 2.36 + }, + "pre_aggregate_ms": { + "n": 9, + "min": 15.281603999999788, + "p50": 22.727642999999944, + "p95": 84.41279499999973, + "max": 84.41279499999973, + "spread_ratio": 5.52 + }, + "speedup_p50": 3.4, + "speedup_range_min_max": [0.6, 7.5] + }, + "gdpr_erasure_drill": { + "mutation": "ALTER TABLE sales DELETE WHERE tenant_id='t_alpha' AND customer_id < 250", + "verdict": "drifted", + "buckets_compared": 48, + "revenue_true_after_erasure": 761580554, + "revenue_pre_aggregate_would_report": 801571002, + "revenue_overstatement": 39990448, + "revenue_overstatement_pct": 5.251, + "worst_bucket": { + "key": "2026-08-28|FI", + "metric": "revenue", + "source_value": 15830779, + "aggregate_value": 16699145, + "absolute_delta": 868366, + "relative_delta": 0.05485301765630106 + }, + "guarded_router_decision": { + "source": "base_table", + "refusal": "state_not_serving", + "detail": "materialization state is \"drifted\"" + }, + "staleness_only_router_would_serve": true, + "staleness_seconds_at_decision": 1, + "note": "The aggregate was NOT stale by any clock: it had just been refreshed, and every freshness check passed. Only the source comparison caught it. This is why staleness alone is not a correctness guarantee." + }, + "after_rebuild": { + "verdict": "match", + "revenue_total": { + "source": 761580554, + "aggregate": 761580554, + "delta": 0 + }, + "routing": { + "source": "materialization", + "materialization_id": "mv_sales_daily", + "target_ref": "opt_p105_mtql9p2f.sales_daily", + "staleness_seconds": 0 + } + } +} diff --git a/scripts/bi-p061-release-race.mjs b/scripts/bi-p061-release-race.mjs new file mode 100644 index 0000000..5e3fa73 --- /dev/null +++ b/scripts/bi-p061-release-race.mjs @@ -0,0 +1,133 @@ +#!/usr/bin/env node +/** + * BI-P061: is "only one published release per dashboard" actually enforced? + * + * The application checks for a live release before publishing, and that check + * is a read-then-write race. This script measures BOTH directions against a + * real Postgres and writes evidence/bi-p061/dashboard-release-race.json: + * + * guard_absent the partial unique index dropped -> both concurrent + * publishes commit, and the dashboard has TWO live versions. + * guard_present the index in place -> the loser fails with 23505. + * + * Without the first run, the second proves nothing: a race that never + * interleaved would also show one published release. + * + * ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25441/opt \ + * node scripts/bi-p061-release-race.mjs + * + * It CREATEs and DROPs its own throwaway schemas and touches nothing else. + */ +import fs from 'node:fs'; +import path from 'node:path'; +import postgres from 'postgres'; + +const URL_ = process.env.ANALYTICS_TEST_DATABASE_URL; +if (!URL_) { + console.error('ANALYTICS_TEST_DATABASE_URL is required. Refusing to invent numbers.'); + process.exit(2); +} + +const STAMP = Date.now().toString(36); +const SCHEMAS = { guard_present: `bi_p061_ev_on_${STAMP}`, guard_absent: `bi_p061_ev_off_${STAMP}` }; +const MIGRATIONS = ['sql/008_analytics_domain_model.sql', 'sql/010_analytics_dashboard_releases.sql']; +const HASH = (c) => c.repeat(64); + +const connect = (schema, max = 1) => + postgres(URL_, { max, onnotice: () => undefined, connection: { search_path: schema } }); + +async function setup(schema, { dropIndex }) { + const admin = postgres(URL_, { max: 1, onnotice: () => undefined }); + await admin.unsafe(`CREATE SCHEMA IF NOT EXISTS ${schema}`); + await admin.end({ timeout: 5 }); + + const c = connect(schema, 2); + for (const f of MIGRATIONS) await c.unsafe(fs.readFileSync(path.join(process.cwd(), f), 'utf8')); + if (dropIndex) await c.unsafe('DROP INDEX opt_analytics_dashboard_release_one_published'); + await c.unsafe(` + INSERT INTO opt_analytics_dashboard_definition + (dashboard_id, tenant_id, project_id, name, current_version, owner_id) + VALUES ('dash_race', 't_alpha', 'p_main', 'Race', 1, 'u_1'); + INSERT INTO opt_analytics_dashboard_version + (dashboard_version_id, dashboard_id, tenant_id, project_id, version, layout_json, layout_hash) + VALUES ('dv_1', 'dash_race', 't_alpha', 'p_main', 1, '{}', '${HASH('a')}'), + ('dv_2', 'dash_race', 't_alpha', 'p_main', 2, '{}', '${HASH('b')}');`); + return c; +} + +/** Both sessions read "nothing is live", then both write. */ +async function race(schema) { + const conns = [connect(schema), connect(schema)]; + const publish = (conn, n) => + conn.begin(async (tx) => { + await tx.unsafe( + `SELECT release_id FROM opt_analytics_dashboard_release + WHERE dashboard_id='dash_race' AND state='published'`, + ); + await new Promise((r) => setTimeout(r, 60)); + await tx.unsafe(` + INSERT INTO opt_analytics_dashboard_release + (release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, + document_hash, published_at) + VALUES ('rel_${n}', 'dash_race', 'dv_${n}', 't_alpha', 'p_main', ${n}, + 'published', '${HASH(n === 1 ? 'a' : 'b')}', now())`); + }); + const results = await Promise.allSettled([publish(conns[0], 1), publish(conns[1], 2)]); + for (const c of conns) await c.end({ timeout: 5 }); + return { + committed: results.filter((r) => r.status === 'fulfilled').length, + rejected: results + .filter((r) => r.status === 'rejected') + .map((r) => ({ code: r.reason?.code ?? null, constraint: r.reason?.constraint_name ?? null })), + }; +} + +async function publishedRows(conn) { + const rows = await conn.unsafe( + `SELECT release_id, version FROM opt_analytics_dashboard_release + WHERE dashboard_id='dash_race' AND state='published' ORDER BY version`, + ); + return rows.map((r) => ({ release_id: r.release_id, version: r.version })); +} + +const out = { recorded_at: new Date().toISOString() }; +const cleanup = []; +try { + for (const [label, schema] of Object.entries(SCHEMAS)) { + const conn = await setup(schema, { dropIndex: label === 'guard_absent' }); + cleanup.push([schema, conn]); + if (!out.postgres_version) { + const v = await conn.unsafe('SHOW server_version'); + out.postgres_version = v[0].server_version; + } + const result = await race(schema); + out[label] = { + partial_unique_index_present: label === 'guard_present', + transactions_committed: result.committed, + rejections: result.rejected, + published_releases_after: await publishedRows(conn), + }; + } + + out.verdict = { + counterfactual_produced_two_live_dashboards: out.guard_absent.published_releases_after.length === 2, + guard_left_exactly_one: out.guard_present.published_releases_after.length === 1, + loser_error_code: out.guard_present.rejections[0]?.code ?? null, + application_check_alone_is_sufficient: false, + note: + 'DashboardReleaseService.publish() reads for a live release before writing. Both sessions in ' + + 'guard_absent passed that check and both committed. The partial unique index is what holds.', + }; + + const dir = path.join(process.cwd(), 'evidence/bi-p061'); + fs.mkdirSync(dir, { recursive: true }); + const file = path.join(dir, 'dashboard-release-race.json'); + fs.writeFileSync(file, `${JSON.stringify(out, null, 2)}\n`); + console.log(`wrote ${file}`); + console.log(JSON.stringify(out.verdict, null, 2)); +} finally { + for (const [schema, conn] of cleanup) { + await conn.unsafe(`DROP SCHEMA IF EXISTS ${schema} CASCADE`); + await conn.end({ timeout: 5 }); + } +} diff --git a/sql/010_analytics_dashboard_releases.sql b/sql/010_analytics_dashboard_releases.sql new file mode 100644 index 0000000..171a7a2 --- /dev/null +++ b/sql/010_analytics_dashboard_releases.sql @@ -0,0 +1,129 @@ +-- 0pt analytics: dashboard releases (BI-P061 — DASHBOARD VERSIONING). +-- +-- Extends sql/008. Additive and idempotent (CREATE TABLE IF NOT EXISTS, +-- CREATE OR REPLACE FUNCTION, DROP TRIGGER IF EXISTS before CREATE TRIGGER), +-- so it replays onto an already-migrated database like 009 does. +-- +-- Ordering: requires sql/008 (it references opt_analytics_dashboard_version). +-- +-- WHY A SEPARATE TABLE RATHER THAN A `state` COLUMN ON THE VERSION. +-- sql/008 puts a BEFORE UPDATE OR DELETE trigger on +-- opt_analytics_dashboard_version, because BI-P002 lists DashboardVersion as +-- immutable. A lifecycle column on that row would need UPDATEs, so adding one +-- would mean weakening the immutability everything else depends on. Instead +-- the frozen DOCUMENT stays immutable in the version table, and the RELEASE — +-- the thing that legitimately moves DRAFT -> PUBLISHED -> SUPERSEDED — lives +-- here with its own, much narrower, mutation rule: +-- +-- * `state` may only advance draft -> published -> superseded. Any other +-- transition RAISEs, including published -> draft ("un-publishing" is how +-- a rollback gets faked without leaving a trace) and superseded -> *. +-- * Every content column is frozen on UPDATE. A release may change state and +-- nothing else; repointing a published release at a different version is +-- the exact edit that would make "immutable release" a lie. +-- * DELETE is refused outright. Release history is a ledger; a rollback is +-- an entry in it, never the removal of one. +-- +-- THE INVARIANT THAT NEEDS THE DATABASE: at most ONE published release per +-- dashboard. An application check reads, decides, then writes, and two +-- requests that interleave between the read and the write both decide "yes". +-- The partial unique index below turns that race into a constraint violation +-- for the loser. tests/analytics/dashboard-release-postgres.realinfra.test.ts +-- runs the race with the index DROPPED first and shows two published releases +-- for one dashboard, so the index is known to be what prevents it. + +-- ── the release ledger ──────────────────────────────────────────────────── +CREATE TABLE IF NOT EXISTS opt_analytics_dashboard_release ( + release_id TEXT PRIMARY KEY, + dashboard_id TEXT NOT NULL REFERENCES opt_analytics_dashboard_definition (dashboard_id), + dashboard_version_id TEXT NOT NULL REFERENCES opt_analytics_dashboard_version (dashboard_version_id), + tenant_id TEXT NOT NULL, + project_id TEXT NOT NULL, + version INTEGER NOT NULL CHECK (version >= 1), + state TEXT NOT NULL CHECK (state IN ('draft', 'published', 'superseded')), + -- The BI-P060 canonical hash of the released document. Copied here so a + -- rollback can be VERIFIED (the new release must carry the hash of the + -- version it restores) without re-reading and re-hashing the document. + document_hash TEXT NOT NULL CHECK (document_hash ~ '^[0-9a-f]{64}$'), + -- Set when this release restores an earlier version. It is a NEW release of + -- a NEW version, never a resurrected old row -- that is what keeps the + -- history linear and the audit answerable ("what was live at 14:05?"). + rollback_of_version INTEGER CHECK (rollback_of_version >= 1), + created_by TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + published_at TIMESTAMPTZ, + superseded_at TIMESTAMPTZ, + superseded_by_release_id TEXT, + UNIQUE (dashboard_id, version), + -- A state's timestamps must match the state. A 'published' row with no + -- published_at is the shape that makes an SLO or an audit answer wrong. + CONSTRAINT opt_analytics_dashboard_release_timestamps CHECK ( + (state = 'draft' AND published_at IS NULL AND superseded_at IS NULL) + OR (state = 'published' AND published_at IS NOT NULL AND superseded_at IS NULL) + OR (state = 'superseded' AND published_at IS NOT NULL AND superseded_at IS NOT NULL) + ) +); + +CREATE INDEX IF NOT EXISTS opt_analytics_dashboard_release_tenant_idx + ON opt_analytics_dashboard_release (tenant_id, project_id, dashboard_id, version); + +-- At most one PUBLISHED release per dashboard. This is the concurrency +-- guarantee; the application's check is only an early, friendlier error. +CREATE UNIQUE INDEX IF NOT EXISTS opt_analytics_dashboard_release_one_published + ON opt_analytics_dashboard_release (dashboard_id) + WHERE state = 'published'; + +-- At most one open DRAFT per dashboard. Two drafts race to publish and the +-- loser's edits vanish into a superseded release nobody asked for. +CREATE UNIQUE INDEX IF NOT EXISTS opt_analytics_dashboard_release_one_draft + ON opt_analytics_dashboard_release (dashboard_id) + WHERE state = 'draft'; + +-- ── transition enforcement ──────────────────────────────────────────────── +CREATE OR REPLACE FUNCTION opt_analytics_dashboard_release_guard() RETURNS trigger AS $$ +BEGIN + IF TG_OP = 'DELETE' THEN + RAISE EXCEPTION + 'BI-P061: dashboard releases are a ledger; delete is refused (release %)', OLD.release_id + USING ERRCODE = 'restrict_violation'; + END IF; + + -- Content is frozen for the life of the release. + IF NEW.dashboard_id IS DISTINCT FROM OLD.dashboard_id + OR NEW.dashboard_version_id IS DISTINCT FROM OLD.dashboard_version_id + OR NEW.tenant_id IS DISTINCT FROM OLD.tenant_id + OR NEW.project_id IS DISTINCT FROM OLD.project_id + OR NEW.version IS DISTINCT FROM OLD.version + OR NEW.document_hash IS DISTINCT FROM OLD.document_hash + OR NEW.rollback_of_version IS DISTINCT FROM OLD.rollback_of_version + OR NEW.created_at IS DISTINCT FROM OLD.created_at THEN + RAISE EXCEPTION + 'BI-P061: a dashboard release is immutable apart from its state (release %)', OLD.release_id + USING ERRCODE = 'restrict_violation'; + END IF; + + IF NEW.state IS DISTINCT FROM OLD.state THEN + IF NOT ((OLD.state = 'draft' AND NEW.state = 'published') + OR (OLD.state = 'published' AND NEW.state = 'superseded')) THEN + RAISE EXCEPTION + 'BI-P061: illegal dashboard release transition % -> % (release %)', + OLD.state, NEW.state, OLD.release_id + USING ERRCODE = 'restrict_violation'; + END IF; + END IF; + + -- A published timestamp, once set, is history. + IF OLD.published_at IS NOT NULL AND NEW.published_at IS DISTINCT FROM OLD.published_at THEN + RAISE EXCEPTION + 'BI-P061: published_at is history and cannot be rewritten (release %)', OLD.release_id + USING ERRCODE = 'restrict_violation'; + END IF; + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS opt_analytics_dashboard_release_guard ON opt_analytics_dashboard_release; +CREATE TRIGGER opt_analytics_dashboard_release_guard + BEFORE UPDATE OR DELETE ON opt_analytics_dashboard_release + FOR EACH ROW EXECUTE FUNCTION opt_analytics_dashboard_release_guard(); diff --git a/src/analytics/bench/benchmark-harness.ts b/src/analytics/bench/benchmark-harness.ts new file mode 100644 index 0000000..9a1a12d --- /dev/null +++ b/src/analytics/bench/benchmark-harness.ts @@ -0,0 +1,341 @@ +/** + * src/analytics/bench/benchmark-harness.ts + * ========================================= + * BI-P104 — BI PERFORMANCE BENCHMARK HARNESS. DoD: "results committed as + * machine-readable evidence." + * + * The harness exists because of a specific way benchmarks lie. Yesterday's + * lane measured the same budget-enforcement overshoot at ~150x on one run and + * ~130x on the next, on the same machine, and correctly reported the SHAPE + * rather than a number. That is the doctrine encoded here: + * + * 1. A SINGLE FIGURE IS NOT A SYSTEM PROPERTY. `summarizeSamples` returns a + * distribution — n, min, p50, p95, p99, max, mean, stdev, CV, IQR — and + * there is no function anywhere in this module that returns one number + * for a measurement. `describeComparison` renders a RANGE and the sample + * size into every sentence it produces. + * + * 2. OVERLAPPING DISTRIBUTIONS ARE INDISTINGUISHABLE, whatever the medians + * say. If the candidate's p95 is not below the baseline's minimum, the + * verdict is `indistinguishable` even when p50 differs by 2x — because a + * 2x median gap inside a 10x spread is noise with a story attached. + * + * 3. TOO FEW SAMPLES IS A REFUSAL. Under MIN_SAMPLES the summary carries + * `stability: 'insufficient_samples'` and comparisons refuse to judge. + * A three-run benchmark is an anecdote. + * + * 4. AN ENGINE THAT WAS NOT RUN IS RECORDED AS NOT RUN. `EngineAvailability` + * carries a reason. The evidence file lists every engine BI-P104 names, + * including the ones nothing measured — an omitted engine reads as "we + * chose not to report it". + */ + +/** The six dataset profiles BI-P104 names, with what makes each one distinct. */ +export const BENCH_DATASETS = [ + { + id: 'small_saas', + label: 'small SaaS', + approx_rows: 100_000, + shape: 'narrow fact table, one tenant, high selectivity', + }, + { + id: 'medium_business', + label: 'medium business', + approx_rows: 2_000_000, + shape: 'multi-tenant fact table with a date and a few dimensions', + }, + { + id: 'vehiq_market', + label: 'VehIQ market', + approx_rows: 1_000_000, + shape: 'vehicle listings: make/model/year/price/mileage, skewed cardinalities', + }, + { + id: 'large_event', + label: 'large event', + approx_rows: 5_000_000, + shape: 'append-only event stream, timestamp-ordered, sparse dimensions', + }, + { + id: 'wide_table', + label: 'wide table', + approx_rows: 200_000, + shape: 'many columns, few read per query — the columnar-store best case', + }, + { + id: 'geospatial', + label: 'geospatial', + approx_rows: 500_000, + shape: 'lat/lon points with bounding-box and distance predicates', + }, +] as const; +export type BenchDatasetId = (typeof BENCH_DATASETS)[number]['id']; + +/** Every engine BI-P104 lists. All of them appear in the evidence, run or not. */ +export const BENCH_ENGINES = [ + 'clickhouse', + 'duckdb', + 'trino', + 'semantic_query', + 'superset', + 'native_charts', + 'perspective', +] as const; +export type BenchEngine = (typeof BENCH_ENGINES)[number]; + +export interface EngineAvailability { + engine: BenchEngine; + available: boolean; + /** Required when unavailable. "Unknown" is not a reason. */ + reason?: string; + version?: string; +} + +export const STABILITY_VERDICTS = ['stable', 'noisy', 'unstable', 'insufficient_samples'] as const; +export type StabilityVerdict = (typeof STABILITY_VERDICTS)[number]; + +/** Below this a run is an anecdote, not a measurement. */ +export const MIN_SAMPLES = 5; +/** Coefficient of variation thresholds. Wide on purpose: real hosts are noisy. */ +const CV_STABLE = 0.15; +const CV_NOISY = 0.5; + +export interface Distribution { + n: number; + min: number; + p50: number; + p95: number; + p99: number; + max: number; + mean: number; + stdev: number; + /** stdev / mean. The number that decides whether a comparison is meaningful. */ + cv: number; + iqr: number; + /** max / min. Reported because it is what a reader feels. */ + spread_ratio: number; + stability: StabilityVerdict; +} + +/** Nearest-rank; no interpolation, so every reported value really happened. */ +function pct(sorted: number[], q: number): number { + const rank = Math.ceil(q * sorted.length); + return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))]; +} + +export function summarizeSamples(samples: readonly number[]): Distribution { + if (samples.length === 0) { + return { + n: 0, + min: 0, + p50: 0, + p95: 0, + p99: 0, + max: 0, + mean: 0, + stdev: 0, + cv: 0, + iqr: 0, + spread_ratio: 0, + stability: 'insufficient_samples', + }; + } + const s = [...samples].sort((a, b) => a - b); + const mean = s.reduce((a, b) => a + b, 0) / s.length; + const variance = s.reduce((acc, v) => acc + (v - mean) ** 2, 0) / s.length; + const stdev = Math.sqrt(variance); + const cv = mean === 0 ? 0 : stdev / mean; + const stability: StabilityVerdict = + s.length < MIN_SAMPLES + ? 'insufficient_samples' + : cv <= CV_STABLE + ? 'stable' + : cv <= CV_NOISY + ? 'noisy' + : 'unstable'; + return { + n: s.length, + min: s[0], + p50: pct(s, 0.5), + p95: pct(s, 0.95), + p99: pct(s, 0.99), + max: s[s.length - 1], + mean, + stdev, + cv, + iqr: pct(s, 0.75) - pct(s, 0.25), + spread_ratio: s[0] === 0 ? Number.POSITIVE_INFINITY : s[s.length - 1] / s[0], + stability, + }; +} + +export const COMPARISON_VERDICTS = ['faster', 'slower', 'indistinguishable', 'not_comparable'] as const; +export type ComparisonVerdict = (typeof COMPARISON_VERDICTS)[number]; + +export interface Comparison { + verdict: ComparisonVerdict; + /** Ratio of medians. Present for information; never the claim on its own. */ + p50_ratio: number; + /** Worst case for the candidate: baseline.min / candidate.max. */ + conservative_ratio: number; + /** Best case for the candidate: baseline.max / candidate.min. */ + optimistic_ratio: number; + /** True when the observed ranges overlap at all. */ + ranges_overlap: boolean; + baseline: Distribution; + candidate: Distribution; +} + +/** + * Compare two measured distributions. + * + * `faster` requires clean separation — the candidate's p95 below the + * baseline's minimum. Anything less is `indistinguishable`, however flattering + * the medians look, because that is exactly the claim a noisy host will + * manufacture for you. + */ +export function compareDistributions(baseline: Distribution, candidate: Distribution): Comparison { + const ratio = (a: number, b: number) => (b === 0 ? Number.POSITIVE_INFINITY : a / b); + const base: Omit = { + p50_ratio: ratio(baseline.p50, candidate.p50), + conservative_ratio: ratio(baseline.min, candidate.max), + optimistic_ratio: ratio(baseline.max, candidate.min), + ranges_overlap: !(candidate.max < baseline.min || baseline.max < candidate.min), + baseline, + candidate, + }; + if (baseline.stability === 'insufficient_samples' || candidate.stability === 'insufficient_samples') { + return { ...base, verdict: 'not_comparable' }; + } + if (candidate.p95 < baseline.min) return { ...base, verdict: 'faster' }; + if (baseline.p95 < candidate.min) return { ...base, verdict: 'slower' }; + return { ...base, verdict: 'indistinguishable' }; +} + +/** + * The sentence a human is allowed to quote. It ALWAYS carries the range, the + * sample size and the stability verdict, so it cannot be trimmed down to "3.4x + * faster" without visibly removing the caveats. + */ +export function describeComparison(c: Comparison, label = 'candidate'): string { + const r = (v: number) => (Number.isFinite(v) ? v.toFixed(1) : '∞'); + if (c.verdict === 'not_comparable') { + return ( + `${label}: NOT COMPARABLE — ${c.baseline.n} baseline and ${c.candidate.n} candidate samples, ` + + `fewer than the ${MIN_SAMPLES} required. A run this short is an anecdote.` + ); + } + if (c.verdict === 'indistinguishable') { + return ( + `${label}: INDISTINGUISHABLE at this sample size. Medians differ by ${r(c.p50_ratio)}x but the ` + + `observed ranges overlap (baseline ${c.baseline.min.toFixed(1)}-${c.baseline.max.toFixed(1)}ms, ` + + `candidate ${c.candidate.min.toFixed(1)}-${c.candidate.max.toFixed(1)}ms; n=${c.baseline.n}/${c.candidate.n}, ` + + `CV ${c.baseline.cv.toFixed(2)}/${c.candidate.cv.toFixed(2)}).` + ); + } + return ( + `${label}: ${c.verdict.toUpperCase()} — ${r(c.conservative_ratio)}x to ${r(c.optimistic_ratio)}x ` + + `(median ${r(c.p50_ratio)}x) over n=${c.baseline.n}/${c.candidate.n}, ` + + `stability ${c.baseline.stability}/${c.candidate.stability}. Warm cache, one host, this query shape.` + ); +} + +export interface BenchmarkRun { + engine: BenchEngine; + dataset: BenchDatasetId; + query_label: string; + /** Discarded from the statistics; recorded so the discard is visible. */ + warmup_ms: number[]; + samples_ms: number[]; + distribution: Distribution; + rows_returned?: number; +} + +/** + * Run one measurement. Warmups are executed and RECORDED but excluded from the + * distribution: a first run that pays for cold marks and page cache is a + * different measurement, and silently folding it in is how a p95 becomes a + * story about disk. + */ +export async function measure(args: { + engine: BenchEngine; + dataset: BenchDatasetId; + query_label: string; + run: () => Promise<{ rows?: number }>; + warmups?: number; + repetitions?: number; + now?: () => number; +}): Promise { + const now = args.now ?? (() => performance.now()); + const warmups = args.warmups ?? 2; + const repetitions = args.repetitions ?? 9; + const warmup_ms: number[] = []; + const samples_ms: number[] = []; + let rows: number | undefined; + + for (let i = 0; i < warmups; i += 1) { + const t0 = now(); + await args.run(); + warmup_ms.push(now() - t0); + } + for (let i = 0; i < repetitions; i += 1) { + const t0 = now(); + const out = await args.run(); + samples_ms.push(now() - t0); + rows = out.rows ?? rows; + } + return { + engine: args.engine, + dataset: args.dataset, + query_label: args.query_label, + warmup_ms, + samples_ms, + distribution: summarizeSamples(samples_ms), + ...(rows !== undefined ? { rows_returned: rows } : {}), + }; +} + +export interface BenchmarkEvidence { + recorded_at: string; + host_note: string; + datasets: typeof BENCH_DATASETS; + engines: EngineAvailability[]; + runs: BenchmarkRun[]; + comparisons: Array<{ dataset: BenchDatasetId; query_label: string; description: string } & Comparison>; + /** Engines BI-P104 names that produced no measurement, each with a reason. */ + not_exercised: EngineAvailability[]; +} + +/** + * Assemble the evidence document. Every engine BI-P104 names appears, and the + * ones with no runs are listed under `not_exercised` with their reason — an + * engine quietly missing from a benchmark table reads as a choice. + */ +export function buildEvidence(args: { + host_note: string; + engines: EngineAvailability[]; + runs: BenchmarkRun[]; + comparisons?: BenchmarkEvidence['comparisons']; + now?: Date; +}): BenchmarkEvidence { + const measured = new Set(args.runs.map((r) => r.engine)); + const declared = new Map(args.engines.map((e) => [e.engine, e])); + return { + recorded_at: (args.now ?? new Date()).toISOString(), + host_note: args.host_note, + datasets: BENCH_DATASETS, + engines: BENCH_ENGINES.map( + (engine) => declared.get(engine) ?? { engine, available: false, reason: 'not declared by this run' }, + ), + runs: args.runs, + comparisons: args.comparisons ?? [], + not_exercised: BENCH_ENGINES.filter((e) => !measured.has(e)).map((engine) => { + const d = declared.get(engine); + if (!d) return { engine, available: false, reason: 'not present in this environment' }; + // An engine declared AVAILABLE that produced no measurement is the + // dangerous case: it reads as "we could have, and chose not to say". + // It gets a reason too, generated here rather than left blank. + return d.reason ? d : { ...d, reason: 'declared available but produced no measurement in this run' }; + }), + }; +} diff --git a/src/analytics/dashboards/dashboard-document.ts b/src/analytics/dashboards/dashboard-document.ts new file mode 100644 index 0000000..788c8c6 --- /dev/null +++ b/src/analytics/dashboards/dashboard-document.ts @@ -0,0 +1,837 @@ +/** + * src/analytics/dashboards/dashboard-document.ts + * =============================================== + * BI-P060 — ANALYTICS DASHBOARD MODEL. + * + * BI-P002 already named `DashboardDefinition` and `DashboardVersion` and gave + * them storage (sql/008). What it did not give them is a BODY: `layout_json` + * is a string, and a string can hold anything — including a Superset export + * that only Superset can read back. This module is that body, and its whole + * point is the DoD: + * + * "A native SIAX dashboard can be reconstructed without Superset." + * + * Reconstruction is not a claim you can make by writing a type. It is provable + * exactly once you can take a document and produce, for every widget, a + * runnable `AnalyticsQueryRequest` (BI-P016) plus a rendering spec, with no + * vendor identifier anywhere in the plan. `planDashboard()` is that function + * and `tests/analytics/dashboard-model.test.ts` runs it over a document + * carrying Superset ids, asserting the plan is complete and vendor-free. + * + * THE VENDOR RULE, stated once: a Superset/Metabase/Cube identifier may be + * carried as `vendor_ref` for round-tripping, and it is + * - never a widget's source of truth (a widget with only a vendor_ref is + * REFUSED — it is precisely the document that cannot be reconstructed), + * - never part of the canonical hash (so re-exporting to a new Superset + * instance does not fabricate a new dashboard version). + * + * NO SQL LIVES HERE either. Widget sources are metric/dimension NAMES that + * resolve through the catalog, and query widgets are validated by + * `parseAnalyticsQueryRequest`, which rejects `sql`, `where`, `tenant_id` and + * every other smuggling key structurally. A dashboard is therefore not a new + * way into the engine; it is a saved set of contract requests. + */ +import crypto from 'node:crypto'; +import { SiaxError } from '../../platform/errors'; +import { + parseAnalyticsQueryRequest, + TIME_GRAINS, + type AnalyticsFilter, + type AnalyticsQueryRequest, + type FilterOperator, + type TimeGrain, +} from '../contract/query-request'; + +// ── vocabulary ───────────────────────────────────────────────────────────── + +/** + * Visualization types. A CLOSED list, deliberately engine-neutral: these are + * SIAX chart kinds (DES1GN BI-P051..P057 renders them), not Superset viz_type + * strings. A document naming `echarts_timeseries` is a Superset document. + */ +export const VISUALIZATION_TYPES = [ + 'big_number', + 'line', + 'area', + 'bar', + 'stacked_bar', + 'pie', + 'table', + 'pivot_table', + 'scatter', + 'heatmap', + 'funnel', + 'map', +] as const; +export type VisualizationType = (typeof VISUALIZATION_TYPES)[number]; + +export const REFRESH_MODES = ['manual', 'interval', 'scheduled'] as const; +export type RefreshMode = (typeof REFRESH_MODES)[number]; + +/** + * Who may open the dashboard. `roles` are cl0ud role names (BI-P033 decides, + * this only declares). There is deliberately NO tenant field: a dashboard is + * stored inside a tenant, and a document that could name one would be a way to + * ask for another tenant's dashboard — the INF0 Wave-2 shape. + */ +export const DASHBOARD_VISIBILITIES = ['private', 'tenant', 'roles'] as const; +export type DashboardVisibility = (typeof DASHBOARD_VISIBILITIES)[number]; + +export const PARAMETER_TYPES = ['string', 'number', 'date', 'enum'] as const; +export type ParameterType = (typeof PARAMETER_TYPES)[number]; + +// ── the document ─────────────────────────────────────────────────────────── + +/** Grid placement. A 12-column grid, matching the portal layout (BI-P062). */ +export interface WidgetLayout { + x: number; + y: number; + w: number; + h: number; +} + +/** A saved dashboard filter. Its VALUES are data; its dimension is a name. */ +export interface DashboardFilter { + filter_id: string; + label: string; + /** Caller-facing dimension name, resolved through the catalog downstream. */ + dimension: string; + operator: FilterOperator; + /** Default values, or `{{param}}` references to a declared parameter. */ + default_values?: string[]; + /** May a viewer change it? A locked filter is how row scoping is saved. */ + locked?: boolean; +} + +/** A declared parameter. Widgets and filters may reference it as `{{name}}`. */ +export interface DashboardParameter { + name: string; + type: ParameterType; + required: boolean; + default_value?: string; + /** Required for `enum`; a closed list is the only way a value can be trusted. */ + allowed_values?: string[]; +} + +/** A widget backed by ONE metric — the common case, and the cheapest to plan. */ +export interface MetricWidgetSource { + kind: 'metric'; + dataset: string; + metric: string; + dimensions?: string[]; + grain?: TimeGrain; + limit?: number; +} + +/** A widget backed by a saved contract request (still no SQL, by construction). */ +export interface QueryWidgetSource { + kind: 'query'; + /** Parsed with `parseAnalyticsQueryRequest` at validation time, not at render time. */ + request: AnalyticsQueryRequest; +} + +export type WidgetSource = MetricWidgetSource | QueryWidgetSource; + +export interface VisualizationSpec { + type: VisualizationType; + /** Field roles by NAME. `x`/`y` name metrics or dimensions in the source. */ + encoding: { + x?: string; + y?: string[]; + series?: string; + value?: string; + }; + /** Presentation-only knobs. Bounded scalars; never code, never a template. */ + options?: Record; +} + +export interface DashboardWidget { + widget_id: string; + title: string; + source: WidgetSource; + visualization: VisualizationSpec; + layout: WidgetLayout; + /** filter_ids this widget honours. Empty means "all dashboard filters". */ + filter_bindings?: string[]; + /** Advisory only: excluded from the canonical hash, never a source of truth. */ + vendor_ref?: VendorRef; +} + +/** A foreign system's id for the same object. Round-tripping aid, nothing more. */ +export interface VendorRef { + system: string; + id: string; +} + +export interface DashboardRefresh { + mode: RefreshMode; + /** Required for `interval`. */ + interval_seconds?: number; + /** Required for `scheduled`. */ + cron?: string; + /** + * The staleness the OWNER accepts. BI-P026 computes actual freshness; this + * is the number that makes a freshness verdict actionable rather than + * decorative, and BI-P105 reads it when deciding whether a pre-aggregate may + * serve this dashboard. + */ + max_staleness_seconds: number; +} + +export interface DashboardPermissions { + visibility: DashboardVisibility; + /** Required (non-empty) when visibility is `roles`. */ + roles?: string[]; + /** Optional cl0ud entitlement key (BI-P079). */ + required_entitlement?: string; +} + +/** BI-P060's canonical document, field-for-field with the program's list. */ +export interface DashboardDocument { + id: string; + title: string; + description?: string; + layout: { grid_columns: number; row_height_px: number }; + filters: DashboardFilter[]; + widgets: DashboardWidget[]; + parameters: DashboardParameter[]; + refresh: DashboardRefresh; + permissions: DashboardPermissions; + /** Document schema version — bumped when THIS shape changes, not per edit. */ + version: number; + vendor_ref?: VendorRef; +} + +export const DASHBOARD_DOCUMENT_KEYS = [ + 'id', + 'title', + 'description', + 'layout', + 'filters', + 'widgets', + 'parameters', + 'refresh', + 'permissions', + 'version', + 'vendor_ref', +] as const; + +/** + * Keys that mean "this document is really a Superset export". Named + * explicitly so the refusal explains the architecture instead of saying + * "unknown key"; the unknown-key rule below catches everything else anyway. + */ +const VENDOR_NATIVE_KEYS = new Set([ + 'slice_id', + 'slice_name', + 'viz_type', + 'datasource', + 'datasource_id', + 'datasource_type', + 'form_data', + 'query_context', + 'chart_id', + 'dashboard_id_superset', + 'position_json', + 'json_metadata', + 'card_id', + 'cube', +]); + +/** Keys that would turn a dashboard into an SQL surface. */ +const SQL_SMELLING_KEYS = new Set([ + 'sql', + 'query_text', + 'statement', + 'expression', + 'where', + 'having', + 'raw', + 'tenant', + 'tenant_id', + 'project_id', + 'database', + 'table', +]); + +const MAX_WIDGETS = 60; +const MAX_FILTERS = 32; +const MAX_PARAMETERS = 24; +const MAX_NAME_LENGTH = 128; +const MAX_TITLE_LENGTH = 200; +const MAX_GRID_COLUMNS = 24; +const MIN_REFRESH_INTERVAL_SECONDS = 10; +const MAX_STALENESS_SECONDS = 30 * 24 * 3600; +/** `{{param}}` — the only templating the document has, and it binds a VALUE. */ +const PARAM_REFERENCE = /^\{\{\s*([A-Za-z][A-Za-z0-9_]{0,63})\s*\}\}$/; +const IDENTIFIER = /^[A-Za-z][A-Za-z0-9_.-]{0,63}$/; + +function invalid(detail: string): never { + throw new SiaxError('INVALID_REQUEST', '0pt.analytics.dashboard', detail); +} + +function asObject(raw: unknown, what: string): Record { + if (raw === null || typeof raw !== 'object' || Array.isArray(raw)) invalid(`${what} must be an object`); + return raw as Record; +} + +function rejectUnknownKeys(obj: Record, allowed: readonly string[], what: string): void { + for (const key of Object.keys(obj)) { + if (allowed.includes(key)) continue; + if (VENDOR_NATIVE_KEYS.has(key)) { + invalid( + `"${key}" in ${what} is a vendor-native field — a SIAX dashboard is defined by metrics, ` + + `queries and a visualization spec, not by a BI tool's internal ids (BI-P060). ` + + `Carry the id in "vendor_ref" if you need to round-trip it.`, + ); + } + if (SQL_SMELLING_KEYS.has(key)) { + invalid(`"${key}" in ${what} is not part of the dashboard model — dashboards do not carry SQL`); + } + invalid(`unknown field "${key}" in ${what}`); + } +} + +function asString(raw: unknown, what: string, max = MAX_NAME_LENGTH): string { + if (typeof raw !== 'string') invalid(`${what} must be a string`); + const v = raw.trim(); + if (!v) invalid(`${what} must not be empty`); + if (v.length > max) invalid(`${what} exceeds ${max} characters`); + return v; +} + +function asIdentifier(raw: unknown, what: string): string { + const v = asString(raw, what); + if (!IDENTIFIER.test(v)) invalid(`${what} is not a valid identifier`); + return v; +} + +function asInteger(raw: unknown, what: string, min: number, max: number): number { + if (typeof raw !== 'number' || !Number.isFinite(raw)) invalid(`${what} must be a finite number`); + if (!Number.isInteger(raw)) invalid(`${what} must be an integer`); + if (raw < min || raw > max) invalid(`${what} must be between ${min} and ${max}`); + return raw; +} + +function parseVendorRef(raw: unknown, what: string): VendorRef { + const obj = asObject(raw, what); + rejectUnknownKeys(obj, ['system', 'id'], what); + return { system: asString(obj.system, `${what}.system`), id: asString(obj.id, `${what}.id`, 256) }; +} + +// ── parsing ──────────────────────────────────────────────────────────────── + +function parseParameter(raw: unknown, i: number): DashboardParameter { + const what = `parameters[${i}]`; + const obj = asObject(raw, what); + rejectUnknownKeys(obj, ['name', 'type', 'required', 'default_value', 'allowed_values'], what); + const name = asIdentifier(obj.name, `${what}.name`); + const type = asString(obj.type, `${what}.type`); + if (!(PARAMETER_TYPES as readonly string[]).includes(type)) invalid(`${what}.type is not a parameter type`); + if (typeof obj.required !== 'boolean') invalid(`${what}.required must be a boolean`); + const p: DashboardParameter = { name, type: type as ParameterType, required: obj.required }; + if (obj.default_value !== undefined) + p.default_value = asString(obj.default_value, `${what}.default_value`, 4096); + if (obj.allowed_values !== undefined) { + if (!Array.isArray(obj.allowed_values) || obj.allowed_values.length === 0) + invalid(`${what}.allowed_values must be a non-empty array`); + p.allowed_values = obj.allowed_values.map((v, j) => asString(v, `${what}.allowed_values[${j}]`, 4096)); + } + if (p.type === 'enum' && !p.allowed_values) { + invalid(`${what} is an enum parameter with no allowed_values — an unbounded enum is a free-text field`); + } + if (p.required && p.default_value !== undefined) { + invalid(`${what} is required AND has a default — a default makes "required" unenforceable`); + } + return p; +} + +function parseFilter(raw: unknown, i: number, parameterNames: Set): DashboardFilter { + const what = `filters[${i}]`; + const obj = asObject(raw, what); + rejectUnknownKeys(obj, ['filter_id', 'label', 'dimension', 'operator', 'default_values', 'locked'], what); + const filter_id = asIdentifier(obj.filter_id, `${what}.filter_id`); + const label = asString(obj.label, `${what}.label`, MAX_TITLE_LENGTH); + const dimension = asString(obj.dimension, `${what}.dimension`); + const operator = asString(obj.operator, `${what}.operator`) as FilterOperator; + const f: DashboardFilter = { filter_id, label, dimension, operator }; + if (obj.default_values !== undefined) { + if (!Array.isArray(obj.default_values)) invalid(`${what}.default_values must be an array`); + f.default_values = obj.default_values.map((v, j) => { + const s = asString(v, `${what}.default_values[${j}]`, 4096); + const m = PARAM_REFERENCE.exec(s); + if (m && !parameterNames.has(m[1])) { + invalid(`${what}.default_values[${j}] references undeclared parameter "${m[1]}"`); + } + return s; + }); + } + if (obj.locked !== undefined) { + if (typeof obj.locked !== 'boolean') invalid(`${what}.locked must be a boolean`); + f.locked = obj.locked; + } + return f; +} + +function parseVisualization(raw: unknown, what: string): VisualizationSpec { + const obj = asObject(raw, what); + rejectUnknownKeys(obj, ['type', 'encoding', 'options'], what); + const type = asString(obj.type, `${what}.type`); + if (!(VISUALIZATION_TYPES as readonly string[]).includes(type)) { + invalid( + `${what}.type "${type}" is not a SIAX visualization type — the canonical list is engine-neutral ` + + `so a chart survives a BI engine swap (BI-P060)`, + ); + } + const encodingRaw = asObject(obj.encoding ?? {}, `${what}.encoding`); + rejectUnknownKeys(encodingRaw, ['x', 'y', 'series', 'value'], `${what}.encoding`); + const encoding: VisualizationSpec['encoding'] = {}; + if (encodingRaw.x !== undefined) encoding.x = asString(encodingRaw.x, `${what}.encoding.x`); + if (encodingRaw.series !== undefined) + encoding.series = asString(encodingRaw.series, `${what}.encoding.series`); + if (encodingRaw.value !== undefined) encoding.value = asString(encodingRaw.value, `${what}.encoding.value`); + if (encodingRaw.y !== undefined) { + if (!Array.isArray(encodingRaw.y)) invalid(`${what}.encoding.y must be an array`); + encoding.y = encodingRaw.y.map((v, j) => asString(v, `${what}.encoding.y[${j}]`)); + } + const spec: VisualizationSpec = { type: type as VisualizationType, encoding }; + if (obj.options !== undefined) { + const options = asObject(obj.options, `${what}.options`); + const out: Record = {}; + for (const [k, v] of Object.entries(options)) { + if (!IDENTIFIER.test(k)) invalid(`${what}.options key "${k}" is not an identifier`); + if (typeof v === 'string') { + if (v.length > 256) invalid(`${what}.options.${k} exceeds 256 characters`); + out[k] = v; + } else if (typeof v === 'number') { + if (!Number.isFinite(v)) invalid(`${what}.options.${k} must be finite`); + out[k] = v; + } else if (typeof v === 'boolean') { + out[k] = v; + } else { + invalid(`${what}.options.${k} must be a string, number or boolean — options are not a payload`); + } + } + spec.options = out; + } + return spec; +} + +function parseSource(raw: unknown, what: string): WidgetSource { + const obj = asObject(raw, what); + const kind = asString(obj.kind, `${what}.kind`); + if (kind === 'metric') { + rejectUnknownKeys(obj, ['kind', 'dataset', 'metric', 'dimensions', 'grain', 'limit'], what); + const source: MetricWidgetSource = { + kind: 'metric', + dataset: asString(obj.dataset, `${what}.dataset`), + metric: asString(obj.metric, `${what}.metric`), + }; + if (obj.dimensions !== undefined) { + if (!Array.isArray(obj.dimensions)) invalid(`${what}.dimensions must be an array`); + source.dimensions = obj.dimensions.map((d, j) => asString(d, `${what}.dimensions[${j}]`)); + } + if (obj.grain !== undefined) { + const grain = asString(obj.grain, `${what}.grain`); + if (!(TIME_GRAINS as readonly string[]).includes(grain)) invalid(`${what}.grain is not a time grain`); + source.grain = grain as TimeGrain; + } + if (obj.limit !== undefined) source.limit = asInteger(obj.limit, `${what}.limit`, 1, 100_000); + return source; + } + if (kind === 'query') { + rejectUnknownKeys(obj, ['kind', 'request'], what); + // parseAnalyticsQueryRequest is the SQL boundary: it is why a query widget + // cannot become an SQL widget by adding a field. + return { kind: 'query', request: parseAnalyticsQueryRequest(obj.request) }; + } + invalid( + `${what}.kind must be "metric" or "query" — a widget with no reconstructable source cannot be ` + + `rendered without the tool that exported it (BI-P060 DoD)`, + ); +} + +function parseWidget(raw: unknown, i: number, gridColumns: number): DashboardWidget { + const what = `widgets[${i}]`; + const obj = asObject(raw, what); + rejectUnknownKeys( + obj, + ['widget_id', 'title', 'source', 'visualization', 'layout', 'filter_bindings', 'vendor_ref'], + what, + ); + const widget_id = asIdentifier(obj.widget_id, `${what}.widget_id`); + const title = asString(obj.title, `${what}.title`, MAX_TITLE_LENGTH); + const source = parseSource(obj.source, `${what}.source`); + const visualization = parseVisualization(obj.visualization, `${what}.visualization`); + + const layoutObj = asObject(obj.layout, `${what}.layout`); + rejectUnknownKeys(layoutObj, ['x', 'y', 'w', 'h'], `${what}.layout`); + const layout: WidgetLayout = { + x: asInteger(layoutObj.x, `${what}.layout.x`, 0, gridColumns - 1), + y: asInteger(layoutObj.y, `${what}.layout.y`, 0, 10_000), + w: asInteger(layoutObj.w, `${what}.layout.w`, 1, gridColumns), + h: asInteger(layoutObj.h, `${what}.layout.h`, 1, 200), + }; + if (layout.x + layout.w > gridColumns) { + invalid(`${what}.layout overflows the ${gridColumns}-column grid (x=${layout.x}, w=${layout.w})`); + } + + const widget: DashboardWidget = { widget_id, title, source, visualization, layout }; + if (obj.filter_bindings !== undefined) { + if (!Array.isArray(obj.filter_bindings)) invalid(`${what}.filter_bindings must be an array`); + widget.filter_bindings = obj.filter_bindings.map((b, j) => + asIdentifier(b, `${what}.filter_bindings[${j}]`), + ); + } + if (obj.vendor_ref !== undefined) widget.vendor_ref = parseVendorRef(obj.vendor_ref, `${what}.vendor_ref`); + return widget; +} + +/** + * Parse an untrusted dashboard document. Fail-closed everywhere: a field this + * function does not understand is a refusal, because "ignored for now" is how + * a Superset-only document ends up stored as canonical truth. + */ +export function parseDashboardDocument(raw: unknown): DashboardDocument { + const obj = asObject(raw, 'dashboard document'); + rejectUnknownKeys(obj, DASHBOARD_DOCUMENT_KEYS, 'dashboard document'); + + const id = asIdentifier(obj.id, 'id'); + const title = asString(obj.title, 'title', MAX_TITLE_LENGTH); + const version = asInteger(obj.version, 'version', 1, 1_000_000); + + const layoutObj = asObject(obj.layout, 'layout'); + rejectUnknownKeys(layoutObj, ['grid_columns', 'row_height_px'], 'layout'); + const layout = { + grid_columns: asInteger(layoutObj.grid_columns, 'layout.grid_columns', 1, MAX_GRID_COLUMNS), + row_height_px: asInteger(layoutObj.row_height_px, 'layout.row_height_px', 8, 400), + }; + + const rawParameters = obj.parameters ?? []; + if (!Array.isArray(rawParameters)) invalid('parameters must be an array'); + if (rawParameters.length > MAX_PARAMETERS) invalid(`at most ${MAX_PARAMETERS} parameters`); + const parameters = rawParameters.map((p, i) => parseParameter(p, i)); + const parameterNames = new Set(parameters.map((p) => p.name)); + if (parameterNames.size !== parameters.length) invalid('parameters contains duplicate names'); + + const rawFilters = obj.filters ?? []; + if (!Array.isArray(rawFilters)) invalid('filters must be an array'); + if (rawFilters.length > MAX_FILTERS) invalid(`at most ${MAX_FILTERS} filters`); + const filters = rawFilters.map((f, i) => parseFilter(f, i, parameterNames)); + const filterIds = new Set(filters.map((f) => f.filter_id)); + if (filterIds.size !== filters.length) invalid('filters contains duplicate filter_id'); + + if (!Array.isArray(obj.widgets)) invalid('widgets must be an array'); + if (obj.widgets.length === 0) invalid('a dashboard with no widgets has nothing to reconstruct'); + if (obj.widgets.length > MAX_WIDGETS) invalid(`at most ${MAX_WIDGETS} widgets`); + const widgets = obj.widgets.map((w, i) => parseWidget(w, i, layout.grid_columns)); + if (new Set(widgets.map((w) => w.widget_id)).size !== widgets.length) { + invalid('widgets contains duplicate widget_id'); + } + for (const w of widgets) { + for (const b of w.filter_bindings ?? []) { + if (!filterIds.has(b)) invalid(`widget "${w.widget_id}" binds undeclared filter "${b}"`); + } + } + + const refreshObj = asObject(obj.refresh, 'refresh'); + rejectUnknownKeys(refreshObj, ['mode', 'interval_seconds', 'cron', 'max_staleness_seconds'], 'refresh'); + const mode = asString(refreshObj.mode, 'refresh.mode'); + if (!(REFRESH_MODES as readonly string[]).includes(mode)) invalid('refresh.mode is not a refresh mode'); + const refresh: DashboardRefresh = { + mode: mode as RefreshMode, + max_staleness_seconds: asInteger( + refreshObj.max_staleness_seconds, + 'refresh.max_staleness_seconds', + 1, + MAX_STALENESS_SECONDS, + ), + }; + if (refresh.mode === 'interval') { + refresh.interval_seconds = asInteger( + refreshObj.interval_seconds, + 'refresh.interval_seconds', + MIN_REFRESH_INTERVAL_SECONDS, + 86_400, + ); + if (refreshObj.cron !== undefined) invalid('refresh.cron is meaningless when mode is "interval"'); + } else if (refresh.mode === 'scheduled') { + refresh.cron = asString(refreshObj.cron, 'refresh.cron', 128); + if (refreshObj.interval_seconds !== undefined) { + invalid('refresh.interval_seconds is meaningless when mode is "scheduled"'); + } + } else if (refreshObj.interval_seconds !== undefined || refreshObj.cron !== undefined) { + invalid('refresh.mode is "manual" but a schedule was supplied'); + } + + const permObj = asObject(obj.permissions, 'permissions'); + rejectUnknownKeys(permObj, ['visibility', 'roles', 'required_entitlement'], 'permissions'); + const visibility = asString(permObj.visibility, 'permissions.visibility'); + if (!(DASHBOARD_VISIBILITIES as readonly string[]).includes(visibility)) { + invalid('permissions.visibility is not a visibility'); + } + const permissions: DashboardPermissions = { visibility: visibility as DashboardVisibility }; + if (permObj.roles !== undefined) { + if (!Array.isArray(permObj.roles)) invalid('permissions.roles must be an array'); + permissions.roles = permObj.roles.map((r, i) => asIdentifier(r, `permissions.roles[${i}]`)); + } + if (permissions.visibility === 'roles' && (permissions.roles ?? []).length === 0) { + invalid( + 'permissions.visibility is "roles" with no roles — an empty role list reads as "deny all" to one ' + + 'implementation and "allow all" to the next; say which', + ); + } + if (permObj.required_entitlement !== undefined) { + permissions.required_entitlement = asIdentifier( + permObj.required_entitlement, + 'permissions.required_entitlement', + ); + } + + const doc: DashboardDocument = { + id, + title, + layout, + filters, + widgets, + parameters, + refresh, + permissions, + version, + }; + if (obj.description !== undefined) doc.description = asString(obj.description, 'description', 2000); + if (obj.vendor_ref !== undefined) doc.vendor_ref = parseVendorRef(obj.vendor_ref, 'vendor_ref'); + return doc; +} + +// ── canonical hash ───────────────────────────────────────────────────────── + +/** + * Stable ordering of every field that MEANS something, with `vendor_ref` + * omitted at both levels. Two documents that render identically hash + * identically even if their JSON key order differs or one of them has been + * round-tripped through a BI tool; a document whose meaning changed hashes + * differently. That is what makes `DashboardVersion.layout_hash` a usable + * identity for BI-P061 rollback. + */ +function canonicalize(doc: DashboardDocument): unknown { + return [ + doc.id, + doc.title, + doc.description ?? '', + doc.version, + [doc.layout.grid_columns, doc.layout.row_height_px], + doc.parameters + .map((p) => [p.name, p.type, p.required, p.default_value ?? '', [...(p.allowed_values ?? [])].sort()]) + .sort((a, b) => String(a[0]).localeCompare(String(b[0]))), + doc.filters + .map((f) => [f.filter_id, f.label, f.dimension, f.operator, f.default_values ?? [], f.locked ?? false]) + .sort((a, b) => String(a[0]).localeCompare(String(b[0]))), + doc.widgets + .map((w) => [ + w.widget_id, + w.title, + w.source.kind === 'metric' + ? [ + 'metric', + w.source.dataset, + w.source.metric, + w.source.dimensions ?? [], + w.source.grain ?? '', + w.source.limit ?? 0, + ] + : ['query', canonicalRequest(w.source.request)], + [ + w.visualization.type, + w.visualization.encoding.x ?? '', + w.visualization.encoding.y ?? [], + w.visualization.encoding.series ?? '', + w.visualization.encoding.value ?? '', + Object.keys(w.visualization.options ?? {}) + .sort() + .map((k) => [k, (w.visualization.options ?? {})[k]]), + ], + [w.layout.x, w.layout.y, w.layout.w, w.layout.h], + [...(w.filter_bindings ?? [])].sort(), + ]) + .sort((a, b) => String(a[0]).localeCompare(String(b[0]))), + [ + doc.refresh.mode, + doc.refresh.interval_seconds ?? 0, + doc.refresh.cron ?? '', + doc.refresh.max_staleness_seconds, + ], + [ + doc.permissions.visibility, + [...(doc.permissions.roles ?? [])].sort(), + doc.permissions.required_entitlement ?? '', + ], + ]; +} + +function canonicalRequest(r: AnalyticsQueryRequest): unknown { + return [ + r.dataset, + [...r.metrics], + [...r.dimensions], + r.filters.map((f) => [f.dimension, f.operator, f.values ?? []]), + r.timeRange + ? [r.timeRange.dimension ?? '', r.timeRange.from ?? '', r.timeRange.to ?? '', r.timeRange.grain ?? ''] + : [], + (r.orderBy ?? []).map((o) => [o.ref, o.direction]), + r.limit ?? 0, + r.format ?? 'json', + ]; +} + +/** sha256 of the canonical form. Fills `DashboardVersion.layout_hash`. */ +export function dashboardDocumentHash(doc: DashboardDocument): string { + return crypto + .createHash('sha256') + .update(JSON.stringify(canonicalize(doc))) + .digest('hex'); +} + +// ── reconstruction (the DoD) ─────────────────────────────────────────────── + +export interface PlannedWidget { + widget_id: string; + title: string; + /** Runnable, contract-valid, engine-neutral. This is the DoD in one field. */ + request: AnalyticsQueryRequest; + visualization: VisualizationSpec; + layout: WidgetLayout; +} + +export interface DashboardPlan { + dashboard_id: string; + title: string; + document_hash: string; + grid_columns: number; + widgets: PlannedWidget[]; + /** Parameter values actually bound, after defaults. Reported, not implied. */ + bound_parameters: Record; +} + +export interface PlanOptions { + /** Caller-supplied parameter values. Validated against the declarations. */ + parameters?: Record; + /** Filter ids a viewer switched off. `locked` filters cannot be switched off. */ + disabled_filter_ids?: string[]; +} + +function bindParameters(doc: DashboardDocument, supplied: Record): Record { + const bound: Record = {}; + const declared = new Set(doc.parameters.map((p) => p.name)); + for (const name of Object.keys(supplied)) { + if (!declared.has(name)) invalid(`parameter "${name}" is not declared by this dashboard`); + } + for (const p of doc.parameters) { + const value = supplied[p.name] ?? p.default_value; + if (value === undefined) { + if (p.required) invalid(`required parameter "${p.name}" was not supplied`); + continue; + } + if (p.allowed_values && !p.allowed_values.includes(value)) { + invalid(`parameter "${p.name}" value is not in its allowed_values`); + } + if (p.type === 'number' && !Number.isFinite(Number(value))) { + invalid(`parameter "${p.name}" is declared number but "${value}" is not numeric`); + } + if (p.type === 'date' && Number.isNaN(Date.parse(value))) { + invalid(`parameter "${p.name}" is declared date but "${value}" is not an ISO date`); + } + bound[p.name] = value; + } + return bound; +} + +function resolveFilterValues(f: DashboardFilter, bound: Record): string[] | undefined { + if (!f.default_values) return undefined; + return f.default_values.map((v) => { + const m = PARAM_REFERENCE.exec(v); + if (!m) return v; + const value = bound[m[1]]; + if (value === undefined) { + invalid(`filter "${f.filter_id}" references parameter "${m[1]}", which has no value`); + } + return value; + }); +} + +/** + * Turn a dashboard document into runnable analytics requests. + * + * This IS BI-P060's DoD. If this function returns a plan, the dashboard can be + * rendered by anything that can execute `AnalyticsQueryRequest` and draw the + * twelve visualization types — Superset not required, not consulted, and not + * present anywhere in the output. + */ +export function planDashboard(doc: DashboardDocument, options: PlanOptions = {}): DashboardPlan { + const bound = bindParameters(doc, options.parameters ?? {}); + const disabled = new Set(options.disabled_filter_ids ?? []); + for (const f of doc.filters) { + if (disabled.has(f.filter_id) && f.locked) { + invalid( + `filter "${f.filter_id}" is locked and cannot be disabled — locked filters are how a saved ` + + `dashboard pins its scope, and a viewer removing one silently widens the result set`, + ); + } + } + + const widgets = doc.widgets.map((w) => { + const applicable = doc.filters.filter( + (f) => + !disabled.has(f.filter_id) && + (w.filter_bindings === undefined || w.filter_bindings.length === 0 + ? true + : w.filter_bindings.includes(f.filter_id)), + ); + const dashboardFilters: AnalyticsFilter[] = applicable.map((f) => { + const values = resolveFilterValues(f, bound); + const af: AnalyticsFilter = { dimension: f.dimension, operator: f.operator }; + if (values !== undefined) af.values = values; + return af; + }); + + const base = + w.source.kind === 'metric' + ? { + dataset: w.source.dataset, + metrics: [w.source.metric], + dimensions: w.source.dimensions ?? [], + filters: [] as AnalyticsFilter[], + ...(w.source.grain ? { timeRange: { grain: w.source.grain } } : {}), + ...(w.source.limit ? { limit: w.source.limit } : {}), + } + : { + ...w.source.request, + timeRange: w.source.request.timeRange, + }; + + // Re-parse the merged request rather than trusting the merge: the filters + // we just added come from the document, and a document is untrusted input + // even after it has been stored. + const request = parseAnalyticsQueryRequest({ + ...base, + filters: [...(w.source.kind === 'query' ? w.source.request.filters : []), ...dashboardFilters], + }); + + return { + widget_id: w.widget_id, + title: w.title, + request, + visualization: w.visualization, + layout: w.layout, + }; + }); + + return { + dashboard_id: doc.id, + title: doc.title, + document_hash: dashboardDocumentHash(doc), + grid_columns: doc.layout.grid_columns, + widgets, + bound_parameters: bound, + }; +} diff --git a/src/analytics/dashboards/dashboard-release.ts b/src/analytics/dashboards/dashboard-release.ts new file mode 100644 index 0000000..e6bcbef --- /dev/null +++ b/src/analytics/dashboards/dashboard-release.ts @@ -0,0 +1,449 @@ +/** + * src/analytics/dashboards/dashboard-release.ts + * ============================================== + * BI-P061 — DASHBOARD VERSIONING. + * + * DRAFT -> PUBLISHED -> SUPERSEDED, and the DoD: "rollback to previous + * dashboard definition". + * + * Three decisions worth stating, because each one is a bug someone would + * otherwise write: + * + * 1. ROLLBACK MINTS A NEW VERSION. It does not resurrect the old release row + * and it does not edit the published one. "We went back to v3" is then a + * fact with a timestamp (v5 = rollback_of_version 3) instead of a hole in + * the history where v4 used to be. `verifyRollback()` proves the restore is + * faithful by comparing BI-P060 canonical hashes, so a rollback that + * silently restored a different document fails loudly. + * + * 2. THE DOCUMENT IS FROZEN, THE RELEASE MOVES. sql/008 makes + * DashboardVersion rows immutable at the database. The release ledger + * (sql/010) carries the state, and its trigger allows exactly two + * transitions and freezes every other column. Un-publishing is refused: + * it is how a rollback gets faked without leaving a trace. + * + * 3. THE APPLICATION CHECK IS NOT THE GUARANTEE. `publish()` checks for an + * existing published release, and that check is a read-then-write race. + * The partial unique index in sql/010 is what actually holds under + * concurrency; the realinfra test drops it, runs the race and shows two + * published releases for one dashboard, so nobody has to take that on + * faith. This module maps the resulting constraint violation onto a + * CONFLICT rather than a 500. + */ +import crypto from 'node:crypto'; +import { SiaxError } from '../../platform/errors'; +import { dashboardDocumentHash, parseDashboardDocument, type DashboardDocument } from './dashboard-document'; + +export const DASHBOARD_RELEASE_STATES = ['draft', 'published', 'superseded'] as const; +export type DashboardReleaseState = (typeof DASHBOARD_RELEASE_STATES)[number]; + +/** The only transitions that exist. Everything else is a refusal, not a no-op. */ +const LEGAL_TRANSITIONS: Readonly> = { + draft: ['published'], + published: ['superseded'], + superseded: [], +}; + +export function isLegalTransition(from: DashboardReleaseState, to: DashboardReleaseState): boolean { + return LEGAL_TRANSITIONS[from].includes(to); +} + +export interface DashboardRelease { + release_id: string; + dashboard_id: string; + dashboard_version_id: string; + tenant_id: string; + project_id: string; + version: number; + state: DashboardReleaseState; + /** BI-P060 canonical hash of the released document. */ + document_hash: string; + /** Set when this release restores an earlier version. */ + rollback_of_version?: number; + created_by?: string; + created_at: string; + published_at?: string; + superseded_at?: string; + superseded_by_release_id?: string; +} + +/** The immutable frozen document behind a release (sql/008's DashboardVersion). */ +export interface DashboardVersionRecord { + dashboard_version_id: string; + dashboard_id: string; + tenant_id: string; + project_id: string; + version: number; + layout_json: string; + layout_hash: string; + created_at: string; + created_by?: string; +} + +export interface TenantScope { + tenant_id: string; + project_id: string; +} + +/** + * The persistence port. Deliberately narrow, and every method is tenant-scoped + * — there is no signature that lets a caller forget the tenant. + */ +export interface DashboardReleaseStore { + /** Highest version number ever minted for this dashboard, or 0. */ + maxVersion(scope: TenantScope, dashboardId: string): Promise; + insertVersionAndRelease(version: DashboardVersionRecord, release: DashboardRelease): Promise; + getRelease(scope: TenantScope, releaseId: string): Promise; + getVersion(scope: TenantScope, dashboardVersionId: string): Promise; + findVersionByNumber( + scope: TenantScope, + dashboardId: string, + version: number, + ): Promise; + findByState( + scope: TenantScope, + dashboardId: string, + state: DashboardReleaseState, + ): Promise; + listReleases(scope: TenantScope, dashboardId: string): Promise; + /** + * Publish `releaseId` and supersede `supersedes` in ONE transaction. Both or + * neither: a supersede that lands without its publish leaves a dashboard + * with no live version, and a publish that lands without its supersede + * leaves two. + */ + publishAtomically(args: { + scope: TenantScope; + releaseId: string; + publishedAt: string; + supersedes?: { release_id: string; superseded_at: string }; + }): Promise; +} + +function conflict(detail: string): never { + throw new SiaxError('CONFLICT', '0pt.analytics.dashboard', detail); +} + +function notFound(detail: string): never { + throw new SiaxError('NOT_FOUND', '0pt.analytics.dashboard', detail); +} + +function assertScope(scope: TenantScope): void { + if (!scope?.tenant_id?.trim() || !scope?.project_id?.trim()) { + throw new SiaxError( + 'UNAUTHENTICATED', + '0pt.analytics.dashboard', + 'dashboard releases require a verified tenant_id + project_id', + ); + } +} + +export interface DraftResult { + release: DashboardRelease; + version: DashboardVersionRecord; +} + +/** + * The lifecycle. Stateless apart from its store and its clock; the clock is + * injectable so a test can pin timestamps instead of sleeping. + */ +export class DashboardReleaseService { + constructor( + private readonly store: DashboardReleaseStore, + private readonly now: () => Date = () => new Date(), + private readonly newId: () => string = () => crypto.randomUUID(), + ) {} + + /** + * Freeze a document as the next version and open a DRAFT release for it. + * The document is re-parsed here even if the caller already parsed it: what + * gets frozen must be what the model accepts, not what a caller assembled. + */ + async draft(args: { + scope: TenantScope; + dashboardId: string; + document: unknown; + createdBy?: string; + rollbackOfVersion?: number; + }): Promise { + assertScope(args.scope); + const document = parseDashboardDocument(args.document); + const existingDraft = await this.store.findByState(args.scope, args.dashboardId, 'draft'); + if (existingDraft) { + conflict( + `dashboard "${args.dashboardId}" already has an open draft (version ${existingDraft.version}); ` + + `publish or abandon it before opening another — two drafts race to publish and the loser's ` + + `edits vanish`, + ); + } + const next = (await this.store.maxVersion(args.scope, args.dashboardId)) + 1; + const at = this.now().toISOString(); + const hash = dashboardDocumentHash(document); + + const version: DashboardVersionRecord = { + dashboard_version_id: this.newId(), + dashboard_id: args.dashboardId, + tenant_id: args.scope.tenant_id, + project_id: args.scope.project_id, + version: next, + layout_json: JSON.stringify(document), + layout_hash: hash, + created_at: at, + ...(args.createdBy ? { created_by: args.createdBy } : {}), + }; + const release: DashboardRelease = { + release_id: this.newId(), + dashboard_id: args.dashboardId, + dashboard_version_id: version.dashboard_version_id, + tenant_id: args.scope.tenant_id, + project_id: args.scope.project_id, + version: next, + state: 'draft', + document_hash: hash, + created_at: at, + ...(args.createdBy ? { created_by: args.createdBy } : {}), + ...(args.rollbackOfVersion ? { rollback_of_version: args.rollbackOfVersion } : {}), + }; + await this.store.insertVersionAndRelease(version, release); + return { release, version }; + } + + /** + * DRAFT -> PUBLISHED, superseding whatever was live, in one transaction. + * The pre-check exists to return a friendly CONFLICT; the guarantee is the + * partial unique index (sql/010). + */ + async publish(args: { + scope: TenantScope; + releaseId: string; + }): Promise<{ published: DashboardRelease; superseded?: DashboardRelease }> { + assertScope(args.scope); + const release = await this.store.getRelease(args.scope, args.releaseId); + if (!release) notFound(`dashboard release "${args.releaseId}" does not exist in this tenant`); + if (release.state !== 'draft') { + conflict( + `release "${args.releaseId}" is ${release.state}, not draft — ${release.state} -> published is ` + + `not a legal transition (BI-P061)`, + ); + } + const live = await this.store.findByState(args.scope, release.dashboard_id, 'published'); + const at = this.now().toISOString(); + await this.store.publishAtomically({ + scope: args.scope, + releaseId: release.release_id, + publishedAt: at, + ...(live ? { supersedes: { release_id: live.release_id, superseded_at: at } } : {}), + }); + + return { + published: { ...release, state: 'published', published_at: at }, + ...(live + ? { + superseded: { + ...live, + state: 'superseded', + superseded_at: at, + superseded_by_release_id: release.release_id, + }, + } + : {}), + }; + } + + /** The release currently serving, or undefined if the dashboard has never been published. */ + async live(scope: TenantScope, dashboardId: string): Promise { + assertScope(scope); + return this.store.findByState(scope, dashboardId, 'published'); + } + + /** The frozen document behind a release, re-parsed on the way out. */ + async documentOf(scope: TenantScope, release: DashboardRelease): Promise { + assertScope(scope); + const version = await this.store.getVersion(scope, release.dashboard_version_id); + if (!version) notFound(`dashboard version "${release.dashboard_version_id}" is missing`); + const document = parseDashboardDocument(JSON.parse(version.layout_json)); + const hash = dashboardDocumentHash(document); + if (hash !== release.document_hash) { + // Storage returned something other than what was released. Refusing is + // the only safe answer: a dashboard that renders a document nobody + // approved is worse than a dashboard that will not open. + throw new SiaxError( + 'PERMANENT_FAILURE', + '0pt.analytics.dashboard', + `stored document for release "${release.release_id}" hashes to ${hash}, not the released ` + + `${release.document_hash} — refusing to serve an unapproved definition`, + ); + } + return document; + } + + /** + * BI-P061's DoD. Restores version `toVersion` by minting a NEW version with + * the SAME document and publishing it. Returns the new release plus the + * verification that the restore is faithful. + */ + async rollback(args: { + scope: TenantScope; + dashboardId: string; + toVersion: number; + createdBy?: string; + }): Promise<{ release: DashboardRelease; restored_from: number; document_hash: string }> { + assertScope(args.scope); + const target = await this.store.findVersionByNumber(args.scope, args.dashboardId, args.toVersion); + if (!target) { + notFound(`dashboard "${args.dashboardId}" has no version ${args.toVersion} in this tenant`); + } + const live = await this.store.findByState(args.scope, args.dashboardId, 'published'); + if (live && live.version === args.toVersion) { + conflict(`version ${args.toVersion} is already live; a rollback to the live version is a no-op`); + } + const document = parseDashboardDocument(JSON.parse(target.layout_json)); + const hash = dashboardDocumentHash(document); + if (hash !== target.layout_hash) { + throw new SiaxError( + 'PERMANENT_FAILURE', + '0pt.analytics.dashboard', + `version ${args.toVersion} hashes to ${hash} but was stored as ${target.layout_hash} — ` + + `refusing to roll back to a definition that has changed underneath us`, + ); + } + + const { release } = await this.draft({ + scope: args.scope, + dashboardId: args.dashboardId, + document, + ...(args.createdBy ? { createdBy: args.createdBy } : {}), + rollbackOfVersion: args.toVersion, + }); + const { published } = await this.publish({ scope: args.scope, releaseId: release.release_id }); + + if (published.document_hash !== target.layout_hash) { + throw new SiaxError( + 'PERMANENT_FAILURE', + '0pt.analytics.dashboard', + `rollback published ${published.document_hash}, not the requested ${target.layout_hash}`, + ); + } + return { release: published, restored_from: args.toVersion, document_hash: published.document_hash }; + } + + /** Full ledger, oldest first. A rollback is an entry here, never a deletion. */ + async history(scope: TenantScope, dashboardId: string): Promise { + assertScope(scope); + const rows = await this.store.listReleases(scope, dashboardId); + return [...rows].sort((a, b) => a.version - b.version); + } +} + +/** + * In-memory store. Used by the unit tests, and it enforces the same two + * uniqueness rules the SQL does — a store that is laxer than the database + * makes the unit tests pass on states production can never reach. + */ +export class InMemoryDashboardReleaseStore implements DashboardReleaseStore { + private readonly versions: DashboardVersionRecord[] = []; + private readonly releases: DashboardRelease[] = []; + + private scoped(scope: TenantScope, rows: T[]): T[] { + return rows.filter((r) => r.tenant_id === scope.tenant_id && r.project_id === scope.project_id); + } + + async maxVersion(scope: TenantScope, dashboardId: string): Promise { + return this.scoped(scope, this.versions) + .filter((v) => v.dashboard_id === dashboardId) + .reduce((max, v) => Math.max(max, v.version), 0); + } + + async insertVersionAndRelease(version: DashboardVersionRecord, release: DashboardRelease): Promise { + const scope = { tenant_id: release.tenant_id, project_id: release.project_id }; + if ( + release.state === 'draft' && + this.scoped(scope, this.releases).some( + (r) => r.dashboard_id === release.dashboard_id && r.state === 'draft', + ) + ) { + conflict('one open draft per dashboard'); + } + if (this.versions.some((v) => v.dashboard_id === version.dashboard_id && v.version === version.version)) { + conflict(`version ${version.version} already exists for "${version.dashboard_id}"`); + } + this.versions.push({ ...version }); + this.releases.push({ ...release }); + } + + async getRelease(scope: TenantScope, releaseId: string): Promise { + const r = this.scoped(scope, this.releases).find((x) => x.release_id === releaseId); + return r ? { ...r } : undefined; + } + + async getVersion(scope: TenantScope, id: string): Promise { + const v = this.scoped(scope, this.versions).find((x) => x.dashboard_version_id === id); + return v ? { ...v } : undefined; + } + + async findVersionByNumber( + scope: TenantScope, + dashboardId: string, + version: number, + ): Promise { + const v = this.scoped(scope, this.versions).find( + (x) => x.dashboard_id === dashboardId && x.version === version, + ); + return v ? { ...v } : undefined; + } + + async findByState( + scope: TenantScope, + dashboardId: string, + state: DashboardReleaseState, + ): Promise { + const r = this.scoped(scope, this.releases).find( + (x) => x.dashboard_id === dashboardId && x.state === state, + ); + return r ? { ...r } : undefined; + } + + async listReleases(scope: TenantScope, dashboardId: string): Promise { + return this.scoped(scope, this.releases) + .filter((r) => r.dashboard_id === dashboardId) + .map((r) => ({ ...r })); + } + + async publishAtomically(args: { + scope: TenantScope; + releaseId: string; + publishedAt: string; + supersedes?: { release_id: string; superseded_at: string }; + }): Promise { + const target = this.releases.find( + (r) => + r.release_id === args.releaseId && + r.tenant_id === args.scope.tenant_id && + r.project_id === args.scope.project_id, + ); + if (!target) notFound(`release "${args.releaseId}" not found`); + if (!isLegalTransition(target.state, 'published')) { + conflict(`illegal transition ${target.state} -> published`); + } + const alreadyLive = this.releases.find( + (r) => + r.dashboard_id === target.dashboard_id && + r.state === 'published' && + r.release_id !== args.supersedes?.release_id, + ); + if (alreadyLive) conflict('one published release per dashboard'); + + if (args.supersedes) { + const live = this.releases.find((r) => r.release_id === args.supersedes?.release_id); + if (!live) notFound(`release "${args.supersedes.release_id}" not found`); + if (!isLegalTransition(live.state, 'superseded')) { + conflict(`illegal transition ${live.state} -> superseded`); + } + live.state = 'superseded'; + live.superseded_at = args.supersedes.superseded_at; + live.superseded_by_release_id = target.release_id; + } + target.state = 'published'; + target.published_at = args.publishedAt; + } +} diff --git a/src/analytics/materialization/drift-detector.ts b/src/analytics/materialization/drift-detector.ts new file mode 100644 index 0000000..e5f348a --- /dev/null +++ b/src/analytics/materialization/drift-detector.ts @@ -0,0 +1,170 @@ +/** + * src/analytics/materialization/drift-detector.ts + * ================================================ + * BI-P105 — is the pre-aggregate still telling the truth? + * + * The comparison is bucket-by-bucket between the aggregate and a recomputation + * from the source over the same window. Three things make it a real detector + * rather than a ceremony: + * + * 1. A MISSING BUCKET IS DRIFT. The nastiest pre-aggregate failure is not a + * wrong value, it is an absent row: late-arriving data, a materialized view + * that was created after the backfill, a refresh that failed silently. Every + * value it DOES hold is correct, so a value-only comparison passes while the + * total is wrong. Set difference is checked first, before any arithmetic. + * + * 2. ZERO BUCKETS IS `insufficient_data`, NEVER `match`. A comparison over an + * empty window finds no discrepancies — the same false green BI-P025 found + * in a quality check over an empty table. The router treats + * `insufficient_data` as "do not serve", not as "fine". + * + * 3. RELATIVE AND ABSOLUTE TOLERANCE BOTH HAVE TO HOLD. A 0.1% tolerance on a + * revenue bucket of 40M is 40 000 kr of invisible error; an absolute-only + * tolerance flags every rounding difference on a large bucket. Requiring + * both means a discrepancy has to be small in kronor AND small in + * proportion. + */ + +import type { DriftTolerance, DriftVerdict, MaterializationStatus } from './materialization'; + +/** One aggregated row, keyed by its dimension tuple. */ +export interface AggregateBucket { + /** Stable key for the dimension tuple, e.g. "2026-09-01|SE". */ + key: string; + /** metric name -> value. */ + values: Record; +} + +export interface DriftDiscrepancy { + key: string; + metric: string; + source_value: number; + aggregate_value: number; + absolute_delta: number; + /** |delta| / |source|, or Infinity when the source value is 0 and the aggregate is not. */ + relative_delta: number; +} + +export interface DriftReport { + verdict: DriftVerdict; + checked_at: string; + buckets_compared: number; + /** In the source recomputation, absent from the aggregate. Always drift. */ + missing_in_aggregate: string[]; + /** In the aggregate, absent from the source. Always drift (deleted rows). */ + extra_in_aggregate: string[]; + discrepancies: DriftDiscrepancy[]; + worst?: DriftDiscrepancy; + tolerance: DriftTolerance; + /** + * What a caller would have been told if it had trusted the aggregate: the + * total the aggregate reports vs the total the source computes, per metric. + * This is the number that makes a drift report actionable — "0.4% off" means + * nothing until it is "1 284 500 kr reported, 1 271 900 kr true". + */ + totals: Record; +} + +const MAX_REPORTED_DISCREPANCIES = 50; + +function relative(sourceValue: number, delta: number): number { + if (sourceValue === 0) return delta === 0 ? 0 : Number.POSITIVE_INFINITY; + return Math.abs(delta / sourceValue); +} + +/** + * Compare a recomputation from source against the pre-aggregate. + * + * `fromSource` is the truth. Both sides must cover the same window; the caller + * is responsible for that, and `missing_in_aggregate` will catch it loudly if + * they do not. + */ +export function detectDrift(args: { + fromSource: AggregateBucket[]; + fromAggregate: AggregateBucket[]; + tolerance: DriftTolerance; + metrics: string[]; + now?: Date; +}): DriftReport { + const { fromSource, fromAggregate, tolerance, metrics } = args; + const checked_at = (args.now ?? new Date()).toISOString(); + const sourceByKey = new Map(fromSource.map((b) => [b.key, b])); + const aggByKey = new Map(fromAggregate.map((b) => [b.key, b])); + + const missing_in_aggregate = [...sourceByKey.keys()].filter((k) => !aggByKey.has(k)).sort(); + const extra_in_aggregate = [...aggByKey.keys()].filter((k) => !sourceByKey.has(k)).sort(); + + const discrepancies: DriftDiscrepancy[] = []; + let buckets_compared = 0; + const totals: DriftReport['totals'] = {}; + for (const m of metrics) totals[m] = { source: 0, aggregate: 0, delta: 0 }; + + for (const [key, sourceBucket] of sourceByKey) { + for (const m of metrics) totals[m].source += sourceBucket.values[m] ?? 0; + const aggBucket = aggByKey.get(key); + if (!aggBucket) continue; + buckets_compared += 1; + for (const m of metrics) { + const sv = sourceBucket.values[m] ?? 0; + const av = aggBucket.values[m] ?? 0; + const delta = av - sv; + const rel = relative(sv, delta); + if (Math.abs(delta) > tolerance.absolute || rel > tolerance.relative) { + discrepancies.push({ + key, + metric: m, + source_value: sv, + aggregate_value: av, + absolute_delta: delta, + relative_delta: rel, + }); + } + } + } + for (const bucket of aggByKey.values()) { + for (const m of metrics) totals[m].aggregate += bucket.values[m] ?? 0; + } + for (const m of metrics) totals[m].delta = totals[m].aggregate - totals[m].source; + + discrepancies.sort((a, b) => Math.abs(b.absolute_delta) - Math.abs(a.absolute_delta)); + + // A comparison that compared nothing has not proved anything. This is the + // BI-P025 false green, restated for pre-aggregates. + const verdict: DriftVerdict = + buckets_compared === 0 && missing_in_aggregate.length === 0 && extra_in_aggregate.length === 0 + ? 'insufficient_data' + : discrepancies.length > 0 || missing_in_aggregate.length > 0 || extra_in_aggregate.length > 0 + ? 'drifted' + : 'match'; + + const report: DriftReport = { + verdict, + checked_at, + buckets_compared, + missing_in_aggregate, + extra_in_aggregate, + discrepancies: discrepancies.slice(0, MAX_REPORTED_DISCREPANCIES), + tolerance, + totals, + }; + if (discrepancies.length > 0) report.worst = discrepancies[0]; + return report; +} + +/** + * Fold a drift report into the runtime status. A drifted aggregate is taken + * OUT of service here rather than merely annotated: leaving it `serving` with + * a bad verdict relies on every future reader remembering to check the + * verdict, and one that forgets serves the wrong number. + * + * `insufficient_data` also stops service. It means the check could not prove + * anything, and an unproven aggregate is not a current one. + */ +export function applyDriftReport(status: MaterializationStatus, report: DriftReport): MaterializationStatus { + return { + ...status, + state: report.verdict === 'match' ? status.state : 'drifted', + last_drift_check_at: report.checked_at, + last_drift_verdict: report.verdict, + }; +} diff --git a/src/analytics/materialization/materialization.ts b/src/analytics/materialization/materialization.ts new file mode 100644 index 0000000..77b5f86 --- /dev/null +++ b/src/analytics/materialization/materialization.ts @@ -0,0 +1,309 @@ +/** + * src/analytics/materialization/materialization.ts + * ================================================= + * BI-P105 — MATERIALIZATION / PRE-AGGREGATION. + * + * A pre-aggregate is the one optimisation that fails SILENTLY. A slow query is + * a complaint; a pre-aggregate that has drifted from its source answers + * instantly, with confidence, and is wrong — no error, no timeout, nothing in + * a log. So this module is built around one rule: + * + * A MATERIALIZATION SERVES ONLY WHILE IT IS PROVABLY CURRENT AND PROVABLY + * UNDRIFTED. Every unknown routes to the source table. + * + * Six refusal reasons, each of which is a real way the aggregate becomes a + * confident liar: + * + * coverage_miss it does not carry the metric/dimension/grain + * asked for (a filter on a dimension the + * aggregate collapsed cannot be applied after + * the fact, and applying it anyway is how a + * pre-aggregate returns another slice's number) + * watermark_unknown nobody knows how current it is. BI-P026 already + * refuses to render UNKNOWN age as current; the + * same doctrine applies here. + * staleness_exceeded it is older than its own freshness contract + * window_beyond_watermark the question reaches past the newest data the + * aggregate contains — it would silently answer + * with a shorter period + * drift_never_verified it has never been compared to its source + * drift_verification_stale the last comparison is older than the interval + * the spec requires + * drift_detected the last comparison FAILED + * + * `drift_never_verified` is the one people leave out, and it is the one that + * matters most on day one: a brand-new materialization is exactly as unproven + * as a drifted one, and a router that defaults new aggregates to "serving" + * ships the bug on the day the feature launches. + * + * The DoD also asks for measured improvement and documented freshness + * semantics: scripts/bi-p105-materialization-drift.mjs measures both against a + * live ClickHouse and writes evidence/bi-p105/materialization-drift.json, + * including what happens WITHOUT this router (the drifted aggregate served). + */ +import { SiaxError } from '../../platform/errors'; +import type { AnalyticsQueryRequest, TimeGrain } from '../contract/query-request'; + +export const MATERIALIZATION_STRATEGIES = [ + 'clickhouse_materialized_view', + 'clickhouse_projection', + 'dbt_incremental', + 'cube_preaggregation', +] as const; +export type MaterializationStrategy = (typeof MATERIALIZATION_STRATEGIES)[number]; + +export const MATERIALIZATION_STATES = ['building', 'serving', 'drifted', 'disabled'] as const; +export type MaterializationState = (typeof MATERIALIZATION_STATES)[number]; + +/** + * Grains ordered coarsest-last. An aggregate at `day` can answer `day` and + * anything coarser by re-aggregating; it can NEVER answer `hour`, because the + * detail was thrown away when the rows were rolled up. + */ +const GRAIN_ORDER: readonly TimeGrain[] = ['hour', 'day', 'week', 'month', 'quarter', 'year']; + +/** + * `week` does not divide `month` and `month` does not divide `quarter` by row + * count, but both roll up cleanly from `day`. Grains that CANNOT be derived by + * re-aggregating a coarser one are listed here so the router refuses rather + * than quietly returning a mis-bucketed series. + */ +function canRollUp(from: TimeGrain, to: TimeGrain): boolean { + if (from === to) return true; + const fi = GRAIN_ORDER.indexOf(from); + const ti = GRAIN_ORDER.indexOf(to); + if (fi < 0 || ti < 0 || ti < fi) return false; + // Weekly buckets do not compose into calendar months/quarters/years. + if (from === 'week' && to !== 'week') return false; + return true; +} + +export interface DriftTolerance { + /** Absolute delta allowed per metric per bucket. */ + absolute: number; + /** Relative delta allowed (0.001 = 0.1%). Both must hold. */ + relative: number; +} + +/** What a materialization IS. Declarative; the engine adapters read it. */ +export interface MaterializationSpec { + materialization_id: string; + tenant_id: string; + project_id: string; + /** The dataset it pre-aggregates (BI-P003). */ + dataset: string; + strategy: MaterializationStrategy; + /** Physical target (ClickHouse table, dbt model, Cube pre-agg name). */ + target_ref: string; + metrics: string[]; + dimensions: string[]; + time_dimension?: string; + grain?: TimeGrain; + /** The freshness contract. Older than this and it stops serving. */ + max_staleness_seconds: number; + /** How often the aggregate must be re-proved against its source. */ + drift_check_interval_seconds: number; + tolerance: DriftTolerance; +} + +export const DRIFT_VERDICTS = ['match', 'drifted', 'insufficient_data'] as const; +export type DriftVerdict = (typeof DRIFT_VERDICTS)[number]; + +/** Runtime state. Separate from the spec: the spec is config, this is fact. */ +export interface MaterializationStatus { + materialization_id: string; + state: MaterializationState; + /** + * The newest source timestamp the aggregate represents. UNDEFINED means + * UNKNOWN, and unknown is never treated as current. + */ + data_watermark?: string; + last_refresh_at?: string; + last_drift_check_at?: string; + last_drift_verdict?: DriftVerdict; +} + +export const ROUTING_REFUSALS = [ + 'coverage_miss', + 'watermark_unknown', + 'staleness_exceeded', + 'window_beyond_watermark', + 'drift_never_verified', + 'drift_verification_stale', + 'drift_detected', + 'state_not_serving', +] as const; +export type RoutingRefusal = (typeof ROUTING_REFUSALS)[number]; + +export type RoutingDecision = + | { + source: 'materialization'; + materialization_id: string; + target_ref: string; + /** Reported so a caller can show it; never inferred, always measured. */ + staleness_seconds: number; + } + | { source: 'base_table'; refusal: RoutingRefusal; detail: string }; + +function refuse(refusal: RoutingRefusal, detail: string): RoutingDecision { + return { source: 'base_table', refusal, detail }; +} + +/** + * Staleness of an aggregate, in seconds, or `undefined` when it cannot be + * computed. Deliberately NOT defaulted to 0: "we could not tell" and "it is + * current" are different facts, and conflating them is the bug. + */ +export function materializationStaleness(status: MaterializationStatus, now: Date): number | undefined { + if (!status.data_watermark) return undefined; + const t = Date.parse(status.data_watermark); + if (Number.isNaN(t)) return undefined; + return Math.max(0, Math.floor((now.getTime() - t) / 1000)); +} + +/** + * Decide whether a request may be answered from the pre-aggregate. + * + * Fail-closed by construction: the function returns `materialization` only + * after every check has passed, and every unknown is a refusal with a named + * reason. Callers get the reason so it can be logged and counted (BI-P103 + * reads the counts). + */ +export function selectQuerySource(args: { + request: AnalyticsQueryRequest; + spec: MaterializationSpec; + status: MaterializationStatus; + now: Date; +}): RoutingDecision { + const { request, spec, status, now } = args; + + if (request.dataset !== spec.dataset) { + return refuse('coverage_miss', `materialization covers "${spec.dataset}", not "${request.dataset}"`); + } + if (status.state !== 'serving') { + return refuse('state_not_serving', `materialization state is "${status.state}"`); + } + + // ── coverage ──────────────────────────────────────────────────────────── + const metrics = new Set(spec.metrics); + for (const m of request.metrics) { + if (!metrics.has(m)) return refuse('coverage_miss', `metric "${m}" is not in the pre-aggregate`); + } + const dimensions = new Set(spec.dimensions); + for (const d of request.dimensions) { + if (!dimensions.has(d)) return refuse('coverage_miss', `dimension "${d}" is not in the pre-aggregate`); + } + for (const f of request.filters) { + if (!dimensions.has(f.dimension)) { + // The aggregate summed across this dimension. Filtering afterwards would + // return a number computed over rows the filter should have excluded. + return refuse( + 'coverage_miss', + `filter on "${f.dimension}" cannot be applied: the pre-aggregate has already summed across it`, + ); + } + } + const requestedGrain = request.timeRange?.grain; + if (requestedGrain) { + if (!spec.grain) return refuse('coverage_miss', 'the pre-aggregate has no time grain'); + if (!canRollUp(spec.grain, requestedGrain)) { + return refuse( + 'coverage_miss', + `a "${spec.grain}" pre-aggregate cannot answer at "${requestedGrain}" grain`, + ); + } + } + + // ── freshness ─────────────────────────────────────────────────────────── + const staleness = materializationStaleness(status, now); + if (staleness === undefined) { + return refuse( + 'watermark_unknown', + 'the pre-aggregate has no data watermark — unknown age is not current age', + ); + } + if (staleness > spec.max_staleness_seconds) { + return refuse( + 'staleness_exceeded', + `${staleness}s stale, contract allows ${spec.max_staleness_seconds}s`, + ); + } + const to = request.timeRange?.to; + if (to) { + const toMs = Date.parse(to); + const watermarkMs = Date.parse(status.data_watermark!); + if (!Number.isNaN(toMs) && toMs > watermarkMs) { + return refuse( + 'window_beyond_watermark', + `the question runs to ${to} but the pre-aggregate only contains data to ${status.data_watermark}`, + ); + } + } + + // ── drift ─────────────────────────────────────────────────────────────── + if (!status.last_drift_check_at || !status.last_drift_verdict) { + return refuse( + 'drift_never_verified', + 'the pre-aggregate has never been compared to its source; a new aggregate is as unproven as a drifted one', + ); + } + if (status.last_drift_verdict !== 'match') { + return refuse('drift_detected', `last drift check returned "${status.last_drift_verdict}"`); + } + const checkedAgo = Math.floor((now.getTime() - Date.parse(status.last_drift_check_at)) / 1000); + if (Number.isNaN(checkedAgo)) { + return refuse('drift_never_verified', 'last_drift_check_at is not a timestamp'); + } + if (checkedAgo > spec.drift_check_interval_seconds) { + return refuse( + 'drift_verification_stale', + `last verified ${checkedAgo}s ago, spec requires every ${spec.drift_check_interval_seconds}s`, + ); + } + + return { + source: 'materialization', + materialization_id: spec.materialization_id, + target_ref: spec.target_ref, + staleness_seconds: staleness, + }; +} + +/** + * The routing decision as a freshness envelope for the result (BI-P026 shape): + * a consumer must be able to see WHICH source answered and how old it was. + * A result that does not say is a result nobody can audit. + */ +export function describeRouting(decision: RoutingDecision): { + served_from: 'materialization' | 'base_table'; + materialization_id?: string; + staleness_seconds?: number; + refusal?: RoutingRefusal; + detail?: string; +} { + return decision.source === 'materialization' + ? { + served_from: 'materialization', + materialization_id: decision.materialization_id, + staleness_seconds: decision.staleness_seconds, + } + : { served_from: 'base_table', refusal: decision.refusal, detail: decision.detail }; +} + +export function assertSpec(spec: MaterializationSpec): void { + const invalid = (d: string): never => { + throw new SiaxError('INVALID_REQUEST', '0pt.analytics.materialization', d); + }; + if (!spec.materialization_id?.trim()) invalid('materialization_id is required'); + if (!spec.tenant_id?.trim() || !spec.project_id?.trim()) invalid('a materialization is tenant-scoped'); + if (spec.metrics.length === 0) invalid('a materialization with no metrics pre-aggregates nothing'); + if (spec.max_staleness_seconds <= 0) invalid('max_staleness_seconds must be positive'); + if (spec.drift_check_interval_seconds <= 0) { + invalid( + 'drift_check_interval_seconds must be positive — an aggregate that is never re-verified is one ' + + 'source mutation away from serving wrong numbers forever', + ); + } + if (spec.tolerance.absolute < 0 || spec.tolerance.relative < 0) invalid('tolerance must be non-negative'); + if (spec.grain && !GRAIN_ORDER.includes(spec.grain)) invalid('grain is not a time grain'); +} diff --git a/src/analytics/slo/bi-slo.ts b/src/analytics/slo/bi-slo.ts new file mode 100644 index 0000000..b4bad4b --- /dev/null +++ b/src/analytics/slo/bi-slo.ts @@ -0,0 +1,501 @@ +/** + * src/analytics/slo/bi-slo.ts + * ============================ + * BI-P103 — BI SLO. DoD: "measured baselines exist". + * + * An SLO you cannot measure from real telemetry is a wish, so every objective + * here declares WHERE its number comes from, and the evaluator refuses to + * produce a number it does not have. Three ways that refusal shows up, each of + * which is a lie the usual implementation tells: + * + * insufficient_data nothing (or too little) was observed in the window. + * An availability objective evaluated over zero events + * is NOT 100% attainment — it is silence. This is the + * single most common false green in SLO reporting, and + * it is the same shape BI-P025 found in a quality + * check over an empty table. + * not_measurable_here the objective is real but these events cannot + * measure it. Dashboard data FRESHNESS is a property + * of the dataset (BI-P026 / BI-P105 watermarks), not + * of a usage event, and REPORT DELIVERY is N0TIFY's + * fact — aud0's seven types carry no delivery event. + * Saying so beats fabricating a proxy. + * low_confidence a p95 or p99 computed from a handful of samples. The + * number is reported WITH its sample size and marked, + * because a p99 over 12 requests is the 12th-slowest + * request wearing a percentile's clothes. + * + * The measurement path is aud0's seven BI usage event types (BI-P076, merged + * in aud0#34), produced by src/analytics/slo/usage-events.ts. There is + * deliberately no second telemetry path: two measurement systems disagree, and + * the disagreement is discovered during an incident. + */ +import type { Aud0BiEvent } from './usage-events'; + +export const BI_WORKLOAD_CLASSES = ['interactive_dashboard', 'exploration', 'report'] as const; +export type BiWorkloadClass = (typeof BI_WORKLOAD_CLASSES)[number]; + +export const SLO_OBJECTIVE_KINDS = [ + 'availability', + 'latency_p95_first', + 'latency_p95_cached', + 'latency_p95', + 'latency_p99', + 'error_rate', + 'freshness', + 'max_result_rows', + 'timeout', + 'completion_sla', + 'delivery_sla', +] as const; +export type SloObjectiveKind = (typeof SLO_OBJECTIVE_KINDS)[number]; + +/** Where the number comes from. `none` is an admission, not a placeholder. */ +export const MEASUREMENT_SOURCES = [ + 'aud0_bi_usage_events', + 'analytics_freshness_watermark', + 'n0tify_delivery_receipt', + 'engine_configuration', +] as const; +export type MeasurementSource = (typeof MEASUREMENT_SOURCES)[number]; + +export interface SloObjective { + objective_id: string; + workload: BiWorkloadClass; + kind: SloObjectiveKind; + /** Target value; unit is fixed by kind (ms, ratio, rows, seconds). */ + target: number; + unit: 'ms' | 'ratio' | 'rows' | 'seconds'; + /** Higher-is-better (availability) vs lower-is-better (latency). */ + direction: 'at_least' | 'at_most'; + measured_from: MeasurementSource; + /** Fewer samples than this and the verdict carries low_confidence. */ + min_samples: number; + rationale: string; +} + +/** + * The catalogue, field-for-field with what BI-P103 lists per workload class. + * Targets are STARTING POINTS chosen to be honest about what has been + * measured: they are the thresholds the evaluator compares against, and + * evidence/bi-p103 records the first baselines rather than pretending the + * targets were derived from a year of production traffic. + */ +export const BI_SLO_CATALOGUE: readonly SloObjective[] = [ + // ── interactive dashboard ──────────────────────────────────────────────── + { + objective_id: 'dash.availability', + workload: 'interactive_dashboard', + kind: 'availability', + target: 0.995, + unit: 'ratio', + direction: 'at_least', + measured_from: 'aud0_bi_usage_events', + min_samples: 100, + rationale: 'completed / (completed + failed) for dashboard-surface queries.', + }, + { + objective_id: 'dash.p95_first_query', + workload: 'interactive_dashboard', + kind: 'latency_p95_first', + target: 3000, + unit: 'ms', + direction: 'at_most', + measured_from: 'aud0_bi_usage_events', + min_samples: 50, + rationale: 'cache_hit=false: what a user waits for when they open the dashboard cold.', + }, + { + objective_id: 'dash.p95_cached', + workload: 'interactive_dashboard', + kind: 'latency_p95_cached', + target: 400, + unit: 'ms', + direction: 'at_most', + measured_from: 'aud0_bi_usage_events', + min_samples: 50, + rationale: + 'cache_hit=true: BI-P020 cache hits, measured separately so a high hit rate cannot hide a slow cold path.', + }, + { + objective_id: 'dash.error_rate', + workload: 'interactive_dashboard', + kind: 'error_rate', + target: 0.005, + unit: 'ratio', + direction: 'at_most', + measured_from: 'aud0_bi_usage_events', + min_samples: 100, + rationale: 'failed / terminal. Cancellations count as failures here: the user did not get an answer.', + }, + { + objective_id: 'dash.freshness', + workload: 'interactive_dashboard', + kind: 'freshness', + target: 900, + unit: 'seconds', + direction: 'at_most', + measured_from: 'analytics_freshness_watermark', + min_samples: 1, + rationale: + 'Data age is a property of the DATASET (BI-P026 freshness, BI-P105 watermark), not of a usage ' + + 'event. Usage events cannot measure it and this evaluator does not pretend to.', + }, + // ── exploration ────────────────────────────────────────────────────────── + { + objective_id: 'explore.p95', + workload: 'exploration', + kind: 'latency_p95', + target: 10_000, + unit: 'ms', + direction: 'at_most', + measured_from: 'aud0_bi_usage_events', + min_samples: 50, + rationale: 'explore-surface queries; an analyst tolerates more than a dashboard viewer.', + }, + { + objective_id: 'explore.p99', + workload: 'exploration', + kind: 'latency_p99', + target: 60_000, + unit: 'ms', + direction: 'at_most', + measured_from: 'aud0_bi_usage_events', + min_samples: 200, + rationale: 'p99 needs a genuinely large sample; below min_samples it is reported as low confidence.', + }, + { + objective_id: 'explore.max_result', + workload: 'exploration', + kind: 'max_result_rows', + target: 1_000_000, + unit: 'rows', + direction: 'at_most', + measured_from: 'aud0_bi_usage_events', + min_samples: 1, + rationale: 'BI-P012 truncates at the engine ceiling; this watches what actually came back.', + }, + { + objective_id: 'explore.timeout', + workload: 'exploration', + kind: 'timeout', + target: 300, + unit: 'seconds', + direction: 'at_most', + measured_from: 'engine_configuration', + min_samples: 1, + rationale: 'The configured ceiling (BI-P012 settings profile), not an observation.', + }, + // ── reports ────────────────────────────────────────────────────────────── + { + objective_id: 'report.completion_sla', + workload: 'report', + kind: 'completion_sla', + target: 900_000, + unit: 'ms', + direction: 'at_most', + measured_from: 'aud0_bi_usage_events', + min_samples: 20, + rationale: 'aud0.report_generated duration_ms: submitted to produced.', + }, + { + objective_id: 'report.delivery_sla', + workload: 'report', + kind: 'delivery_sla', + target: 300_000, + unit: 'ms', + direction: 'at_most', + measured_from: 'n0tify_delivery_receipt', + min_samples: 20, + rationale: + "Produced is not delivered. aud0 has no delivery event; the receipt is N0TIFY's, so this " + + 'objective is declared and reported unmeasured rather than proxied by completion.', + }, +]; + +export const SLO_STATUSES = ['met', 'breached', 'insufficient_data', 'not_measurable_here'] as const; +export type SloStatus = (typeof SLO_STATUSES)[number]; + +export interface SloResult { + objective_id: string; + workload: BiWorkloadClass; + kind: SloObjectiveKind; + status: SloStatus; + target: number; + unit: SloObjective['unit']; + /** The measured value, or undefined when nothing could be measured. */ + observed?: number; + sample_size: number; + /** True when a percentile was computed from fewer samples than min_samples. */ + low_confidence?: boolean; + detail: string; +} + +export interface SloReport { + window_from: string; + window_to: string; + results: SloResult[]; + /** Queries that started and never reached a terminal event in the window. */ + unresolved_queries: number; + /** True only when EVERY objective produced a real number. */ + fully_measured: boolean; +} + +interface QueryFact { + query_id: string; + surface?: string; + started_at: number; + outcome?: 'completed' | 'failed'; + duration_ms?: number; + row_count?: number; + cache_hit?: boolean; +} + +function num(v: unknown): number | undefined { + return typeof v === 'number' && Number.isFinite(v) ? v : undefined; +} + +function str(v: unknown): string | undefined { + return typeof v === 'string' ? v : undefined; +} + +/** + * Nearest-rank percentile. No interpolation: with small samples an + * interpolated p99 invents a value that no request ever experienced. + */ +export function percentile(sorted: number[], q: number): number | undefined { + if (sorted.length === 0) return undefined; + const rank = Math.ceil(q * sorted.length); + return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))]; +} + +/** Fold the seven event types into per-query facts. */ +function foldQueries(events: Aud0BiEvent[]): Map { + const facts = new Map(); + for (const e of events) { + const qid = str(e.payload.query_id); + if (!qid) continue; + const at = Date.parse(e.occurred_at); + const existing = facts.get(qid) ?? { query_id: qid, started_at: Number.isNaN(at) ? 0 : at }; + if (e.payload.surface !== undefined) existing.surface = str(e.payload.surface); + if (e.event_type === 'aud0.analytics_query_started' && !Number.isNaN(at)) existing.started_at = at; + if (e.event_type === 'aud0.analytics_query_completed') { + existing.outcome = 'completed'; + existing.duration_ms = num(e.payload.duration_ms); + existing.row_count = num(e.payload.row_count); + existing.cache_hit = e.payload.cache_hit === true; + } + if (e.event_type === 'aud0.analytics_query_failed') { + existing.outcome = 'failed'; + existing.duration_ms = num(e.payload.duration_ms); + } + facts.set(qid, existing); + } + return facts; +} + +function result( + o: SloObjective, + status: SloStatus, + sample_size: number, + detail: string, + observed?: number, + low_confidence?: boolean, +): SloResult { + const r: SloResult = { + objective_id: o.objective_id, + workload: o.workload, + kind: o.kind, + status, + target: o.target, + unit: o.unit, + sample_size, + detail, + }; + if (observed !== undefined) r.observed = observed; + if (low_confidence) r.low_confidence = true; + return r; +} + +function compare(o: SloObjective, observed: number): SloStatus { + return o.direction === 'at_least' + ? observed >= o.target + ? 'met' + : 'breached' + : observed <= o.target + ? 'met' + : 'breached'; +} + +/** + * Evaluate the catalogue over a window of AUD0 BI usage events. + * + * Events are DATA: they arrive from aud0's read model, they were written by + * other services, and nothing in them is treated as an instruction or as a + * schema. Unknown event types and unknown payload keys are ignored, never + * spread. + */ +export function evaluateBiSlos(args: { + events: Aud0BiEvent[]; + from: Date; + to: Date; + objectives?: readonly SloObjective[]; +}): SloReport { + const objectives = args.objectives ?? BI_SLO_CATALOGUE; + const fromMs = args.from.getTime(); + const toMs = args.to.getTime(); + const inWindow = args.events.filter((e) => { + const t = Date.parse(e.occurred_at); + return !Number.isNaN(t) && t >= fromMs && t <= toMs; + }); + + const queries = [...foldQueries(inWindow).values()]; + const dashboardQueries = queries.filter((q) => q.surface === 'dashboard'); + const exploreQueries = queries.filter((q) => q.surface === 'explore'); + const terminal = (qs: QueryFact[]) => qs.filter((q) => q.outcome !== undefined); + const durations = (qs: QueryFact[]) => + qs + .map((q) => q.duration_ms) + .filter((d): d is number => d !== undefined) + .sort((a, b) => a - b); + + const reportDurations = inWindow + .filter((e) => e.event_type === 'aud0.report_generated') + .map((e) => num(e.payload.duration_ms)) + .filter((d): d is number => d !== undefined) + .sort((a, b) => a - b); + + const results = objectives.map((o) => { + if (o.measured_from !== 'aud0_bi_usage_events') { + return result( + o, + 'not_measurable_here', + 0, + `measured from ${o.measured_from}; BI usage events cannot answer it — ${o.rationale}`, + ); + } + + switch (o.kind) { + case 'availability': + case 'error_rate': { + const t = terminal(dashboardQueries); + if (t.length === 0) { + return result(o, 'insufficient_data', 0, 'no terminal dashboard query events in the window'); + } + const failed = t.filter((q) => q.outcome === 'failed').length; + const observed = o.kind === 'availability' ? (t.length - failed) / t.length : failed / t.length; + const status = compare(o, observed); + return result( + o, + t.length < o.min_samples ? 'insufficient_data' : status, + t.length, + t.length < o.min_samples + ? `${t.length} terminal queries is below the ${o.min_samples} this objective requires; ` + + `the ratio ${observed.toFixed(4)} is reported but not judged` + : `${failed} failed of ${t.length} terminal queries`, + observed, + t.length < o.min_samples, + ); + } + case 'latency_p95_first': + case 'latency_p95_cached': { + const wanted = o.kind === 'latency_p95_cached'; + const sample = durations(dashboardQueries.filter((q) => (q.cache_hit === true) === wanted)); + const p95 = percentile(sample, 0.95); + if (p95 === undefined) { + return result( + o, + 'insufficient_data', + 0, + `no ${wanted ? 'cached' : 'uncached'} dashboard durations`, + ); + } + const low = sample.length < o.min_samples; + return result( + o, + low ? 'insufficient_data' : compare(o, p95), + sample.length, + low + ? `p95 over ${sample.length} samples (needs ${o.min_samples}) — reported, not judged` + : `p95 over ${sample.length} samples`, + p95, + low, + ); + } + case 'latency_p95': + case 'latency_p99': { + const sample = durations(exploreQueries); + const q = o.kind === 'latency_p99' ? 0.99 : 0.95; + const value = percentile(sample, q); + if (value === undefined) return result(o, 'insufficient_data', 0, 'no exploration durations'); + const low = sample.length < o.min_samples; + return result( + o, + low ? 'insufficient_data' : compare(o, value), + sample.length, + low + ? `p${q * 100} over ${sample.length} samples (needs ${o.min_samples}); with this few samples ` + + `it is the ${Math.ceil((1 - q) * sample.length) || 1}-slowest request, not a percentile` + : `p${q * 100} over ${sample.length} samples`, + value, + low, + ); + } + case 'max_result_rows': { + const rows = exploreQueries + .map((q) => q.row_count) + .filter((r): r is number => r !== undefined) + .sort((a, b) => b - a); + if (rows.length === 0) return result(o, 'insufficient_data', 0, 'no exploration row counts'); + return result(o, compare(o, rows[0]), rows.length, `largest result seen in the window`, rows[0]); + } + case 'completion_sla': { + const p95 = percentile(reportDurations, 0.95); + if (p95 === undefined) return result(o, 'insufficient_data', 0, 'no report_generated events'); + const low = reportDurations.length < o.min_samples; + return result( + o, + low ? 'insufficient_data' : compare(o, p95), + reportDurations.length, + low ? `p95 over ${reportDurations.length} reports (needs ${o.min_samples})` : 'p95 report duration', + p95, + low, + ); + } + default: + return result(o, 'not_measurable_here', 0, `no evaluator for kind "${o.kind}"`); + } + }); + + const unresolved = queries.filter((q) => q.outcome === undefined).length; + return { + window_from: args.from.toISOString(), + window_to: args.to.toISOString(), + results, + unresolved_queries: unresolved, + fully_measured: results.every((r) => r.status === 'met' || r.status === 'breached'), + }; +} + +/** + * One line an operator can read: how many objectives are actually being + * measured. "9 of 11 met" is meaningless if the other two were never + * measured, so the summary counts them separately and never rolls them in. + */ +export function summariseSloReport(report: SloReport): { + met: number; + breached: number; + insufficient_data: number; + not_measurable_here: number; + measured_fraction: number; +} { + const count = (s: SloStatus) => report.results.filter((r) => r.status === s).length; + const measured = count('met') + count('breached'); + return { + met: count('met'), + breached: count('breached'), + insufficient_data: count('insufficient_data'), + not_measurable_here: count('not_measurable_here'), + measured_fraction: report.results.length === 0 ? 0 : measured / report.results.length, + }; +} diff --git a/src/analytics/slo/usage-events.ts b/src/analytics/slo/usage-events.ts new file mode 100644 index 0000000..dec22ba --- /dev/null +++ b/src/analytics/slo/usage-events.ts @@ -0,0 +1,209 @@ +/** + * src/analytics/slo/usage-events.ts + * ================================== + * BI-P103's measurement path: 0pt emits the AUD0 BI usage events (aud0#34, + * BI-P076) that the SLO evaluator then reads. + * + * aud0 says it plainly in docs/BI_ANALYTICS_USAGE.md: "Zero real producers + * emit these seven types today." An SLO computed from a second, private + * telemetry path would be a number nobody else can audit, so 0pt becomes a + * producer instead of inventing one. + * + * THE PAYLOAD IS AN ALLOWLIST, BUILT FIELD BY NAMED FIELD. AUD0's ingest + * refuses result-bearing payloads with a 400 and its schemas are `.strict()`, + * so a leak here is a 400 rather than a breach — but a producer that has to be + * refused is a producer that is broken. `biUsageEventsForExecution` therefore + * never spreads an execution record; it names each field, and + * tests/analytics/bi-slo.test.ts feeds it a record carrying SQL text, an error + * message, a tenant id and a result sample, then asserts none of it appears in + * the output. + * + * WHAT IS DELIBERATELY NOT SENT: `compiled_query_hash` goes out as + * `query_hash` (aud0 requires a 64-hex digest and nothing else), the statement + * never leaves 0pt at all, `error_code` travels but `error_message` does not, + * and `row_count` travels but no row does. + */ + +/** The seven event types, named exactly as aud0 registers them. */ +export const AUD0_BI_EVENT_TYPES = [ + 'aud0.analytics_query_started', + 'aud0.analytics_query_completed', + 'aud0.analytics_query_failed', + 'aud0.dashboard_viewed', + 'aud0.report_generated', + 'aud0.dataset_exported', + 'aud0.metric_queried', +] as const; +export type Aud0BiEventType = (typeof AUD0_BI_EVENT_TYPES)[number]; + +/** aud0's closed surface list (BI_SURFACES). */ +export const BI_SURFACES = ['dashboard', 'report', 'explore', 'api', 'scheduled', 'notebook'] as const; +export type BiSurface = (typeof BI_SURFACES)[number]; + +export interface Aud0BiEvent { + event_type: Aud0BiEventType; + /** Bounded, allowlisted metadata. Never a result, a statement or a message. */ + payload: Record; + /** Occurrence time, used by the SLO window. */ + occurred_at: string; +} + +/** + * The 0pt-side view of one execution. Intentionally a SUPERSET of what may be + * emitted — it carries the fields 0pt genuinely has, including the ones that + * must never leave — so that the mapping below is a real filter rather than a + * rename. + */ +export interface ExecutionTelemetry { + query_id: string; + /** 64-hex; anything else is dropped rather than reshaped. */ + compiled_query_hash?: string; + /** NEVER emitted. Present here because 0pt has it. */ + compiled_sql?: string; + /** NEVER emitted. */ + error_message?: string; + error_code?: string; + /** NEVER emitted — a consumer must not be able to name a tenant, BI-P016. */ + tenant_id?: string; + dataset_id?: string; + engine?: string; + surface?: BiSurface; + state: 'succeeded' | 'failed' | 'cancelled' | 'running'; + started_at: string; + finished_at?: string; + duration_ms?: number; + row_count?: number; + bytes_scanned?: number; + cache_hit?: boolean; + filter_keys?: string[]; + /** NEVER emitted: filter VALUES are facts about a subject. */ + filter_values?: Record; + /** NEVER emitted. */ + result_sample?: unknown; + metric_name?: string; + dimension_keys?: string[]; + granularity?: string; + dashboard_id?: string; + widget_count?: number; +} + +const SHA256 = /^[0-9a-f]{64}$/; +const MAX_KEY_NAMES = 50; +const MAX_SHORT_TEXT = 120; + +function shortText(v: string | undefined): string | undefined { + if (!v) return undefined; + return v.slice(0, MAX_SHORT_TEXT); +} + +function keyNames(v: string[] | undefined): string[] | undefined { + if (!v || v.length === 0) return undefined; + return v.slice(0, MAX_KEY_NAMES).map((k) => k.slice(0, MAX_SHORT_TEXT)); +} + +function put( + payload: Record, + key: string, + value: string | number | boolean | string[] | undefined, +): void { + if (value === undefined) return; + payload[key] = value; +} + +/** + * Map one execution to its AUD0 usage events. + * + * A running execution emits only `started`: a query that has not finished is + * neither a success nor a failure, and the SLO evaluator counts it as + * unresolved rather than quietly as one or the other. + */ +export function biUsageEventsForExecution(t: ExecutionTelemetry): Aud0BiEvent[] { + const base: Record = {}; + put(base, 'query_id', t.query_id); + if (t.compiled_query_hash && SHA256.test(t.compiled_query_hash)) { + put(base, 'query_hash', t.compiled_query_hash); + } + put(base, 'dataset_id', t.dataset_id); + put(base, 'engine', shortText(t.engine)); + put(base, 'surface', t.surface); + put(base, 'filter_keys', keyNames(t.filter_keys)); + + const events: Aud0BiEvent[] = [ + { event_type: 'aud0.analytics_query_started', payload: { ...base }, occurred_at: t.started_at }, + ]; + + if (t.state === 'succeeded') { + const payload = { ...base }; + put(payload, 'duration_ms', t.duration_ms); + put(payload, 'row_count', t.row_count); + put(payload, 'bytes_scanned', t.bytes_scanned); + put(payload, 'cache_hit', t.cache_hit); + events.push({ + event_type: 'aud0.analytics_query_completed', + payload, + occurred_at: t.finished_at ?? t.started_at, + }); + } else if (t.state === 'failed' || t.state === 'cancelled') { + const payload = { ...base }; + put(payload, 'duration_ms', t.duration_ms); + // A CODE. `error_message` is never mapped, at any length. + put( + payload, + 'error_code', + shortText(t.error_code ?? (t.state === 'cancelled' ? 'CANCELLED' : undefined)), + ); + events.push({ + event_type: 'aud0.analytics_query_failed', + payload, + occurred_at: t.finished_at ?? t.started_at, + }); + } + + if (t.metric_name) { + const payload: Record = {}; + put(payload, 'metric_name', shortText(t.metric_name)); + put(payload, 'dataset_id', t.dataset_id); + put(payload, 'surface', t.surface); + put(payload, 'engine', shortText(t.engine)); + put(payload, 'dimension_keys', keyNames(t.dimension_keys)); + put(payload, 'granularity', shortText(t.granularity)); + put(payload, 'row_count', t.row_count); + put(payload, 'duration_ms', t.duration_ms); + events.push({ + event_type: 'aud0.metric_queried', + payload, + occurred_at: t.finished_at ?? t.started_at, + }); + } + + if (t.dashboard_id) { + const payload: Record = {}; + put(payload, 'dashboard_id', t.dashboard_id); + put(payload, 'widget_count', t.widget_count); + put(payload, 'duration_ms', t.duration_ms); + put(payload, 'surface', t.surface ?? 'dashboard'); + events.push({ event_type: 'aud0.dashboard_viewed', payload, occurred_at: t.started_at }); + } + + return events; +} + +/** Every payload key this producer can ever emit. Pinned by test. */ +export const EMITTABLE_PAYLOAD_KEYS = [ + 'query_id', + 'query_hash', + 'dataset_id', + 'engine', + 'surface', + 'filter_keys', + 'duration_ms', + 'row_count', + 'bytes_scanned', + 'cache_hit', + 'error_code', + 'metric_name', + 'dimension_keys', + 'granularity', + 'dashboard_id', + 'widget_count', +] as const; diff --git a/tests/analytics/benchmark-harness.test.ts b/tests/analytics/benchmark-harness.test.ts new file mode 100644 index 0000000..c47751f --- /dev/null +++ b/tests/analytics/benchmark-harness.test.ts @@ -0,0 +1,194 @@ +/** + * tests/analytics/benchmark-harness.test.ts + * ========================================== + * BI-P104 — the harness's job is to stop a benchmark from lying. + * + * The tests are therefore attacks on the reporting, not on the timing: + * + * - two overlapping distributions whose MEDIANS differ 2x must come back + * `indistinguishable`, because that is the claim a noisy host manufactures; + * - the real 150x/130x pair from yesterday's budget-enforcement lane is fed + * in as data, and the harness must describe a RANGE rather than pick one; + * - a three-sample run must refuse to be compared at all; + * - every sentence `describeComparison` produces must carry the range, the + * sample size and the stability, so it cannot be quoted as a bare figure; + * - an engine that was not run must appear in the evidence with a reason. + */ +import { describe, expect, it } from 'vitest'; +import { + BENCH_ENGINES, + buildEvidence, + compareDistributions, + describeComparison, + measure, + MIN_SAMPLES, + summarizeSamples, +} from '@/src/analytics/bench/benchmark-harness'; + +describe('BI-P104 distribution summary', () => { + it('reports the shape, and never a single number', () => { + const d = summarizeSamples([10, 11, 12, 13, 14, 15, 16, 40]); + expect(d.n).toBe(8); + expect(d.min).toBe(10); + expect(d.max).toBe(40); + expect(d.p50).toBe(13); + expect(d.spread_ratio).toBe(4); + expect(d.cv).toBeGreaterThan(0); + // Every field a reader needs to judge the p50 is present next to it. + for (const key of [ + 'min', + 'p50', + 'p95', + 'p99', + 'max', + 'stdev', + 'cv', + 'iqr', + 'spread_ratio', + 'stability', + ]) { + expect(Object.keys(d)).toContain(key); + } + }); + + it('calls a tight run stable and a spread run unstable', () => { + expect(summarizeSamples([100, 101, 99, 100, 102, 98, 100]).stability).toBe('stable'); + expect(summarizeSamples([10, 200, 15, 400, 12, 350, 11]).stability).toBe('unstable'); + }); + + it('refuses to characterise fewer than MIN_SAMPLES', () => { + const d = summarizeSamples([10, 11, 12]); + expect(d.n).toBe(3); + expect(d.stability).toBe('insufficient_samples'); + expect(MIN_SAMPLES).toBeGreaterThan(3); + }); +}); + +describe('BI-P104 comparison — variance beats a flattering median', () => { + it('overlapping ranges are INDISTINGUISHABLE even when medians differ 2x', () => { + const baseline = summarizeSamples([50, 60, 200, 70, 300, 55, 65, 280]); + const candidate = summarizeSamples([25, 30, 190, 35, 260, 28, 33, 240]); + const c = compareDistributions(baseline, candidate); + + expect(c.p50_ratio).toBeGreaterThan(1.5); // the tempting headline + expect(c.ranges_overlap).toBe(true); + expect(c.verdict).toBe('indistinguishable'); // the honest verdict + expect(describeComparison(c)).toMatch(/INDISTINGUISHABLE/); + }); + + it('clean separation is reported as faster, WITH the range', () => { + const baseline = summarizeSamples([500, 520, 510, 530, 505, 515, 525]); + const candidate = summarizeSamples([50, 55, 52, 58, 51, 53, 54]); + const c = compareDistributions(baseline, candidate); + + expect(c.verdict).toBe('faster'); + const sentence = describeComparison(c, 'pre-aggregate'); + expect(sentence).toMatch(/FASTER/); + expect(sentence).toMatch(/x to /); // a range, not a number + expect(sentence).toMatch(/n=7\/7/); // sample sizes + expect(sentence).toMatch(/stability/); + }); + + it('the real 150x / 130x overshoot pair is described as a range, not a figure', () => { + // Two runs of the same measurement on the same machine, from the BI-P021 + // budget-enforcement lane. A harness that picks one of these is lying. + const overshoot = summarizeSamples([153, 131, 149, 128, 144, 137, 152, 126, 141]); + + // Worth being precise about: by the CV test this run is STABLE (0.07). The + // problem was never instability — it is that a stable run still spans + // 126x-153x, so any single figure quoted from it is ~20% wrong. The + // harness's answer is to keep the ends attached to the middle. + expect(overshoot.stability).toBe('stable'); + expect(overshoot.min).toBe(126); + expect(overshoot.p50).toBe(141); + expect(overshoot.max).toBe(153); + expect(overshoot.spread_ratio).toBeCloseTo(1.21, 2); + + const sentence = describeComparison( + compareDistributions(summarizeSamples([1, 1, 1, 1, 1, 1, 1, 1, 1]), overshoot), + 'budget overshoot factor', + ); + expect(sentence).toMatch(/x to /); // a range survives into the prose + expect(sentence).toMatch(/n=9\/9/); + }); + + it('a three-sample run cannot be compared at all', () => { + const c = compareDistributions(summarizeSamples([10, 11, 12]), summarizeSamples([1, 2, 3])); + expect(c.verdict).toBe('not_comparable'); + expect(describeComparison(c)).toMatch(/anecdote/); + }); + + it('slower is reported as slower — the harness has no favourite', () => { + const baseline = summarizeSamples([50, 51, 52, 53, 54, 55, 56]); + const candidate = summarizeSamples([500, 510, 520, 530, 540, 550, 560]); + expect(compareDistributions(baseline, candidate).verdict).toBe('slower'); + }); +}); + +describe('BI-P104 measurement', () => { + it('records warmups but excludes them from the statistics', async () => { + let call = 0; + let clock = 0; + const run = await measure({ + engine: 'duckdb', + dataset: 'small_saas', + query_label: 'group by day', + warmups: 2, + repetitions: 6, + // A cold first pair (1000ms) then a warm steady state (10ms). + now: () => { + const t = clock; + clock += call <= 2 ? 1000 : 10; + call += 1; + return t; + }, + run: async () => ({ rows: 42 }), + }); + + expect(run.warmup_ms).toHaveLength(2); + expect(run.samples_ms).toHaveLength(6); + expect(run.distribution.n).toBe(6); + expect(run.distribution.max).toBeLessThan(1000); // the cold pair is not in it + expect(run.rows_returned).toBe(42); + }); +}); + +describe('BI-P104 evidence', () => { + it('lists every engine BI-P104 names, and says why the unrun ones were not run', () => { + const evidence = buildEvidence({ + host_note: 'one idle host, warm cache', + engines: [ + { engine: 'clickhouse', available: true, version: '26.3' }, + { engine: 'duckdb', available: true, version: '1.5.5' }, + { engine: 'trino', available: false, reason: 'the running Trino belongs to another lane' }, + ], + runs: [ + { + engine: 'clickhouse', + dataset: 'small_saas', + query_label: 'q', + warmup_ms: [1], + samples_ms: [1, 2, 3, 4, 5], + distribution: summarizeSamples([1, 2, 3, 4, 5]), + }, + ], + }); + + expect(evidence.engines.map((e) => e.engine)).toEqual([...BENCH_ENGINES]); + const notRun = evidence.not_exercised.map((e) => e.engine); + expect(notRun).toContain('superset'); + expect(notRun).toContain('perspective'); + expect(notRun).toContain('duckdb'); // declared available but produced no run + for (const e of evidence.not_exercised) { + expect(e.reason, e.engine).toBeTruthy(); + } + expect(evidence.datasets.map((d) => d.id)).toEqual([ + 'small_saas', + 'medium_business', + 'vehiq_market', + 'large_event', + 'wide_table', + 'geospatial', + ]); + }); +}); diff --git a/tests/analytics/bi-slo.test.ts b/tests/analytics/bi-slo.test.ts new file mode 100644 index 0000000..1dc5df2 --- /dev/null +++ b/tests/analytics/bi-slo.test.ts @@ -0,0 +1,293 @@ +/** + * tests/analytics/bi-slo.test.ts + * =============================== + * BI-P103 — BI SLO, and the AUD0 usage events it is measured from. + * + * Two failure modes are attacked directly, because both look like a healthy + * dashboard: + * + * 1. AN SLO REPORT OVER SILENCE. Zero events must not read as 100% + * availability. Every objective in an empty window must come back + * `insufficient_data`, and `fully_measured` must be false. + * 2. A PRODUCER THAT LEAKS. The emitter is handed an execution record + * carrying SQL text, a database error message quoting a personnummer, + * filter VALUES, a tenant id and a result sample — everything AUD0's + * BI-P076 ingest exists to refuse — and none of it may appear in any + * emitted payload. The test greps the serialized events, and first + * proves the grep finds those strings in the INPUT so a clean result + * cannot be a broken detector. + */ +import { describe, expect, it } from 'vitest'; +import { + BI_SLO_CATALOGUE, + evaluateBiSlos, + percentile, + summariseSloReport, + type SloResult, +} from '@/src/analytics/slo/bi-slo'; +import { + biUsageEventsForExecution, + EMITTABLE_PAYLOAD_KEYS, + type Aud0BiEvent, + type ExecutionTelemetry, +} from '@/src/analytics/slo/usage-events'; + +const FROM = new Date('2026-09-07T00:00:00.000Z'); +const TO = new Date('2026-09-07T23:59:59.000Z'); + +function queryEvents(args: { + n: number; + surface: 'dashboard' | 'explore'; + outcome: 'completed' | 'failed' | 'running'; + durationMs: (i: number) => number; + cacheHit?: boolean; + idPrefix?: string; +}): Aud0BiEvent[] { + const out: Aud0BiEvent[] = []; + for (let i = 0; i < args.n; i += 1) { + const t: ExecutionTelemetry = { + query_id: `${args.idPrefix ?? args.surface}_${i}`, + surface: args.surface, + state: args.outcome === 'completed' ? 'succeeded' : args.outcome === 'failed' ? 'failed' : 'running', + started_at: new Date(FROM.getTime() + i * 1000).toISOString(), + finished_at: new Date(FROM.getTime() + i * 1000 + args.durationMs(i)).toISOString(), + duration_ms: args.durationMs(i), + ...(args.cacheHit !== undefined ? { cache_hit: args.cacheHit } : {}), + ...(args.outcome === 'failed' ? { error_code: 'TIMEOUT' } : {}), + }; + out.push(...biUsageEventsForExecution(t)); + } + return out; +} + +function byId(results: SloResult[], id: string): SloResult { + const r = results.find((x) => x.objective_id === id); + if (!r) throw new Error(`no result for ${id}`); + return r; +} + +describe('BI-P103 — an SLO over silence is not a met SLO', () => { + it('an empty window makes every measurable objective insufficient_data, never 100%', () => { + const report = evaluateBiSlos({ events: [], from: FROM, to: TO }); + expect(report.fully_measured).toBe(false); + for (const r of report.results) { + expect(['insufficient_data', 'not_measurable_here']).toContain(r.status); + expect(r.status).not.toBe('met'); + } + const summary = summariseSloReport(report); + expect(summary.met).toBe(0); + expect(summary.measured_fraction).toBe(0); + }); + + it('a started query that never finished is unresolved, not a success', () => { + const events = queryEvents({ n: 200, surface: 'dashboard', outcome: 'running', durationMs: () => 100 }); + const report = evaluateBiSlos({ events, from: FROM, to: TO }); + expect(report.unresolved_queries).toBe(200); + expect(byId(report.results, 'dash.availability').status).toBe('insufficient_data'); + expect(byId(report.results, 'dash.availability').observed).toBeUndefined(); + }); + + it('a percentile below min_samples is reported WITH its sample size and not judged', () => { + const events = queryEvents({ + n: 12, + surface: 'explore', + outcome: 'completed', + durationMs: (i) => 1000 + i * 10, + }); + const p99 = byId(evaluateBiSlos({ events, from: FROM, to: TO }).results, 'explore.p99'); + expect(p99.status).toBe('insufficient_data'); + expect(p99.low_confidence).toBe(true); + expect(p99.sample_size).toBe(12); + expect(p99.observed).toBeDefined(); // reported… + expect(p99.detail).toMatch(/not a percentile/); // …with the caveat attached + }); + + it('freshness and delivery are declared not_measurable_here rather than proxied', () => { + const report = evaluateBiSlos({ events: [], from: FROM, to: TO }); + const freshness = byId(report.results, 'dash.freshness'); + const delivery = byId(report.results, 'report.delivery_sla'); + expect(freshness.status).toBe('not_measurable_here'); + expect(freshness.detail).toMatch(/analytics_freshness_watermark/); + expect(delivery.status).toBe('not_measurable_here'); + expect(delivery.detail).toMatch(/n0tify_delivery_receipt/); + }); +}); + +describe('BI-P103 — real measurement from real events', () => { + it('computes availability, error rate and both dashboard latencies', () => { + const events = [ + ...queryEvents({ + n: 180, + surface: 'dashboard', + outcome: 'completed', + cacheHit: false, + durationMs: (i) => 500 + i * 10, + idPrefix: 'cold', + }), + ...queryEvents({ + n: 120, + surface: 'dashboard', + outcome: 'completed', + cacheHit: true, + durationMs: () => 120, + idPrefix: 'warm', + }), + ...queryEvents({ + n: 3, + surface: 'dashboard', + outcome: 'failed', + durationMs: () => 900, + idPrefix: 'bad', + }), + ]; + const { results } = evaluateBiSlos({ events, from: FROM, to: TO }); + + const availability = byId(results, 'dash.availability'); + expect(availability.sample_size).toBe(303); + expect(availability.observed).toBeCloseTo(300 / 303, 6); + expect(availability.status).toBe('breached'); // 0.9901 < 0.995 target + + const errorRate = byId(results, 'dash.error_rate'); + expect(errorRate.observed).toBeCloseTo(3 / 303, 6); + + const cold = byId(results, 'dash.p95_first_query'); + expect(cold.sample_size).toBe(183); // 180 cold + 3 failures, which had no cache hit + expect(cold.status).toBe('met'); + + const cached = byId(results, 'dash.p95_cached'); + expect(cached.sample_size).toBe(120); + expect(cached.observed).toBe(120); + expect(cached.status).toBe('met'); + }); + + it('a breach is reported as a breach', () => { + const events = queryEvents({ + n: 60, + surface: 'dashboard', + outcome: 'completed', + cacheHit: false, + durationMs: () => 9000, + }); + expect( + byId(evaluateBiSlos({ events, from: FROM, to: TO }).results, 'dash.p95_first_query'), + ).toMatchObject({ status: 'breached', observed: 9000 }); + }); + + it('events outside the window do not count', () => { + const events = queryEvents({ n: 200, surface: 'dashboard', outcome: 'completed', durationMs: () => 100 }); + const report = evaluateBiSlos({ + events, + from: new Date('2026-09-08T00:00:00.000Z'), + to: new Date('2026-09-08T23:59:59.000Z'), + }); + expect(byId(report.results, 'dash.availability').status).toBe('insufficient_data'); + }); + + it('percentile is nearest-rank — it never invents a value nobody experienced', () => { + const sample = [10, 20, 30, 40]; + expect(percentile(sample, 0.95)).toBe(40); + expect(percentile(sample, 0.5)).toBe(20); + expect(percentile([], 0.95)).toBeUndefined(); + }); + + it('the catalogue covers all three workload classes and every field BI-P103 names', () => { + const kinds = new Set(BI_SLO_CATALOGUE.map((o) => `${o.workload}:${o.kind}`)); + for (const expected of [ + 'interactive_dashboard:availability', + 'interactive_dashboard:latency_p95_first', + 'interactive_dashboard:latency_p95_cached', + 'interactive_dashboard:error_rate', + 'interactive_dashboard:freshness', + 'exploration:latency_p95', + 'exploration:latency_p99', + 'exploration:max_result_rows', + 'exploration:timeout', + 'report:completion_sla', + 'report:delivery_sla', + ]) { + expect(kinds.has(expected), expected).toBe(true); + } + }); +}); + +describe('BI-P103 — the producer cannot leak a result into AUD0', () => { + const hostile: ExecutionTelemetry = { + query_id: 'q_1', + compiled_query_hash: 'f'.repeat(64), + compiled_sql: "SELECT * FROM customers WHERE personnummer = '19800101-1234'", + error_message: 'duplicate key value violates unique constraint (personnummer)=(19800101-1234)', + error_code: 'CONFLICT', + tenant_id: 't_alpha', + dataset_id: 'ds_sales', + engine: 'clickhouse', + surface: 'dashboard', + state: 'failed', + started_at: '2026-09-07T10:00:00.000Z', + finished_at: '2026-09-07T10:00:02.000Z', + duration_ms: 2000, + filter_keys: ['region'], + filter_values: { region: ['Gotland'] }, + result_sample: [{ customer: 'Anna Andersson', revenue: 42 }], + metric_name: 'revenue', + dashboard_id: 'dash_sales', + widget_count: 4, + }; + + it('emits none of the forbidden material', () => { + const events = biUsageEventsForExecution(hostile); + const serialized = JSON.stringify(events); + + // The detector must be able to find these strings — they are in the input. + const input = JSON.stringify(hostile); + for (const secret of ['19800101-1234', 'Anna Andersson', 'Gotland', 't_alpha', 'SELECT *']) { + expect(input, `detector check for ${secret}`).toContain(secret); + } + for (const secret of [ + '19800101-1234', + 'Anna Andersson', + 'Gotland', + 't_alpha', + 'SELECT *', + 'duplicate key', + ]) { + expect(serialized, secret).not.toContain(secret); + } + }); + + it('every emitted key is on the allowlist AUD0 accepts', () => { + const events = biUsageEventsForExecution(hostile); + for (const e of events) { + for (const key of Object.keys(e.payload)) { + expect(EMITTABLE_PAYLOAD_KEYS, `${e.event_type}.${key}`).toContain(key); + } + } + }); + + it('a query_hash that is not a sha256 digest is dropped, not reshaped', () => { + const events = biUsageEventsForExecution({ ...hostile, compiled_query_hash: 'SELECT 1' }); + for (const e of events) expect(e.payload.query_hash).toBeUndefined(); + }); + + it('a running query emits only "started" — it is neither a success nor a failure', () => { + const events = biUsageEventsForExecution({ + ...hostile, + state: 'running', + metric_name: undefined, + dashboard_id: undefined, + }); + expect(events.map((e) => e.event_type)).toEqual(['aud0.analytics_query_started']); + }); + + it('a cancelled query is reported as failed with a CODE', () => { + const events = biUsageEventsForExecution({ + ...hostile, + state: 'cancelled', + error_code: undefined, + metric_name: undefined, + dashboard_id: undefined, + }); + const failed = events.find((e) => e.event_type === 'aud0.analytics_query_failed'); + expect(failed?.payload.error_code).toBe('CANCELLED'); + expect(failed?.payload.error_message).toBeUndefined(); + }); +}); diff --git a/tests/analytics/dashboard-model.test.ts b/tests/analytics/dashboard-model.test.ts new file mode 100644 index 0000000..7c93bcf --- /dev/null +++ b/tests/analytics/dashboard-model.test.ts @@ -0,0 +1,330 @@ +/** + * tests/analytics/dashboard-model.test.ts + * ======================================== + * BI-P060 — ANALYTICS DASHBOARD MODEL. + * + * The DoD is "a native SIAX dashboard can be reconstructed without Superset", + * so the central test takes a document that CAME from Superset (it still + * carries the slice ids in `vendor_ref`), plans it, and then greps the whole + * plan for any trace of the vendor. A plan that still needs Superset to be + * understood fails here. + * + * The counterfactual matters as much: `vendor-only-widget-is-refused` builds + * the document that CANNOT be reconstructed — a widget whose only content is a + * Superset id — and requires the model to refuse it. Without that test the + * vendor-free assertion would pass trivially for documents nobody could + * render. + */ +import { describe, expect, it } from 'vitest'; +import { SiaxError } from '@/src/platform/errors'; +import { + dashboardDocumentHash, + parseDashboardDocument, + planDashboard, + VISUALIZATION_TYPES, + type DashboardDocument, +} from '@/src/analytics/dashboards/dashboard-document'; + +/** A dashboard exported from Superset and rewritten into the canonical model. */ +function supersetImportedDocument(): unknown { + return { + id: 'dash_sales_overview', + title: 'Sales overview', + description: 'Migrated from Superset dashboard 41.', + version: 1, + layout: { grid_columns: 12, row_height_px: 40 }, + parameters: [ + { + name: 'region', + type: 'enum', + required: false, + default_value: 'SE', + allowed_values: ['SE', 'NO', 'DK'], + }, + { name: 'since', type: 'date', required: true }, + ], + filters: [ + { + filter_id: 'f_region', + label: 'Region', + dimension: 'region', + operator: 'eq', + default_values: ['{{region}}'], + }, + { + filter_id: 'f_channel', + label: 'Channel', + dimension: 'channel', + operator: 'in', + default_values: ['web'], + }, + { + filter_id: 'f_tenant_scope', + label: 'Business unit', + dimension: 'business_unit', + operator: 'eq', + default_values: ['bu_north'], + locked: true, + }, + ], + widgets: [ + { + widget_id: 'w_revenue_total', + title: 'Revenue', + source: { kind: 'metric', dataset: 'ds_sales', metric: 'revenue' }, + visualization: { type: 'big_number', encoding: { value: 'revenue' } }, + layout: { x: 0, y: 0, w: 3, h: 2 }, + vendor_ref: { system: 'superset', id: 'slice-8812' }, + }, + { + widget_id: 'w_revenue_by_day', + title: 'Revenue by day', + source: { + kind: 'metric', + dataset: 'ds_sales', + metric: 'revenue', + dimensions: ['order_date'], + grain: 'day', + }, + visualization: { + type: 'line', + encoding: { x: 'order_date', y: ['revenue'] }, + options: { stacked: false }, + }, + layout: { x: 3, y: 0, w: 9, h: 4 }, + filter_bindings: ['f_region', 'f_tenant_scope'], + vendor_ref: { system: 'superset', id: 'slice-8813' }, + }, + { + widget_id: 'w_top_products', + title: 'Top products', + source: { + kind: 'query', + request: { + dataset: 'ds_sales', + metrics: ['revenue', 'orders'], + dimensions: ['product'], + filters: [{ dimension: 'status', operator: 'eq', values: ['completed'] }], + orderBy: [{ ref: 'revenue', direction: 'desc' }], + limit: 20, + }, + }, + visualization: { type: 'table', encoding: { x: 'product', y: ['revenue', 'orders'] } }, + layout: { x: 0, y: 4, w: 12, h: 6 }, + }, + ], + refresh: { mode: 'interval', interval_seconds: 300, max_staleness_seconds: 900 }, + permissions: { + visibility: 'roles', + roles: ['analyst', 'sales_lead'], + required_entitlement: 'bi_dashboards', + }, + vendor_ref: { system: 'superset', id: 'dashboard-41' }, + }; +} + +describe('BI-P060 dashboard model — reconstruction without Superset', () => { + it('plans every widget into a runnable analytics request', () => { + const doc = parseDashboardDocument(supersetImportedDocument()); + const plan = planDashboard(doc, { parameters: { since: '2026-01-01', region: 'NO' } }); + + expect(plan.widgets).toHaveLength(3); + for (const w of plan.widgets) { + expect(w.request.dataset).toBe('ds_sales'); + expect(w.request.metrics.length).toBeGreaterThan(0); + expect(VISUALIZATION_TYPES).toContain(w.visualization.type); + } + // The parameter reference was BOUND, not left as a template string. + const byDay = plan.widgets.find((w) => w.widget_id === 'w_revenue_by_day'); + expect(byDay?.request.filters.find((f) => f.dimension === 'region')?.values).toEqual(['NO']); + expect(plan.bound_parameters).toEqual({ region: 'NO', since: '2026-01-01' }); + }); + + it('the plan contains no vendor identifier anywhere', () => { + const doc = parseDashboardDocument(supersetImportedDocument()); + const plan = planDashboard(doc, { parameters: { since: '2026-01-01' } }); + const serialized = JSON.stringify(plan).toLowerCase(); + + // The detector must be capable of finding one: the document DOES carry + // these ids, so a grep that finds nothing in the plan means the plan is + // clean, not that the grep is broken. + expect(JSON.stringify(doc).toLowerCase()).toContain('slice-8812'); + + for (const trace of ['superset', 'slice-8812', 'slice-8813', 'dashboard-41', 'viz_type', 'form_data']) { + expect(serialized).not.toContain(trace); + } + }); + + it('vendor-only widget is refused — it is the document that cannot be reconstructed', () => { + const raw = supersetImportedDocument() as { widgets: unknown[] }; + raw.widgets = [ + { + widget_id: 'w_mystery', + title: 'Whatever Superset had', + source: { kind: 'vendor', system: 'superset', id: 'slice-9001' }, + visualization: { type: 'line', encoding: {} }, + layout: { x: 0, y: 0, w: 4, h: 3 }, + }, + ]; + expect(() => parseDashboardDocument(raw)).toThrow(SiaxError); + expect(() => parseDashboardDocument(raw)).toThrow(/reconstruct/i); + }); + + it('vendor-native fields are refused at the top level and inside a widget', () => { + const top = { ...(supersetImportedDocument() as object), position_json: '{}' }; + expect(() => parseDashboardDocument(top)).toThrow(/vendor-native/); + + const raw = supersetImportedDocument() as { widgets: Record[] }; + raw.widgets[0].viz_type = 'echarts_timeseries'; + expect(() => parseDashboardDocument(raw)).toThrow(/vendor-native/); + }); + + it('a Superset viz_type value is not a SIAX visualization type', () => { + const raw = supersetImportedDocument() as { widgets: Record[] }; + raw.widgets[0].visualization = { type: 'echarts_timeseries', encoding: {} }; + expect(() => parseDashboardDocument(raw)).toThrow(/not a SIAX visualization type/); + }); +}); + +describe('BI-P060 dashboard model — no SQL surface', () => { + it('a widget cannot carry SQL, however it is spelled', () => { + for (const key of ['sql', 'query_text', 'statement', 'where', 'expression']) { + const raw = supersetImportedDocument() as { widgets: Record[] }; + raw.widgets[0][key] = 'SELECT 1'; + expect(() => parseDashboardDocument(raw), key).toThrow(SiaxError); + } + }); + + it('a query widget goes through the BI-P016 contract, so tenant selection is impossible', () => { + const raw = supersetImportedDocument() as { widgets: Record[] }; + raw.widgets[2].source = { + kind: 'query', + request: { + dataset: 'ds_sales', + metrics: ['revenue'], + dimensions: [], + filters: [], + tenant_id: 't_victim', + }, + }; + expect(() => parseDashboardDocument(raw)).toThrow(/does not accept SQL, catalogs, or tenant selection/); + }); + + it('the document itself cannot name a tenant', () => { + const raw = { ...(supersetImportedDocument() as object), tenant_id: 't_victim' }; + expect(() => parseDashboardDocument(raw)).toThrow(/dashboards do not carry SQL|unknown field/); + }); +}); + +describe('BI-P060 dashboard model — validation that would otherwise bite at render time', () => { + it('a widget binding an undeclared filter is refused', () => { + const raw = supersetImportedDocument() as { widgets: Record[] }; + raw.widgets[1].filter_bindings = ['f_nonexistent']; + expect(() => parseDashboardDocument(raw)).toThrow(/undeclared filter/); + }); + + it('a filter defaulting to an undeclared parameter is refused', () => { + const raw = supersetImportedDocument() as { filters: Record[] }; + raw.filters[0].default_values = ['{{not_declared}}']; + expect(() => parseDashboardDocument(raw)).toThrow(/undeclared parameter/); + }); + + it('a widget overflowing the grid is refused', () => { + const raw = supersetImportedDocument() as { widgets: Record[] }; + raw.widgets[1].layout = { x: 9, y: 0, w: 6, h: 3 }; + expect(() => parseDashboardDocument(raw)).toThrow(/overflows the 12-column grid/); + }); + + it('an interval refresh with a cron, and a manual refresh with a schedule, are both refused', () => { + const a = supersetImportedDocument() as Record; + a.refresh = { mode: 'interval', interval_seconds: 300, cron: '0 * * * *', max_staleness_seconds: 900 }; + expect(() => parseDashboardDocument(a)).toThrow(/meaningless/); + + const b = supersetImportedDocument() as Record; + b.refresh = { mode: 'manual', interval_seconds: 300, max_staleness_seconds: 900 }; + expect(() => parseDashboardDocument(b)).toThrow(/manual/); + }); + + it('visibility "roles" with no roles is refused rather than guessed', () => { + const raw = supersetImportedDocument() as Record; + raw.permissions = { visibility: 'roles', roles: [] }; + expect(() => parseDashboardDocument(raw)).toThrow(/deny all/); + }); + + it('a required parameter that was not supplied fails the plan, not the render', () => { + const doc = parseDashboardDocument(supersetImportedDocument()); + expect(() => planDashboard(doc, { parameters: {} })).toThrow(/required parameter "since"/); + }); + + it('a locked filter cannot be switched off by a viewer', () => { + const doc = parseDashboardDocument(supersetImportedDocument()); + expect(() => + planDashboard(doc, { parameters: { since: '2026-01-01' }, disabled_filter_ids: ['f_tenant_scope'] }), + ).toThrow(/locked/); + // An unlocked one can be, and disappears from the compiled request. + const plan = planDashboard(doc, { + parameters: { since: '2026-01-01' }, + disabled_filter_ids: ['f_channel'], + }); + const table = plan.widgets.find((w) => w.widget_id === 'w_top_products'); + expect(table?.request.filters.map((f) => f.dimension)).not.toContain('channel'); + }); + + it('an enum parameter without allowed_values is refused', () => { + const raw = supersetImportedDocument() as { parameters: Record[] }; + raw.parameters[0] = { name: 'region', type: 'enum', required: false, default_value: 'SE' }; + expect(() => parseDashboardDocument(raw)).toThrow(/unbounded enum/); + }); +}); + +/** Rebuild an object graph with every object's keys in reverse order. */ +function shuffleKeys(value: unknown): unknown { + if (Array.isArray(value)) return value.map(shuffleKeys); + if (value !== null && typeof value === 'object') { + const out: Record = {}; + for (const k of Object.keys(value as object).reverse()) { + out[k] = shuffleKeys((value as Record)[k]); + } + return out; + } + return value; +} + +describe('BI-P060 dashboard model — canonical hash', () => { + function parsed(): DashboardDocument { + return parseDashboardDocument(supersetImportedDocument()); + } + + it('is stable across key order and vendor re-export', () => { + const a = parsed(); + const b = parsed(); + b.vendor_ref = { system: 'superset', id: 'dashboard-999-after-migration' }; + b.widgets[0].vendor_ref = { system: 'metabase', id: 'card-3' }; + expect(dashboardDocumentHash(b)).toBe(dashboardDocumentHash(a)); + + // Key order in the source JSON must not change identity either. + expect(dashboardDocumentHash(parseDashboardDocument(shuffleKeys(supersetImportedDocument())))).toBe( + dashboardDocumentHash(a), + ); + }); + + it('changes when meaning changes', () => { + const base = dashboardDocumentHash(parsed()); + + const retitled = parsed(); + retitled.widgets[0].title = 'Net revenue'; + expect(dashboardDocumentHash(retitled)).not.toBe(base); + + const remetric = parsed(); + (remetric.widgets[0].source as { metric: string }).metric = 'revenue_net'; + expect(dashboardDocumentHash(remetric)).not.toBe(base); + + const moved = parsed(); + moved.widgets[1].layout = { x: 3, y: 1, w: 9, h: 4 }; + expect(dashboardDocumentHash(moved)).not.toBe(base); + + const refiltered = parsed(); + refiltered.filters[1].default_values = ['store']; + expect(dashboardDocumentHash(refiltered)).not.toBe(base); + }); +}); diff --git a/tests/analytics/dashboard-release-postgres.realinfra.test.ts b/tests/analytics/dashboard-release-postgres.realinfra.test.ts new file mode 100644 index 0000000..31ec1b4 --- /dev/null +++ b/tests/analytics/dashboard-release-postgres.realinfra.test.ts @@ -0,0 +1,346 @@ +/** + * tests/analytics/dashboard-release-postgres.realinfra.test.ts + * ============================================================= + * sql/010 (BI-P061) against a REAL Postgres. + * + * The interesting claim in sql/010 is not "there is a state machine" — it is + * "TWO PEOPLE CANNOT PUBLISH AT ONCE". That claim lives in a partial unique + * index, and an index nobody raced is an index nobody has tested. So the + * concurrency test runs BOTH directions, in two separate schemas so neither + * can contaminate the other: + * + * 1. WITHOUT the index (a schema where it is dropped): two sessions publish + * a release of the same dashboard concurrently, each having first read + * "nothing is live" — the exact interleaving an application-level check + * cannot see. The database ends up with TWO published releases. This is + * the counterfactual: it proves the read-then-write check in + * DashboardReleaseService.publish() is NOT the guarantee, and that this + * test can detect the failure it claims to prevent. + * 2. WITH the index: the same race, and the loser fails with + * unique_violation (23505). Exactly one published release remains. + * + * The transition trigger is attacked the same way: every illegal transition is + * attempted against the database, not against the TypeScript. + * + * Gated on ANALYTICS_TEST_DATABASE_URL, matching the convention in + * analytics-governance-postgres.realinfra.test.ts. With it set and Postgres + * down this FAILS rather than skipping. + * + * docker run -d --name pg -p 25441:5432 -e POSTGRES_PASSWORD=test \ + * -e POSTGRES_DB=opt postgres:16-alpine + * ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25441/opt \ + * pnpm vitest run tests/analytics/dashboard-release-postgres.realinfra.test.ts + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import postgres from 'postgres'; + +const TEST_DATABASE_URL = process.env.ANALYTICS_TEST_DATABASE_URL; +const run = TEST_DATABASE_URL ? describe : describe.skip; +const STAMP = Date.now().toString(36); +const SCHEMA = `bi_p061_${STAMP}`; +/** The counterfactual lives in its own schema: no cleanup, no contamination. */ +const SCHEMA_NOGUARD = `bi_p061_noguard_${STAMP}`; +const HASH_A = 'a'.repeat(64); +const HASH_B = 'b'.repeat(64); +const MIGRATIONS = ['sql/008_analytics_domain_model.sql', 'sql/010_analytics_dashboard_releases.sql']; + +let sql: ReturnType; + +function connect(schema: string, max = 1): ReturnType { + return postgres(TEST_DATABASE_URL!, { + max, + onnotice: () => undefined, + connection: { search_path: schema }, + }); +} + +async function refusalOf(fn: () => Promise): Promise { + try { + await fn(); + return undefined; + } catch (e) { + return String((e as Error).message); + } +} + +async function migrate(conn: ReturnType): Promise { + for (const file of MIGRATIONS) { + await conn.unsafe(fs.readFileSync(path.join(process.cwd(), file), 'utf8')); + } +} + +async function seedDashboard(conn: ReturnType, id: string): Promise { + await conn.unsafe(` + INSERT INTO opt_analytics_dashboard_definition + (dashboard_id, tenant_id, project_id, name, current_version, owner_id) + VALUES ('${id}', 't_alpha', 'p_main', '${id}', 1, 'u_1') + ON CONFLICT DO NOTHING`); +} + +async function seedVersion( + conn: ReturnType, + versionId: string, + dashboardId: string, + version: number, + hash = HASH_A, +): Promise { + await conn.unsafe(` + INSERT INTO opt_analytics_dashboard_version + (dashboard_version_id, dashboard_id, tenant_id, project_id, version, layout_json, layout_hash) + VALUES ('${versionId}', '${dashboardId}', 't_alpha', 'p_main', ${version}, '{}', '${hash}')`); +} + +async function seedDraftRelease( + conn: ReturnType, + releaseId: string, + versionId: string, + dashboardId: string, + version: number, + hash = HASH_A, +): Promise { + await conn.unsafe(` + INSERT INTO opt_analytics_dashboard_release + (release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, document_hash) + VALUES ('${releaseId}', '${dashboardId}', '${versionId}', 't_alpha', 'p_main', ${version}, 'draft', '${hash}')`); +} + +async function publishedCount(conn: ReturnType, dashboardId: string): Promise { + const rows = (await conn.unsafe( + `SELECT count(*)::int AS n FROM opt_analytics_dashboard_release + WHERE dashboard_id = '${dashboardId}' AND state = 'published'`, + )) as unknown as Array<{ n: number }>; + return rows[0].n; +} + +/** + * Two sessions publish a release of the SAME dashboard at the same time, each + * having first read "nothing is live". This is the create-and-publish path: + * a new version is minted and released in one transaction, which is what a + * "save and publish" button does. + */ +async function racePublish(schema: string, dashboardId: string): Promise<{ ok: number; codes: string[] }> { + const a = connect(schema); + const b = connect(schema); + const publish = (conn: ReturnType, n: number) => + conn.begin(async (tx) => { + await tx.unsafe( + `SELECT release_id FROM opt_analytics_dashboard_release + WHERE dashboard_id = '${dashboardId}' AND state = 'published'`, + ); + // A tiny pause so both transactions have certainly done their read + // before either writes. Without it the race may not interleave and the + // counterfactual could pass for the wrong reason. + await new Promise((r) => setTimeout(r, 60)); + await tx.unsafe(` + INSERT INTO opt_analytics_dashboard_release + (release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, + document_hash, published_at) + VALUES ('rel_race_${n}', '${dashboardId}', 'dv_race_${n}', 't_alpha', 'p_main', ${400 + n}, + 'published', '${n === 1 ? HASH_A : HASH_B}', now())`); + }); + const results = await Promise.allSettled([publish(a, 1), publish(b, 2)]); + await a.end({ timeout: 5 }); + await b.end({ timeout: 5 }); + return { + ok: results.filter((r) => r.status === 'fulfilled').length, + codes: results.flatMap((r) => + r.status === 'rejected' ? [String((r.reason as { code?: string })?.code ?? r.reason)] : [], + ), + }; +} + +beforeAll(async () => { + if (!TEST_DATABASE_URL) return; + const admin = postgres(TEST_DATABASE_URL, { max: 1, onnotice: () => undefined }); + await admin.unsafe(`CREATE SCHEMA IF NOT EXISTS ${SCHEMA}`); + await admin.unsafe(`CREATE SCHEMA IF NOT EXISTS ${SCHEMA_NOGUARD}`); + await admin.end({ timeout: 5 }); + + sql = connect(SCHEMA, 3); + await migrate(sql); + for (const d of ['dash_txn', 'dash_race', 'dash_draft']) await seedDashboard(sql, d); +}, 120_000); + +afterAll(async () => { + if (!TEST_DATABASE_URL || !sql) return; + await sql.unsafe(`DROP SCHEMA IF EXISTS ${SCHEMA} CASCADE`); + await sql.unsafe(`DROP SCHEMA IF EXISTS ${SCHEMA_NOGUARD} CASCADE`); + await sql.end({ timeout: 5 }); +}); + +run('sql/010 applies and replays', () => { + it('is idempotent — a replay onto an already-migrated schema succeeds', async () => { + const text = fs.readFileSync( + path.join(process.cwd(), 'sql/010_analytics_dashboard_releases.sql'), + 'utf8', + ); + await expect(sql.unsafe(text)).resolves.toBeDefined(); + }); + + it('has no column that could hold a document body — the document lives in the version row', async () => { + const rows = (await sql.unsafe( + `SELECT column_name FROM information_schema.columns + WHERE table_schema = '${SCHEMA}' AND table_name = 'opt_analytics_dashboard_release'`, + )) as unknown as Array<{ column_name: string }>; + const names = new Set(rows.map((r) => r.column_name)); + for (const forbidden of ['layout_json', 'document', 'definition', 'payload', 'sql']) { + expect(names.has(forbidden), forbidden).toBe(false); + } + expect(names.has('document_hash')).toBe(true); + }); +}); + +run('BI-P061 transitions are enforced by the database', () => { + beforeAll(async () => { + await seedVersion(sql, 'dv_t1', 'dash_txn', 101); + await seedVersion(sql, 'dv_t3', 'dash_txn', 103, HASH_B); + await seedDraftRelease(sql, 'rel_t1', 'dv_t1', 'dash_txn', 101); + }); + + it('draft -> published is allowed', async () => { + await sql.unsafe( + `UPDATE opt_analytics_dashboard_release SET state='published', published_at=now() WHERE release_id='rel_t1'`, + ); + const rows = (await sql.unsafe( + `SELECT state FROM opt_analytics_dashboard_release WHERE release_id='rel_t1'`, + )) as unknown as Array<{ state: string }>; + expect(rows[0].state).toBe('published'); + }); + + it('published -> draft (un-publishing) is REFUSED', async () => { + const msg = await refusalOf(() => + sql.unsafe( + `UPDATE opt_analytics_dashboard_release SET state='draft', published_at=NULL WHERE release_id='rel_t1'`, + ), + ); + expect(msg).toMatch(/illegal dashboard release transition published -> draft/); + }); + + it('repointing a release at a different version is REFUSED', async () => { + const msg = await refusalOf(() => + sql.unsafe( + `UPDATE opt_analytics_dashboard_release SET dashboard_version_id='dv_t3' WHERE release_id='rel_t1'`, + ), + ); + expect(msg).toMatch(/immutable apart from its state/); + }); + + it('rewriting document_hash on a release is REFUSED', async () => { + const msg = await refusalOf(() => + sql.unsafe( + `UPDATE opt_analytics_dashboard_release SET document_hash='${HASH_B}' WHERE release_id='rel_t1'`, + ), + ); + expect(msg).toMatch(/immutable apart from its state/); + }); + + it('rewriting published_at is REFUSED — a published timestamp is history', async () => { + const msg = await refusalOf(() => + sql.unsafe( + `UPDATE opt_analytics_dashboard_release SET published_at = now() - interval '10 days' + WHERE release_id='rel_t1'`, + ), + ); + expect(msg).toMatch(/published_at is history/); + }); + + it('published -> superseded is allowed, and superseded -> published is REFUSED', async () => { + await sql.unsafe( + `UPDATE opt_analytics_dashboard_release SET state='superseded', superseded_at=now() WHERE release_id='rel_t1'`, + ); + const msg = await refusalOf(() => + sql.unsafe(`UPDATE opt_analytics_dashboard_release SET state='published' WHERE release_id='rel_t1'`), + ); + expect(msg).toMatch(/illegal dashboard release transition superseded -> published/); + }); + + it('deleting a release is REFUSED — the ledger has no eraser', async () => { + const msg = await refusalOf(() => + sql.unsafe(`DELETE FROM opt_analytics_dashboard_release WHERE release_id='rel_t1'`), + ); + expect(msg).toMatch(/ledger; delete is refused/); + }); + + it('a published release with no published_at is REFUSED by the CHECK', async () => { + const msg = await refusalOf(() => + sql.unsafe(` + INSERT INTO opt_analytics_dashboard_release + (release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, document_hash) + VALUES ('rel_bad', 'dash_txn', 'dv_t3', 't_alpha', 'p_main', 900, 'published', '${HASH_A}')`), + ); + expect(msg).toMatch(/opt_analytics_dashboard_release_timestamps/); + }); + + it('a document_hash that is not a sha256 digest is REFUSED', async () => { + const msg = await refusalOf(() => + sql.unsafe(` + INSERT INTO opt_analytics_dashboard_release + (release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, document_hash) + VALUES ('rel_bad2', 'dash_txn', 'dv_t3', 't_alpha', 'p_main', 901, 'draft', 'not-a-hash')`), + ); + expect(msg).toMatch(/document_hash/); + }); + + it('the sql/008 immutability trigger still forbids editing the frozen document', async () => { + // Assert the row exists FIRST: an UPDATE that matches nothing raises + // nothing, which would read as "refused" to a careless test. + const before = (await sql.unsafe( + `SELECT count(*)::int AS n FROM opt_analytics_dashboard_version WHERE dashboard_version_id='dv_t3'`, + )) as unknown as Array<{ n: number }>; + expect(before[0].n).toBe(1); + + const msg = await refusalOf(() => + sql.unsafe( + `UPDATE opt_analytics_dashboard_version SET layout_json='{"tampered":true}' + WHERE dashboard_version_id='dv_t3'`, + ), + ); + expect(msg).toMatch(/rows are immutable/); + }); + + it('a second open draft for one dashboard is REFUSED', async () => { + await seedVersion(sql, 'dv_d1', 'dash_draft', 301); + await seedVersion(sql, 'dv_d2', 'dash_draft', 302); + await seedDraftRelease(sql, 'rel_d1', 'dv_d1', 'dash_draft', 301); + const msg = await refusalOf(() => seedDraftRelease(sql, 'rel_d2', 'dv_d2', 'dash_draft', 302)); + expect(msg).toMatch(/one_draft/); + }); +}); + +run('BI-P061 concurrency: the partial unique index is the guarantee', () => { + let noguard: ReturnType; + + beforeAll(async () => { + noguard = connect(SCHEMA_NOGUARD, 3); + await migrate(noguard); + await noguard.unsafe(`DROP INDEX opt_analytics_dashboard_release_one_published`); + await seedDashboard(noguard, 'dash_race'); + await seedVersion(noguard, 'dv_race_1', 'dash_race', 401); + await seedVersion(noguard, 'dv_race_2', 'dash_race', 402, HASH_B); + + await seedVersion(sql, 'dv_race_1', 'dash_race', 401); + await seedVersion(sql, 'dv_race_2', 'dash_race', 402, HASH_B); + }, 60_000); + + afterAll(async () => { + if (noguard) await noguard.end({ timeout: 5 }); + }); + + it('WITHOUT the index, two concurrent publishes BOTH succeed — two live dashboards', async () => { + const race = await racePublish(SCHEMA_NOGUARD, 'dash_race'); + expect(race.ok).toBe(2); + expect(await publishedCount(noguard, 'dash_race')).toBe(2); + }, 30_000); + + it('WITH the index, the same race leaves exactly one published release', async () => { + expect(await publishedCount(sql, 'dash_race')).toBe(0); + + const race = await racePublish(SCHEMA, 'dash_race'); + + expect(race.ok).toBe(1); + expect(race.codes.join(' ')).toContain('23505'); + expect(await publishedCount(sql, 'dash_race')).toBe(1); + }, 30_000); +}); diff --git a/tests/analytics/dashboard-release.test.ts b/tests/analytics/dashboard-release.test.ts new file mode 100644 index 0000000..3e5c703 --- /dev/null +++ b/tests/analytics/dashboard-release.test.ts @@ -0,0 +1,212 @@ +/** + * tests/analytics/dashboard-release.test.ts + * ========================================== + * BI-P061 — DASHBOARD VERSIONING. DoD: "rollback to previous dashboard + * definition". + * + * A rollback test that only asserts "the service returned version 4" proves + * nothing: the interesting failure is a rollback that publishes a DIFFERENT + * document while reporting success. So every rollback assertion here compares + * BI-P060 canonical hashes and re-plans the restored dashboard, and one test + * corrupts the stored document underneath a release to show that serving it + * is refused rather than silently rendered. + */ +import { describe, expect, it } from 'vitest'; +import { SiaxError } from '@/src/platform/errors'; +import { + dashboardDocumentHash, + parseDashboardDocument, + planDashboard, +} from '@/src/analytics/dashboards/dashboard-document'; +import { + DashboardReleaseService, + InMemoryDashboardReleaseStore, + isLegalTransition, + DASHBOARD_RELEASE_STATES, + type DashboardReleaseState, +} from '@/src/analytics/dashboards/dashboard-release'; + +const SCOPE = { tenant_id: 't_alpha', project_id: 'p_main' }; + +function doc(title: string, metric = 'revenue') { + return { + id: 'dash_sales', + title, + version: 1, + layout: { grid_columns: 12, row_height_px: 40 }, + parameters: [], + filters: [], + widgets: [ + { + widget_id: 'w_1', + title, + source: { kind: 'metric', dataset: 'ds_sales', metric }, + visualization: { type: 'big_number', encoding: { value: metric } }, + layout: { x: 0, y: 0, w: 4, h: 2 }, + }, + ], + refresh: { mode: 'manual', max_staleness_seconds: 3600 }, + permissions: { visibility: 'tenant' }, + }; +} + +function service() { + let n = 0; + let t = Date.parse('2026-09-07T10:00:00.000Z'); + return new DashboardReleaseService( + new InMemoryDashboardReleaseStore(), + () => new Date((t += 1000)), + () => `id_${++n}`, + ); +} + +describe('BI-P061 release lifecycle', () => { + it('the transition table is exactly DRAFT -> PUBLISHED -> SUPERSEDED', () => { + const legal: Array<[DashboardReleaseState, DashboardReleaseState]> = [ + ['draft', 'published'], + ['published', 'superseded'], + ]; + for (const from of DASHBOARD_RELEASE_STATES) { + for (const to of DASHBOARD_RELEASE_STATES) { + const expected = legal.some(([f, t]) => f === from && t === to); + expect(isLegalTransition(from, to), `${from} -> ${to}`).toBe(expected); + } + } + // Named explicitly because each is a real request someone will make. + expect(isLegalTransition('published', 'draft')).toBe(false); + expect(isLegalTransition('superseded', 'published')).toBe(false); + expect(isLegalTransition('draft', 'superseded')).toBe(false); + }); + + it('publishing supersedes the previous release, and only one is ever live', async () => { + const svc = service(); + const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') }); + await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id }); + const v2 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v2') }); + const result = await svc.publish({ scope: SCOPE, releaseId: v2.release.release_id }); + + expect(result.published.version).toBe(2); + expect(result.superseded?.version).toBe(1); + expect(result.superseded?.superseded_by_release_id).toBe(v2.release.release_id); + + const history = await svc.history(SCOPE, 'dash_sales'); + expect(history.map((r) => `${r.version}:${r.state}`)).toEqual(['1:superseded', '2:published']); + expect((await svc.live(SCOPE, 'dash_sales'))?.version).toBe(2); + }); + + it('a second open draft is refused rather than racing the first', async () => { + const svc = service(); + await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') }); + await expect( + svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1-again') }), + ).rejects.toThrow(/already has an open draft/); + }); + + it('a superseded release cannot be re-published', async () => { + const svc = service(); + const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') }); + await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id }); + const v2 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v2') }); + await svc.publish({ scope: SCOPE, releaseId: v2.release.release_id }); + + await expect(svc.publish({ scope: SCOPE, releaseId: v1.release.release_id })).rejects.toThrow( + /is superseded, not draft/, + ); + }); + + it('another tenant cannot see, publish or roll back this tenant’s releases', async () => { + const svc = service(); + const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') }); + const other = { tenant_id: 't_beta', project_id: 'p_main' }; + + await expect(svc.publish({ scope: other, releaseId: v1.release.release_id })).rejects.toThrow( + /does not exist in this tenant/, + ); + expect(await svc.live(other, 'dash_sales')).toBeUndefined(); + await expect(svc.rollback({ scope: other, dashboardId: 'dash_sales', toVersion: 1 })).rejects.toThrow( + /no version 1 in this tenant/, + ); + }); +}); + +describe('BI-P061 rollback — the DoD', () => { + it('restores the earlier definition byte-for-byte, as a NEW version', async () => { + const svc = service(); + const original = doc('Sales v1', 'revenue'); + const originalHash = dashboardDocumentHash(parseDashboardDocument(original)); + + const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: original }); + await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id }); + const v2 = await svc.draft({ + scope: SCOPE, + dashboardId: 'dash_sales', + document: doc('Sales v2 (broken)', 'revenue_net'), + }); + await svc.publish({ scope: SCOPE, releaseId: v2.release.release_id }); + expect((await svc.live(SCOPE, 'dash_sales'))?.document_hash).not.toBe(originalHash); + + const rolled = await svc.rollback({ scope: SCOPE, dashboardId: 'dash_sales', toVersion: 1 }); + + // Faithful: same canonical document. + expect(rolled.document_hash).toBe(originalHash); + // New version, not a resurrected row — the history stays linear. + expect(rolled.release.version).toBe(3); + expect(rolled.release.rollback_of_version).toBe(1); + + const history = await svc.history(SCOPE, 'dash_sales'); + expect(history.map((r) => `${r.version}:${r.state}`)).toEqual([ + '1:superseded', + '2:superseded', + '3:published', + ]); + + // And the restored dashboard actually renders as the original did. + const live = await svc.live(SCOPE, 'dash_sales'); + const restored = await svc.documentOf(SCOPE, live!); + expect(planDashboard(restored)).toEqual(planDashboard(parseDashboardDocument(original))); + }); + + it('rolling back to the live version is refused as a no-op', async () => { + const svc = service(); + const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') }); + await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id }); + await expect(svc.rollback({ scope: SCOPE, dashboardId: 'dash_sales', toVersion: 1 })).rejects.toThrow( + /already live/, + ); + }); + + it('a stored document that no longer matches its released hash is refused, not rendered', async () => { + const store = new InMemoryDashboardReleaseStore(); + const svc = new DashboardReleaseService( + store, + () => new Date(), + () => crypto.randomUUID(), + ); + const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') }); + const { published } = await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id }); + + // Tamper with storage the way a bad migration or a manual UPDATE would. + const stored = await store.getVersion(SCOPE, published.dashboard_version_id); + const tampered = JSON.parse(stored!.layout_json); + tampered.widgets[0].source.metric = 'revenue_including_vat'; + await store.insertVersionAndRelease( + { ...stored!, dashboard_version_id: 'tampered', version: 99, layout_json: JSON.stringify(tampered) }, + { + ...published, + release_id: 'r_tampered', + dashboard_version_id: 'tampered', + version: 99, + state: 'draft', + published_at: undefined, + }, + ); + + // documentOf must compare hashes, not trust the row. + await expect(svc.documentOf(SCOPE, { ...published, dashboard_version_id: 'tampered' })).rejects.toThrow( + /refusing to serve an unapproved definition/, + ); + await expect(svc.documentOf(SCOPE, { ...published, dashboard_version_id: 'tampered' })).rejects.toThrow( + SiaxError, + ); + }); +}); diff --git a/tests/analytics/materialization-drift.test.ts b/tests/analytics/materialization-drift.test.ts new file mode 100644 index 0000000..dadef13 --- /dev/null +++ b/tests/analytics/materialization-drift.test.ts @@ -0,0 +1,327 @@ +/** + * tests/analytics/materialization-drift.test.ts + * ============================================== + * BI-P105 — MATERIALIZATION / PRE-AGGREGATION. + * + * The DoD asks for measured improvement and documented freshness semantics. + * The failure mode it does not name, and the one that actually costs money, is + * a pre-aggregate that returns a confident wrong number. So every test here is + * written as an attack on the router: + * + * - a drifted aggregate must NOT serve, and the counterfactual (a naive + * router that only checks staleness) must serve it — otherwise the guard + * is not known to be doing anything; + * - a MISSING bucket must count as drift even though every value present is + * correct; + * - a comparison over an empty window must be `insufficient_data`, never + * `match`; + * - an aggregate that has never been verified must not serve, because on + * launch day it is exactly as unproven as a drifted one. + */ +import { describe, expect, it } from 'vitest'; +import { + applyDriftReport, + detectDrift, + type AggregateBucket, +} from '@/src/analytics/materialization/drift-detector'; +import { + assertSpec, + describeRouting, + materializationStaleness, + selectQuerySource, + type MaterializationSpec, + type MaterializationStatus, +} from '@/src/analytics/materialization/materialization'; +import type { AnalyticsQueryRequest } from '@/src/analytics/contract/query-request'; + +const NOW = new Date('2026-09-07T12:00:00.000Z'); +const TOLERANCE = { absolute: 0.5, relative: 0.001 }; + +const SPEC: MaterializationSpec = { + materialization_id: 'mv_sales_daily', + tenant_id: 't_alpha', + project_id: 'p_main', + dataset: 'ds_sales', + strategy: 'clickhouse_materialized_view', + target_ref: 'opt_sales_daily', + metrics: ['revenue', 'orders'], + dimensions: ['order_date', 'region'], + time_dimension: 'order_date', + grain: 'day', + max_staleness_seconds: 3600, + drift_check_interval_seconds: 86_400, + tolerance: TOLERANCE, +}; + +/** Fresh, recently verified, matching. The only state that may serve. */ +function healthyStatus(): MaterializationStatus { + return { + materialization_id: SPEC.materialization_id, + state: 'serving', + data_watermark: '2026-09-07T11:45:00.000Z', + last_refresh_at: '2026-09-07T11:46:00.000Z', + last_drift_check_at: '2026-09-07T06:00:00.000Z', + last_drift_verdict: 'match', + }; +} + +const REQUEST: AnalyticsQueryRequest = { + dataset: 'ds_sales', + metrics: ['revenue'], + dimensions: ['order_date'], + filters: [{ dimension: 'region', operator: 'eq', values: ['SE'] }], + timeRange: { from: '2026-09-01T00:00:00.000Z', to: '2026-09-07T00:00:00.000Z', grain: 'day' }, +}; + +/** + * The router someone writes when they have thought about staleness but not + * about correctness. It exists here as the counterfactual: it must SERVE the + * drifted aggregate that `selectQuerySource` refuses. + */ +function naiveRouter(status: MaterializationStatus, now: Date): 'materialization' | 'base_table' { + const staleness = materializationStaleness(status, now) ?? 0; + return staleness <= SPEC.max_staleness_seconds ? 'materialization' : 'base_table'; +} + +function buckets(spec: Record): AggregateBucket[] { + return Object.entries(spec).map(([key, [revenue, orders]]) => ({ key, values: { revenue, orders } })); +} + +const SOURCE = buckets({ + '2026-09-01|SE': [120_000, 410], + '2026-09-02|SE': [131_500, 442], + '2026-09-03|SE': [98_250, 305], + '2026-09-04|SE': [140_000, 480], +}); + +describe('BI-P105 drift detection', () => { + it('an identical aggregate matches', () => { + const report = detectDrift({ + fromSource: SOURCE, + fromAggregate: SOURCE, + tolerance: TOLERANCE, + metrics: ['revenue', 'orders'], + now: NOW, + }); + expect(report.verdict).toBe('match'); + expect(report.buckets_compared).toBe(4); + expect(report.totals.revenue.delta).toBe(0); + }); + + it('a value beyond tolerance is drift, and the report says what a caller would have been told', () => { + const drifted = buckets({ + '2026-09-01|SE': [120_000, 410], + '2026-09-02|SE': [131_500, 442], + '2026-09-03|SE': [98_250, 305], + // A source row was deleted (GDPR erasure) but the aggregate still counts it. + '2026-09-04|SE': [152_400, 495], + }); + const report = detectDrift({ + fromSource: SOURCE, + fromAggregate: drifted, + tolerance: TOLERANCE, + metrics: ['revenue', 'orders'], + now: NOW, + }); + expect(report.verdict).toBe('drifted'); + expect(report.worst?.key).toBe('2026-09-04|SE'); + expect(report.worst?.absolute_delta).toBe(12_400); + expect(report.totals.revenue.aggregate - report.totals.revenue.source).toBe(12_400); + }); + + it('a MISSING bucket is drift even though every value present is correct', () => { + const lateArriving = SOURCE.filter((b) => b.key !== '2026-09-04|SE'); + const report = detectDrift({ + fromSource: SOURCE, + fromAggregate: lateArriving, + tolerance: TOLERANCE, + metrics: ['revenue', 'orders'], + now: NOW, + }); + expect(report.discrepancies).toHaveLength(0); // every compared value agrees + expect(report.verdict).toBe('drifted'); // and it is still wrong + expect(report.missing_in_aggregate).toEqual(['2026-09-04|SE']); + expect(report.totals.revenue.delta).toBe(-140_000); + }); + + it('an EXTRA bucket is drift — the source rows are gone, the aggregate is not', () => { + const stale = [...SOURCE, { key: '2026-08-31|SE', values: { revenue: 90_000, orders: 300 } }]; + const report = detectDrift({ + fromSource: SOURCE, + fromAggregate: stale, + tolerance: TOLERANCE, + metrics: ['revenue', 'orders'], + now: NOW, + }); + expect(report.verdict).toBe('drifted'); + expect(report.extra_in_aggregate).toEqual(['2026-08-31|SE']); + }); + + it('an EMPTY comparison is insufficient_data, not match', () => { + const report = detectDrift({ + fromSource: [], + fromAggregate: [], + tolerance: TOLERANCE, + metrics: ['revenue'], + now: NOW, + }); + expect(report.verdict).toBe('insufficient_data'); + expect(report.buckets_compared).toBe(0); + // And it must not be allowed to serve on the strength of that. + const status = applyDriftReport(healthyStatus(), report); + expect(status.state).toBe('drifted'); + }); + + it('both tolerances must hold — a large bucket cannot hide behind a relative bound', () => { + const source = buckets({ 'b|SE': [40_000_000, 1] }); + const off = buckets({ 'b|SE': [40_020_000, 1] }); // 0.05%: inside relative, far outside absolute + const report = detectDrift({ + fromSource: source, + fromAggregate: off, + tolerance: { absolute: 1000, relative: 0.001 }, + metrics: ['revenue'], + now: NOW, + }); + expect(report.verdict).toBe('drifted'); + expect(report.worst?.relative_delta).toBeCloseTo(0.0005, 6); + }); + + it('a bucket that should be zero and is not is drift, not a division error', () => { + const report = detectDrift({ + fromSource: buckets({ 'b|SE': [0, 0] }), + fromAggregate: buckets({ 'b|SE': [12, 1] }), + tolerance: TOLERANCE, + metrics: ['revenue'], + now: NOW, + }); + expect(report.verdict).toBe('drifted'); + expect(report.worst?.relative_delta).toBe(Number.POSITIVE_INFINITY); + }); +}); + +describe('BI-P105 routing — a pre-aggregate serves only while it is provably current and undrifted', () => { + it('serves when everything is proved', () => { + const decision = selectQuerySource({ request: REQUEST, spec: SPEC, status: healthyStatus(), now: NOW }); + expect(decision.source).toBe('materialization'); + expect(describeRouting(decision)).toMatchObject({ + served_from: 'materialization', + materialization_id: 'mv_sales_daily', + staleness_seconds: 900, + }); + }); + + it('a DRIFTED aggregate is refused — and the naive router serves it', () => { + const drifted = applyDriftReport(healthyStatus(), { + verdict: 'drifted', + checked_at: '2026-09-07T11:00:00.000Z', + buckets_compared: 4, + missing_in_aggregate: [], + extra_in_aggregate: [], + discrepancies: [], + tolerance: TOLERANCE, + totals: {}, + }); + + const guarded = selectQuerySource({ request: REQUEST, spec: SPEC, status: drifted, now: NOW }); + expect(guarded.source).toBe('base_table'); + expect(guarded).toMatchObject({ refusal: 'state_not_serving' }); + + // The counterfactual: the same state, routed by staleness alone. + expect(naiveRouter(drifted, NOW)).toBe('materialization'); + }); + + it('a NEVER-VERIFIED aggregate is refused — and the naive router serves it', () => { + const brandNew: MaterializationStatus = { + materialization_id: SPEC.materialization_id, + state: 'serving', + data_watermark: '2026-09-07T11:59:00.000Z', + last_refresh_at: '2026-09-07T11:59:00.000Z', + }; + const guarded = selectQuerySource({ request: REQUEST, spec: SPEC, status: brandNew, now: NOW }); + expect(guarded).toMatchObject({ source: 'base_table', refusal: 'drift_never_verified' }); + expect(naiveRouter(brandNew, NOW)).toBe('materialization'); + }); + + it('a verification older than the required interval is refused', () => { + const status = { ...healthyStatus(), last_drift_check_at: '2026-09-05T06:00:00.000Z' }; + expect(selectQuerySource({ request: REQUEST, spec: SPEC, status, now: NOW })).toMatchObject({ + refusal: 'drift_verification_stale', + }); + }); + + it('an UNKNOWN watermark is refused rather than treated as current', () => { + const status = { ...healthyStatus(), data_watermark: undefined }; + expect(materializationStaleness(status, NOW)).toBeUndefined(); + expect(selectQuerySource({ request: REQUEST, spec: SPEC, status, now: NOW })).toMatchObject({ + refusal: 'watermark_unknown', + }); + }); + + it('an aggregate older than its freshness contract is refused', () => { + const status = { ...healthyStatus(), data_watermark: '2026-09-07T09:00:00.000Z' }; + expect(selectQuerySource({ request: REQUEST, spec: SPEC, status, now: NOW })).toMatchObject({ + refusal: 'staleness_exceeded', + }); + }); + + it('a question that reaches past the watermark is refused, not silently shortened', () => { + const request = { ...REQUEST, timeRange: { ...REQUEST.timeRange, to: '2026-09-07T12:00:00.000Z' } }; + expect(selectQuerySource({ request, spec: SPEC, status: healthyStatus(), now: NOW })).toMatchObject({ + refusal: 'window_beyond_watermark', + }); + }); + + it('a filter on a dimension the aggregate summed across is refused', () => { + const request = { + ...REQUEST, + filters: [{ dimension: 'sales_channel', operator: 'eq' as const, values: ['web'] }], + }; + const decision = selectQuerySource({ request, spec: SPEC, status: healthyStatus(), now: NOW }); + expect(decision).toMatchObject({ source: 'base_table', refusal: 'coverage_miss' }); + expect((decision as { detail: string }).detail).toMatch(/already summed across it/); + }); + + it('a metric or dimension the aggregate does not carry is refused', () => { + expect( + selectQuerySource({ + request: { ...REQUEST, metrics: ['margin'] }, + spec: SPEC, + status: healthyStatus(), + now: NOW, + }), + ).toMatchObject({ refusal: 'coverage_miss' }); + expect( + selectQuerySource({ + request: { ...REQUEST, dimensions: ['product'] }, + spec: SPEC, + status: healthyStatus(), + now: NOW, + }), + ).toMatchObject({ refusal: 'coverage_miss' }); + }); + + it('a daily aggregate cannot answer an hourly question, but can answer a monthly one', () => { + const hourly = { ...REQUEST, timeRange: { ...REQUEST.timeRange, grain: 'hour' as const } }; + expect( + selectQuerySource({ request: hourly, spec: SPEC, status: healthyStatus(), now: NOW }), + ).toMatchObject({ refusal: 'coverage_miss' }); + + const monthly = { ...REQUEST, timeRange: { ...REQUEST.timeRange, grain: 'month' as const } }; + expect( + selectQuerySource({ request: monthly, spec: SPEC, status: healthyStatus(), now: NOW }).source, + ).toBe('materialization'); + }); + + it('a WEEKLY aggregate cannot answer a monthly question — weeks do not compose into months', () => { + const weekly = { ...SPEC, grain: 'week' as const }; + const monthly = { ...REQUEST, timeRange: { ...REQUEST.timeRange, grain: 'month' as const } }; + expect( + selectQuerySource({ request: monthly, spec: weekly, status: healthyStatus(), now: NOW }), + ).toMatchObject({ refusal: 'coverage_miss' }); + }); + + it('a spec with no drift check interval is refused at declaration time', () => { + expect(() => assertSpec({ ...SPEC, drift_check_interval_seconds: 0 })).toThrow(/never re-verified/); + expect(() => assertSpec(SPEC)).not.toThrow(); + }); +}); diff --git a/tests/integration/bi-p104-benchmark.integration.test.ts b/tests/integration/bi-p104-benchmark.integration.test.ts new file mode 100644 index 0000000..7b3d415 --- /dev/null +++ b/tests/integration/bi-p104-benchmark.integration.test.ts @@ -0,0 +1,344 @@ +/** + * tests/integration/bi-p104-benchmark.integration.test.ts + * ======================================================== + * BI-P104 — the benchmark itself, against real engines. Skips honestly when + * CLICKHOUSE_URL is unset; DuckDB additionally needs DUCKDB_PYTHON (an + * interpreter with the `duckdb` package), matching + * tests/duckdb/duckdb-provider.live.test.ts. + * + * Six dataset profiles, one representative question each, the same logical + * question on both engines. Deliberately NO engine-specific functions in the + * geospatial case (bounding box + grid arithmetic rather than each engine's + * GIS library) so the comparison is about the storage and execution engine + * rather than about who ships the nicer geo function. + * + * Statistics come from src/analytics/bench/benchmark-harness.ts — the shipped + * code, not a copy — so the evidence file is produced by the thing under test. + * + * Trino, Superset, native charts and Perspective are NOT exercised, and each + * appears in the evidence with its reason. An engine omitted from a benchmark + * table reads as a choice, so nothing is omitted. + * + * CLICKHOUSE_URL=http://127.0.0.1:28123 CLICKHOUSE_AUTH=default:test \ + * DUCKDB_PYTHON=/path/to/venv/bin/python \ + * pnpm vitest run tests/integration/bi-p104-benchmark.integration.test.ts + * + * It CREATEs and DROPs its own ClickHouse database and uses an in-memory + * DuckDB. Never point it at the b0ts ClickHouse or any quarantined data. + */ +import { execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { + buildEvidence, + compareDistributions, + describeComparison, + summarizeSamples, + type BenchDatasetId, + type BenchmarkRun, + type EngineAvailability, +} from '@/src/analytics/bench/benchmark-harness'; + +const CLICKHOUSE_URL = process.env.CLICKHOUSE_URL; +const CLICKHOUSE_AUTH = process.env.CLICKHOUSE_AUTH; +const DUCKDB_PYTHON = process.env.DUCKDB_PYTHON; +const run = CLICKHOUSE_URL ? describe : describe.skip; +const DB = `opt_p104_${Date.now().toString(36)}`; +const WARMUPS = 2; +const REPETITIONS = 9; +const WIDE_COLUMNS = 50; + +const headers: Record = CLICKHOUSE_AUTH + ? { Authorization: `Basic ${Buffer.from(CLICKHOUSE_AUTH).toString('base64')}` } + : {}; + +async function ch(query: string): Promise { + const res = await fetch(CLICKHOUSE_URL ?? 'http://skipped.invalid', { + method: 'POST', + headers, + body: query, + }); + const text = await res.text(); + if (!res.ok) throw new Error(`ClickHouse ${res.status}: ${text.slice(0, 300)}`); + return text; +} + +/** The six questions, one per profile, phrased for each engine. */ +const QUERIES: Record = { + small_saas: { + label: 'revenue by day', + clickhouse: `SELECT d, sum(amount) AS revenue, count() AS n FROM ${DB}.small_saas WHERE tenant_id='t_1' GROUP BY d ORDER BY d`, + duckdb: `SELECT d, sum(amount) AS revenue, count(*) AS n FROM small_saas WHERE tenant_id='t_1' GROUP BY d ORDER BY d`, + }, + medium_business: { + label: 'revenue by month and region for one tenant', + clickhouse: `SELECT toStartOfMonth(d) AS m, region, sum(amount) AS revenue FROM ${DB}.medium_business WHERE tenant_id='t_2' GROUP BY m, region ORDER BY m, region`, + duckdb: `SELECT date_trunc('month', d) AS m, region, sum(amount) AS revenue FROM medium_business WHERE tenant_id='t_2' GROUP BY m, region ORDER BY m, region`, + }, + vehiq_market: { + label: 'average asking price by make and model year', + clickhouse: `SELECT make, model_year, avg(price) AS avg_price, count() AS listings FROM ${DB}.vehiq_market WHERE mileage < 150000 GROUP BY make, model_year ORDER BY make, model_year`, + duckdb: `SELECT make, model_year, avg(price) AS avg_price, count(*) AS listings FROM vehiq_market WHERE mileage < 150000 GROUP BY make, model_year ORDER BY make, model_year`, + }, + large_event: { + label: 'hourly event volume for one event type', + clickhouse: `SELECT toStartOfHour(ts) AS h, count() AS n, uniqExact(user_id) AS users FROM ${DB}.large_event WHERE event_type='checkout' GROUP BY h ORDER BY h`, + duckdb: `SELECT date_trunc('hour', ts) AS h, count(*) AS n, count(DISTINCT user_id) AS users FROM large_event WHERE event_type='checkout' GROUP BY h ORDER BY h`, + }, + wide_table: { + label: 'three columns of fifty, grouped', + clickhouse: `SELECT grp, sum(c7) AS a, sum(c23) AS b, sum(c41) AS c FROM ${DB}.wide_table GROUP BY grp ORDER BY grp`, + duckdb: `SELECT grp, sum(c7) AS a, sum(c23) AS b, sum(c41) AS c FROM wide_table GROUP BY grp ORDER BY grp`, + }, + geospatial: { + label: 'bounding box, aggregated to a grid cell', + clickhouse: `SELECT floor(lat*10)/10 AS cell_lat, floor(lon*10)/10 AS cell_lon, count() AS n FROM ${DB}.geospatial WHERE lat BETWEEN 57.5 AND 59.5 AND lon BETWEEN 11.5 AND 18.5 GROUP BY cell_lat, cell_lon ORDER BY cell_lat, cell_lon`, + duckdb: `SELECT floor(lat*10)/10 AS cell_lat, floor(lon*10)/10 AS cell_lon, count(*) AS n FROM geospatial WHERE lat BETWEEN 57.5 AND 59.5 AND lon BETWEEN 11.5 AND 18.5 GROUP BY cell_lat, cell_lon ORDER BY cell_lat, cell_lon`, + }, +}; + +const CH_DDL: Array<[BenchDatasetId, string]> = [ + [ + 'small_saas', + `CREATE TABLE ${DB}.small_saas ENGINE = MergeTree ORDER BY (tenant_id, d) AS + SELECT concat('t_', toString(number % 3)) AS tenant_id, + toDate('2026-01-01') + toIntervalDay(number % 365) AS d, + ['free','pro','enterprise'][(number % 3) + 1] AS plan, + toDecimal64((number % 499) + 9, 2) AS amount + FROM numbers(100000)`, + ], + [ + 'medium_business', + `CREATE TABLE ${DB}.medium_business ENGINE = MergeTree ORDER BY (tenant_id, d) AS + SELECT concat('t_', toString(number % 8)) AS tenant_id, + toDate('2025-01-01') + toIntervalDay(number % 730) AS d, + ['SE','NO','DK','FI','DE'][(number % 5) + 1] AS region, + toDecimal64((number % 977) + 13, 2) AS amount + FROM numbers(2000000)`, + ], + [ + 'vehiq_market', + `CREATE TABLE ${DB}.vehiq_market ENGINE = MergeTree ORDER BY (make, model_year) AS + SELECT ['Volvo','VW','Toyota','BMW','Kia','Tesla'][(number % 6) + 1] AS make, + toUInt16(2005 + (number % 20)) AS model_year, + toUInt32(30000 + (number % 900000)) AS price, + toUInt32((number * 7) % 400000) AS mileage, + ['diesel','petrol','hybrid','electric'][(number % 4) + 1] AS fuel + FROM numbers(1000000)`, + ], + [ + 'large_event', + `CREATE TABLE ${DB}.large_event ENGINE = MergeTree ORDER BY (event_type, ts) AS + SELECT toDateTime('2026-08-01 00:00:00') + toIntervalSecond(number % 2592000) AS ts, + toUInt32(number % 250000) AS user_id, + ['view','click','checkout','signup'][(number % 4) + 1] AS event_type, + toFloat64(number % 1000) AS value + FROM numbers(5000000)`, + ], + [ + 'wide_table', + `CREATE TABLE ${DB}.wide_table ENGINE = MergeTree ORDER BY grp AS + SELECT toUInt16(number % 500) AS grp, + ${Array.from({ length: WIDE_COLUMNS }, (_, i) => `toFloat64((number + ${i}) % 1000) AS c${i}`).join(',\n ')} + FROM numbers(200000)`, + ], + [ + 'geospatial', + `CREATE TABLE ${DB}.geospatial ENGINE = MergeTree ORDER BY (lat, lon) AS + SELECT 55.0 + (number % 10000) / 1000.0 AS lat, + 10.0 + ((number * 13) % 10000) / 1000.0 AS lon, + toUInt32(number) AS point_id + FROM numbers(500000)`, + ], +]; + +/** Runs all six DuckDB measurements inside ONE python process. */ +const DUCKDB_PROGRAM = ` +import duckdb, json, time, sys +con = duckdb.connect() +# The progress bar writes to STDOUT and corrupts the JSON transcript. It cost +# one run to find, so it is turned off explicitly rather than filtered out. +con.execute("SET enable_progress_bar=false") +WIDE = ${WIDE_COLUMNS} +wide_cols = ", ".join(f"(i + {j}) % 1000.0 AS c{j}" for j in range(WIDE)) +ddl = { + "small_saas": "CREATE TABLE small_saas AS SELECT 't_' || (i % 3) AS tenant_id, DATE '2026-01-01' + INTERVAL (i % 365) DAY AS d, ['free','pro','enterprise'][(i % 3) + 1] AS plan, ((i % 499) + 9)::DECIMAL(18,2) AS amount FROM range(100000) t(i)", + "medium_business": "CREATE TABLE medium_business AS SELECT 't_' || (i % 8) AS tenant_id, DATE '2025-01-01' + INTERVAL (i % 730) DAY AS d, ['SE','NO','DK','FI','DE'][(i % 5) + 1] AS region, ((i % 977) + 13)::DECIMAL(18,2) AS amount FROM range(2000000) t(i)", + "vehiq_market": "CREATE TABLE vehiq_market AS SELECT ['Volvo','VW','Toyota','BMW','Kia','Tesla'][(i % 6) + 1] AS make, (2005 + (i % 20))::USMALLINT AS model_year, (30000 + (i % 900000))::UINTEGER AS price, ((i * 7) % 400000)::UINTEGER AS mileage, ['diesel','petrol','hybrid','electric'][(i % 4) + 1] AS fuel FROM range(1000000) t(i)", + "large_event": "CREATE TABLE large_event AS SELECT TIMESTAMP '2026-08-01 00:00:00' + INTERVAL (i % 2592000) SECOND AS ts, (i % 250000)::UINTEGER AS user_id, ['view','click','checkout','signup'][(i % 4) + 1] AS event_type, (i % 1000)::DOUBLE AS value FROM range(5000000) t(i)", + "wide_table": f"CREATE TABLE wide_table AS SELECT (i % 500)::USMALLINT AS grp, {wide_cols} FROM range(200000) t(i)", + "geospatial": "CREATE TABLE geospatial AS SELECT 55.0 + (i % 10000) / 1000.0 AS lat, 10.0 + ((i * 13) % 10000) / 1000.0 AS lon, i::UINTEGER AS point_id FROM range(500000) t(i)", +} +queries = json.loads(sys.stdin.read()) +out = {"version": duckdb.__version__, "runs": {}} +for name, sql in ddl.items(): + con.execute(sql) +for name, q in queries.items(): + warm = [] + for _ in range(${WARMUPS}): + t0 = time.perf_counter(); rows = con.execute(q).fetchall(); warm.append((time.perf_counter() - t0) * 1000) + samples = [] + for _ in range(${REPETITIONS}): + t0 = time.perf_counter(); rows = con.execute(q).fetchall(); samples.append((time.perf_counter() - t0) * 1000) + out["runs"][name] = {"warmup_ms": warm, "samples_ms": samples, "rows": len(rows)} +sys.stdout.write("<<>>" + json.dumps(out)) +`; + +const runs: BenchmarkRun[] = []; +const engines: EngineAvailability[] = []; +let duckdbResult: + | { version: string; runs: Record } + | undefined; + +run('BI-P104 benchmark harness against real engines', () => { + beforeAll(async () => { + const version = (await ch('SELECT version()')).trim(); + engines.push({ engine: 'clickhouse', available: true, version }); + await ch(`CREATE DATABASE IF NOT EXISTS ${DB}`); + for (const [, ddl] of CH_DDL) await ch(ddl); + }, 900_000); + + afterAll(async () => { + if (!CLICKHOUSE_URL) return; + try { + await ch(`DROP DATABASE IF EXISTS ${DB}`); + } finally { + const comparisons = duckdbResult + ? (Object.keys(QUERIES) as BenchDatasetId[]).flatMap((dataset) => { + const chRun = runs.find((r) => r.engine === 'clickhouse' && r.dataset === dataset); + const duckRun = runs.find((r) => r.engine === 'duckdb' && r.dataset === dataset); + if (!chRun || !duckRun) return []; + const c = compareDistributions(chRun.distribution, duckRun.distribution); + return [ + { + dataset, + query_label: QUERIES[dataset].label, + description: describeComparison(c, `duckdb vs clickhouse (${dataset})`), + ...c, + }, + ]; + }) + : []; + const evidence = buildEvidence({ + host_note: + 'One idle host, warm caches, both engines local, single-threaded client. These numbers ' + + 'describe these query shapes on this data on this machine. They are not engine rankings.', + engines, + runs, + comparisons, + }); + const dir = path.join(process.cwd(), 'evidence/bi-p104'); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(path.join(dir, 'benchmark.json'), `${JSON.stringify(evidence, null, 2)}\n`); + } + }, 120_000); + + it('measures all six dataset profiles on ClickHouse', async () => { + for (const dataset of Object.keys(QUERIES) as BenchDatasetId[]) { + const q = QUERIES[dataset]; + const warmup_ms: number[] = []; + const samples_ms: number[] = []; + let rows = 0; + for (let i = 0; i < WARMUPS; i += 1) { + const t0 = performance.now(); + await ch(`${q.clickhouse} FORMAT JSONCompact`); + warmup_ms.push(performance.now() - t0); + } + for (let i = 0; i < REPETITIONS; i += 1) { + const t0 = performance.now(); + const text = await ch(`${q.clickhouse} FORMAT JSONCompact`); + samples_ms.push(performance.now() - t0); + rows = (JSON.parse(text) as { rows: number }).rows; + } + runs.push({ + engine: 'clickhouse', + dataset, + query_label: q.label, + warmup_ms, + samples_ms, + distribution: summarizeSamples(samples_ms), + rows_returned: rows, + }); + } + expect(runs.filter((r) => r.engine === 'clickhouse')).toHaveLength(6); + for (const r of runs) { + expect(r.distribution.n).toBe(REPETITIONS); + expect(r.distribution.stability).not.toBe('insufficient_samples'); + } + }, 900_000); + + it('measures the same six on DuckDB, or records why it did not', async () => { + if (!DUCKDB_PYTHON) { + engines.push({ + engine: 'duckdb', + available: false, + reason: 'DUCKDB_PYTHON not set on this runner', + }); + expect(engines.find((e) => e.engine === 'duckdb')?.reason).toBeTruthy(); + return; + } + const payload = Object.fromEntries( + (Object.keys(QUERIES) as BenchDatasetId[]).map((d) => [d, QUERIES[d].duckdb]), + ); + const stdout = execFileSync(DUCKDB_PYTHON, ['-c', DUCKDB_PROGRAM], { + input: JSON.stringify(payload), + encoding: 'utf8', + maxBuffer: 32 * 1024 * 1024, + timeout: 900_000, + }); + // Parse from the sentinel: anything DuckDB writes to stdout before it + // (a progress bar, a warning) must not be able to corrupt the transcript. + const marker = stdout.lastIndexOf('<<>>'); + expect(marker, 'duckdb transcript marker missing').toBeGreaterThanOrEqual(0); + duckdbResult = JSON.parse(stdout.slice(marker + '<<>>'.length)); + engines.push({ engine: 'duckdb', available: true, version: duckdbResult!.version }); + + for (const [dataset, r] of Object.entries(duckdbResult!.runs)) { + runs.push({ + engine: 'duckdb', + dataset: dataset as BenchDatasetId, + query_label: QUERIES[dataset as BenchDatasetId].label, + warmup_ms: r.warmup_ms, + samples_ms: r.samples_ms, + distribution: summarizeSamples(r.samples_ms), + rows_returned: r.rows, + }); + } + expect(runs.filter((r) => r.engine === 'duckdb')).toHaveLength(6); + }, 900_000); + + it('records the engines BI-P104 names that nothing measured, with reasons', () => { + engines.push( + { + engine: 'trino', + available: false, + reason: + 'a Trino is running on this host but belongs to another lane (st0re-verify-trino); ' + + 'benchmarking through someone else’s container would measure their configuration', + }, + { + engine: 'semantic_query', + available: false, + reason: 'MetricFlow has never been executed in 0pt (BI-P029 honest status)', + }, + { engine: 'superset', available: false, reason: 'no Superset deployment in the estate yet' }, + { engine: 'native_charts', available: false, reason: 'render-side; DES1GN owns it (BI-P051..P057)' }, + { engine: 'perspective', available: false, reason: 'not deployed' }, + ); + const evidence = buildEvidence({ host_note: 'x', engines, runs }); + for (const e of evidence.not_exercised) expect(e.reason, e.engine).toBeTruthy(); + expect(evidence.engines).toHaveLength(7); + }); + + it('reports every comparison as a range with its stability, never a bare figure', () => { + if (!duckdbResult) return; + for (const dataset of Object.keys(QUERIES) as BenchDatasetId[]) { + const a = runs.find((r) => r.engine === 'clickhouse' && r.dataset === dataset)!; + const b = runs.find((r) => r.engine === 'duckdb' && r.dataset === dataset)!; + const sentence = describeComparison(compareDistributions(a.distribution, b.distribution), dataset); + expect(sentence).toMatch(/n=9\/9/); + expect(sentence).toMatch(/INDISTINGUISHABLE|x to /); + } + }); +}); diff --git a/tests/integration/materialization-drift-clickhouse.integration.test.ts b/tests/integration/materialization-drift-clickhouse.integration.test.ts new file mode 100644 index 0000000..a8f70b3 --- /dev/null +++ b/tests/integration/materialization-drift-clickhouse.integration.test.ts @@ -0,0 +1,348 @@ +/** + * tests/integration/materialization-drift-clickhouse.integration.test.ts + * ======================================================================= + * BI-P105 against a LIVE ClickHouse. Skips honestly when CLICKHOUSE_URL is + * unset; with it set and the server down, it FAILS rather than skipping. + * + * BI-P105's DoD is "performance improvement measured and freshness semantics + * documented", and both halves are done here against real data rather than + * asserted. What the DoD does not say, and what this file is really about, is + * that a pre-aggregate fails SILENTLY: it answers instantly and wrongly. + * + * The drill drives a real ClickHouse materialized view through the exact + * sequence that produces a confident wrong number in production: + * + * 1. 2M source rows, a SummingMergeTree target and a materialized view. + * 2. Backfill, verify with the SHIPPED detector: verdict `match`. + * 3. Measure both paths repeatedly, and report the DISTRIBUTION — a single + * speed-up figure quoted as a system property is misleading (BI-P104). + * 4. `ALTER TABLE ... DELETE` on the source: a GDPR erasure, the single most + * common real cause of pre-aggregate drift. A ClickHouse materialized + * view is an INSERT trigger; it does not see mutations, so the target + * keeps the deleted revenue forever and nothing errors. + * 5. The detector must catch it, and `selectQuerySource` must refuse to + * serve. The counterfactual — a router that checks only staleness — is + * run on the same state and MUST serve, with the wrong number recorded + * in kronor. + * + * Everything measured lands in evidence/bi-p105/materialization-drift.json. + * + * docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_PASSWORD=test \ + * clickhouse/clickhouse-server:26.3 + * CLICKHOUSE_URL=http://127.0.0.1:28123 CLICKHOUSE_AUTH=default:test \ + * pnpm vitest run tests/integration/materialization-drift-clickhouse.integration.test.ts + * + * It CREATEs and DROPs its own database. Never point it at the b0ts + * ClickHouse or any quarantined market data. + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { + applyDriftReport, + detectDrift, + type AggregateBucket, + type DriftReport, +} from '@/src/analytics/materialization/drift-detector'; +import { + materializationStaleness, + selectQuerySource, + type MaterializationSpec, + type MaterializationStatus, +} from '@/src/analytics/materialization/materialization'; +import type { AnalyticsQueryRequest } from '@/src/analytics/contract/query-request'; + +const CLICKHOUSE_URL = process.env.CLICKHOUSE_URL; +const CLICKHOUSE_AUTH = process.env.CLICKHOUSE_AUTH; +const run = CLICKHOUSE_URL ? describe : describe.skip; +const DB = `opt_p105_${Date.now().toString(36)}`; +const ROWS = 2_000_000; +const REPETITIONS = 9; + +const headers: Record = CLICKHOUSE_AUTH + ? { Authorization: `Basic ${Buffer.from(CLICKHOUSE_AUTH).toString('base64')}` } + : {}; + +async function ch(query: string): Promise { + const res = await fetch(CLICKHOUSE_URL ?? 'http://skipped.invalid', { + method: 'POST', + headers, + body: query, + }); + const text = await res.text(); + if (!res.ok) throw new Error(`ClickHouse ${res.status}: ${text.slice(0, 400)}`); + return text; +} + +/** Rows as buckets keyed "date|region", which is what the detector compares. */ +async function bucketsFrom(query: string): Promise { + const text = await ch(`${query} FORMAT JSONEachRow`); + return text + .trim() + .split('\n') + .filter(Boolean) + .map((line) => { + const r = JSON.parse(line) as { d: string; region: string; revenue: number; orders: number }; + return { key: `${r.d}|${r.region}`, values: { revenue: Number(r.revenue), orders: Number(r.orders) } }; + }); +} + +/** Median and the spread around it. A single number is not a system property. */ +function distribution(samples: number[]): { + n: number; + min: number; + p50: number; + p95: number; + max: number; + spread_ratio: number; +} { + const s = [...samples].sort((a, b) => a - b); + const at = (q: number) => s[Math.min(s.length - 1, Math.floor(q * s.length))]; + return { + n: s.length, + min: s[0], + p50: at(0.5), + p95: at(0.95), + max: s[s.length - 1], + spread_ratio: Number((s[s.length - 1] / Math.max(s[0], 1e-9)).toFixed(2)), + }; +} + +async function timeQuery(query: string): Promise { + const t0 = performance.now(); + await ch(`${query} FORMAT JSONEachRow`); + return performance.now() - t0; +} + +const SOURCE_AGG = ` + SELECT toString(order_date) AS d, region, sum(amount) AS revenue, count() AS orders + FROM ${DB}.sales WHERE tenant_id = 't_alpha' GROUP BY d, region ORDER BY d, region`; +const TARGET_AGG = ` + SELECT toString(order_date) AS d, region, sum(revenue) AS revenue, sum(orders) AS orders + FROM ${DB}.sales_daily WHERE tenant_id = 't_alpha' GROUP BY d, region ORDER BY d, region`; + +const SPEC: MaterializationSpec = { + materialization_id: 'mv_sales_daily', + tenant_id: 't_alpha', + project_id: 'p_main', + dataset: 'ds_sales', + strategy: 'clickhouse_materialized_view', + target_ref: `${DB}.sales_daily`, + metrics: ['revenue', 'orders'], + dimensions: ['order_date', 'region'], + time_dimension: 'order_date', + grain: 'day', + max_staleness_seconds: 3600, + drift_check_interval_seconds: 86_400, + tolerance: { absolute: 0.5, relative: 0.0001 }, +}; + +const REQUEST: AnalyticsQueryRequest = { + dataset: 'ds_sales', + metrics: ['revenue'], + dimensions: ['order_date'], + filters: [], + timeRange: { grain: 'day' }, +}; + +const evidence: Record = {}; +let status: MaterializationStatus; + +run('BI-P105 materialization against a LIVE ClickHouse', () => { + beforeAll(async () => { + evidence.clickhouse_version = (await ch('SELECT version()')).trim(); + evidence.recorded_at = new Date().toISOString(); + evidence.source_rows = ROWS; + + await ch(`CREATE DATABASE IF NOT EXISTS ${DB}`); + await ch(` + CREATE TABLE ${DB}.sales ( + tenant_id LowCardinality(String), + order_date Date, + region LowCardinality(String), + customer_id UInt32, + amount Decimal(18, 2) + ) ENGINE = MergeTree ORDER BY (tenant_id, order_date, region)`); + await ch(` + INSERT INTO ${DB}.sales + SELECT + if(number % 5 = 0, 't_beta', 't_alpha') AS tenant_id, + toDate('2026-08-01') + toIntervalDay(number % 30) AS order_date, + ['SE','NO','DK','FI'][(number % 4) + 1] AS region, + toUInt32(number % 5000) AS customer_id, + toDecimal64((number % 977) + 13, 2) AS amount + FROM numbers(${ROWS})`); + + await ch(` + CREATE TABLE ${DB}.sales_daily ( + tenant_id LowCardinality(String), + order_date Date, + region LowCardinality(String), + revenue Decimal(38, 2), + orders UInt64 + ) ENGINE = SummingMergeTree ORDER BY (tenant_id, order_date, region)`); + await ch(` + CREATE MATERIALIZED VIEW ${DB}.sales_daily_mv TO ${DB}.sales_daily AS + SELECT tenant_id, order_date, region, sum(amount) AS revenue, count() AS orders + FROM ${DB}.sales GROUP BY tenant_id, order_date, region`); + // The MV is an INSERT trigger: it saw nothing that was already there, so + // the backfill is explicit. (This is drift cause #1, and it is why the + // first check below has to be able to fail.) + await ch(` + INSERT INTO ${DB}.sales_daily + SELECT tenant_id, order_date, region, sum(amount) AS revenue, count() AS orders + FROM ${DB}.sales GROUP BY tenant_id, order_date, region`); + await ch(`OPTIMIZE TABLE ${DB}.sales_daily FINAL`); + + status = { + materialization_id: SPEC.materialization_id, + state: 'serving', + data_watermark: new Date().toISOString(), + last_refresh_at: new Date().toISOString(), + }; + }, 300_000); + + afterAll(async () => { + if (!CLICKHOUSE_URL) return; + try { + await ch(`DROP DATABASE IF EXISTS ${DB}`); + } finally { + const dir = path.join(process.cwd(), 'evidence/bi-p105'); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync( + path.join(dir, 'materialization-drift.json'), + `${JSON.stringify(evidence, null, 2)}\n`, + ); + } + }, 60_000); + + it('the freshly backfilled aggregate MATCHES its source', async () => { + const report = detectDrift({ + fromSource: await bucketsFrom(SOURCE_AGG), + fromAggregate: await bucketsFrom(TARGET_AGG), + tolerance: SPEC.tolerance, + metrics: ['revenue', 'orders'], + }); + expect(report.buckets_compared).toBeGreaterThan(0); // an empty compare proves nothing + expect(report.verdict).toBe('match'); + + status = applyDriftReport(status, report); + evidence.after_backfill = { + verdict: report.verdict, + buckets_compared: report.buckets_compared, + revenue_total: report.totals.revenue, + }; + }, 120_000); + + it('measures the improvement as a DISTRIBUTION, not a single number', async () => { + const base: number[] = []; + const agg: number[] = []; + for (let i = 0; i < REPETITIONS; i += 1) { + // Interleaved so a machine that gets busy halfway through penalises both. + base.push(await timeQuery(SOURCE_AGG)); + agg.push(await timeQuery(TARGET_AGG)); + } + const baseD = distribution(base); + const aggD = distribution(agg); + evidence.performance = { + note: + 'ClickHouse caches marks and pages between repetitions, so these are WARM numbers on one host ' + + 'with nothing else running. They describe this shape of query on this data, not a system property.', + repetitions: REPETITIONS, + base_table_ms: baseD, + pre_aggregate_ms: aggD, + speedup_p50: Number((baseD.p50 / Math.max(aggD.p50, 1e-9)).toFixed(1)), + speedup_range_min_max: [ + Number((baseD.min / Math.max(aggD.max, 1e-9)).toFixed(1)), + Number((baseD.max / Math.max(aggD.min, 1e-9)).toFixed(1)), + ], + }; + // The claim under test is only that the pre-aggregate is not SLOWER; the + // magnitude is reported, never asserted as a threshold. + expect(aggD.p50).toBeLessThanOrEqual(baseD.p50); + }, 300_000); + + it('a GDPR erasure on the source DRIFTS the aggregate, and the drift is caught', async () => { + const before = await bucketsFrom(TARGET_AGG); + + // The mutation a right-to-erasure request produces. A materialized view is + // an insert trigger — it never sees this. + await ch( + `ALTER TABLE ${DB}.sales DELETE WHERE tenant_id = 't_alpha' AND customer_id < 250 SETTINGS mutations_sync = 2`, + ); + + const fromSource = await bucketsFrom(SOURCE_AGG); + const fromAggregate = await bucketsFrom(TARGET_AGG); + expect(fromAggregate).toEqual(before); // the aggregate genuinely did not move + + const report: DriftReport = detectDrift({ + fromSource, + fromAggregate, + tolerance: SPEC.tolerance, + metrics: ['revenue', 'orders'], + }); + expect(report.verdict).toBe('drifted'); + expect(report.worst).toBeDefined(); + + const drifted = applyDriftReport(status, report); + const guarded = selectQuerySource({ request: REQUEST, spec: SPEC, status: drifted, now: new Date() }); + expect(guarded.source).toBe('base_table'); + + // COUNTERFACTUAL: a router that checks only staleness serves it, and this + // is the number a dashboard would have shown. + const stalenessOnlyServes = + (materializationStaleness(drifted, new Date()) ?? 0) <= SPEC.max_staleness_seconds; + expect(stalenessOnlyServes).toBe(true); + + evidence.gdpr_erasure_drill = { + mutation: "ALTER TABLE sales DELETE WHERE tenant_id='t_alpha' AND customer_id < 250", + verdict: report.verdict, + buckets_compared: report.buckets_compared, + revenue_true_after_erasure: report.totals.revenue.source, + revenue_pre_aggregate_would_report: report.totals.revenue.aggregate, + revenue_overstatement: report.totals.revenue.delta, + revenue_overstatement_pct: Number( + ((report.totals.revenue.delta / report.totals.revenue.source) * 100).toFixed(3), + ), + worst_bucket: report.worst, + guarded_router_decision: guarded, + staleness_only_router_would_serve: stalenessOnlyServes, + staleness_seconds_at_decision: materializationStaleness(drifted, new Date()), + note: + 'The aggregate was NOT stale by any clock: it had just been refreshed, and every freshness check ' + + 'passed. Only the source comparison caught it. This is why staleness alone is not a correctness ' + + 'guarantee.', + }; + status = drifted; + }, 300_000); + + it('and a refresh that re-derives the aggregate clears the drift', async () => { + await ch(`TRUNCATE TABLE ${DB}.sales_daily`); + await ch(` + INSERT INTO ${DB}.sales_daily + SELECT tenant_id, order_date, region, sum(amount) AS revenue, count() AS orders + FROM ${DB}.sales GROUP BY tenant_id, order_date, region`); + await ch(`OPTIMIZE TABLE ${DB}.sales_daily FINAL`); + + const report = detectDrift({ + fromSource: await bucketsFrom(SOURCE_AGG), + fromAggregate: await bucketsFrom(TARGET_AGG), + tolerance: SPEC.tolerance, + metrics: ['revenue', 'orders'], + }); + expect(report.verdict).toBe('match'); + + const healed: MaterializationStatus = applyDriftReport( + { ...status, state: 'serving', data_watermark: new Date().toISOString() }, + report, + ); + const decision = selectQuerySource({ request: REQUEST, spec: SPEC, status: healed, now: new Date() }); + expect(decision.source).toBe('materialization'); + + evidence.after_rebuild = { + verdict: report.verdict, + revenue_total: report.totals.revenue, + routing: decision, + }; + }, 300_000); +});