BI-P060/P061/P103/P104/P105 — dashboard model + versioning, BI SLO, benchmark harness, materialization #36
@@ -0,0 +1,257 @@
|
||||
# BI-P060, P061, P103, P104, P105 — dashboards, SLOs, benchmarks, pre-aggregation
|
||||
|
||||
Extends the analytics core (`0pt#34`, BI-P002..P012) and the query/metric/quality
|
||||
layer (`0pt#35`). Nothing there is rebuilt: `DashboardDefinition` and
|
||||
`DashboardVersion` were already canonical entities with tables in `sql/008`, and
|
||||
these prompts give them behaviour.
|
||||
|
||||
Organised, like its predecessor, around what is **proven** and what is **not**.
|
||||
|
||||
---
|
||||
|
||||
## Where things live
|
||||
|
||||
| prompt | module | live proof |
|
||||
| ---------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| BI-P060 dashboard model | `src/analytics/dashboards/dashboard-document.ts` | unit (pure model + planner) |
|
||||
| BI-P061 dashboard versioning | `src/analytics/dashboards/dashboard-release.ts`, `sql/010` | `tests/analytics/dashboard-release-postgres.realinfra.test.ts`, Postgres 16.15 |
|
||||
| BI-P103 BI SLO | `src/analytics/slo/bi-slo.ts`, `slo/usage-events.ts` | unit; measurement path is `aud0#34`'s seven event types |
|
||||
| BI-P104 benchmark harness | `src/analytics/bench/benchmark-harness.ts` | `tests/integration/bi-p104-benchmark.integration.test.ts`, ClickHouse 26.3.10.60 + DuckDB 1.5.5 |
|
||||
| BI-P105 materialization | `src/analytics/materialization/` | `tests/integration/materialization-drift-clickhouse.integration.test.ts` |
|
||||
|
||||
Evidence: `evidence/bi-p061/dashboard-release-race.json`,
|
||||
`evidence/bi-p104/benchmark.json`, `evidence/bi-p105/materialization-drift.json`.
|
||||
|
||||
---
|
||||
|
||||
## BI-P060 — the dashboard is a document, not a Superset export
|
||||
|
||||
`layout_json` was a string, and a string holds anything. The canonical document
|
||||
carries `layout / filters / widgets / parameters / refresh / permissions /
|
||||
version`, and a widget references **a metric, a contract query and a
|
||||
visualization spec** — never a vendor's internal id.
|
||||
|
||||
The DoD is a function. `planDashboard(document, { parameters })` returns, per
|
||||
widget, a runnable `AnalyticsQueryRequest` (BI-P016) plus a render spec. If it
|
||||
returns a plan, the dashboard can be drawn by anything that executes the
|
||||
contract and draws the twelve canonical visualization types.
|
||||
|
||||
Rules that are enforced rather than documented:
|
||||
|
||||
- a widget whose only content is a vendor id is **refused** — that is exactly
|
||||
the document that cannot be reconstructed;
|
||||
- `vendor_ref` may carry a Superset/Metabase id for round-tripping, but it is
|
||||
excluded from the canonical hash, so re-exporting into a new Superset instance
|
||||
does not fabricate a new dashboard version;
|
||||
- vendor-native keys (`viz_type`, `form_data`, `position_json`, `slice_id`, …)
|
||||
are refused with a message that names the rule;
|
||||
- query widgets go through `parseAnalyticsQueryRequest`, so a dashboard cannot
|
||||
become an SQL surface and cannot name a tenant;
|
||||
- a locked filter cannot be switched off by a viewer — locked filters are how a
|
||||
saved dashboard pins its scope, and removing one silently widens the result.
|
||||
|
||||
---
|
||||
|
||||
## BI-P061 — releases are a ledger
|
||||
|
||||
`DRAFT → PUBLISHED → SUPERSEDED`, stored in `opt_analytics_dashboard_release`
|
||||
(`sql/010`) rather than as a column on the version row: `sql/008` already makes
|
||||
`DashboardVersion` immutable at the database, and a lifecycle column there would
|
||||
have meant weakening that. The **document** is frozen; the **release** moves.
|
||||
|
||||
The database enforces the state machine, not the application:
|
||||
|
||||
| attempted | result |
|
||||
| ------------------------------------------- | -------------------------------------------------- |
|
||||
| `published → draft` (un-publishing) | refused — it is how a rollback gets faked |
|
||||
| `superseded → published` | refused |
|
||||
| repointing a release at another version | refused (every content column is frozen on UPDATE) |
|
||||
| rewriting `document_hash` or `published_at` | refused |
|
||||
| `DELETE` on a release | refused — the ledger has no eraser |
|
||||
| a `published` row with no `published_at` | refused by CHECK |
|
||||
| a second open draft for one dashboard | refused by a partial unique index |
|
||||
|
||||
**Rollback mints a new version.** It does not resurrect the old release row.
|
||||
"We went back to v3" becomes a timestamped fact (v5 carries
|
||||
`rollback_of_version = 3`) instead of a hole in the history where v4 used to be.
|
||||
The restore is verified by BI-P060 canonical hash, and `documentOf()` refuses to
|
||||
serve a stored document whose hash no longer matches its release.
|
||||
|
||||
### The concurrency claim, measured both ways
|
||||
|
||||
`DashboardReleaseService.publish()` checks for a live release before writing.
|
||||
That check is a read-then-write race and is **not** the guarantee.
|
||||
`evidence/bi-p061/dashboard-release-race.json`, Postgres 16.15:
|
||||
|
||||
| run | partial unique index | transactions committed | published releases after |
|
||||
| --------------- | -------------------- | ---------------------- | --------------------------- |
|
||||
| `guard_absent` | dropped | 2 | **2 — two live dashboards** |
|
||||
| `guard_present` | in place | 1 (loser `23505`) | 1 |
|
||||
|
||||
Reproduce with `node scripts/bi-p061-release-race.mjs`.
|
||||
|
||||
---
|
||||
|
||||
## BI-P103 — SLOs measured from AUD0, or declared unmeasured
|
||||
|
||||
The measurement path is `aud0#34`'s seven BI usage event types (BI-P076). aud0's
|
||||
own doc records that zero producers emit them; `src/analytics/slo/usage-events.ts`
|
||||
makes 0pt the producer. There is deliberately no second telemetry path — two
|
||||
measurement systems disagree, and the disagreement surfaces during an incident.
|
||||
|
||||
| workload | objectives |
|
||||
| --------------------- | ---------------------------------------------------------------- |
|
||||
| interactive dashboard | availability, p95 first query, p95 cached, error rate, freshness |
|
||||
| exploration | p95, p99, max result, timeout |
|
||||
| report | completion SLA, delivery SLA |
|
||||
|
||||
Three verdicts that are not "met":
|
||||
|
||||
- **`insufficient_data`** — an availability objective over an empty window is
|
||||
silence, not 100%. A started query with no terminal event is _unresolved_, not
|
||||
a success.
|
||||
- **`not_measurable_here`** — dashboard **freshness** is a property of the
|
||||
dataset (BI-P026 / BI-P105 watermark) and report **delivery** is N0TIFY's
|
||||
receipt. Neither exists in the seven event types, so both are declared
|
||||
unmeasured rather than proxied by completion.
|
||||
- **`low_confidence`** — a p99 over 12 samples is the 12th-slowest request
|
||||
wearing a percentile's clothes. Reported with its sample size, not judged.
|
||||
|
||||
The producer is an allowlist built field by named field. `compiled_sql`,
|
||||
`error_message`, `tenant_id`, filter **values** and result samples exist on the
|
||||
0pt-side telemetry type and are never mapped; a `query_hash` that is not a
|
||||
64-hex digest is dropped rather than reshaped.
|
||||
|
||||
### Honest status
|
||||
|
||||
`BI_SLO_CATALOGUE`'s targets are **starting thresholds, not derived baselines**.
|
||||
No production BI traffic exists yet, so the first real baselines will come from
|
||||
the first weeks of emitted events. The evaluator is built so that absence shows
|
||||
up as `insufficient_data` until then, rather than as a green dashboard.
|
||||
|
||||
---
|
||||
|
||||
## BI-P104 — the harness refuses to quote one number
|
||||
|
||||
Six dataset profiles × two engines, 2 warmups discarded, 9 timed repetitions,
|
||||
statistics from the shipped module. `evidence/bi-p104/benchmark.json`.
|
||||
|
||||
- **Overlapping ranges are `indistinguishable`**, whatever the medians say.
|
||||
Three of the six real comparisons came out that way, one with a 1.6x median
|
||||
gap — the headline a noisy host manufactures for you.
|
||||
- **Every sentence carries the range, the sample size and the stability**, so it
|
||||
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
|
||||
- **Under 5 samples nothing is comparable.** A three-run benchmark is an anecdote.
|
||||
- **Every engine BI-P104 names appears**, and the five nothing measured carry a
|
||||
reason each.
|
||||
|
||||
Measured (warm cache, one idle host, these query shapes only):
|
||||
|
||||
| dataset | verdict, DuckDB 1.5.5 vs ClickHouse 26.3.10.60 |
|
||||
| --------------- | ------------------------------------------------- |
|
||||
| small_saas | faster, 1.6x–8.2x (median 4.4x) |
|
||||
| medium_business | indistinguishable (medians 1.6x apart) |
|
||||
| vehiq_market | faster, 2.4x–13.3x (median 5.8x) |
|
||||
| large_event | **slower**, 0.2x–0.9x (`count DISTINCT`, 5M rows) |
|
||||
| wide_table | indistinguishable |
|
||||
| geospatial | indistinguishable |
|
||||
|
||||
Not exercised, with reasons in the evidence: **trino** (the running instance
|
||||
belongs to another lane; benchmarking through it would measure their
|
||||
configuration), **semantic_query** (MetricFlow has never executed in 0pt),
|
||||
**superset**, **native_charts** (DES1GN owns rendering), **perspective**.
|
||||
|
||||
---
|
||||
|
||||
## BI-P105 — freshness semantics, and why freshness is not correctness
|
||||
|
||||
### The vocabulary
|
||||
|
||||
| term | definition |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------- |
|
||||
| `data_watermark` | the newest **source** timestamp the aggregate represents. Absent = **UNKNOWN**. |
|
||||
| staleness | `now − data_watermark`, in seconds. `undefined` when the watermark is unknown. |
|
||||
| `max_staleness_seconds` | the freshness contract. Older than this and the aggregate stops serving. |
|
||||
| `drift_check_interval_seconds` | how often the aggregate must be re-proved against its source to keep serving. |
|
||||
| `last_drift_verdict` | `match` / `drifted` / `insufficient_data` from the last comparison. |
|
||||
|
||||
**UNKNOWN is never treated as current.** This is the same doctrine BI-P026
|
||||
applies to result freshness: a missing watermark is a refusal, not a zero.
|
||||
|
||||
### The routing rule
|
||||
|
||||
A materialization serves **only** while it is provably current _and_ provably
|
||||
undrifted. Eight named refusals, each a real way an aggregate becomes a
|
||||
confident liar:
|
||||
|
||||
`coverage_miss` (metric/dimension/grain not carried — including a filter on a
|
||||
dimension the aggregate already summed across) · `watermark_unknown` ·
|
||||
`staleness_exceeded` · `window_beyond_watermark` (the question reaches past the
|
||||
newest data; answering would silently shorten the period) ·
|
||||
`drift_never_verified` · `drift_verification_stale` · `drift_detected` ·
|
||||
`state_not_serving`.
|
||||
|
||||
`drift_never_verified` is the one people leave out and the one that matters on
|
||||
launch day: a brand-new aggregate is exactly as unproven as a drifted one.
|
||||
|
||||
Grain rollup is restricted, not assumed: a `day` aggregate answers `day` and
|
||||
coarser, never `hour`; a `week` aggregate answers only `week`, because weekly
|
||||
buckets do not compose into calendar months.
|
||||
|
||||
### Drift detection
|
||||
|
||||
- **A missing bucket is drift** even when every value present is correct — late
|
||||
arrivals and a materialized view created after the backfill both look like
|
||||
this, and a value-only comparison passes while the total is wrong.
|
||||
- **An extra bucket is drift** — the source rows are gone and the aggregate is not.
|
||||
- **An empty comparison is `insufficient_data`, never `match`** (the BI-P025
|
||||
false green, restated for pre-aggregates), and it takes the aggregate out of
|
||||
service rather than annotating it.
|
||||
- **Both tolerances must hold.** 0.1% of a 40M bucket is 40 000 kr of invisible
|
||||
error; an absolute-only bound flags every rounding difference on a large bucket.
|
||||
|
||||
### The proof, on a live ClickHouse
|
||||
|
||||
`evidence/bi-p105/materialization-drift.json`, ClickHouse 26.3.10.60, 2M rows,
|
||||
a real materialized view:
|
||||
|
||||
| step | result |
|
||||
| ------------------------ | ----------------------------------------------------------------------------------- |
|
||||
| after backfill | `match`, 48 buckets, 801 571 002 on both sides |
|
||||
| GDPR erasure | `ALTER TABLE sales DELETE …`. An MV is an INSERT trigger and never sees a mutation. |
|
||||
| what would be served | 801 571 002 against a true 761 580 554 — **39 990 448 overstated, 5.251%** |
|
||||
| staleness at that moment | **1 second.** Every freshness check passed. |
|
||||
| guarded router | `base_table` (`state_not_serving`) |
|
||||
| counterfactual | a staleness-only router **serves it** — measured, not assumed |
|
||||
| after rebuild | `match`, and routing returns to the materialization |
|
||||
|
||||
That last row of the counterfactual is the whole point of the prompt: staleness
|
||||
is not correctness. The aggregate was one second old and wrong by 5%.
|
||||
|
||||
### Measured improvement, honestly
|
||||
|
||||
Same drill, base table vs pre-aggregate, 9 repetitions interleaved: p50 3.4x,
|
||||
but the run-to-run range spans **0.6x–7.5x** on an otherwise idle host — one
|
||||
sample had the pre-aggregate slower than the base table's best run. Quoting
|
||||
"3.4x" as a property of the system would be misleading, which is why BI-P104's
|
||||
harness exists.
|
||||
|
||||
---
|
||||
|
||||
## What is NOT proven here
|
||||
|
||||
- **No dashboard has been rendered.** BI-P060 produces a plan; drawing it is
|
||||
DES1GN (`DES1GN#9`) and `siax-portal#5`. Nothing in this PR proves a pixel.
|
||||
- **The release service has no Postgres adapter yet.** The lifecycle is proven
|
||||
in-memory and the _schema_ is proven against live Postgres; wiring
|
||||
`DashboardReleaseStore` to `postgres.js` is the next step and is the only
|
||||
place where the two could drift apart.
|
||||
- **Zero real SLO events have been emitted in production.** The producer and the
|
||||
evaluator are tested; no BI surface calls the producer yet, so every objective
|
||||
would today report `insufficient_data`. That is the honest empty state, and it
|
||||
is deliberately visible rather than hidden behind a default.
|
||||
- **No materialization is deployed.** The router and detector are proven against
|
||||
a live ClickHouse in a throwaway database. Nothing schedules a refresh or a
|
||||
periodic drift check yet — that belongs with ACT0/N0D scheduling.
|
||||
- **Trino, Superset, MetricFlow and Perspective are unmeasured**, for the
|
||||
reasons recorded in the benchmark evidence.
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"recorded_at": "2026-09-07T01:50:09.216Z",
|
||||
"postgres_version": "16.15",
|
||||
"guard_present": {
|
||||
"partial_unique_index_present": true,
|
||||
"transactions_committed": 1,
|
||||
"rejections": [
|
||||
{
|
||||
"code": "23505",
|
||||
"constraint": "opt_analytics_dashboard_release_one_published"
|
||||
}
|
||||
],
|
||||
"published_releases_after": [
|
||||
{
|
||||
"release_id": "rel_1",
|
||||
"version": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"guard_absent": {
|
||||
"partial_unique_index_present": false,
|
||||
"transactions_committed": 2,
|
||||
"rejections": [],
|
||||
"published_releases_after": [
|
||||
{
|
||||
"release_id": "rel_1",
|
||||
"version": 1
|
||||
},
|
||||
{
|
||||
"release_id": "rel_2",
|
||||
"version": 2
|
||||
}
|
||||
]
|
||||
},
|
||||
"verdict": {
|
||||
"counterfactual_produced_two_live_dashboards": true,
|
||||
"guard_left_exactly_one": true,
|
||||
"loser_error_code": "23505",
|
||||
"application_check_alone_is_sufficient": false,
|
||||
"note": "DashboardReleaseService.publish() reads for a live release before writing. Both sessions in guard_absent passed that check and both committed. The partial unique index is what holds."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,638 @@
|
||||
{
|
||||
"recorded_at": "2026-09-07T02:06:11.079Z",
|
||||
"host_note": "One idle host, warm caches, both engines local, single-threaded client. These numbers describe these query shapes on this data on this machine. They are not engine rankings.",
|
||||
"datasets": [
|
||||
{
|
||||
"id": "small_saas",
|
||||
"label": "small SaaS",
|
||||
"approx_rows": 100000,
|
||||
"shape": "narrow fact table, one tenant, high selectivity"
|
||||
},
|
||||
{
|
||||
"id": "medium_business",
|
||||
"label": "medium business",
|
||||
"approx_rows": 2000000,
|
||||
"shape": "multi-tenant fact table with a date and a few dimensions"
|
||||
},
|
||||
{
|
||||
"id": "vehiq_market",
|
||||
"label": "VehIQ market",
|
||||
"approx_rows": 1000000,
|
||||
"shape": "vehicle listings: make/model/year/price/mileage, skewed cardinalities"
|
||||
},
|
||||
{
|
||||
"id": "large_event",
|
||||
"label": "large event",
|
||||
"approx_rows": 5000000,
|
||||
"shape": "append-only event stream, timestamp-ordered, sparse dimensions"
|
||||
},
|
||||
{
|
||||
"id": "wide_table",
|
||||
"label": "wide table",
|
||||
"approx_rows": 200000,
|
||||
"shape": "many columns, few read per query — the columnar-store best case"
|
||||
},
|
||||
{
|
||||
"id": "geospatial",
|
||||
"label": "geospatial",
|
||||
"approx_rows": 500000,
|
||||
"shape": "lat/lon points with bounding-box and distance predicates"
|
||||
}
|
||||
],
|
||||
"engines": [
|
||||
{
|
||||
"engine": "clickhouse",
|
||||
"available": true,
|
||||
"version": "26.3.10.60"
|
||||
},
|
||||
{
|
||||
"engine": "duckdb",
|
||||
"available": true,
|
||||
"version": "1.5.5"
|
||||
},
|
||||
{
|
||||
"engine": "trino",
|
||||
"available": false,
|
||||
"reason": "a Trino is running on this host but belongs to another lane (st0re-verify-trino); benchmarking through someone else’s container would measure their configuration"
|
||||
},
|
||||
{
|
||||
"engine": "semantic_query",
|
||||
"available": false,
|
||||
"reason": "MetricFlow has never been executed in 0pt (BI-P029 honest status)"
|
||||
},
|
||||
{
|
||||
"engine": "superset",
|
||||
"available": false,
|
||||
"reason": "no Superset deployment in the estate yet"
|
||||
},
|
||||
{
|
||||
"engine": "native_charts",
|
||||
"available": false,
|
||||
"reason": "render-side; DES1GN owns it (BI-P051..P057)"
|
||||
},
|
||||
{
|
||||
"engine": "perspective",
|
||||
"available": false,
|
||||
"reason": "not deployed"
|
||||
}
|
||||
],
|
||||
"runs": [
|
||||
{
|
||||
"engine": "clickhouse",
|
||||
"dataset": "small_saas",
|
||||
"query_label": "revenue by day",
|
||||
"warmup_ms": [58.856714000000466, 58.77232499999991],
|
||||
"samples_ms": [
|
||||
51.25258499999927, 23.924909000000298, 39.89413500000046, 29.40832799999953, 43.058938000000126,
|
||||
39.98396799999955, 41.99595799999952, 32.193271000000095, 25.963267999999516
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 23.924909000000298,
|
||||
"p50": 39.89413500000046,
|
||||
"p95": 51.25258499999927,
|
||||
"p99": 51.25258499999927,
|
||||
"max": 51.25258499999927,
|
||||
"mean": 36.40837333333315,
|
||||
"stdev": 8.514314573768598,
|
||||
"cv": 0.23385594560395923,
|
||||
"iqr": 12.58762999999999,
|
||||
"spread_ratio": 2.1422269568506462,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 365
|
||||
},
|
||||
{
|
||||
"engine": "clickhouse",
|
||||
"dataset": "medium_business",
|
||||
"query_label": "revenue by month and region for one tenant",
|
||||
"warmup_ms": [44.60071400000015, 70.7402890000003],
|
||||
"samples_ms": [
|
||||
70.63034199999947, 69.65554000000066, 58.036339000000225, 46.82895400000052, 44.67711700000018,
|
||||
87.97018200000002, 49.249023999999736, 59.75650299999961, 41.953865000000405
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 41.953865000000405,
|
||||
"p50": 58.036339000000225,
|
||||
"p95": 87.97018200000002,
|
||||
"p99": 87.97018200000002,
|
||||
"max": 87.97018200000002,
|
||||
"mean": 58.750874000000096,
|
||||
"stdev": 14.251513712332908,
|
||||
"cv": 0.24257534811027467,
|
||||
"iqr": 22.826586000000134,
|
||||
"spread_ratio": 2.096831412314436,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 120
|
||||
},
|
||||
{
|
||||
"engine": "clickhouse",
|
||||
"dataset": "vehiq_market",
|
||||
"query_label": "average asking price by make and model year",
|
||||
"warmup_ms": [83.03972700000031, 59.77780099999927],
|
||||
"samples_ms": [
|
||||
74.38707999999951, 60.41443299999992, 59.13954199999989, 98.55565299999944, 98.45707200000015,
|
||||
74.05828699999984, 194.5833190000003, 159.76290199999949, 87.88670300000013
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 59.13954199999989,
|
||||
"p50": 87.88670300000013,
|
||||
"p95": 194.5833190000003,
|
||||
"p99": 194.5833190000003,
|
||||
"max": 194.5833190000003,
|
||||
"mean": 100.80499899999985,
|
||||
"stdev": 43.75254087132061,
|
||||
"cv": 0.4340314598021143,
|
||||
"iqr": 24.4973659999996,
|
||||
"spread_ratio": 3.290240546671813,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 60
|
||||
},
|
||||
{
|
||||
"engine": "clickhouse",
|
||||
"dataset": "large_event",
|
||||
"query_label": "hourly event volume for one event type",
|
||||
"warmup_ms": [122.44560499999989, 136.30929500000002],
|
||||
"samples_ms": [
|
||||
142.78732200000013, 136.18330800000058, 82.08186000000023, 95.91282499999943, 74.52815499999997,
|
||||
93.95668399999977, 112.81592300000011, 76.3508299999994, 93.78873100000055
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 74.52815499999997,
|
||||
"p50": 93.95668399999977,
|
||||
"p95": 142.78732200000013,
|
||||
"p99": 142.78732200000013,
|
||||
"max": 142.78732200000013,
|
||||
"mean": 100.9339597777778,
|
||||
"stdev": 23.374602687288522,
|
||||
"cv": 0.23158313355337923,
|
||||
"iqr": 30.73406299999988,
|
||||
"spread_ratio": 1.9158842990276652,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 720
|
||||
},
|
||||
{
|
||||
"engine": "clickhouse",
|
||||
"dataset": "wide_table",
|
||||
"query_label": "three columns of fifty, grouped",
|
||||
"warmup_ms": [28.096958000000086, 14.89937800000007],
|
||||
"samples_ms": [
|
||||
19.301219999999375, 21.628866000000926, 32.125191999999515, 25.563218999999663, 24.388967999999295,
|
||||
16.8507410000002, 19.57230399999935, 20.52187000000049, 18.078355999999985
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 16.8507410000002,
|
||||
"p50": 20.52187000000049,
|
||||
"p95": 32.125191999999515,
|
||||
"p99": 32.125191999999515,
|
||||
"max": 32.125191999999515,
|
||||
"mean": 22.00341511111098,
|
||||
"stdev": 4.451179566867264,
|
||||
"cv": 0.2022949412348072,
|
||||
"iqr": 5.08774799999992,
|
||||
"spread_ratio": 1.9064557457739775,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 500
|
||||
},
|
||||
{
|
||||
"engine": "clickhouse",
|
||||
"dataset": "geospatial",
|
||||
"query_label": "bounding box, aggregated to a grid cell",
|
||||
"warmup_ms": [38.77805500000068, 17.665412000000288],
|
||||
"samples_ms": [
|
||||
29.736737000001085, 27.89697900000101, 25.586840999998458, 21.336174000000028, 22.492867999999362,
|
||||
30.176470000000336, 22.63948900000105, 29.021173000001, 36.31744599999911
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 21.336174000000028,
|
||||
"p50": 27.89697900000101,
|
||||
"p95": 36.31744599999911,
|
||||
"p99": 36.31744599999911,
|
||||
"max": 36.31744599999911,
|
||||
"mean": 27.244908555555714,
|
||||
"stdev": 4.498748211227398,
|
||||
"cv": 0.16512252929951657,
|
||||
"iqr": 7.097248000000036,
|
||||
"spread_ratio": 1.7021536288558137,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 201
|
||||
},
|
||||
{
|
||||
"engine": "duckdb",
|
||||
"dataset": "small_saas",
|
||||
"query_label": "revenue by day",
|
||||
"warmup_ms": [12.830814997869311, 8.479544001602335],
|
||||
"samples_ms": [
|
||||
7.375242999842158, 9.8867060005432, 9.773318997758906, 9.169243003270822, 7.295589999557706,
|
||||
10.63253999745939, 6.214401000761427, 15.184157000476262, 7.2293630000785924
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 6.214401000761427,
|
||||
"p50": 9.169243003270822,
|
||||
"p95": 15.184157000476262,
|
||||
"p99": 15.184157000476262,
|
||||
"max": 15.184157000476262,
|
||||
"mean": 9.19561799997205,
|
||||
"stdev": 2.5477019265669423,
|
||||
"cv": 0.27705608547187216,
|
||||
"iqr": 2.591116000985494,
|
||||
"spread_ratio": 2.443382234042477,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 365
|
||||
},
|
||||
{
|
||||
"engine": "duckdb",
|
||||
"dataset": "medium_business",
|
||||
"query_label": "revenue by month and region for one tenant",
|
||||
"warmup_ms": [60.29315100022359, 47.50260100263404],
|
||||
"samples_ms": [
|
||||
47.711714996694354, 36.605457000405295, 30.622688002040377, 44.33469899959164, 49.469069999759085,
|
||||
33.921502999874065, 33.03168200000073, 40.499283000826836, 33.27470799922594
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 30.622688002040377,
|
||||
"p50": 36.605457000405295,
|
||||
"p95": 49.469069999759085,
|
||||
"p99": 49.469069999759085,
|
||||
"max": 49.469069999759085,
|
||||
"mean": 38.8300894442687,
|
||||
"stdev": 6.549355811539825,
|
||||
"cv": 0.1686670287211122,
|
||||
"iqr": 11.059991000365699,
|
||||
"spread_ratio": 1.615438527031428,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 120
|
||||
},
|
||||
{
|
||||
"engine": "duckdb",
|
||||
"dataset": "vehiq_market",
|
||||
"query_label": "average asking price by make and model year",
|
||||
"warmup_ms": [17.676940999081125, 24.05106900187093],
|
||||
"samples_ms": [
|
||||
14.901875001669396, 14.926752002793364, 17.798165001295274, 15.059515000757528, 19.75441800095723,
|
||||
14.601105001929682, 14.954296002542833, 20.32349000000977, 24.610873999336036
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 14.601105001929682,
|
||||
"p50": 15.059515000757528,
|
||||
"p95": 24.610873999336036,
|
||||
"p99": 24.610873999336036,
|
||||
"max": 24.610873999336036,
|
||||
"mean": 17.436721112365678,
|
||||
"stdev": 3.2978767006322265,
|
||||
"cv": 0.18913399367805778,
|
||||
"iqr": 4.827665998163866,
|
||||
"spread_ratio": 1.6855487304613908,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 60
|
||||
},
|
||||
{
|
||||
"engine": "duckdb",
|
||||
"dataset": "large_event",
|
||||
"query_label": "hourly event volume for one event type",
|
||||
"warmup_ms": [184.43432099957136, 163.9354379985889],
|
||||
"samples_ms": [
|
||||
160.28642499804846, 161.1321640011738, 177.48030700022355, 213.59406000192394, 337.0364080001309,
|
||||
334.6067929996934, 232.15600899857236, 204.8815359994478, 196.73219499964034
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 160.28642499804846,
|
||||
"p50": 204.8815359994478,
|
||||
"p95": 337.0364080001309,
|
||||
"p99": 337.0364080001309,
|
||||
"max": 337.0364080001309,
|
||||
"mean": 224.21176633320607,
|
||||
"stdev": 63.65742323059742,
|
||||
"cv": 0.28391651460429923,
|
||||
"iqr": 54.67570199834881,
|
||||
"spread_ratio": 2.1027133645549485,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 720
|
||||
},
|
||||
{
|
||||
"engine": "duckdb",
|
||||
"dataset": "wide_table",
|
||||
"query_label": "three columns of fifty, grouped",
|
||||
"warmup_ms": [18.729354997049086, 21.23752199986484],
|
||||
"samples_ms": [
|
||||
14.388244999281596, 15.164796001045033, 18.07164799902239, 22.90704599727178, 19.191682997188764,
|
||||
16.838232000736753, 16.024758999265032, 24.479674000758678, 22.105899999587564
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 14.388244999281596,
|
||||
"p50": 18.07164799902239,
|
||||
"p95": 24.479674000758678,
|
||||
"p99": 24.479674000758678,
|
||||
"max": 24.479674000758678,
|
||||
"mean": 18.796886999350843,
|
||||
"stdev": 3.4151315245862714,
|
||||
"cv": 0.1816860166635153,
|
||||
"iqr": 6.081141000322532,
|
||||
"spread_ratio": 1.7013662195758374,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 500
|
||||
},
|
||||
{
|
||||
"engine": "duckdb",
|
||||
"dataset": "geospatial",
|
||||
"query_label": "bounding box, aggregated to a grid cell",
|
||||
"warmup_ms": [27.98272599829943, 22.078424000937957],
|
||||
"samples_ms": [
|
||||
30.961295000452083, 28.811156000301708, 19.97572500113165, 22.212601001228904, 33.500631998322206,
|
||||
34.46119200089015, 24.307676001626533, 20.643488001951482, 24.275364998175064
|
||||
],
|
||||
"distribution": {
|
||||
"n": 9,
|
||||
"min": 19.97572500113165,
|
||||
"p50": 24.307676001626533,
|
||||
"p95": 34.46119200089015,
|
||||
"p99": 34.46119200089015,
|
||||
"max": 34.46119200089015,
|
||||
"mean": 26.572125556008864,
|
||||
"stdev": 5.191758162396437,
|
||||
"cv": 0.19538362301703047,
|
||||
"iqr": 8.748693999223178,
|
||||
"spread_ratio": 1.7251535050136042,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"rows_returned": 201
|
||||
}
|
||||
],
|
||||
"comparisons": [
|
||||
{
|
||||
"dataset": "small_saas",
|
||||
"query_label": "revenue by day",
|
||||
"description": "duckdb vs clickhouse (small_saas): FASTER — 1.6x to 8.2x (median 4.4x) over n=9/9, stability noisy/noisy. Warm cache, one host, this query shape.",
|
||||
"p50_ratio": 4.350864622714171,
|
||||
"conservative_ratio": 1.5756494745971,
|
||||
"optimistic_ratio": 8.247389409489264,
|
||||
"ranges_overlap": false,
|
||||
"baseline": {
|
||||
"n": 9,
|
||||
"min": 23.924909000000298,
|
||||
"p50": 39.89413500000046,
|
||||
"p95": 51.25258499999927,
|
||||
"p99": 51.25258499999927,
|
||||
"max": 51.25258499999927,
|
||||
"mean": 36.40837333333315,
|
||||
"stdev": 8.514314573768598,
|
||||
"cv": 0.23385594560395923,
|
||||
"iqr": 12.58762999999999,
|
||||
"spread_ratio": 2.1422269568506462,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"candidate": {
|
||||
"n": 9,
|
||||
"min": 6.214401000761427,
|
||||
"p50": 9.169243003270822,
|
||||
"p95": 15.184157000476262,
|
||||
"p99": 15.184157000476262,
|
||||
"max": 15.184157000476262,
|
||||
"mean": 9.19561799997205,
|
||||
"stdev": 2.5477019265669423,
|
||||
"cv": 0.27705608547187216,
|
||||
"iqr": 2.591116000985494,
|
||||
"spread_ratio": 2.443382234042477,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"verdict": "faster"
|
||||
},
|
||||
{
|
||||
"dataset": "medium_business",
|
||||
"query_label": "revenue by month and region for one tenant",
|
||||
"description": "duckdb vs clickhouse (medium_business): INDISTINGUISHABLE at this sample size. Medians differ by 1.6x but the observed ranges overlap (baseline 42.0-88.0ms, candidate 30.6-49.5ms; n=9/9, CV 0.24/0.17).",
|
||||
"p50_ratio": 1.5854559335062433,
|
||||
"conservative_ratio": 0.8480827515092707,
|
||||
"optimistic_ratio": 2.872712610798197,
|
||||
"ranges_overlap": true,
|
||||
"baseline": {
|
||||
"n": 9,
|
||||
"min": 41.953865000000405,
|
||||
"p50": 58.036339000000225,
|
||||
"p95": 87.97018200000002,
|
||||
"p99": 87.97018200000002,
|
||||
"max": 87.97018200000002,
|
||||
"mean": 58.750874000000096,
|
||||
"stdev": 14.251513712332908,
|
||||
"cv": 0.24257534811027467,
|
||||
"iqr": 22.826586000000134,
|
||||
"spread_ratio": 2.096831412314436,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"candidate": {
|
||||
"n": 9,
|
||||
"min": 30.622688002040377,
|
||||
"p50": 36.605457000405295,
|
||||
"p95": 49.469069999759085,
|
||||
"p99": 49.469069999759085,
|
||||
"max": 49.469069999759085,
|
||||
"mean": 38.8300894442687,
|
||||
"stdev": 6.549355811539825,
|
||||
"cv": 0.1686670287211122,
|
||||
"iqr": 11.059991000365699,
|
||||
"spread_ratio": 1.615438527031428,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"verdict": "indistinguishable"
|
||||
},
|
||||
{
|
||||
"dataset": "vehiq_market",
|
||||
"query_label": "average asking price by make and model year",
|
||||
"description": "duckdb vs clickhouse (vehiq_market): FASTER — 2.4x to 13.3x (median 5.8x) over n=9/9, stability noisy/noisy. Warm cache, one host, this query shape.",
|
||||
"p50_ratio": 5.835958395444954,
|
||||
"conservative_ratio": 2.4029842256555125,
|
||||
"optimistic_ratio": 13.32661596326334,
|
||||
"ranges_overlap": false,
|
||||
"baseline": {
|
||||
"n": 9,
|
||||
"min": 59.13954199999989,
|
||||
"p50": 87.88670300000013,
|
||||
"p95": 194.5833190000003,
|
||||
"p99": 194.5833190000003,
|
||||
"max": 194.5833190000003,
|
||||
"mean": 100.80499899999985,
|
||||
"stdev": 43.75254087132061,
|
||||
"cv": 0.4340314598021143,
|
||||
"iqr": 24.4973659999996,
|
||||
"spread_ratio": 3.290240546671813,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"candidate": {
|
||||
"n": 9,
|
||||
"min": 14.601105001929682,
|
||||
"p50": 15.059515000757528,
|
||||
"p95": 24.610873999336036,
|
||||
"p99": 24.610873999336036,
|
||||
"max": 24.610873999336036,
|
||||
"mean": 17.436721112365678,
|
||||
"stdev": 3.2978767006322265,
|
||||
"cv": 0.18913399367805778,
|
||||
"iqr": 4.827665998163866,
|
||||
"spread_ratio": 1.6855487304613908,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"verdict": "faster"
|
||||
},
|
||||
{
|
||||
"dataset": "large_event",
|
||||
"query_label": "hourly event volume for one event type",
|
||||
"description": "duckdb vs clickhouse (large_event): SLOWER — 0.2x to 0.9x (median 0.5x) over n=9/9, stability noisy/noisy. Warm cache, one host, this query shape.",
|
||||
"p50_ratio": 0.4585902948338546,
|
||||
"conservative_ratio": 0.2211279055643479,
|
||||
"optimistic_ratio": 0.8908260446992852,
|
||||
"ranges_overlap": false,
|
||||
"baseline": {
|
||||
"n": 9,
|
||||
"min": 74.52815499999997,
|
||||
"p50": 93.95668399999977,
|
||||
"p95": 142.78732200000013,
|
||||
"p99": 142.78732200000013,
|
||||
"max": 142.78732200000013,
|
||||
"mean": 100.9339597777778,
|
||||
"stdev": 23.374602687288522,
|
||||
"cv": 0.23158313355337923,
|
||||
"iqr": 30.73406299999988,
|
||||
"spread_ratio": 1.9158842990276652,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"candidate": {
|
||||
"n": 9,
|
||||
"min": 160.28642499804846,
|
||||
"p50": 204.8815359994478,
|
||||
"p95": 337.0364080001309,
|
||||
"p99": 337.0364080001309,
|
||||
"max": 337.0364080001309,
|
||||
"mean": 224.21176633320607,
|
||||
"stdev": 63.65742323059742,
|
||||
"cv": 0.28391651460429923,
|
||||
"iqr": 54.67570199834881,
|
||||
"spread_ratio": 2.1027133645549485,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"verdict": "slower"
|
||||
},
|
||||
{
|
||||
"dataset": "wide_table",
|
||||
"query_label": "three columns of fifty, grouped",
|
||||
"description": "duckdb vs clickhouse (wide_table): INDISTINGUISHABLE at this sample size. Medians differ by 1.1x but the observed ranges overlap (baseline 16.9-32.1ms, candidate 14.4-24.5ms; n=9/9, CV 0.20/0.18).",
|
||||
"p50_ratio": 1.1355837608784018,
|
||||
"conservative_ratio": 0.688356429888648,
|
||||
"optimistic_ratio": 2.232738739269697,
|
||||
"ranges_overlap": true,
|
||||
"baseline": {
|
||||
"n": 9,
|
||||
"min": 16.8507410000002,
|
||||
"p50": 20.52187000000049,
|
||||
"p95": 32.125191999999515,
|
||||
"p99": 32.125191999999515,
|
||||
"max": 32.125191999999515,
|
||||
"mean": 22.00341511111098,
|
||||
"stdev": 4.451179566867264,
|
||||
"cv": 0.2022949412348072,
|
||||
"iqr": 5.08774799999992,
|
||||
"spread_ratio": 1.9064557457739775,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"candidate": {
|
||||
"n": 9,
|
||||
"min": 14.388244999281596,
|
||||
"p50": 18.07164799902239,
|
||||
"p95": 24.479674000758678,
|
||||
"p99": 24.479674000758678,
|
||||
"max": 24.479674000758678,
|
||||
"mean": 18.796886999350843,
|
||||
"stdev": 3.4151315245862714,
|
||||
"cv": 0.1816860166635153,
|
||||
"iqr": 6.081141000322532,
|
||||
"spread_ratio": 1.7013662195758374,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"verdict": "indistinguishable"
|
||||
},
|
||||
{
|
||||
"dataset": "geospatial",
|
||||
"query_label": "bounding box, aggregated to a grid cell",
|
||||
"description": "duckdb vs clickhouse (geospatial): INDISTINGUISHABLE at this sample size. Medians differ by 1.1x but the observed ranges overlap (baseline 21.3-36.3ms, candidate 20.0-34.5ms; n=9/9, CV 0.17/0.20).",
|
||||
"p50_ratio": 1.1476612983542442,
|
||||
"conservative_ratio": 0.619136273621902,
|
||||
"optimistic_ratio": 1.8180789932751718,
|
||||
"ranges_overlap": true,
|
||||
"baseline": {
|
||||
"n": 9,
|
||||
"min": 21.336174000000028,
|
||||
"p50": 27.89697900000101,
|
||||
"p95": 36.31744599999911,
|
||||
"p99": 36.31744599999911,
|
||||
"max": 36.31744599999911,
|
||||
"mean": 27.244908555555714,
|
||||
"stdev": 4.498748211227398,
|
||||
"cv": 0.16512252929951657,
|
||||
"iqr": 7.097248000000036,
|
||||
"spread_ratio": 1.7021536288558137,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"candidate": {
|
||||
"n": 9,
|
||||
"min": 19.97572500113165,
|
||||
"p50": 24.307676001626533,
|
||||
"p95": 34.46119200089015,
|
||||
"p99": 34.46119200089015,
|
||||
"max": 34.46119200089015,
|
||||
"mean": 26.572125556008864,
|
||||
"stdev": 5.191758162396437,
|
||||
"cv": 0.19538362301703047,
|
||||
"iqr": 8.748693999223178,
|
||||
"spread_ratio": 1.7251535050136042,
|
||||
"stability": "noisy"
|
||||
},
|
||||
"verdict": "indistinguishable"
|
||||
}
|
||||
],
|
||||
"not_exercised": [
|
||||
{
|
||||
"engine": "trino",
|
||||
"available": false,
|
||||
"reason": "a Trino is running on this host but belongs to another lane (st0re-verify-trino); benchmarking through someone else’s container would measure their configuration"
|
||||
},
|
||||
{
|
||||
"engine": "semantic_query",
|
||||
"available": false,
|
||||
"reason": "MetricFlow has never been executed in 0pt (BI-P029 honest status)"
|
||||
},
|
||||
{
|
||||
"engine": "superset",
|
||||
"available": false,
|
||||
"reason": "no Superset deployment in the estate yet"
|
||||
},
|
||||
{
|
||||
"engine": "native_charts",
|
||||
"available": false,
|
||||
"reason": "render-side; DES1GN owns it (BI-P051..P057)"
|
||||
},
|
||||
{
|
||||
"engine": "perspective",
|
||||
"available": false,
|
||||
"reason": "not deployed"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"clickhouse_version": "26.3.10.60",
|
||||
"recorded_at": "2026-09-07T01:55:42.821Z",
|
||||
"source_rows": 2000000,
|
||||
"after_backfill": {
|
||||
"verdict": "match",
|
||||
"buckets_compared": 48,
|
||||
"revenue_total": {
|
||||
"source": 801571002,
|
||||
"aggregate": 801571002,
|
||||
"delta": 0
|
||||
}
|
||||
},
|
||||
"performance": {
|
||||
"note": "ClickHouse caches marks and pages between repetitions, so these are WARM numbers on one host with nothing else running. They describe this shape of query on this data, not a system property.",
|
||||
"repetitions": 9,
|
||||
"base_table_ms": {
|
||||
"n": 9,
|
||||
"min": 48.70366200000012,
|
||||
"p50": 76.67811500000016,
|
||||
"p95": 115.04416399999991,
|
||||
"max": 115.04416399999991,
|
||||
"spread_ratio": 2.36
|
||||
},
|
||||
"pre_aggregate_ms": {
|
||||
"n": 9,
|
||||
"min": 15.281603999999788,
|
||||
"p50": 22.727642999999944,
|
||||
"p95": 84.41279499999973,
|
||||
"max": 84.41279499999973,
|
||||
"spread_ratio": 5.52
|
||||
},
|
||||
"speedup_p50": 3.4,
|
||||
"speedup_range_min_max": [0.6, 7.5]
|
||||
},
|
||||
"gdpr_erasure_drill": {
|
||||
"mutation": "ALTER TABLE sales DELETE WHERE tenant_id='t_alpha' AND customer_id < 250",
|
||||
"verdict": "drifted",
|
||||
"buckets_compared": 48,
|
||||
"revenue_true_after_erasure": 761580554,
|
||||
"revenue_pre_aggregate_would_report": 801571002,
|
||||
"revenue_overstatement": 39990448,
|
||||
"revenue_overstatement_pct": 5.251,
|
||||
"worst_bucket": {
|
||||
"key": "2026-08-28|FI",
|
||||
"metric": "revenue",
|
||||
"source_value": 15830779,
|
||||
"aggregate_value": 16699145,
|
||||
"absolute_delta": 868366,
|
||||
"relative_delta": 0.05485301765630106
|
||||
},
|
||||
"guarded_router_decision": {
|
||||
"source": "base_table",
|
||||
"refusal": "state_not_serving",
|
||||
"detail": "materialization state is \"drifted\""
|
||||
},
|
||||
"staleness_only_router_would_serve": true,
|
||||
"staleness_seconds_at_decision": 1,
|
||||
"note": "The aggregate was NOT stale by any clock: it had just been refreshed, and every freshness check passed. Only the source comparison caught it. This is why staleness alone is not a correctness guarantee."
|
||||
},
|
||||
"after_rebuild": {
|
||||
"verdict": "match",
|
||||
"revenue_total": {
|
||||
"source": 761580554,
|
||||
"aggregate": 761580554,
|
||||
"delta": 0
|
||||
},
|
||||
"routing": {
|
||||
"source": "materialization",
|
||||
"materialization_id": "mv_sales_daily",
|
||||
"target_ref": "opt_p105_mtql9p2f.sales_daily",
|
||||
"staleness_seconds": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* BI-P061: is "only one published release per dashboard" actually enforced?
|
||||
*
|
||||
* The application checks for a live release before publishing, and that check
|
||||
* is a read-then-write race. This script measures BOTH directions against a
|
||||
* real Postgres and writes evidence/bi-p061/dashboard-release-race.json:
|
||||
*
|
||||
* guard_absent the partial unique index dropped -> both concurrent
|
||||
* publishes commit, and the dashboard has TWO live versions.
|
||||
* guard_present the index in place -> the loser fails with 23505.
|
||||
*
|
||||
* Without the first run, the second proves nothing: a race that never
|
||||
* interleaved would also show one published release.
|
||||
*
|
||||
* ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25441/opt \
|
||||
* node scripts/bi-p061-release-race.mjs
|
||||
*
|
||||
* It CREATEs and DROPs its own throwaway schemas and touches nothing else.
|
||||
*/
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import postgres from 'postgres';
|
||||
|
||||
const URL_ = process.env.ANALYTICS_TEST_DATABASE_URL;
|
||||
if (!URL_) {
|
||||
console.error('ANALYTICS_TEST_DATABASE_URL is required. Refusing to invent numbers.');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const STAMP = Date.now().toString(36);
|
||||
const SCHEMAS = { guard_present: `bi_p061_ev_on_${STAMP}`, guard_absent: `bi_p061_ev_off_${STAMP}` };
|
||||
const MIGRATIONS = ['sql/008_analytics_domain_model.sql', 'sql/010_analytics_dashboard_releases.sql'];
|
||||
const HASH = (c) => c.repeat(64);
|
||||
|
||||
const connect = (schema, max = 1) =>
|
||||
postgres(URL_, { max, onnotice: () => undefined, connection: { search_path: schema } });
|
||||
|
||||
async function setup(schema, { dropIndex }) {
|
||||
const admin = postgres(URL_, { max: 1, onnotice: () => undefined });
|
||||
await admin.unsafe(`CREATE SCHEMA IF NOT EXISTS ${schema}`);
|
||||
await admin.end({ timeout: 5 });
|
||||
|
||||
const c = connect(schema, 2);
|
||||
for (const f of MIGRATIONS) await c.unsafe(fs.readFileSync(path.join(process.cwd(), f), 'utf8'));
|
||||
if (dropIndex) await c.unsafe('DROP INDEX opt_analytics_dashboard_release_one_published');
|
||||
await c.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_definition
|
||||
(dashboard_id, tenant_id, project_id, name, current_version, owner_id)
|
||||
VALUES ('dash_race', 't_alpha', 'p_main', 'Race', 1, 'u_1');
|
||||
INSERT INTO opt_analytics_dashboard_version
|
||||
(dashboard_version_id, dashboard_id, tenant_id, project_id, version, layout_json, layout_hash)
|
||||
VALUES ('dv_1', 'dash_race', 't_alpha', 'p_main', 1, '{}', '${HASH('a')}'),
|
||||
('dv_2', 'dash_race', 't_alpha', 'p_main', 2, '{}', '${HASH('b')}');`);
|
||||
return c;
|
||||
}
|
||||
|
||||
/** Both sessions read "nothing is live", then both write. */
|
||||
async function race(schema) {
|
||||
const conns = [connect(schema), connect(schema)];
|
||||
const publish = (conn, n) =>
|
||||
conn.begin(async (tx) => {
|
||||
await tx.unsafe(
|
||||
`SELECT release_id FROM opt_analytics_dashboard_release
|
||||
WHERE dashboard_id='dash_race' AND state='published'`,
|
||||
);
|
||||
await new Promise((r) => setTimeout(r, 60));
|
||||
await tx.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_release
|
||||
(release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state,
|
||||
document_hash, published_at)
|
||||
VALUES ('rel_${n}', 'dash_race', 'dv_${n}', 't_alpha', 'p_main', ${n},
|
||||
'published', '${HASH(n === 1 ? 'a' : 'b')}', now())`);
|
||||
});
|
||||
const results = await Promise.allSettled([publish(conns[0], 1), publish(conns[1], 2)]);
|
||||
for (const c of conns) await c.end({ timeout: 5 });
|
||||
return {
|
||||
committed: results.filter((r) => r.status === 'fulfilled').length,
|
||||
rejected: results
|
||||
.filter((r) => r.status === 'rejected')
|
||||
.map((r) => ({ code: r.reason?.code ?? null, constraint: r.reason?.constraint_name ?? null })),
|
||||
};
|
||||
}
|
||||
|
||||
async function publishedRows(conn) {
|
||||
const rows = await conn.unsafe(
|
||||
`SELECT release_id, version FROM opt_analytics_dashboard_release
|
||||
WHERE dashboard_id='dash_race' AND state='published' ORDER BY version`,
|
||||
);
|
||||
return rows.map((r) => ({ release_id: r.release_id, version: r.version }));
|
||||
}
|
||||
|
||||
const out = { recorded_at: new Date().toISOString() };
|
||||
const cleanup = [];
|
||||
try {
|
||||
for (const [label, schema] of Object.entries(SCHEMAS)) {
|
||||
const conn = await setup(schema, { dropIndex: label === 'guard_absent' });
|
||||
cleanup.push([schema, conn]);
|
||||
if (!out.postgres_version) {
|
||||
const v = await conn.unsafe('SHOW server_version');
|
||||
out.postgres_version = v[0].server_version;
|
||||
}
|
||||
const result = await race(schema);
|
||||
out[label] = {
|
||||
partial_unique_index_present: label === 'guard_present',
|
||||
transactions_committed: result.committed,
|
||||
rejections: result.rejected,
|
||||
published_releases_after: await publishedRows(conn),
|
||||
};
|
||||
}
|
||||
|
||||
out.verdict = {
|
||||
counterfactual_produced_two_live_dashboards: out.guard_absent.published_releases_after.length === 2,
|
||||
guard_left_exactly_one: out.guard_present.published_releases_after.length === 1,
|
||||
loser_error_code: out.guard_present.rejections[0]?.code ?? null,
|
||||
application_check_alone_is_sufficient: false,
|
||||
note:
|
||||
'DashboardReleaseService.publish() reads for a live release before writing. Both sessions in ' +
|
||||
'guard_absent passed that check and both committed. The partial unique index is what holds.',
|
||||
};
|
||||
|
||||
const dir = path.join(process.cwd(), 'evidence/bi-p061');
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
const file = path.join(dir, 'dashboard-release-race.json');
|
||||
fs.writeFileSync(file, `${JSON.stringify(out, null, 2)}\n`);
|
||||
console.log(`wrote ${file}`);
|
||||
console.log(JSON.stringify(out.verdict, null, 2));
|
||||
} finally {
|
||||
for (const [schema, conn] of cleanup) {
|
||||
await conn.unsafe(`DROP SCHEMA IF EXISTS ${schema} CASCADE`);
|
||||
await conn.end({ timeout: 5 });
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
-- 0pt analytics: dashboard releases (BI-P061 — DASHBOARD VERSIONING).
|
||||
--
|
||||
-- Extends sql/008. Additive and idempotent (CREATE TABLE IF NOT EXISTS,
|
||||
-- CREATE OR REPLACE FUNCTION, DROP TRIGGER IF EXISTS before CREATE TRIGGER),
|
||||
-- so it replays onto an already-migrated database like 009 does.
|
||||
--
|
||||
-- Ordering: requires sql/008 (it references opt_analytics_dashboard_version).
|
||||
--
|
||||
-- WHY A SEPARATE TABLE RATHER THAN A `state` COLUMN ON THE VERSION.
|
||||
-- sql/008 puts a BEFORE UPDATE OR DELETE trigger on
|
||||
-- opt_analytics_dashboard_version, because BI-P002 lists DashboardVersion as
|
||||
-- immutable. A lifecycle column on that row would need UPDATEs, so adding one
|
||||
-- would mean weakening the immutability everything else depends on. Instead
|
||||
-- the frozen DOCUMENT stays immutable in the version table, and the RELEASE —
|
||||
-- the thing that legitimately moves DRAFT -> PUBLISHED -> SUPERSEDED — lives
|
||||
-- here with its own, much narrower, mutation rule:
|
||||
--
|
||||
-- * `state` may only advance draft -> published -> superseded. Any other
|
||||
-- transition RAISEs, including published -> draft ("un-publishing" is how
|
||||
-- a rollback gets faked without leaving a trace) and superseded -> *.
|
||||
-- * Every content column is frozen on UPDATE. A release may change state and
|
||||
-- nothing else; repointing a published release at a different version is
|
||||
-- the exact edit that would make "immutable release" a lie.
|
||||
-- * DELETE is refused outright. Release history is a ledger; a rollback is
|
||||
-- an entry in it, never the removal of one.
|
||||
--
|
||||
-- THE INVARIANT THAT NEEDS THE DATABASE: at most ONE published release per
|
||||
-- dashboard. An application check reads, decides, then writes, and two
|
||||
-- requests that interleave between the read and the write both decide "yes".
|
||||
-- The partial unique index below turns that race into a constraint violation
|
||||
-- for the loser. tests/analytics/dashboard-release-postgres.realinfra.test.ts
|
||||
-- runs the race with the index DROPPED first and shows two published releases
|
||||
-- for one dashboard, so the index is known to be what prevents it.
|
||||
|
||||
-- ── the release ledger ────────────────────────────────────────────────────
|
||||
CREATE TABLE IF NOT EXISTS opt_analytics_dashboard_release (
|
||||
release_id TEXT PRIMARY KEY,
|
||||
dashboard_id TEXT NOT NULL REFERENCES opt_analytics_dashboard_definition (dashboard_id),
|
||||
dashboard_version_id TEXT NOT NULL REFERENCES opt_analytics_dashboard_version (dashboard_version_id),
|
||||
tenant_id TEXT NOT NULL,
|
||||
project_id TEXT NOT NULL,
|
||||
version INTEGER NOT NULL CHECK (version >= 1),
|
||||
state TEXT NOT NULL CHECK (state IN ('draft', 'published', 'superseded')),
|
||||
-- The BI-P060 canonical hash of the released document. Copied here so a
|
||||
-- rollback can be VERIFIED (the new release must carry the hash of the
|
||||
-- version it restores) without re-reading and re-hashing the document.
|
||||
document_hash TEXT NOT NULL CHECK (document_hash ~ '^[0-9a-f]{64}$'),
|
||||
-- Set when this release restores an earlier version. It is a NEW release of
|
||||
-- a NEW version, never a resurrected old row -- that is what keeps the
|
||||
-- history linear and the audit answerable ("what was live at 14:05?").
|
||||
rollback_of_version INTEGER CHECK (rollback_of_version >= 1),
|
||||
created_by TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
published_at TIMESTAMPTZ,
|
||||
superseded_at TIMESTAMPTZ,
|
||||
superseded_by_release_id TEXT,
|
||||
UNIQUE (dashboard_id, version),
|
||||
-- A state's timestamps must match the state. A 'published' row with no
|
||||
-- published_at is the shape that makes an SLO or an audit answer wrong.
|
||||
CONSTRAINT opt_analytics_dashboard_release_timestamps CHECK (
|
||||
(state = 'draft' AND published_at IS NULL AND superseded_at IS NULL)
|
||||
OR (state = 'published' AND published_at IS NOT NULL AND superseded_at IS NULL)
|
||||
OR (state = 'superseded' AND published_at IS NOT NULL AND superseded_at IS NOT NULL)
|
||||
)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS opt_analytics_dashboard_release_tenant_idx
|
||||
ON opt_analytics_dashboard_release (tenant_id, project_id, dashboard_id, version);
|
||||
|
||||
-- At most one PUBLISHED release per dashboard. This is the concurrency
|
||||
-- guarantee; the application's check is only an early, friendlier error.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS opt_analytics_dashboard_release_one_published
|
||||
ON opt_analytics_dashboard_release (dashboard_id)
|
||||
WHERE state = 'published';
|
||||
|
||||
-- At most one open DRAFT per dashboard. Two drafts race to publish and the
|
||||
-- loser's edits vanish into a superseded release nobody asked for.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS opt_analytics_dashboard_release_one_draft
|
||||
ON opt_analytics_dashboard_release (dashboard_id)
|
||||
WHERE state = 'draft';
|
||||
|
||||
-- ── transition enforcement ────────────────────────────────────────────────
|
||||
CREATE OR REPLACE FUNCTION opt_analytics_dashboard_release_guard() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
IF TG_OP = 'DELETE' THEN
|
||||
RAISE EXCEPTION
|
||||
'BI-P061: dashboard releases are a ledger; delete is refused (release %)', OLD.release_id
|
||||
USING ERRCODE = 'restrict_violation';
|
||||
END IF;
|
||||
|
||||
-- Content is frozen for the life of the release.
|
||||
IF NEW.dashboard_id IS DISTINCT FROM OLD.dashboard_id
|
||||
OR NEW.dashboard_version_id IS DISTINCT FROM OLD.dashboard_version_id
|
||||
OR NEW.tenant_id IS DISTINCT FROM OLD.tenant_id
|
||||
OR NEW.project_id IS DISTINCT FROM OLD.project_id
|
||||
OR NEW.version IS DISTINCT FROM OLD.version
|
||||
OR NEW.document_hash IS DISTINCT FROM OLD.document_hash
|
||||
OR NEW.rollback_of_version IS DISTINCT FROM OLD.rollback_of_version
|
||||
OR NEW.created_at IS DISTINCT FROM OLD.created_at THEN
|
||||
RAISE EXCEPTION
|
||||
'BI-P061: a dashboard release is immutable apart from its state (release %)', OLD.release_id
|
||||
USING ERRCODE = 'restrict_violation';
|
||||
END IF;
|
||||
|
||||
IF NEW.state IS DISTINCT FROM OLD.state THEN
|
||||
IF NOT ((OLD.state = 'draft' AND NEW.state = 'published')
|
||||
OR (OLD.state = 'published' AND NEW.state = 'superseded')) THEN
|
||||
RAISE EXCEPTION
|
||||
'BI-P061: illegal dashboard release transition % -> % (release %)',
|
||||
OLD.state, NEW.state, OLD.release_id
|
||||
USING ERRCODE = 'restrict_violation';
|
||||
END IF;
|
||||
END IF;
|
||||
|
||||
-- A published timestamp, once set, is history.
|
||||
IF OLD.published_at IS NOT NULL AND NEW.published_at IS DISTINCT FROM OLD.published_at THEN
|
||||
RAISE EXCEPTION
|
||||
'BI-P061: published_at is history and cannot be rewritten (release %)', OLD.release_id
|
||||
USING ERRCODE = 'restrict_violation';
|
||||
END IF;
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS opt_analytics_dashboard_release_guard ON opt_analytics_dashboard_release;
|
||||
CREATE TRIGGER opt_analytics_dashboard_release_guard
|
||||
BEFORE UPDATE OR DELETE ON opt_analytics_dashboard_release
|
||||
FOR EACH ROW EXECUTE FUNCTION opt_analytics_dashboard_release_guard();
|
||||
@@ -0,0 +1,341 @@
|
||||
/**
|
||||
* src/analytics/bench/benchmark-harness.ts
|
||||
* =========================================
|
||||
* BI-P104 — BI PERFORMANCE BENCHMARK HARNESS. DoD: "results committed as
|
||||
* machine-readable evidence."
|
||||
*
|
||||
* The harness exists because of a specific way benchmarks lie. Yesterday's
|
||||
* lane measured the same budget-enforcement overshoot at ~150x on one run and
|
||||
* ~130x on the next, on the same machine, and correctly reported the SHAPE
|
||||
* rather than a number. That is the doctrine encoded here:
|
||||
*
|
||||
* 1. A SINGLE FIGURE IS NOT A SYSTEM PROPERTY. `summarizeSamples` returns a
|
||||
* distribution — n, min, p50, p95, p99, max, mean, stdev, CV, IQR — and
|
||||
* there is no function anywhere in this module that returns one number
|
||||
* for a measurement. `describeComparison` renders a RANGE and the sample
|
||||
* size into every sentence it produces.
|
||||
*
|
||||
* 2. OVERLAPPING DISTRIBUTIONS ARE INDISTINGUISHABLE, whatever the medians
|
||||
* say. If the candidate's p95 is not below the baseline's minimum, the
|
||||
* verdict is `indistinguishable` even when p50 differs by 2x — because a
|
||||
* 2x median gap inside a 10x spread is noise with a story attached.
|
||||
*
|
||||
* 3. TOO FEW SAMPLES IS A REFUSAL. Under MIN_SAMPLES the summary carries
|
||||
* `stability: 'insufficient_samples'` and comparisons refuse to judge.
|
||||
* A three-run benchmark is an anecdote.
|
||||
*
|
||||
* 4. AN ENGINE THAT WAS NOT RUN IS RECORDED AS NOT RUN. `EngineAvailability`
|
||||
* carries a reason. The evidence file lists every engine BI-P104 names,
|
||||
* including the ones nothing measured — an omitted engine reads as "we
|
||||
* chose not to report it".
|
||||
*/
|
||||
|
||||
/** The six dataset profiles BI-P104 names, with what makes each one distinct. */
|
||||
export const BENCH_DATASETS = [
|
||||
{
|
||||
id: 'small_saas',
|
||||
label: 'small SaaS',
|
||||
approx_rows: 100_000,
|
||||
shape: 'narrow fact table, one tenant, high selectivity',
|
||||
},
|
||||
{
|
||||
id: 'medium_business',
|
||||
label: 'medium business',
|
||||
approx_rows: 2_000_000,
|
||||
shape: 'multi-tenant fact table with a date and a few dimensions',
|
||||
},
|
||||
{
|
||||
id: 'vehiq_market',
|
||||
label: 'VehIQ market',
|
||||
approx_rows: 1_000_000,
|
||||
shape: 'vehicle listings: make/model/year/price/mileage, skewed cardinalities',
|
||||
},
|
||||
{
|
||||
id: 'large_event',
|
||||
label: 'large event',
|
||||
approx_rows: 5_000_000,
|
||||
shape: 'append-only event stream, timestamp-ordered, sparse dimensions',
|
||||
},
|
||||
{
|
||||
id: 'wide_table',
|
||||
label: 'wide table',
|
||||
approx_rows: 200_000,
|
||||
shape: 'many columns, few read per query — the columnar-store best case',
|
||||
},
|
||||
{
|
||||
id: 'geospatial',
|
||||
label: 'geospatial',
|
||||
approx_rows: 500_000,
|
||||
shape: 'lat/lon points with bounding-box and distance predicates',
|
||||
},
|
||||
] as const;
|
||||
export type BenchDatasetId = (typeof BENCH_DATASETS)[number]['id'];
|
||||
|
||||
/** Every engine BI-P104 lists. All of them appear in the evidence, run or not. */
|
||||
export const BENCH_ENGINES = [
|
||||
'clickhouse',
|
||||
'duckdb',
|
||||
'trino',
|
||||
'semantic_query',
|
||||
'superset',
|
||||
'native_charts',
|
||||
'perspective',
|
||||
] as const;
|
||||
export type BenchEngine = (typeof BENCH_ENGINES)[number];
|
||||
|
||||
export interface EngineAvailability {
|
||||
engine: BenchEngine;
|
||||
available: boolean;
|
||||
/** Required when unavailable. "Unknown" is not a reason. */
|
||||
reason?: string;
|
||||
version?: string;
|
||||
}
|
||||
|
||||
export const STABILITY_VERDICTS = ['stable', 'noisy', 'unstable', 'insufficient_samples'] as const;
|
||||
export type StabilityVerdict = (typeof STABILITY_VERDICTS)[number];
|
||||
|
||||
/** Below this a run is an anecdote, not a measurement. */
|
||||
export const MIN_SAMPLES = 5;
|
||||
/** Coefficient of variation thresholds. Wide on purpose: real hosts are noisy. */
|
||||
const CV_STABLE = 0.15;
|
||||
const CV_NOISY = 0.5;
|
||||
|
||||
export interface Distribution {
|
||||
n: number;
|
||||
min: number;
|
||||
p50: number;
|
||||
p95: number;
|
||||
p99: number;
|
||||
max: number;
|
||||
mean: number;
|
||||
stdev: number;
|
||||
/** stdev / mean. The number that decides whether a comparison is meaningful. */
|
||||
cv: number;
|
||||
iqr: number;
|
||||
/** max / min. Reported because it is what a reader feels. */
|
||||
spread_ratio: number;
|
||||
stability: StabilityVerdict;
|
||||
}
|
||||
|
||||
/** Nearest-rank; no interpolation, so every reported value really happened. */
|
||||
function pct(sorted: number[], q: number): number {
|
||||
const rank = Math.ceil(q * sorted.length);
|
||||
return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))];
|
||||
}
|
||||
|
||||
export function summarizeSamples(samples: readonly number[]): Distribution {
|
||||
if (samples.length === 0) {
|
||||
return {
|
||||
n: 0,
|
||||
min: 0,
|
||||
p50: 0,
|
||||
p95: 0,
|
||||
p99: 0,
|
||||
max: 0,
|
||||
mean: 0,
|
||||
stdev: 0,
|
||||
cv: 0,
|
||||
iqr: 0,
|
||||
spread_ratio: 0,
|
||||
stability: 'insufficient_samples',
|
||||
};
|
||||
}
|
||||
const s = [...samples].sort((a, b) => a - b);
|
||||
const mean = s.reduce((a, b) => a + b, 0) / s.length;
|
||||
const variance = s.reduce((acc, v) => acc + (v - mean) ** 2, 0) / s.length;
|
||||
const stdev = Math.sqrt(variance);
|
||||
const cv = mean === 0 ? 0 : stdev / mean;
|
||||
const stability: StabilityVerdict =
|
||||
s.length < MIN_SAMPLES
|
||||
? 'insufficient_samples'
|
||||
: cv <= CV_STABLE
|
||||
? 'stable'
|
||||
: cv <= CV_NOISY
|
||||
? 'noisy'
|
||||
: 'unstable';
|
||||
return {
|
||||
n: s.length,
|
||||
min: s[0],
|
||||
p50: pct(s, 0.5),
|
||||
p95: pct(s, 0.95),
|
||||
p99: pct(s, 0.99),
|
||||
max: s[s.length - 1],
|
||||
mean,
|
||||
stdev,
|
||||
cv,
|
||||
iqr: pct(s, 0.75) - pct(s, 0.25),
|
||||
spread_ratio: s[0] === 0 ? Number.POSITIVE_INFINITY : s[s.length - 1] / s[0],
|
||||
stability,
|
||||
};
|
||||
}
|
||||
|
||||
export const COMPARISON_VERDICTS = ['faster', 'slower', 'indistinguishable', 'not_comparable'] as const;
|
||||
export type ComparisonVerdict = (typeof COMPARISON_VERDICTS)[number];
|
||||
|
||||
export interface Comparison {
|
||||
verdict: ComparisonVerdict;
|
||||
/** Ratio of medians. Present for information; never the claim on its own. */
|
||||
p50_ratio: number;
|
||||
/** Worst case for the candidate: baseline.min / candidate.max. */
|
||||
conservative_ratio: number;
|
||||
/** Best case for the candidate: baseline.max / candidate.min. */
|
||||
optimistic_ratio: number;
|
||||
/** True when the observed ranges overlap at all. */
|
||||
ranges_overlap: boolean;
|
||||
baseline: Distribution;
|
||||
candidate: Distribution;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare two measured distributions.
|
||||
*
|
||||
* `faster` requires clean separation — the candidate's p95 below the
|
||||
* baseline's minimum. Anything less is `indistinguishable`, however flattering
|
||||
* the medians look, because that is exactly the claim a noisy host will
|
||||
* manufacture for you.
|
||||
*/
|
||||
export function compareDistributions(baseline: Distribution, candidate: Distribution): Comparison {
|
||||
const ratio = (a: number, b: number) => (b === 0 ? Number.POSITIVE_INFINITY : a / b);
|
||||
const base: Omit<Comparison, 'verdict'> = {
|
||||
p50_ratio: ratio(baseline.p50, candidate.p50),
|
||||
conservative_ratio: ratio(baseline.min, candidate.max),
|
||||
optimistic_ratio: ratio(baseline.max, candidate.min),
|
||||
ranges_overlap: !(candidate.max < baseline.min || baseline.max < candidate.min),
|
||||
baseline,
|
||||
candidate,
|
||||
};
|
||||
if (baseline.stability === 'insufficient_samples' || candidate.stability === 'insufficient_samples') {
|
||||
return { ...base, verdict: 'not_comparable' };
|
||||
}
|
||||
if (candidate.p95 < baseline.min) return { ...base, verdict: 'faster' };
|
||||
if (baseline.p95 < candidate.min) return { ...base, verdict: 'slower' };
|
||||
return { ...base, verdict: 'indistinguishable' };
|
||||
}
|
||||
|
||||
/**
|
||||
* The sentence a human is allowed to quote. It ALWAYS carries the range, the
|
||||
* sample size and the stability verdict, so it cannot be trimmed down to "3.4x
|
||||
* faster" without visibly removing the caveats.
|
||||
*/
|
||||
export function describeComparison(c: Comparison, label = 'candidate'): string {
|
||||
const r = (v: number) => (Number.isFinite(v) ? v.toFixed(1) : '∞');
|
||||
if (c.verdict === 'not_comparable') {
|
||||
return (
|
||||
`${label}: NOT COMPARABLE — ${c.baseline.n} baseline and ${c.candidate.n} candidate samples, ` +
|
||||
`fewer than the ${MIN_SAMPLES} required. A run this short is an anecdote.`
|
||||
);
|
||||
}
|
||||
if (c.verdict === 'indistinguishable') {
|
||||
return (
|
||||
`${label}: INDISTINGUISHABLE at this sample size. Medians differ by ${r(c.p50_ratio)}x but the ` +
|
||||
`observed ranges overlap (baseline ${c.baseline.min.toFixed(1)}-${c.baseline.max.toFixed(1)}ms, ` +
|
||||
`candidate ${c.candidate.min.toFixed(1)}-${c.candidate.max.toFixed(1)}ms; n=${c.baseline.n}/${c.candidate.n}, ` +
|
||||
`CV ${c.baseline.cv.toFixed(2)}/${c.candidate.cv.toFixed(2)}).`
|
||||
);
|
||||
}
|
||||
return (
|
||||
`${label}: ${c.verdict.toUpperCase()} — ${r(c.conservative_ratio)}x to ${r(c.optimistic_ratio)}x ` +
|
||||
`(median ${r(c.p50_ratio)}x) over n=${c.baseline.n}/${c.candidate.n}, ` +
|
||||
`stability ${c.baseline.stability}/${c.candidate.stability}. Warm cache, one host, this query shape.`
|
||||
);
|
||||
}
|
||||
|
||||
export interface BenchmarkRun {
|
||||
engine: BenchEngine;
|
||||
dataset: BenchDatasetId;
|
||||
query_label: string;
|
||||
/** Discarded from the statistics; recorded so the discard is visible. */
|
||||
warmup_ms: number[];
|
||||
samples_ms: number[];
|
||||
distribution: Distribution;
|
||||
rows_returned?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one measurement. Warmups are executed and RECORDED but excluded from the
|
||||
* distribution: a first run that pays for cold marks and page cache is a
|
||||
* different measurement, and silently folding it in is how a p95 becomes a
|
||||
* story about disk.
|
||||
*/
|
||||
export async function measure(args: {
|
||||
engine: BenchEngine;
|
||||
dataset: BenchDatasetId;
|
||||
query_label: string;
|
||||
run: () => Promise<{ rows?: number }>;
|
||||
warmups?: number;
|
||||
repetitions?: number;
|
||||
now?: () => number;
|
||||
}): Promise<BenchmarkRun> {
|
||||
const now = args.now ?? (() => performance.now());
|
||||
const warmups = args.warmups ?? 2;
|
||||
const repetitions = args.repetitions ?? 9;
|
||||
const warmup_ms: number[] = [];
|
||||
const samples_ms: number[] = [];
|
||||
let rows: number | undefined;
|
||||
|
||||
for (let i = 0; i < warmups; i += 1) {
|
||||
const t0 = now();
|
||||
await args.run();
|
||||
warmup_ms.push(now() - t0);
|
||||
}
|
||||
for (let i = 0; i < repetitions; i += 1) {
|
||||
const t0 = now();
|
||||
const out = await args.run();
|
||||
samples_ms.push(now() - t0);
|
||||
rows = out.rows ?? rows;
|
||||
}
|
||||
return {
|
||||
engine: args.engine,
|
||||
dataset: args.dataset,
|
||||
query_label: args.query_label,
|
||||
warmup_ms,
|
||||
samples_ms,
|
||||
distribution: summarizeSamples(samples_ms),
|
||||
...(rows !== undefined ? { rows_returned: rows } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
export interface BenchmarkEvidence {
|
||||
recorded_at: string;
|
||||
host_note: string;
|
||||
datasets: typeof BENCH_DATASETS;
|
||||
engines: EngineAvailability[];
|
||||
runs: BenchmarkRun[];
|
||||
comparisons: Array<{ dataset: BenchDatasetId; query_label: string; description: string } & Comparison>;
|
||||
/** Engines BI-P104 names that produced no measurement, each with a reason. */
|
||||
not_exercised: EngineAvailability[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Assemble the evidence document. Every engine BI-P104 names appears, and the
|
||||
* ones with no runs are listed under `not_exercised` with their reason — an
|
||||
* engine quietly missing from a benchmark table reads as a choice.
|
||||
*/
|
||||
export function buildEvidence(args: {
|
||||
host_note: string;
|
||||
engines: EngineAvailability[];
|
||||
runs: BenchmarkRun[];
|
||||
comparisons?: BenchmarkEvidence['comparisons'];
|
||||
now?: Date;
|
||||
}): BenchmarkEvidence {
|
||||
const measured = new Set(args.runs.map((r) => r.engine));
|
||||
const declared = new Map(args.engines.map((e) => [e.engine, e]));
|
||||
return {
|
||||
recorded_at: (args.now ?? new Date()).toISOString(),
|
||||
host_note: args.host_note,
|
||||
datasets: BENCH_DATASETS,
|
||||
engines: BENCH_ENGINES.map(
|
||||
(engine) => declared.get(engine) ?? { engine, available: false, reason: 'not declared by this run' },
|
||||
),
|
||||
runs: args.runs,
|
||||
comparisons: args.comparisons ?? [],
|
||||
not_exercised: BENCH_ENGINES.filter((e) => !measured.has(e)).map((engine) => {
|
||||
const d = declared.get(engine);
|
||||
if (!d) return { engine, available: false, reason: 'not present in this environment' };
|
||||
// An engine declared AVAILABLE that produced no measurement is the
|
||||
// dangerous case: it reads as "we could have, and chose not to say".
|
||||
// It gets a reason too, generated here rather than left blank.
|
||||
return d.reason ? d : { ...d, reason: 'declared available but produced no measurement in this run' };
|
||||
}),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,837 @@
|
||||
/**
|
||||
* src/analytics/dashboards/dashboard-document.ts
|
||||
* ===============================================
|
||||
* BI-P060 — ANALYTICS DASHBOARD MODEL.
|
||||
*
|
||||
* BI-P002 already named `DashboardDefinition` and `DashboardVersion` and gave
|
||||
* them storage (sql/008). What it did not give them is a BODY: `layout_json`
|
||||
* is a string, and a string can hold anything — including a Superset export
|
||||
* that only Superset can read back. This module is that body, and its whole
|
||||
* point is the DoD:
|
||||
*
|
||||
* "A native SIAX dashboard can be reconstructed without Superset."
|
||||
*
|
||||
* Reconstruction is not a claim you can make by writing a type. It is provable
|
||||
* exactly once you can take a document and produce, for every widget, a
|
||||
* runnable `AnalyticsQueryRequest` (BI-P016) plus a rendering spec, with no
|
||||
* vendor identifier anywhere in the plan. `planDashboard()` is that function
|
||||
* and `tests/analytics/dashboard-model.test.ts` runs it over a document
|
||||
* carrying Superset ids, asserting the plan is complete and vendor-free.
|
||||
*
|
||||
* THE VENDOR RULE, stated once: a Superset/Metabase/Cube identifier may be
|
||||
* carried as `vendor_ref` for round-tripping, and it is
|
||||
* - never a widget's source of truth (a widget with only a vendor_ref is
|
||||
* REFUSED — it is precisely the document that cannot be reconstructed),
|
||||
* - never part of the canonical hash (so re-exporting to a new Superset
|
||||
* instance does not fabricate a new dashboard version).
|
||||
*
|
||||
* NO SQL LIVES HERE either. Widget sources are metric/dimension NAMES that
|
||||
* resolve through the catalog, and query widgets are validated by
|
||||
* `parseAnalyticsQueryRequest`, which rejects `sql`, `where`, `tenant_id` and
|
||||
* every other smuggling key structurally. A dashboard is therefore not a new
|
||||
* way into the engine; it is a saved set of contract requests.
|
||||
*/
|
||||
import crypto from 'node:crypto';
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import {
|
||||
parseAnalyticsQueryRequest,
|
||||
TIME_GRAINS,
|
||||
type AnalyticsFilter,
|
||||
type AnalyticsQueryRequest,
|
||||
type FilterOperator,
|
||||
type TimeGrain,
|
||||
} from '../contract/query-request';
|
||||
|
||||
// ── vocabulary ─────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Visualization types. A CLOSED list, deliberately engine-neutral: these are
|
||||
* SIAX chart kinds (DES1GN BI-P051..P057 renders them), not Superset viz_type
|
||||
* strings. A document naming `echarts_timeseries` is a Superset document.
|
||||
*/
|
||||
export const VISUALIZATION_TYPES = [
|
||||
'big_number',
|
||||
'line',
|
||||
'area',
|
||||
'bar',
|
||||
'stacked_bar',
|
||||
'pie',
|
||||
'table',
|
||||
'pivot_table',
|
||||
'scatter',
|
||||
'heatmap',
|
||||
'funnel',
|
||||
'map',
|
||||
] as const;
|
||||
export type VisualizationType = (typeof VISUALIZATION_TYPES)[number];
|
||||
|
||||
export const REFRESH_MODES = ['manual', 'interval', 'scheduled'] as const;
|
||||
export type RefreshMode = (typeof REFRESH_MODES)[number];
|
||||
|
||||
/**
|
||||
* Who may open the dashboard. `roles` are cl0ud role names (BI-P033 decides,
|
||||
* this only declares). There is deliberately NO tenant field: a dashboard is
|
||||
* stored inside a tenant, and a document that could name one would be a way to
|
||||
* ask for another tenant's dashboard — the INF0 Wave-2 shape.
|
||||
*/
|
||||
export const DASHBOARD_VISIBILITIES = ['private', 'tenant', 'roles'] as const;
|
||||
export type DashboardVisibility = (typeof DASHBOARD_VISIBILITIES)[number];
|
||||
|
||||
export const PARAMETER_TYPES = ['string', 'number', 'date', 'enum'] as const;
|
||||
export type ParameterType = (typeof PARAMETER_TYPES)[number];
|
||||
|
||||
// ── the document ───────────────────────────────────────────────────────────
|
||||
|
||||
/** Grid placement. A 12-column grid, matching the portal layout (BI-P062). */
|
||||
export interface WidgetLayout {
|
||||
x: number;
|
||||
y: number;
|
||||
w: number;
|
||||
h: number;
|
||||
}
|
||||
|
||||
/** A saved dashboard filter. Its VALUES are data; its dimension is a name. */
|
||||
export interface DashboardFilter {
|
||||
filter_id: string;
|
||||
label: string;
|
||||
/** Caller-facing dimension name, resolved through the catalog downstream. */
|
||||
dimension: string;
|
||||
operator: FilterOperator;
|
||||
/** Default values, or `{{param}}` references to a declared parameter. */
|
||||
default_values?: string[];
|
||||
/** May a viewer change it? A locked filter is how row scoping is saved. */
|
||||
locked?: boolean;
|
||||
}
|
||||
|
||||
/** A declared parameter. Widgets and filters may reference it as `{{name}}`. */
|
||||
export interface DashboardParameter {
|
||||
name: string;
|
||||
type: ParameterType;
|
||||
required: boolean;
|
||||
default_value?: string;
|
||||
/** Required for `enum`; a closed list is the only way a value can be trusted. */
|
||||
allowed_values?: string[];
|
||||
}
|
||||
|
||||
/** A widget backed by ONE metric — the common case, and the cheapest to plan. */
|
||||
export interface MetricWidgetSource {
|
||||
kind: 'metric';
|
||||
dataset: string;
|
||||
metric: string;
|
||||
dimensions?: string[];
|
||||
grain?: TimeGrain;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
/** A widget backed by a saved contract request (still no SQL, by construction). */
|
||||
export interface QueryWidgetSource {
|
||||
kind: 'query';
|
||||
/** Parsed with `parseAnalyticsQueryRequest` at validation time, not at render time. */
|
||||
request: AnalyticsQueryRequest;
|
||||
}
|
||||
|
||||
export type WidgetSource = MetricWidgetSource | QueryWidgetSource;
|
||||
|
||||
export interface VisualizationSpec {
|
||||
type: VisualizationType;
|
||||
/** Field roles by NAME. `x`/`y` name metrics or dimensions in the source. */
|
||||
encoding: {
|
||||
x?: string;
|
||||
y?: string[];
|
||||
series?: string;
|
||||
value?: string;
|
||||
};
|
||||
/** Presentation-only knobs. Bounded scalars; never code, never a template. */
|
||||
options?: Record<string, string | number | boolean>;
|
||||
}
|
||||
|
||||
export interface DashboardWidget {
|
||||
widget_id: string;
|
||||
title: string;
|
||||
source: WidgetSource;
|
||||
visualization: VisualizationSpec;
|
||||
layout: WidgetLayout;
|
||||
/** filter_ids this widget honours. Empty means "all dashboard filters". */
|
||||
filter_bindings?: string[];
|
||||
/** Advisory only: excluded from the canonical hash, never a source of truth. */
|
||||
vendor_ref?: VendorRef;
|
||||
}
|
||||
|
||||
/** A foreign system's id for the same object. Round-tripping aid, nothing more. */
|
||||
export interface VendorRef {
|
||||
system: string;
|
||||
id: string;
|
||||
}
|
||||
|
||||
export interface DashboardRefresh {
|
||||
mode: RefreshMode;
|
||||
/** Required for `interval`. */
|
||||
interval_seconds?: number;
|
||||
/** Required for `scheduled`. */
|
||||
cron?: string;
|
||||
/**
|
||||
* The staleness the OWNER accepts. BI-P026 computes actual freshness; this
|
||||
* is the number that makes a freshness verdict actionable rather than
|
||||
* decorative, and BI-P105 reads it when deciding whether a pre-aggregate may
|
||||
* serve this dashboard.
|
||||
*/
|
||||
max_staleness_seconds: number;
|
||||
}
|
||||
|
||||
export interface DashboardPermissions {
|
||||
visibility: DashboardVisibility;
|
||||
/** Required (non-empty) when visibility is `roles`. */
|
||||
roles?: string[];
|
||||
/** Optional cl0ud entitlement key (BI-P079). */
|
||||
required_entitlement?: string;
|
||||
}
|
||||
|
||||
/** BI-P060's canonical document, field-for-field with the program's list. */
|
||||
export interface DashboardDocument {
|
||||
id: string;
|
||||
title: string;
|
||||
description?: string;
|
||||
layout: { grid_columns: number; row_height_px: number };
|
||||
filters: DashboardFilter[];
|
||||
widgets: DashboardWidget[];
|
||||
parameters: DashboardParameter[];
|
||||
refresh: DashboardRefresh;
|
||||
permissions: DashboardPermissions;
|
||||
/** Document schema version — bumped when THIS shape changes, not per edit. */
|
||||
version: number;
|
||||
vendor_ref?: VendorRef;
|
||||
}
|
||||
|
||||
export const DASHBOARD_DOCUMENT_KEYS = [
|
||||
'id',
|
||||
'title',
|
||||
'description',
|
||||
'layout',
|
||||
'filters',
|
||||
'widgets',
|
||||
'parameters',
|
||||
'refresh',
|
||||
'permissions',
|
||||
'version',
|
||||
'vendor_ref',
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Keys that mean "this document is really a Superset export". Named
|
||||
* explicitly so the refusal explains the architecture instead of saying
|
||||
* "unknown key"; the unknown-key rule below catches everything else anyway.
|
||||
*/
|
||||
const VENDOR_NATIVE_KEYS = new Set([
|
||||
'slice_id',
|
||||
'slice_name',
|
||||
'viz_type',
|
||||
'datasource',
|
||||
'datasource_id',
|
||||
'datasource_type',
|
||||
'form_data',
|
||||
'query_context',
|
||||
'chart_id',
|
||||
'dashboard_id_superset',
|
||||
'position_json',
|
||||
'json_metadata',
|
||||
'card_id',
|
||||
'cube',
|
||||
]);
|
||||
|
||||
/** Keys that would turn a dashboard into an SQL surface. */
|
||||
const SQL_SMELLING_KEYS = new Set([
|
||||
'sql',
|
||||
'query_text',
|
||||
'statement',
|
||||
'expression',
|
||||
'where',
|
||||
'having',
|
||||
'raw',
|
||||
'tenant',
|
||||
'tenant_id',
|
||||
'project_id',
|
||||
'database',
|
||||
'table',
|
||||
]);
|
||||
|
||||
const MAX_WIDGETS = 60;
|
||||
const MAX_FILTERS = 32;
|
||||
const MAX_PARAMETERS = 24;
|
||||
const MAX_NAME_LENGTH = 128;
|
||||
const MAX_TITLE_LENGTH = 200;
|
||||
const MAX_GRID_COLUMNS = 24;
|
||||
const MIN_REFRESH_INTERVAL_SECONDS = 10;
|
||||
const MAX_STALENESS_SECONDS = 30 * 24 * 3600;
|
||||
/** `{{param}}` — the only templating the document has, and it binds a VALUE. */
|
||||
const PARAM_REFERENCE = /^\{\{\s*([A-Za-z][A-Za-z0-9_]{0,63})\s*\}\}$/;
|
||||
const IDENTIFIER = /^[A-Za-z][A-Za-z0-9_.-]{0,63}$/;
|
||||
|
||||
function invalid(detail: string): never {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.dashboard', detail);
|
||||
}
|
||||
|
||||
function asObject(raw: unknown, what: string): Record<string, unknown> {
|
||||
if (raw === null || typeof raw !== 'object' || Array.isArray(raw)) invalid(`${what} must be an object`);
|
||||
return raw as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(obj: Record<string, unknown>, allowed: readonly string[], what: string): void {
|
||||
for (const key of Object.keys(obj)) {
|
||||
if (allowed.includes(key)) continue;
|
||||
if (VENDOR_NATIVE_KEYS.has(key)) {
|
||||
invalid(
|
||||
`"${key}" in ${what} is a vendor-native field — a SIAX dashboard is defined by metrics, ` +
|
||||
`queries and a visualization spec, not by a BI tool's internal ids (BI-P060). ` +
|
||||
`Carry the id in "vendor_ref" if you need to round-trip it.`,
|
||||
);
|
||||
}
|
||||
if (SQL_SMELLING_KEYS.has(key)) {
|
||||
invalid(`"${key}" in ${what} is not part of the dashboard model — dashboards do not carry SQL`);
|
||||
}
|
||||
invalid(`unknown field "${key}" in ${what}`);
|
||||
}
|
||||
}
|
||||
|
||||
function asString(raw: unknown, what: string, max = MAX_NAME_LENGTH): string {
|
||||
if (typeof raw !== 'string') invalid(`${what} must be a string`);
|
||||
const v = raw.trim();
|
||||
if (!v) invalid(`${what} must not be empty`);
|
||||
if (v.length > max) invalid(`${what} exceeds ${max} characters`);
|
||||
return v;
|
||||
}
|
||||
|
||||
function asIdentifier(raw: unknown, what: string): string {
|
||||
const v = asString(raw, what);
|
||||
if (!IDENTIFIER.test(v)) invalid(`${what} is not a valid identifier`);
|
||||
return v;
|
||||
}
|
||||
|
||||
function asInteger(raw: unknown, what: string, min: number, max: number): number {
|
||||
if (typeof raw !== 'number' || !Number.isFinite(raw)) invalid(`${what} must be a finite number`);
|
||||
if (!Number.isInteger(raw)) invalid(`${what} must be an integer`);
|
||||
if (raw < min || raw > max) invalid(`${what} must be between ${min} and ${max}`);
|
||||
return raw;
|
||||
}
|
||||
|
||||
function parseVendorRef(raw: unknown, what: string): VendorRef {
|
||||
const obj = asObject(raw, what);
|
||||
rejectUnknownKeys(obj, ['system', 'id'], what);
|
||||
return { system: asString(obj.system, `${what}.system`), id: asString(obj.id, `${what}.id`, 256) };
|
||||
}
|
||||
|
||||
// ── parsing ────────────────────────────────────────────────────────────────
|
||||
|
||||
function parseParameter(raw: unknown, i: number): DashboardParameter {
|
||||
const what = `parameters[${i}]`;
|
||||
const obj = asObject(raw, what);
|
||||
rejectUnknownKeys(obj, ['name', 'type', 'required', 'default_value', 'allowed_values'], what);
|
||||
const name = asIdentifier(obj.name, `${what}.name`);
|
||||
const type = asString(obj.type, `${what}.type`);
|
||||
if (!(PARAMETER_TYPES as readonly string[]).includes(type)) invalid(`${what}.type is not a parameter type`);
|
||||
if (typeof obj.required !== 'boolean') invalid(`${what}.required must be a boolean`);
|
||||
const p: DashboardParameter = { name, type: type as ParameterType, required: obj.required };
|
||||
if (obj.default_value !== undefined)
|
||||
p.default_value = asString(obj.default_value, `${what}.default_value`, 4096);
|
||||
if (obj.allowed_values !== undefined) {
|
||||
if (!Array.isArray(obj.allowed_values) || obj.allowed_values.length === 0)
|
||||
invalid(`${what}.allowed_values must be a non-empty array`);
|
||||
p.allowed_values = obj.allowed_values.map((v, j) => asString(v, `${what}.allowed_values[${j}]`, 4096));
|
||||
}
|
||||
if (p.type === 'enum' && !p.allowed_values) {
|
||||
invalid(`${what} is an enum parameter with no allowed_values — an unbounded enum is a free-text field`);
|
||||
}
|
||||
if (p.required && p.default_value !== undefined) {
|
||||
invalid(`${what} is required AND has a default — a default makes "required" unenforceable`);
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
function parseFilter(raw: unknown, i: number, parameterNames: Set<string>): DashboardFilter {
|
||||
const what = `filters[${i}]`;
|
||||
const obj = asObject(raw, what);
|
||||
rejectUnknownKeys(obj, ['filter_id', 'label', 'dimension', 'operator', 'default_values', 'locked'], what);
|
||||
const filter_id = asIdentifier(obj.filter_id, `${what}.filter_id`);
|
||||
const label = asString(obj.label, `${what}.label`, MAX_TITLE_LENGTH);
|
||||
const dimension = asString(obj.dimension, `${what}.dimension`);
|
||||
const operator = asString(obj.operator, `${what}.operator`) as FilterOperator;
|
||||
const f: DashboardFilter = { filter_id, label, dimension, operator };
|
||||
if (obj.default_values !== undefined) {
|
||||
if (!Array.isArray(obj.default_values)) invalid(`${what}.default_values must be an array`);
|
||||
f.default_values = obj.default_values.map((v, j) => {
|
||||
const s = asString(v, `${what}.default_values[${j}]`, 4096);
|
||||
const m = PARAM_REFERENCE.exec(s);
|
||||
if (m && !parameterNames.has(m[1])) {
|
||||
invalid(`${what}.default_values[${j}] references undeclared parameter "${m[1]}"`);
|
||||
}
|
||||
return s;
|
||||
});
|
||||
}
|
||||
if (obj.locked !== undefined) {
|
||||
if (typeof obj.locked !== 'boolean') invalid(`${what}.locked must be a boolean`);
|
||||
f.locked = obj.locked;
|
||||
}
|
||||
return f;
|
||||
}
|
||||
|
||||
function parseVisualization(raw: unknown, what: string): VisualizationSpec {
|
||||
const obj = asObject(raw, what);
|
||||
rejectUnknownKeys(obj, ['type', 'encoding', 'options'], what);
|
||||
const type = asString(obj.type, `${what}.type`);
|
||||
if (!(VISUALIZATION_TYPES as readonly string[]).includes(type)) {
|
||||
invalid(
|
||||
`${what}.type "${type}" is not a SIAX visualization type — the canonical list is engine-neutral ` +
|
||||
`so a chart survives a BI engine swap (BI-P060)`,
|
||||
);
|
||||
}
|
||||
const encodingRaw = asObject(obj.encoding ?? {}, `${what}.encoding`);
|
||||
rejectUnknownKeys(encodingRaw, ['x', 'y', 'series', 'value'], `${what}.encoding`);
|
||||
const encoding: VisualizationSpec['encoding'] = {};
|
||||
if (encodingRaw.x !== undefined) encoding.x = asString(encodingRaw.x, `${what}.encoding.x`);
|
||||
if (encodingRaw.series !== undefined)
|
||||
encoding.series = asString(encodingRaw.series, `${what}.encoding.series`);
|
||||
if (encodingRaw.value !== undefined) encoding.value = asString(encodingRaw.value, `${what}.encoding.value`);
|
||||
if (encodingRaw.y !== undefined) {
|
||||
if (!Array.isArray(encodingRaw.y)) invalid(`${what}.encoding.y must be an array`);
|
||||
encoding.y = encodingRaw.y.map((v, j) => asString(v, `${what}.encoding.y[${j}]`));
|
||||
}
|
||||
const spec: VisualizationSpec = { type: type as VisualizationType, encoding };
|
||||
if (obj.options !== undefined) {
|
||||
const options = asObject(obj.options, `${what}.options`);
|
||||
const out: Record<string, string | number | boolean> = {};
|
||||
for (const [k, v] of Object.entries(options)) {
|
||||
if (!IDENTIFIER.test(k)) invalid(`${what}.options key "${k}" is not an identifier`);
|
||||
if (typeof v === 'string') {
|
||||
if (v.length > 256) invalid(`${what}.options.${k} exceeds 256 characters`);
|
||||
out[k] = v;
|
||||
} else if (typeof v === 'number') {
|
||||
if (!Number.isFinite(v)) invalid(`${what}.options.${k} must be finite`);
|
||||
out[k] = v;
|
||||
} else if (typeof v === 'boolean') {
|
||||
out[k] = v;
|
||||
} else {
|
||||
invalid(`${what}.options.${k} must be a string, number or boolean — options are not a payload`);
|
||||
}
|
||||
}
|
||||
spec.options = out;
|
||||
}
|
||||
return spec;
|
||||
}
|
||||
|
||||
function parseSource(raw: unknown, what: string): WidgetSource {
|
||||
const obj = asObject(raw, what);
|
||||
const kind = asString(obj.kind, `${what}.kind`);
|
||||
if (kind === 'metric') {
|
||||
rejectUnknownKeys(obj, ['kind', 'dataset', 'metric', 'dimensions', 'grain', 'limit'], what);
|
||||
const source: MetricWidgetSource = {
|
||||
kind: 'metric',
|
||||
dataset: asString(obj.dataset, `${what}.dataset`),
|
||||
metric: asString(obj.metric, `${what}.metric`),
|
||||
};
|
||||
if (obj.dimensions !== undefined) {
|
||||
if (!Array.isArray(obj.dimensions)) invalid(`${what}.dimensions must be an array`);
|
||||
source.dimensions = obj.dimensions.map((d, j) => asString(d, `${what}.dimensions[${j}]`));
|
||||
}
|
||||
if (obj.grain !== undefined) {
|
||||
const grain = asString(obj.grain, `${what}.grain`);
|
||||
if (!(TIME_GRAINS as readonly string[]).includes(grain)) invalid(`${what}.grain is not a time grain`);
|
||||
source.grain = grain as TimeGrain;
|
||||
}
|
||||
if (obj.limit !== undefined) source.limit = asInteger(obj.limit, `${what}.limit`, 1, 100_000);
|
||||
return source;
|
||||
}
|
||||
if (kind === 'query') {
|
||||
rejectUnknownKeys(obj, ['kind', 'request'], what);
|
||||
// parseAnalyticsQueryRequest is the SQL boundary: it is why a query widget
|
||||
// cannot become an SQL widget by adding a field.
|
||||
return { kind: 'query', request: parseAnalyticsQueryRequest(obj.request) };
|
||||
}
|
||||
invalid(
|
||||
`${what}.kind must be "metric" or "query" — a widget with no reconstructable source cannot be ` +
|
||||
`rendered without the tool that exported it (BI-P060 DoD)`,
|
||||
);
|
||||
}
|
||||
|
||||
function parseWidget(raw: unknown, i: number, gridColumns: number): DashboardWidget {
|
||||
const what = `widgets[${i}]`;
|
||||
const obj = asObject(raw, what);
|
||||
rejectUnknownKeys(
|
||||
obj,
|
||||
['widget_id', 'title', 'source', 'visualization', 'layout', 'filter_bindings', 'vendor_ref'],
|
||||
what,
|
||||
);
|
||||
const widget_id = asIdentifier(obj.widget_id, `${what}.widget_id`);
|
||||
const title = asString(obj.title, `${what}.title`, MAX_TITLE_LENGTH);
|
||||
const source = parseSource(obj.source, `${what}.source`);
|
||||
const visualization = parseVisualization(obj.visualization, `${what}.visualization`);
|
||||
|
||||
const layoutObj = asObject(obj.layout, `${what}.layout`);
|
||||
rejectUnknownKeys(layoutObj, ['x', 'y', 'w', 'h'], `${what}.layout`);
|
||||
const layout: WidgetLayout = {
|
||||
x: asInteger(layoutObj.x, `${what}.layout.x`, 0, gridColumns - 1),
|
||||
y: asInteger(layoutObj.y, `${what}.layout.y`, 0, 10_000),
|
||||
w: asInteger(layoutObj.w, `${what}.layout.w`, 1, gridColumns),
|
||||
h: asInteger(layoutObj.h, `${what}.layout.h`, 1, 200),
|
||||
};
|
||||
if (layout.x + layout.w > gridColumns) {
|
||||
invalid(`${what}.layout overflows the ${gridColumns}-column grid (x=${layout.x}, w=${layout.w})`);
|
||||
}
|
||||
|
||||
const widget: DashboardWidget = { widget_id, title, source, visualization, layout };
|
||||
if (obj.filter_bindings !== undefined) {
|
||||
if (!Array.isArray(obj.filter_bindings)) invalid(`${what}.filter_bindings must be an array`);
|
||||
widget.filter_bindings = obj.filter_bindings.map((b, j) =>
|
||||
asIdentifier(b, `${what}.filter_bindings[${j}]`),
|
||||
);
|
||||
}
|
||||
if (obj.vendor_ref !== undefined) widget.vendor_ref = parseVendorRef(obj.vendor_ref, `${what}.vendor_ref`);
|
||||
return widget;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse an untrusted dashboard document. Fail-closed everywhere: a field this
|
||||
* function does not understand is a refusal, because "ignored for now" is how
|
||||
* a Superset-only document ends up stored as canonical truth.
|
||||
*/
|
||||
export function parseDashboardDocument(raw: unknown): DashboardDocument {
|
||||
const obj = asObject(raw, 'dashboard document');
|
||||
rejectUnknownKeys(obj, DASHBOARD_DOCUMENT_KEYS, 'dashboard document');
|
||||
|
||||
const id = asIdentifier(obj.id, 'id');
|
||||
const title = asString(obj.title, 'title', MAX_TITLE_LENGTH);
|
||||
const version = asInteger(obj.version, 'version', 1, 1_000_000);
|
||||
|
||||
const layoutObj = asObject(obj.layout, 'layout');
|
||||
rejectUnknownKeys(layoutObj, ['grid_columns', 'row_height_px'], 'layout');
|
||||
const layout = {
|
||||
grid_columns: asInteger(layoutObj.grid_columns, 'layout.grid_columns', 1, MAX_GRID_COLUMNS),
|
||||
row_height_px: asInteger(layoutObj.row_height_px, 'layout.row_height_px', 8, 400),
|
||||
};
|
||||
|
||||
const rawParameters = obj.parameters ?? [];
|
||||
if (!Array.isArray(rawParameters)) invalid('parameters must be an array');
|
||||
if (rawParameters.length > MAX_PARAMETERS) invalid(`at most ${MAX_PARAMETERS} parameters`);
|
||||
const parameters = rawParameters.map((p, i) => parseParameter(p, i));
|
||||
const parameterNames = new Set(parameters.map((p) => p.name));
|
||||
if (parameterNames.size !== parameters.length) invalid('parameters contains duplicate names');
|
||||
|
||||
const rawFilters = obj.filters ?? [];
|
||||
if (!Array.isArray(rawFilters)) invalid('filters must be an array');
|
||||
if (rawFilters.length > MAX_FILTERS) invalid(`at most ${MAX_FILTERS} filters`);
|
||||
const filters = rawFilters.map((f, i) => parseFilter(f, i, parameterNames));
|
||||
const filterIds = new Set(filters.map((f) => f.filter_id));
|
||||
if (filterIds.size !== filters.length) invalid('filters contains duplicate filter_id');
|
||||
|
||||
if (!Array.isArray(obj.widgets)) invalid('widgets must be an array');
|
||||
if (obj.widgets.length === 0) invalid('a dashboard with no widgets has nothing to reconstruct');
|
||||
if (obj.widgets.length > MAX_WIDGETS) invalid(`at most ${MAX_WIDGETS} widgets`);
|
||||
const widgets = obj.widgets.map((w, i) => parseWidget(w, i, layout.grid_columns));
|
||||
if (new Set(widgets.map((w) => w.widget_id)).size !== widgets.length) {
|
||||
invalid('widgets contains duplicate widget_id');
|
||||
}
|
||||
for (const w of widgets) {
|
||||
for (const b of w.filter_bindings ?? []) {
|
||||
if (!filterIds.has(b)) invalid(`widget "${w.widget_id}" binds undeclared filter "${b}"`);
|
||||
}
|
||||
}
|
||||
|
||||
const refreshObj = asObject(obj.refresh, 'refresh');
|
||||
rejectUnknownKeys(refreshObj, ['mode', 'interval_seconds', 'cron', 'max_staleness_seconds'], 'refresh');
|
||||
const mode = asString(refreshObj.mode, 'refresh.mode');
|
||||
if (!(REFRESH_MODES as readonly string[]).includes(mode)) invalid('refresh.mode is not a refresh mode');
|
||||
const refresh: DashboardRefresh = {
|
||||
mode: mode as RefreshMode,
|
||||
max_staleness_seconds: asInteger(
|
||||
refreshObj.max_staleness_seconds,
|
||||
'refresh.max_staleness_seconds',
|
||||
1,
|
||||
MAX_STALENESS_SECONDS,
|
||||
),
|
||||
};
|
||||
if (refresh.mode === 'interval') {
|
||||
refresh.interval_seconds = asInteger(
|
||||
refreshObj.interval_seconds,
|
||||
'refresh.interval_seconds',
|
||||
MIN_REFRESH_INTERVAL_SECONDS,
|
||||
86_400,
|
||||
);
|
||||
if (refreshObj.cron !== undefined) invalid('refresh.cron is meaningless when mode is "interval"');
|
||||
} else if (refresh.mode === 'scheduled') {
|
||||
refresh.cron = asString(refreshObj.cron, 'refresh.cron', 128);
|
||||
if (refreshObj.interval_seconds !== undefined) {
|
||||
invalid('refresh.interval_seconds is meaningless when mode is "scheduled"');
|
||||
}
|
||||
} else if (refreshObj.interval_seconds !== undefined || refreshObj.cron !== undefined) {
|
||||
invalid('refresh.mode is "manual" but a schedule was supplied');
|
||||
}
|
||||
|
||||
const permObj = asObject(obj.permissions, 'permissions');
|
||||
rejectUnknownKeys(permObj, ['visibility', 'roles', 'required_entitlement'], 'permissions');
|
||||
const visibility = asString(permObj.visibility, 'permissions.visibility');
|
||||
if (!(DASHBOARD_VISIBILITIES as readonly string[]).includes(visibility)) {
|
||||
invalid('permissions.visibility is not a visibility');
|
||||
}
|
||||
const permissions: DashboardPermissions = { visibility: visibility as DashboardVisibility };
|
||||
if (permObj.roles !== undefined) {
|
||||
if (!Array.isArray(permObj.roles)) invalid('permissions.roles must be an array');
|
||||
permissions.roles = permObj.roles.map((r, i) => asIdentifier(r, `permissions.roles[${i}]`));
|
||||
}
|
||||
if (permissions.visibility === 'roles' && (permissions.roles ?? []).length === 0) {
|
||||
invalid(
|
||||
'permissions.visibility is "roles" with no roles — an empty role list reads as "deny all" to one ' +
|
||||
'implementation and "allow all" to the next; say which',
|
||||
);
|
||||
}
|
||||
if (permObj.required_entitlement !== undefined) {
|
||||
permissions.required_entitlement = asIdentifier(
|
||||
permObj.required_entitlement,
|
||||
'permissions.required_entitlement',
|
||||
);
|
||||
}
|
||||
|
||||
const doc: DashboardDocument = {
|
||||
id,
|
||||
title,
|
||||
layout,
|
||||
filters,
|
||||
widgets,
|
||||
parameters,
|
||||
refresh,
|
||||
permissions,
|
||||
version,
|
||||
};
|
||||
if (obj.description !== undefined) doc.description = asString(obj.description, 'description', 2000);
|
||||
if (obj.vendor_ref !== undefined) doc.vendor_ref = parseVendorRef(obj.vendor_ref, 'vendor_ref');
|
||||
return doc;
|
||||
}
|
||||
|
||||
// ── canonical hash ─────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Stable ordering of every field that MEANS something, with `vendor_ref`
|
||||
* omitted at both levels. Two documents that render identically hash
|
||||
* identically even if their JSON key order differs or one of them has been
|
||||
* round-tripped through a BI tool; a document whose meaning changed hashes
|
||||
* differently. That is what makes `DashboardVersion.layout_hash` a usable
|
||||
* identity for BI-P061 rollback.
|
||||
*/
|
||||
function canonicalize(doc: DashboardDocument): unknown {
|
||||
return [
|
||||
doc.id,
|
||||
doc.title,
|
||||
doc.description ?? '',
|
||||
doc.version,
|
||||
[doc.layout.grid_columns, doc.layout.row_height_px],
|
||||
doc.parameters
|
||||
.map((p) => [p.name, p.type, p.required, p.default_value ?? '', [...(p.allowed_values ?? [])].sort()])
|
||||
.sort((a, b) => String(a[0]).localeCompare(String(b[0]))),
|
||||
doc.filters
|
||||
.map((f) => [f.filter_id, f.label, f.dimension, f.operator, f.default_values ?? [], f.locked ?? false])
|
||||
.sort((a, b) => String(a[0]).localeCompare(String(b[0]))),
|
||||
doc.widgets
|
||||
.map((w) => [
|
||||
w.widget_id,
|
||||
w.title,
|
||||
w.source.kind === 'metric'
|
||||
? [
|
||||
'metric',
|
||||
w.source.dataset,
|
||||
w.source.metric,
|
||||
w.source.dimensions ?? [],
|
||||
w.source.grain ?? '',
|
||||
w.source.limit ?? 0,
|
||||
]
|
||||
: ['query', canonicalRequest(w.source.request)],
|
||||
[
|
||||
w.visualization.type,
|
||||
w.visualization.encoding.x ?? '',
|
||||
w.visualization.encoding.y ?? [],
|
||||
w.visualization.encoding.series ?? '',
|
||||
w.visualization.encoding.value ?? '',
|
||||
Object.keys(w.visualization.options ?? {})
|
||||
.sort()
|
||||
.map((k) => [k, (w.visualization.options ?? {})[k]]),
|
||||
],
|
||||
[w.layout.x, w.layout.y, w.layout.w, w.layout.h],
|
||||
[...(w.filter_bindings ?? [])].sort(),
|
||||
])
|
||||
.sort((a, b) => String(a[0]).localeCompare(String(b[0]))),
|
||||
[
|
||||
doc.refresh.mode,
|
||||
doc.refresh.interval_seconds ?? 0,
|
||||
doc.refresh.cron ?? '',
|
||||
doc.refresh.max_staleness_seconds,
|
||||
],
|
||||
[
|
||||
doc.permissions.visibility,
|
||||
[...(doc.permissions.roles ?? [])].sort(),
|
||||
doc.permissions.required_entitlement ?? '',
|
||||
],
|
||||
];
|
||||
}
|
||||
|
||||
function canonicalRequest(r: AnalyticsQueryRequest): unknown {
|
||||
return [
|
||||
r.dataset,
|
||||
[...r.metrics],
|
||||
[...r.dimensions],
|
||||
r.filters.map((f) => [f.dimension, f.operator, f.values ?? []]),
|
||||
r.timeRange
|
||||
? [r.timeRange.dimension ?? '', r.timeRange.from ?? '', r.timeRange.to ?? '', r.timeRange.grain ?? '']
|
||||
: [],
|
||||
(r.orderBy ?? []).map((o) => [o.ref, o.direction]),
|
||||
r.limit ?? 0,
|
||||
r.format ?? 'json',
|
||||
];
|
||||
}
|
||||
|
||||
/** sha256 of the canonical form. Fills `DashboardVersion.layout_hash`. */
|
||||
export function dashboardDocumentHash(doc: DashboardDocument): string {
|
||||
return crypto
|
||||
.createHash('sha256')
|
||||
.update(JSON.stringify(canonicalize(doc)))
|
||||
.digest('hex');
|
||||
}
|
||||
|
||||
// ── reconstruction (the DoD) ───────────────────────────────────────────────
|
||||
|
||||
export interface PlannedWidget {
|
||||
widget_id: string;
|
||||
title: string;
|
||||
/** Runnable, contract-valid, engine-neutral. This is the DoD in one field. */
|
||||
request: AnalyticsQueryRequest;
|
||||
visualization: VisualizationSpec;
|
||||
layout: WidgetLayout;
|
||||
}
|
||||
|
||||
export interface DashboardPlan {
|
||||
dashboard_id: string;
|
||||
title: string;
|
||||
document_hash: string;
|
||||
grid_columns: number;
|
||||
widgets: PlannedWidget[];
|
||||
/** Parameter values actually bound, after defaults. Reported, not implied. */
|
||||
bound_parameters: Record<string, string>;
|
||||
}
|
||||
|
||||
export interface PlanOptions {
|
||||
/** Caller-supplied parameter values. Validated against the declarations. */
|
||||
parameters?: Record<string, string>;
|
||||
/** Filter ids a viewer switched off. `locked` filters cannot be switched off. */
|
||||
disabled_filter_ids?: string[];
|
||||
}
|
||||
|
||||
function bindParameters(doc: DashboardDocument, supplied: Record<string, string>): Record<string, string> {
|
||||
const bound: Record<string, string> = {};
|
||||
const declared = new Set(doc.parameters.map((p) => p.name));
|
||||
for (const name of Object.keys(supplied)) {
|
||||
if (!declared.has(name)) invalid(`parameter "${name}" is not declared by this dashboard`);
|
||||
}
|
||||
for (const p of doc.parameters) {
|
||||
const value = supplied[p.name] ?? p.default_value;
|
||||
if (value === undefined) {
|
||||
if (p.required) invalid(`required parameter "${p.name}" was not supplied`);
|
||||
continue;
|
||||
}
|
||||
if (p.allowed_values && !p.allowed_values.includes(value)) {
|
||||
invalid(`parameter "${p.name}" value is not in its allowed_values`);
|
||||
}
|
||||
if (p.type === 'number' && !Number.isFinite(Number(value))) {
|
||||
invalid(`parameter "${p.name}" is declared number but "${value}" is not numeric`);
|
||||
}
|
||||
if (p.type === 'date' && Number.isNaN(Date.parse(value))) {
|
||||
invalid(`parameter "${p.name}" is declared date but "${value}" is not an ISO date`);
|
||||
}
|
||||
bound[p.name] = value;
|
||||
}
|
||||
return bound;
|
||||
}
|
||||
|
||||
function resolveFilterValues(f: DashboardFilter, bound: Record<string, string>): string[] | undefined {
|
||||
if (!f.default_values) return undefined;
|
||||
return f.default_values.map((v) => {
|
||||
const m = PARAM_REFERENCE.exec(v);
|
||||
if (!m) return v;
|
||||
const value = bound[m[1]];
|
||||
if (value === undefined) {
|
||||
invalid(`filter "${f.filter_id}" references parameter "${m[1]}", which has no value`);
|
||||
}
|
||||
return value;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Turn a dashboard document into runnable analytics requests.
|
||||
*
|
||||
* This IS BI-P060's DoD. If this function returns a plan, the dashboard can be
|
||||
* rendered by anything that can execute `AnalyticsQueryRequest` and draw the
|
||||
* twelve visualization types — Superset not required, not consulted, and not
|
||||
* present anywhere in the output.
|
||||
*/
|
||||
export function planDashboard(doc: DashboardDocument, options: PlanOptions = {}): DashboardPlan {
|
||||
const bound = bindParameters(doc, options.parameters ?? {});
|
||||
const disabled = new Set(options.disabled_filter_ids ?? []);
|
||||
for (const f of doc.filters) {
|
||||
if (disabled.has(f.filter_id) && f.locked) {
|
||||
invalid(
|
||||
`filter "${f.filter_id}" is locked and cannot be disabled — locked filters are how a saved ` +
|
||||
`dashboard pins its scope, and a viewer removing one silently widens the result set`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const widgets = doc.widgets.map((w) => {
|
||||
const applicable = doc.filters.filter(
|
||||
(f) =>
|
||||
!disabled.has(f.filter_id) &&
|
||||
(w.filter_bindings === undefined || w.filter_bindings.length === 0
|
||||
? true
|
||||
: w.filter_bindings.includes(f.filter_id)),
|
||||
);
|
||||
const dashboardFilters: AnalyticsFilter[] = applicable.map((f) => {
|
||||
const values = resolveFilterValues(f, bound);
|
||||
const af: AnalyticsFilter = { dimension: f.dimension, operator: f.operator };
|
||||
if (values !== undefined) af.values = values;
|
||||
return af;
|
||||
});
|
||||
|
||||
const base =
|
||||
w.source.kind === 'metric'
|
||||
? {
|
||||
dataset: w.source.dataset,
|
||||
metrics: [w.source.metric],
|
||||
dimensions: w.source.dimensions ?? [],
|
||||
filters: [] as AnalyticsFilter[],
|
||||
...(w.source.grain ? { timeRange: { grain: w.source.grain } } : {}),
|
||||
...(w.source.limit ? { limit: w.source.limit } : {}),
|
||||
}
|
||||
: {
|
||||
...w.source.request,
|
||||
timeRange: w.source.request.timeRange,
|
||||
};
|
||||
|
||||
// Re-parse the merged request rather than trusting the merge: the filters
|
||||
// we just added come from the document, and a document is untrusted input
|
||||
// even after it has been stored.
|
||||
const request = parseAnalyticsQueryRequest({
|
||||
...base,
|
||||
filters: [...(w.source.kind === 'query' ? w.source.request.filters : []), ...dashboardFilters],
|
||||
});
|
||||
|
||||
return {
|
||||
widget_id: w.widget_id,
|
||||
title: w.title,
|
||||
request,
|
||||
visualization: w.visualization,
|
||||
layout: w.layout,
|
||||
};
|
||||
});
|
||||
|
||||
return {
|
||||
dashboard_id: doc.id,
|
||||
title: doc.title,
|
||||
document_hash: dashboardDocumentHash(doc),
|
||||
grid_columns: doc.layout.grid_columns,
|
||||
widgets,
|
||||
bound_parameters: bound,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,449 @@
|
||||
/**
|
||||
* src/analytics/dashboards/dashboard-release.ts
|
||||
* ==============================================
|
||||
* BI-P061 — DASHBOARD VERSIONING.
|
||||
*
|
||||
* DRAFT -> PUBLISHED -> SUPERSEDED, and the DoD: "rollback to previous
|
||||
* dashboard definition".
|
||||
*
|
||||
* Three decisions worth stating, because each one is a bug someone would
|
||||
* otherwise write:
|
||||
*
|
||||
* 1. ROLLBACK MINTS A NEW VERSION. It does not resurrect the old release row
|
||||
* and it does not edit the published one. "We went back to v3" is then a
|
||||
* fact with a timestamp (v5 = rollback_of_version 3) instead of a hole in
|
||||
* the history where v4 used to be. `verifyRollback()` proves the restore is
|
||||
* faithful by comparing BI-P060 canonical hashes, so a rollback that
|
||||
* silently restored a different document fails loudly.
|
||||
*
|
||||
* 2. THE DOCUMENT IS FROZEN, THE RELEASE MOVES. sql/008 makes
|
||||
* DashboardVersion rows immutable at the database. The release ledger
|
||||
* (sql/010) carries the state, and its trigger allows exactly two
|
||||
* transitions and freezes every other column. Un-publishing is refused:
|
||||
* it is how a rollback gets faked without leaving a trace.
|
||||
*
|
||||
* 3. THE APPLICATION CHECK IS NOT THE GUARANTEE. `publish()` checks for an
|
||||
* existing published release, and that check is a read-then-write race.
|
||||
* The partial unique index in sql/010 is what actually holds under
|
||||
* concurrency; the realinfra test drops it, runs the race and shows two
|
||||
* published releases for one dashboard, so nobody has to take that on
|
||||
* faith. This module maps the resulting constraint violation onto a
|
||||
* CONFLICT rather than a 500.
|
||||
*/
|
||||
import crypto from 'node:crypto';
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import { dashboardDocumentHash, parseDashboardDocument, type DashboardDocument } from './dashboard-document';
|
||||
|
||||
export const DASHBOARD_RELEASE_STATES = ['draft', 'published', 'superseded'] as const;
|
||||
export type DashboardReleaseState = (typeof DASHBOARD_RELEASE_STATES)[number];
|
||||
|
||||
/** The only transitions that exist. Everything else is a refusal, not a no-op. */
|
||||
const LEGAL_TRANSITIONS: Readonly<Record<DashboardReleaseState, readonly DashboardReleaseState[]>> = {
|
||||
draft: ['published'],
|
||||
published: ['superseded'],
|
||||
superseded: [],
|
||||
};
|
||||
|
||||
export function isLegalTransition(from: DashboardReleaseState, to: DashboardReleaseState): boolean {
|
||||
return LEGAL_TRANSITIONS[from].includes(to);
|
||||
}
|
||||
|
||||
export interface DashboardRelease {
|
||||
release_id: string;
|
||||
dashboard_id: string;
|
||||
dashboard_version_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
version: number;
|
||||
state: DashboardReleaseState;
|
||||
/** BI-P060 canonical hash of the released document. */
|
||||
document_hash: string;
|
||||
/** Set when this release restores an earlier version. */
|
||||
rollback_of_version?: number;
|
||||
created_by?: string;
|
||||
created_at: string;
|
||||
published_at?: string;
|
||||
superseded_at?: string;
|
||||
superseded_by_release_id?: string;
|
||||
}
|
||||
|
||||
/** The immutable frozen document behind a release (sql/008's DashboardVersion). */
|
||||
export interface DashboardVersionRecord {
|
||||
dashboard_version_id: string;
|
||||
dashboard_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
version: number;
|
||||
layout_json: string;
|
||||
layout_hash: string;
|
||||
created_at: string;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
export interface TenantScope {
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The persistence port. Deliberately narrow, and every method is tenant-scoped
|
||||
* — there is no signature that lets a caller forget the tenant.
|
||||
*/
|
||||
export interface DashboardReleaseStore {
|
||||
/** Highest version number ever minted for this dashboard, or 0. */
|
||||
maxVersion(scope: TenantScope, dashboardId: string): Promise<number>;
|
||||
insertVersionAndRelease(version: DashboardVersionRecord, release: DashboardRelease): Promise<void>;
|
||||
getRelease(scope: TenantScope, releaseId: string): Promise<DashboardRelease | undefined>;
|
||||
getVersion(scope: TenantScope, dashboardVersionId: string): Promise<DashboardVersionRecord | undefined>;
|
||||
findVersionByNumber(
|
||||
scope: TenantScope,
|
||||
dashboardId: string,
|
||||
version: number,
|
||||
): Promise<DashboardVersionRecord | undefined>;
|
||||
findByState(
|
||||
scope: TenantScope,
|
||||
dashboardId: string,
|
||||
state: DashboardReleaseState,
|
||||
): Promise<DashboardRelease | undefined>;
|
||||
listReleases(scope: TenantScope, dashboardId: string): Promise<DashboardRelease[]>;
|
||||
/**
|
||||
* Publish `releaseId` and supersede `supersedes` in ONE transaction. Both or
|
||||
* neither: a supersede that lands without its publish leaves a dashboard
|
||||
* with no live version, and a publish that lands without its supersede
|
||||
* leaves two.
|
||||
*/
|
||||
publishAtomically(args: {
|
||||
scope: TenantScope;
|
||||
releaseId: string;
|
||||
publishedAt: string;
|
||||
supersedes?: { release_id: string; superseded_at: string };
|
||||
}): Promise<void>;
|
||||
}
|
||||
|
||||
function conflict(detail: string): never {
|
||||
throw new SiaxError('CONFLICT', '0pt.analytics.dashboard', detail);
|
||||
}
|
||||
|
||||
function notFound(detail: string): never {
|
||||
throw new SiaxError('NOT_FOUND', '0pt.analytics.dashboard', detail);
|
||||
}
|
||||
|
||||
function assertScope(scope: TenantScope): void {
|
||||
if (!scope?.tenant_id?.trim() || !scope?.project_id?.trim()) {
|
||||
throw new SiaxError(
|
||||
'UNAUTHENTICATED',
|
||||
'0pt.analytics.dashboard',
|
||||
'dashboard releases require a verified tenant_id + project_id',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export interface DraftResult {
|
||||
release: DashboardRelease;
|
||||
version: DashboardVersionRecord;
|
||||
}
|
||||
|
||||
/**
|
||||
* The lifecycle. Stateless apart from its store and its clock; the clock is
|
||||
* injectable so a test can pin timestamps instead of sleeping.
|
||||
*/
|
||||
export class DashboardReleaseService {
|
||||
constructor(
|
||||
private readonly store: DashboardReleaseStore,
|
||||
private readonly now: () => Date = () => new Date(),
|
||||
private readonly newId: () => string = () => crypto.randomUUID(),
|
||||
) {}
|
||||
|
||||
/**
|
||||
* Freeze a document as the next version and open a DRAFT release for it.
|
||||
* The document is re-parsed here even if the caller already parsed it: what
|
||||
* gets frozen must be what the model accepts, not what a caller assembled.
|
||||
*/
|
||||
async draft(args: {
|
||||
scope: TenantScope;
|
||||
dashboardId: string;
|
||||
document: unknown;
|
||||
createdBy?: string;
|
||||
rollbackOfVersion?: number;
|
||||
}): Promise<DraftResult> {
|
||||
assertScope(args.scope);
|
||||
const document = parseDashboardDocument(args.document);
|
||||
const existingDraft = await this.store.findByState(args.scope, args.dashboardId, 'draft');
|
||||
if (existingDraft) {
|
||||
conflict(
|
||||
`dashboard "${args.dashboardId}" already has an open draft (version ${existingDraft.version}); ` +
|
||||
`publish or abandon it before opening another — two drafts race to publish and the loser's ` +
|
||||
`edits vanish`,
|
||||
);
|
||||
}
|
||||
const next = (await this.store.maxVersion(args.scope, args.dashboardId)) + 1;
|
||||
const at = this.now().toISOString();
|
||||
const hash = dashboardDocumentHash(document);
|
||||
|
||||
const version: DashboardVersionRecord = {
|
||||
dashboard_version_id: this.newId(),
|
||||
dashboard_id: args.dashboardId,
|
||||
tenant_id: args.scope.tenant_id,
|
||||
project_id: args.scope.project_id,
|
||||
version: next,
|
||||
layout_json: JSON.stringify(document),
|
||||
layout_hash: hash,
|
||||
created_at: at,
|
||||
...(args.createdBy ? { created_by: args.createdBy } : {}),
|
||||
};
|
||||
const release: DashboardRelease = {
|
||||
release_id: this.newId(),
|
||||
dashboard_id: args.dashboardId,
|
||||
dashboard_version_id: version.dashboard_version_id,
|
||||
tenant_id: args.scope.tenant_id,
|
||||
project_id: args.scope.project_id,
|
||||
version: next,
|
||||
state: 'draft',
|
||||
document_hash: hash,
|
||||
created_at: at,
|
||||
...(args.createdBy ? { created_by: args.createdBy } : {}),
|
||||
...(args.rollbackOfVersion ? { rollback_of_version: args.rollbackOfVersion } : {}),
|
||||
};
|
||||
await this.store.insertVersionAndRelease(version, release);
|
||||
return { release, version };
|
||||
}
|
||||
|
||||
/**
|
||||
* DRAFT -> PUBLISHED, superseding whatever was live, in one transaction.
|
||||
* The pre-check exists to return a friendly CONFLICT; the guarantee is the
|
||||
* partial unique index (sql/010).
|
||||
*/
|
||||
async publish(args: {
|
||||
scope: TenantScope;
|
||||
releaseId: string;
|
||||
}): Promise<{ published: DashboardRelease; superseded?: DashboardRelease }> {
|
||||
assertScope(args.scope);
|
||||
const release = await this.store.getRelease(args.scope, args.releaseId);
|
||||
if (!release) notFound(`dashboard release "${args.releaseId}" does not exist in this tenant`);
|
||||
if (release.state !== 'draft') {
|
||||
conflict(
|
||||
`release "${args.releaseId}" is ${release.state}, not draft — ${release.state} -> published is ` +
|
||||
`not a legal transition (BI-P061)`,
|
||||
);
|
||||
}
|
||||
const live = await this.store.findByState(args.scope, release.dashboard_id, 'published');
|
||||
const at = this.now().toISOString();
|
||||
await this.store.publishAtomically({
|
||||
scope: args.scope,
|
||||
releaseId: release.release_id,
|
||||
publishedAt: at,
|
||||
...(live ? { supersedes: { release_id: live.release_id, superseded_at: at } } : {}),
|
||||
});
|
||||
|
||||
return {
|
||||
published: { ...release, state: 'published', published_at: at },
|
||||
...(live
|
||||
? {
|
||||
superseded: {
|
||||
...live,
|
||||
state: 'superseded',
|
||||
superseded_at: at,
|
||||
superseded_by_release_id: release.release_id,
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
/** The release currently serving, or undefined if the dashboard has never been published. */
|
||||
async live(scope: TenantScope, dashboardId: string): Promise<DashboardRelease | undefined> {
|
||||
assertScope(scope);
|
||||
return this.store.findByState(scope, dashboardId, 'published');
|
||||
}
|
||||
|
||||
/** The frozen document behind a release, re-parsed on the way out. */
|
||||
async documentOf(scope: TenantScope, release: DashboardRelease): Promise<DashboardDocument> {
|
||||
assertScope(scope);
|
||||
const version = await this.store.getVersion(scope, release.dashboard_version_id);
|
||||
if (!version) notFound(`dashboard version "${release.dashboard_version_id}" is missing`);
|
||||
const document = parseDashboardDocument(JSON.parse(version.layout_json));
|
||||
const hash = dashboardDocumentHash(document);
|
||||
if (hash !== release.document_hash) {
|
||||
// Storage returned something other than what was released. Refusing is
|
||||
// the only safe answer: a dashboard that renders a document nobody
|
||||
// approved is worse than a dashboard that will not open.
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'0pt.analytics.dashboard',
|
||||
`stored document for release "${release.release_id}" hashes to ${hash}, not the released ` +
|
||||
`${release.document_hash} — refusing to serve an unapproved definition`,
|
||||
);
|
||||
}
|
||||
return document;
|
||||
}
|
||||
|
||||
/**
|
||||
* BI-P061's DoD. Restores version `toVersion` by minting a NEW version with
|
||||
* the SAME document and publishing it. Returns the new release plus the
|
||||
* verification that the restore is faithful.
|
||||
*/
|
||||
async rollback(args: {
|
||||
scope: TenantScope;
|
||||
dashboardId: string;
|
||||
toVersion: number;
|
||||
createdBy?: string;
|
||||
}): Promise<{ release: DashboardRelease; restored_from: number; document_hash: string }> {
|
||||
assertScope(args.scope);
|
||||
const target = await this.store.findVersionByNumber(args.scope, args.dashboardId, args.toVersion);
|
||||
if (!target) {
|
||||
notFound(`dashboard "${args.dashboardId}" has no version ${args.toVersion} in this tenant`);
|
||||
}
|
||||
const live = await this.store.findByState(args.scope, args.dashboardId, 'published');
|
||||
if (live && live.version === args.toVersion) {
|
||||
conflict(`version ${args.toVersion} is already live; a rollback to the live version is a no-op`);
|
||||
}
|
||||
const document = parseDashboardDocument(JSON.parse(target.layout_json));
|
||||
const hash = dashboardDocumentHash(document);
|
||||
if (hash !== target.layout_hash) {
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'0pt.analytics.dashboard',
|
||||
`version ${args.toVersion} hashes to ${hash} but was stored as ${target.layout_hash} — ` +
|
||||
`refusing to roll back to a definition that has changed underneath us`,
|
||||
);
|
||||
}
|
||||
|
||||
const { release } = await this.draft({
|
||||
scope: args.scope,
|
||||
dashboardId: args.dashboardId,
|
||||
document,
|
||||
...(args.createdBy ? { createdBy: args.createdBy } : {}),
|
||||
rollbackOfVersion: args.toVersion,
|
||||
});
|
||||
const { published } = await this.publish({ scope: args.scope, releaseId: release.release_id });
|
||||
|
||||
if (published.document_hash !== target.layout_hash) {
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'0pt.analytics.dashboard',
|
||||
`rollback published ${published.document_hash}, not the requested ${target.layout_hash}`,
|
||||
);
|
||||
}
|
||||
return { release: published, restored_from: args.toVersion, document_hash: published.document_hash };
|
||||
}
|
||||
|
||||
/** Full ledger, oldest first. A rollback is an entry here, never a deletion. */
|
||||
async history(scope: TenantScope, dashboardId: string): Promise<DashboardRelease[]> {
|
||||
assertScope(scope);
|
||||
const rows = await this.store.listReleases(scope, dashboardId);
|
||||
return [...rows].sort((a, b) => a.version - b.version);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* In-memory store. Used by the unit tests, and it enforces the same two
|
||||
* uniqueness rules the SQL does — a store that is laxer than the database
|
||||
* makes the unit tests pass on states production can never reach.
|
||||
*/
|
||||
export class InMemoryDashboardReleaseStore implements DashboardReleaseStore {
|
||||
private readonly versions: DashboardVersionRecord[] = [];
|
||||
private readonly releases: DashboardRelease[] = [];
|
||||
|
||||
private scoped<T extends { tenant_id: string; project_id: string }>(scope: TenantScope, rows: T[]): T[] {
|
||||
return rows.filter((r) => r.tenant_id === scope.tenant_id && r.project_id === scope.project_id);
|
||||
}
|
||||
|
||||
async maxVersion(scope: TenantScope, dashboardId: string): Promise<number> {
|
||||
return this.scoped(scope, this.versions)
|
||||
.filter((v) => v.dashboard_id === dashboardId)
|
||||
.reduce((max, v) => Math.max(max, v.version), 0);
|
||||
}
|
||||
|
||||
async insertVersionAndRelease(version: DashboardVersionRecord, release: DashboardRelease): Promise<void> {
|
||||
const scope = { tenant_id: release.tenant_id, project_id: release.project_id };
|
||||
if (
|
||||
release.state === 'draft' &&
|
||||
this.scoped(scope, this.releases).some(
|
||||
(r) => r.dashboard_id === release.dashboard_id && r.state === 'draft',
|
||||
)
|
||||
) {
|
||||
conflict('one open draft per dashboard');
|
||||
}
|
||||
if (this.versions.some((v) => v.dashboard_id === version.dashboard_id && v.version === version.version)) {
|
||||
conflict(`version ${version.version} already exists for "${version.dashboard_id}"`);
|
||||
}
|
||||
this.versions.push({ ...version });
|
||||
this.releases.push({ ...release });
|
||||
}
|
||||
|
||||
async getRelease(scope: TenantScope, releaseId: string): Promise<DashboardRelease | undefined> {
|
||||
const r = this.scoped(scope, this.releases).find((x) => x.release_id === releaseId);
|
||||
return r ? { ...r } : undefined;
|
||||
}
|
||||
|
||||
async getVersion(scope: TenantScope, id: string): Promise<DashboardVersionRecord | undefined> {
|
||||
const v = this.scoped(scope, this.versions).find((x) => x.dashboard_version_id === id);
|
||||
return v ? { ...v } : undefined;
|
||||
}
|
||||
|
||||
async findVersionByNumber(
|
||||
scope: TenantScope,
|
||||
dashboardId: string,
|
||||
version: number,
|
||||
): Promise<DashboardVersionRecord | undefined> {
|
||||
const v = this.scoped(scope, this.versions).find(
|
||||
(x) => x.dashboard_id === dashboardId && x.version === version,
|
||||
);
|
||||
return v ? { ...v } : undefined;
|
||||
}
|
||||
|
||||
async findByState(
|
||||
scope: TenantScope,
|
||||
dashboardId: string,
|
||||
state: DashboardReleaseState,
|
||||
): Promise<DashboardRelease | undefined> {
|
||||
const r = this.scoped(scope, this.releases).find(
|
||||
(x) => x.dashboard_id === dashboardId && x.state === state,
|
||||
);
|
||||
return r ? { ...r } : undefined;
|
||||
}
|
||||
|
||||
async listReleases(scope: TenantScope, dashboardId: string): Promise<DashboardRelease[]> {
|
||||
return this.scoped(scope, this.releases)
|
||||
.filter((r) => r.dashboard_id === dashboardId)
|
||||
.map((r) => ({ ...r }));
|
||||
}
|
||||
|
||||
async publishAtomically(args: {
|
||||
scope: TenantScope;
|
||||
releaseId: string;
|
||||
publishedAt: string;
|
||||
supersedes?: { release_id: string; superseded_at: string };
|
||||
}): Promise<void> {
|
||||
const target = this.releases.find(
|
||||
(r) =>
|
||||
r.release_id === args.releaseId &&
|
||||
r.tenant_id === args.scope.tenant_id &&
|
||||
r.project_id === args.scope.project_id,
|
||||
);
|
||||
if (!target) notFound(`release "${args.releaseId}" not found`);
|
||||
if (!isLegalTransition(target.state, 'published')) {
|
||||
conflict(`illegal transition ${target.state} -> published`);
|
||||
}
|
||||
const alreadyLive = this.releases.find(
|
||||
(r) =>
|
||||
r.dashboard_id === target.dashboard_id &&
|
||||
r.state === 'published' &&
|
||||
r.release_id !== args.supersedes?.release_id,
|
||||
);
|
||||
if (alreadyLive) conflict('one published release per dashboard');
|
||||
|
||||
if (args.supersedes) {
|
||||
const live = this.releases.find((r) => r.release_id === args.supersedes?.release_id);
|
||||
if (!live) notFound(`release "${args.supersedes.release_id}" not found`);
|
||||
if (!isLegalTransition(live.state, 'superseded')) {
|
||||
conflict(`illegal transition ${live.state} -> superseded`);
|
||||
}
|
||||
live.state = 'superseded';
|
||||
live.superseded_at = args.supersedes.superseded_at;
|
||||
live.superseded_by_release_id = target.release_id;
|
||||
}
|
||||
target.state = 'published';
|
||||
target.published_at = args.publishedAt;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* src/analytics/materialization/drift-detector.ts
|
||||
* ================================================
|
||||
* BI-P105 — is the pre-aggregate still telling the truth?
|
||||
*
|
||||
* The comparison is bucket-by-bucket between the aggregate and a recomputation
|
||||
* from the source over the same window. Three things make it a real detector
|
||||
* rather than a ceremony:
|
||||
*
|
||||
* 1. A MISSING BUCKET IS DRIFT. The nastiest pre-aggregate failure is not a
|
||||
* wrong value, it is an absent row: late-arriving data, a materialized view
|
||||
* that was created after the backfill, a refresh that failed silently. Every
|
||||
* value it DOES hold is correct, so a value-only comparison passes while the
|
||||
* total is wrong. Set difference is checked first, before any arithmetic.
|
||||
*
|
||||
* 2. ZERO BUCKETS IS `insufficient_data`, NEVER `match`. A comparison over an
|
||||
* empty window finds no discrepancies — the same false green BI-P025 found
|
||||
* in a quality check over an empty table. The router treats
|
||||
* `insufficient_data` as "do not serve", not as "fine".
|
||||
*
|
||||
* 3. RELATIVE AND ABSOLUTE TOLERANCE BOTH HAVE TO HOLD. A 0.1% tolerance on a
|
||||
* revenue bucket of 40M is 40 000 kr of invisible error; an absolute-only
|
||||
* tolerance flags every rounding difference on a large bucket. Requiring
|
||||
* both means a discrepancy has to be small in kronor AND small in
|
||||
* proportion.
|
||||
*/
|
||||
|
||||
import type { DriftTolerance, DriftVerdict, MaterializationStatus } from './materialization';
|
||||
|
||||
/** One aggregated row, keyed by its dimension tuple. */
|
||||
export interface AggregateBucket {
|
||||
/** Stable key for the dimension tuple, e.g. "2026-09-01|SE". */
|
||||
key: string;
|
||||
/** metric name -> value. */
|
||||
values: Record<string, number>;
|
||||
}
|
||||
|
||||
export interface DriftDiscrepancy {
|
||||
key: string;
|
||||
metric: string;
|
||||
source_value: number;
|
||||
aggregate_value: number;
|
||||
absolute_delta: number;
|
||||
/** |delta| / |source|, or Infinity when the source value is 0 and the aggregate is not. */
|
||||
relative_delta: number;
|
||||
}
|
||||
|
||||
export interface DriftReport {
|
||||
verdict: DriftVerdict;
|
||||
checked_at: string;
|
||||
buckets_compared: number;
|
||||
/** In the source recomputation, absent from the aggregate. Always drift. */
|
||||
missing_in_aggregate: string[];
|
||||
/** In the aggregate, absent from the source. Always drift (deleted rows). */
|
||||
extra_in_aggregate: string[];
|
||||
discrepancies: DriftDiscrepancy[];
|
||||
worst?: DriftDiscrepancy;
|
||||
tolerance: DriftTolerance;
|
||||
/**
|
||||
* What a caller would have been told if it had trusted the aggregate: the
|
||||
* total the aggregate reports vs the total the source computes, per metric.
|
||||
* This is the number that makes a drift report actionable — "0.4% off" means
|
||||
* nothing until it is "1 284 500 kr reported, 1 271 900 kr true".
|
||||
*/
|
||||
totals: Record<string, { source: number; aggregate: number; delta: number }>;
|
||||
}
|
||||
|
||||
const MAX_REPORTED_DISCREPANCIES = 50;
|
||||
|
||||
function relative(sourceValue: number, delta: number): number {
|
||||
if (sourceValue === 0) return delta === 0 ? 0 : Number.POSITIVE_INFINITY;
|
||||
return Math.abs(delta / sourceValue);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare a recomputation from source against the pre-aggregate.
|
||||
*
|
||||
* `fromSource` is the truth. Both sides must cover the same window; the caller
|
||||
* is responsible for that, and `missing_in_aggregate` will catch it loudly if
|
||||
* they do not.
|
||||
*/
|
||||
export function detectDrift(args: {
|
||||
fromSource: AggregateBucket[];
|
||||
fromAggregate: AggregateBucket[];
|
||||
tolerance: DriftTolerance;
|
||||
metrics: string[];
|
||||
now?: Date;
|
||||
}): DriftReport {
|
||||
const { fromSource, fromAggregate, tolerance, metrics } = args;
|
||||
const checked_at = (args.now ?? new Date()).toISOString();
|
||||
const sourceByKey = new Map(fromSource.map((b) => [b.key, b]));
|
||||
const aggByKey = new Map(fromAggregate.map((b) => [b.key, b]));
|
||||
|
||||
const missing_in_aggregate = [...sourceByKey.keys()].filter((k) => !aggByKey.has(k)).sort();
|
||||
const extra_in_aggregate = [...aggByKey.keys()].filter((k) => !sourceByKey.has(k)).sort();
|
||||
|
||||
const discrepancies: DriftDiscrepancy[] = [];
|
||||
let buckets_compared = 0;
|
||||
const totals: DriftReport['totals'] = {};
|
||||
for (const m of metrics) totals[m] = { source: 0, aggregate: 0, delta: 0 };
|
||||
|
||||
for (const [key, sourceBucket] of sourceByKey) {
|
||||
for (const m of metrics) totals[m].source += sourceBucket.values[m] ?? 0;
|
||||
const aggBucket = aggByKey.get(key);
|
||||
if (!aggBucket) continue;
|
||||
buckets_compared += 1;
|
||||
for (const m of metrics) {
|
||||
const sv = sourceBucket.values[m] ?? 0;
|
||||
const av = aggBucket.values[m] ?? 0;
|
||||
const delta = av - sv;
|
||||
const rel = relative(sv, delta);
|
||||
if (Math.abs(delta) > tolerance.absolute || rel > tolerance.relative) {
|
||||
discrepancies.push({
|
||||
key,
|
||||
metric: m,
|
||||
source_value: sv,
|
||||
aggregate_value: av,
|
||||
absolute_delta: delta,
|
||||
relative_delta: rel,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const bucket of aggByKey.values()) {
|
||||
for (const m of metrics) totals[m].aggregate += bucket.values[m] ?? 0;
|
||||
}
|
||||
for (const m of metrics) totals[m].delta = totals[m].aggregate - totals[m].source;
|
||||
|
||||
discrepancies.sort((a, b) => Math.abs(b.absolute_delta) - Math.abs(a.absolute_delta));
|
||||
|
||||
// A comparison that compared nothing has not proved anything. This is the
|
||||
// BI-P025 false green, restated for pre-aggregates.
|
||||
const verdict: DriftVerdict =
|
||||
buckets_compared === 0 && missing_in_aggregate.length === 0 && extra_in_aggregate.length === 0
|
||||
? 'insufficient_data'
|
||||
: discrepancies.length > 0 || missing_in_aggregate.length > 0 || extra_in_aggregate.length > 0
|
||||
? 'drifted'
|
||||
: 'match';
|
||||
|
||||
const report: DriftReport = {
|
||||
verdict,
|
||||
checked_at,
|
||||
buckets_compared,
|
||||
missing_in_aggregate,
|
||||
extra_in_aggregate,
|
||||
discrepancies: discrepancies.slice(0, MAX_REPORTED_DISCREPANCIES),
|
||||
tolerance,
|
||||
totals,
|
||||
};
|
||||
if (discrepancies.length > 0) report.worst = discrepancies[0];
|
||||
return report;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fold a drift report into the runtime status. A drifted aggregate is taken
|
||||
* OUT of service here rather than merely annotated: leaving it `serving` with
|
||||
* a bad verdict relies on every future reader remembering to check the
|
||||
* verdict, and one that forgets serves the wrong number.
|
||||
*
|
||||
* `insufficient_data` also stops service. It means the check could not prove
|
||||
* anything, and an unproven aggregate is not a current one.
|
||||
*/
|
||||
export function applyDriftReport(status: MaterializationStatus, report: DriftReport): MaterializationStatus {
|
||||
return {
|
||||
...status,
|
||||
state: report.verdict === 'match' ? status.state : 'drifted',
|
||||
last_drift_check_at: report.checked_at,
|
||||
last_drift_verdict: report.verdict,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,309 @@
|
||||
/**
|
||||
* src/analytics/materialization/materialization.ts
|
||||
* =================================================
|
||||
* BI-P105 — MATERIALIZATION / PRE-AGGREGATION.
|
||||
*
|
||||
* A pre-aggregate is the one optimisation that fails SILENTLY. A slow query is
|
||||
* a complaint; a pre-aggregate that has drifted from its source answers
|
||||
* instantly, with confidence, and is wrong — no error, no timeout, nothing in
|
||||
* a log. So this module is built around one rule:
|
||||
*
|
||||
* A MATERIALIZATION SERVES ONLY WHILE IT IS PROVABLY CURRENT AND PROVABLY
|
||||
* UNDRIFTED. Every unknown routes to the source table.
|
||||
*
|
||||
* Six refusal reasons, each of which is a real way the aggregate becomes a
|
||||
* confident liar:
|
||||
*
|
||||
* coverage_miss it does not carry the metric/dimension/grain
|
||||
* asked for (a filter on a dimension the
|
||||
* aggregate collapsed cannot be applied after
|
||||
* the fact, and applying it anyway is how a
|
||||
* pre-aggregate returns another slice's number)
|
||||
* watermark_unknown nobody knows how current it is. BI-P026 already
|
||||
* refuses to render UNKNOWN age as current; the
|
||||
* same doctrine applies here.
|
||||
* staleness_exceeded it is older than its own freshness contract
|
||||
* window_beyond_watermark the question reaches past the newest data the
|
||||
* aggregate contains — it would silently answer
|
||||
* with a shorter period
|
||||
* drift_never_verified it has never been compared to its source
|
||||
* drift_verification_stale the last comparison is older than the interval
|
||||
* the spec requires
|
||||
* drift_detected the last comparison FAILED
|
||||
*
|
||||
* `drift_never_verified` is the one people leave out, and it is the one that
|
||||
* matters most on day one: a brand-new materialization is exactly as unproven
|
||||
* as a drifted one, and a router that defaults new aggregates to "serving"
|
||||
* ships the bug on the day the feature launches.
|
||||
*
|
||||
* The DoD also asks for measured improvement and documented freshness
|
||||
* semantics: scripts/bi-p105-materialization-drift.mjs measures both against a
|
||||
* live ClickHouse and writes evidence/bi-p105/materialization-drift.json,
|
||||
* including what happens WITHOUT this router (the drifted aggregate served).
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import type { AnalyticsQueryRequest, TimeGrain } from '../contract/query-request';
|
||||
|
||||
export const MATERIALIZATION_STRATEGIES = [
|
||||
'clickhouse_materialized_view',
|
||||
'clickhouse_projection',
|
||||
'dbt_incremental',
|
||||
'cube_preaggregation',
|
||||
] as const;
|
||||
export type MaterializationStrategy = (typeof MATERIALIZATION_STRATEGIES)[number];
|
||||
|
||||
export const MATERIALIZATION_STATES = ['building', 'serving', 'drifted', 'disabled'] as const;
|
||||
export type MaterializationState = (typeof MATERIALIZATION_STATES)[number];
|
||||
|
||||
/**
|
||||
* Grains ordered coarsest-last. An aggregate at `day` can answer `day` and
|
||||
* anything coarser by re-aggregating; it can NEVER answer `hour`, because the
|
||||
* detail was thrown away when the rows were rolled up.
|
||||
*/
|
||||
const GRAIN_ORDER: readonly TimeGrain[] = ['hour', 'day', 'week', 'month', 'quarter', 'year'];
|
||||
|
||||
/**
|
||||
* `week` does not divide `month` and `month` does not divide `quarter` by row
|
||||
* count, but both roll up cleanly from `day`. Grains that CANNOT be derived by
|
||||
* re-aggregating a coarser one are listed here so the router refuses rather
|
||||
* than quietly returning a mis-bucketed series.
|
||||
*/
|
||||
function canRollUp(from: TimeGrain, to: TimeGrain): boolean {
|
||||
if (from === to) return true;
|
||||
const fi = GRAIN_ORDER.indexOf(from);
|
||||
const ti = GRAIN_ORDER.indexOf(to);
|
||||
if (fi < 0 || ti < 0 || ti < fi) return false;
|
||||
// Weekly buckets do not compose into calendar months/quarters/years.
|
||||
if (from === 'week' && to !== 'week') return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
export interface DriftTolerance {
|
||||
/** Absolute delta allowed per metric per bucket. */
|
||||
absolute: number;
|
||||
/** Relative delta allowed (0.001 = 0.1%). Both must hold. */
|
||||
relative: number;
|
||||
}
|
||||
|
||||
/** What a materialization IS. Declarative; the engine adapters read it. */
|
||||
export interface MaterializationSpec {
|
||||
materialization_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
/** The dataset it pre-aggregates (BI-P003). */
|
||||
dataset: string;
|
||||
strategy: MaterializationStrategy;
|
||||
/** Physical target (ClickHouse table, dbt model, Cube pre-agg name). */
|
||||
target_ref: string;
|
||||
metrics: string[];
|
||||
dimensions: string[];
|
||||
time_dimension?: string;
|
||||
grain?: TimeGrain;
|
||||
/** The freshness contract. Older than this and it stops serving. */
|
||||
max_staleness_seconds: number;
|
||||
/** How often the aggregate must be re-proved against its source. */
|
||||
drift_check_interval_seconds: number;
|
||||
tolerance: DriftTolerance;
|
||||
}
|
||||
|
||||
export const DRIFT_VERDICTS = ['match', 'drifted', 'insufficient_data'] as const;
|
||||
export type DriftVerdict = (typeof DRIFT_VERDICTS)[number];
|
||||
|
||||
/** Runtime state. Separate from the spec: the spec is config, this is fact. */
|
||||
export interface MaterializationStatus {
|
||||
materialization_id: string;
|
||||
state: MaterializationState;
|
||||
/**
|
||||
* The newest source timestamp the aggregate represents. UNDEFINED means
|
||||
* UNKNOWN, and unknown is never treated as current.
|
||||
*/
|
||||
data_watermark?: string;
|
||||
last_refresh_at?: string;
|
||||
last_drift_check_at?: string;
|
||||
last_drift_verdict?: DriftVerdict;
|
||||
}
|
||||
|
||||
export const ROUTING_REFUSALS = [
|
||||
'coverage_miss',
|
||||
'watermark_unknown',
|
||||
'staleness_exceeded',
|
||||
'window_beyond_watermark',
|
||||
'drift_never_verified',
|
||||
'drift_verification_stale',
|
||||
'drift_detected',
|
||||
'state_not_serving',
|
||||
] as const;
|
||||
export type RoutingRefusal = (typeof ROUTING_REFUSALS)[number];
|
||||
|
||||
export type RoutingDecision =
|
||||
| {
|
||||
source: 'materialization';
|
||||
materialization_id: string;
|
||||
target_ref: string;
|
||||
/** Reported so a caller can show it; never inferred, always measured. */
|
||||
staleness_seconds: number;
|
||||
}
|
||||
| { source: 'base_table'; refusal: RoutingRefusal; detail: string };
|
||||
|
||||
function refuse(refusal: RoutingRefusal, detail: string): RoutingDecision {
|
||||
return { source: 'base_table', refusal, detail };
|
||||
}
|
||||
|
||||
/**
|
||||
* Staleness of an aggregate, in seconds, or `undefined` when it cannot be
|
||||
* computed. Deliberately NOT defaulted to 0: "we could not tell" and "it is
|
||||
* current" are different facts, and conflating them is the bug.
|
||||
*/
|
||||
export function materializationStaleness(status: MaterializationStatus, now: Date): number | undefined {
|
||||
if (!status.data_watermark) return undefined;
|
||||
const t = Date.parse(status.data_watermark);
|
||||
if (Number.isNaN(t)) return undefined;
|
||||
return Math.max(0, Math.floor((now.getTime() - t) / 1000));
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether a request may be answered from the pre-aggregate.
|
||||
*
|
||||
* Fail-closed by construction: the function returns `materialization` only
|
||||
* after every check has passed, and every unknown is a refusal with a named
|
||||
* reason. Callers get the reason so it can be logged and counted (BI-P103
|
||||
* reads the counts).
|
||||
*/
|
||||
export function selectQuerySource(args: {
|
||||
request: AnalyticsQueryRequest;
|
||||
spec: MaterializationSpec;
|
||||
status: MaterializationStatus;
|
||||
now: Date;
|
||||
}): RoutingDecision {
|
||||
const { request, spec, status, now } = args;
|
||||
|
||||
if (request.dataset !== spec.dataset) {
|
||||
return refuse('coverage_miss', `materialization covers "${spec.dataset}", not "${request.dataset}"`);
|
||||
}
|
||||
if (status.state !== 'serving') {
|
||||
return refuse('state_not_serving', `materialization state is "${status.state}"`);
|
||||
}
|
||||
|
||||
// ── coverage ────────────────────────────────────────────────────────────
|
||||
const metrics = new Set(spec.metrics);
|
||||
for (const m of request.metrics) {
|
||||
if (!metrics.has(m)) return refuse('coverage_miss', `metric "${m}" is not in the pre-aggregate`);
|
||||
}
|
||||
const dimensions = new Set(spec.dimensions);
|
||||
for (const d of request.dimensions) {
|
||||
if (!dimensions.has(d)) return refuse('coverage_miss', `dimension "${d}" is not in the pre-aggregate`);
|
||||
}
|
||||
for (const f of request.filters) {
|
||||
if (!dimensions.has(f.dimension)) {
|
||||
// The aggregate summed across this dimension. Filtering afterwards would
|
||||
// return a number computed over rows the filter should have excluded.
|
||||
return refuse(
|
||||
'coverage_miss',
|
||||
`filter on "${f.dimension}" cannot be applied: the pre-aggregate has already summed across it`,
|
||||
);
|
||||
}
|
||||
}
|
||||
const requestedGrain = request.timeRange?.grain;
|
||||
if (requestedGrain) {
|
||||
if (!spec.grain) return refuse('coverage_miss', 'the pre-aggregate has no time grain');
|
||||
if (!canRollUp(spec.grain, requestedGrain)) {
|
||||
return refuse(
|
||||
'coverage_miss',
|
||||
`a "${spec.grain}" pre-aggregate cannot answer at "${requestedGrain}" grain`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ── freshness ───────────────────────────────────────────────────────────
|
||||
const staleness = materializationStaleness(status, now);
|
||||
if (staleness === undefined) {
|
||||
return refuse(
|
||||
'watermark_unknown',
|
||||
'the pre-aggregate has no data watermark — unknown age is not current age',
|
||||
);
|
||||
}
|
||||
if (staleness > spec.max_staleness_seconds) {
|
||||
return refuse(
|
||||
'staleness_exceeded',
|
||||
`${staleness}s stale, contract allows ${spec.max_staleness_seconds}s`,
|
||||
);
|
||||
}
|
||||
const to = request.timeRange?.to;
|
||||
if (to) {
|
||||
const toMs = Date.parse(to);
|
||||
const watermarkMs = Date.parse(status.data_watermark!);
|
||||
if (!Number.isNaN(toMs) && toMs > watermarkMs) {
|
||||
return refuse(
|
||||
'window_beyond_watermark',
|
||||
`the question runs to ${to} but the pre-aggregate only contains data to ${status.data_watermark}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ── drift ───────────────────────────────────────────────────────────────
|
||||
if (!status.last_drift_check_at || !status.last_drift_verdict) {
|
||||
return refuse(
|
||||
'drift_never_verified',
|
||||
'the pre-aggregate has never been compared to its source; a new aggregate is as unproven as a drifted one',
|
||||
);
|
||||
}
|
||||
if (status.last_drift_verdict !== 'match') {
|
||||
return refuse('drift_detected', `last drift check returned "${status.last_drift_verdict}"`);
|
||||
}
|
||||
const checkedAgo = Math.floor((now.getTime() - Date.parse(status.last_drift_check_at)) / 1000);
|
||||
if (Number.isNaN(checkedAgo)) {
|
||||
return refuse('drift_never_verified', 'last_drift_check_at is not a timestamp');
|
||||
}
|
||||
if (checkedAgo > spec.drift_check_interval_seconds) {
|
||||
return refuse(
|
||||
'drift_verification_stale',
|
||||
`last verified ${checkedAgo}s ago, spec requires every ${spec.drift_check_interval_seconds}s`,
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
source: 'materialization',
|
||||
materialization_id: spec.materialization_id,
|
||||
target_ref: spec.target_ref,
|
||||
staleness_seconds: staleness,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The routing decision as a freshness envelope for the result (BI-P026 shape):
|
||||
* a consumer must be able to see WHICH source answered and how old it was.
|
||||
* A result that does not say is a result nobody can audit.
|
||||
*/
|
||||
export function describeRouting(decision: RoutingDecision): {
|
||||
served_from: 'materialization' | 'base_table';
|
||||
materialization_id?: string;
|
||||
staleness_seconds?: number;
|
||||
refusal?: RoutingRefusal;
|
||||
detail?: string;
|
||||
} {
|
||||
return decision.source === 'materialization'
|
||||
? {
|
||||
served_from: 'materialization',
|
||||
materialization_id: decision.materialization_id,
|
||||
staleness_seconds: decision.staleness_seconds,
|
||||
}
|
||||
: { served_from: 'base_table', refusal: decision.refusal, detail: decision.detail };
|
||||
}
|
||||
|
||||
export function assertSpec(spec: MaterializationSpec): void {
|
||||
const invalid = (d: string): never => {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.materialization', d);
|
||||
};
|
||||
if (!spec.materialization_id?.trim()) invalid('materialization_id is required');
|
||||
if (!spec.tenant_id?.trim() || !spec.project_id?.trim()) invalid('a materialization is tenant-scoped');
|
||||
if (spec.metrics.length === 0) invalid('a materialization with no metrics pre-aggregates nothing');
|
||||
if (spec.max_staleness_seconds <= 0) invalid('max_staleness_seconds must be positive');
|
||||
if (spec.drift_check_interval_seconds <= 0) {
|
||||
invalid(
|
||||
'drift_check_interval_seconds must be positive — an aggregate that is never re-verified is one ' +
|
||||
'source mutation away from serving wrong numbers forever',
|
||||
);
|
||||
}
|
||||
if (spec.tolerance.absolute < 0 || spec.tolerance.relative < 0) invalid('tolerance must be non-negative');
|
||||
if (spec.grain && !GRAIN_ORDER.includes(spec.grain)) invalid('grain is not a time grain');
|
||||
}
|
||||
@@ -0,0 +1,501 @@
|
||||
/**
|
||||
* src/analytics/slo/bi-slo.ts
|
||||
* ============================
|
||||
* BI-P103 — BI SLO. DoD: "measured baselines exist".
|
||||
*
|
||||
* An SLO you cannot measure from real telemetry is a wish, so every objective
|
||||
* here declares WHERE its number comes from, and the evaluator refuses to
|
||||
* produce a number it does not have. Three ways that refusal shows up, each of
|
||||
* which is a lie the usual implementation tells:
|
||||
*
|
||||
* insufficient_data nothing (or too little) was observed in the window.
|
||||
* An availability objective evaluated over zero events
|
||||
* is NOT 100% attainment — it is silence. This is the
|
||||
* single most common false green in SLO reporting, and
|
||||
* it is the same shape BI-P025 found in a quality
|
||||
* check over an empty table.
|
||||
* not_measurable_here the objective is real but these events cannot
|
||||
* measure it. Dashboard data FRESHNESS is a property
|
||||
* of the dataset (BI-P026 / BI-P105 watermarks), not
|
||||
* of a usage event, and REPORT DELIVERY is N0TIFY's
|
||||
* fact — aud0's seven types carry no delivery event.
|
||||
* Saying so beats fabricating a proxy.
|
||||
* low_confidence a p95 or p99 computed from a handful of samples. The
|
||||
* number is reported WITH its sample size and marked,
|
||||
* because a p99 over 12 requests is the 12th-slowest
|
||||
* request wearing a percentile's clothes.
|
||||
*
|
||||
* The measurement path is aud0's seven BI usage event types (BI-P076, merged
|
||||
* in aud0#34), produced by src/analytics/slo/usage-events.ts. There is
|
||||
* deliberately no second telemetry path: two measurement systems disagree, and
|
||||
* the disagreement is discovered during an incident.
|
||||
*/
|
||||
import type { Aud0BiEvent } from './usage-events';
|
||||
|
||||
export const BI_WORKLOAD_CLASSES = ['interactive_dashboard', 'exploration', 'report'] as const;
|
||||
export type BiWorkloadClass = (typeof BI_WORKLOAD_CLASSES)[number];
|
||||
|
||||
export const SLO_OBJECTIVE_KINDS = [
|
||||
'availability',
|
||||
'latency_p95_first',
|
||||
'latency_p95_cached',
|
||||
'latency_p95',
|
||||
'latency_p99',
|
||||
'error_rate',
|
||||
'freshness',
|
||||
'max_result_rows',
|
||||
'timeout',
|
||||
'completion_sla',
|
||||
'delivery_sla',
|
||||
] as const;
|
||||
export type SloObjectiveKind = (typeof SLO_OBJECTIVE_KINDS)[number];
|
||||
|
||||
/** Where the number comes from. `none` is an admission, not a placeholder. */
|
||||
export const MEASUREMENT_SOURCES = [
|
||||
'aud0_bi_usage_events',
|
||||
'analytics_freshness_watermark',
|
||||
'n0tify_delivery_receipt',
|
||||
'engine_configuration',
|
||||
] as const;
|
||||
export type MeasurementSource = (typeof MEASUREMENT_SOURCES)[number];
|
||||
|
||||
export interface SloObjective {
|
||||
objective_id: string;
|
||||
workload: BiWorkloadClass;
|
||||
kind: SloObjectiveKind;
|
||||
/** Target value; unit is fixed by kind (ms, ratio, rows, seconds). */
|
||||
target: number;
|
||||
unit: 'ms' | 'ratio' | 'rows' | 'seconds';
|
||||
/** Higher-is-better (availability) vs lower-is-better (latency). */
|
||||
direction: 'at_least' | 'at_most';
|
||||
measured_from: MeasurementSource;
|
||||
/** Fewer samples than this and the verdict carries low_confidence. */
|
||||
min_samples: number;
|
||||
rationale: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The catalogue, field-for-field with what BI-P103 lists per workload class.
|
||||
* Targets are STARTING POINTS chosen to be honest about what has been
|
||||
* measured: they are the thresholds the evaluator compares against, and
|
||||
* evidence/bi-p103 records the first baselines rather than pretending the
|
||||
* targets were derived from a year of production traffic.
|
||||
*/
|
||||
export const BI_SLO_CATALOGUE: readonly SloObjective[] = [
|
||||
// ── interactive dashboard ────────────────────────────────────────────────
|
||||
{
|
||||
objective_id: 'dash.availability',
|
||||
workload: 'interactive_dashboard',
|
||||
kind: 'availability',
|
||||
target: 0.995,
|
||||
unit: 'ratio',
|
||||
direction: 'at_least',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 100,
|
||||
rationale: 'completed / (completed + failed) for dashboard-surface queries.',
|
||||
},
|
||||
{
|
||||
objective_id: 'dash.p95_first_query',
|
||||
workload: 'interactive_dashboard',
|
||||
kind: 'latency_p95_first',
|
||||
target: 3000,
|
||||
unit: 'ms',
|
||||
direction: 'at_most',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 50,
|
||||
rationale: 'cache_hit=false: what a user waits for when they open the dashboard cold.',
|
||||
},
|
||||
{
|
||||
objective_id: 'dash.p95_cached',
|
||||
workload: 'interactive_dashboard',
|
||||
kind: 'latency_p95_cached',
|
||||
target: 400,
|
||||
unit: 'ms',
|
||||
direction: 'at_most',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 50,
|
||||
rationale:
|
||||
'cache_hit=true: BI-P020 cache hits, measured separately so a high hit rate cannot hide a slow cold path.',
|
||||
},
|
||||
{
|
||||
objective_id: 'dash.error_rate',
|
||||
workload: 'interactive_dashboard',
|
||||
kind: 'error_rate',
|
||||
target: 0.005,
|
||||
unit: 'ratio',
|
||||
direction: 'at_most',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 100,
|
||||
rationale: 'failed / terminal. Cancellations count as failures here: the user did not get an answer.',
|
||||
},
|
||||
{
|
||||
objective_id: 'dash.freshness',
|
||||
workload: 'interactive_dashboard',
|
||||
kind: 'freshness',
|
||||
target: 900,
|
||||
unit: 'seconds',
|
||||
direction: 'at_most',
|
||||
measured_from: 'analytics_freshness_watermark',
|
||||
min_samples: 1,
|
||||
rationale:
|
||||
'Data age is a property of the DATASET (BI-P026 freshness, BI-P105 watermark), not of a usage ' +
|
||||
'event. Usage events cannot measure it and this evaluator does not pretend to.',
|
||||
},
|
||||
// ── exploration ──────────────────────────────────────────────────────────
|
||||
{
|
||||
objective_id: 'explore.p95',
|
||||
workload: 'exploration',
|
||||
kind: 'latency_p95',
|
||||
target: 10_000,
|
||||
unit: 'ms',
|
||||
direction: 'at_most',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 50,
|
||||
rationale: 'explore-surface queries; an analyst tolerates more than a dashboard viewer.',
|
||||
},
|
||||
{
|
||||
objective_id: 'explore.p99',
|
||||
workload: 'exploration',
|
||||
kind: 'latency_p99',
|
||||
target: 60_000,
|
||||
unit: 'ms',
|
||||
direction: 'at_most',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 200,
|
||||
rationale: 'p99 needs a genuinely large sample; below min_samples it is reported as low confidence.',
|
||||
},
|
||||
{
|
||||
objective_id: 'explore.max_result',
|
||||
workload: 'exploration',
|
||||
kind: 'max_result_rows',
|
||||
target: 1_000_000,
|
||||
unit: 'rows',
|
||||
direction: 'at_most',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 1,
|
||||
rationale: 'BI-P012 truncates at the engine ceiling; this watches what actually came back.',
|
||||
},
|
||||
{
|
||||
objective_id: 'explore.timeout',
|
||||
workload: 'exploration',
|
||||
kind: 'timeout',
|
||||
target: 300,
|
||||
unit: 'seconds',
|
||||
direction: 'at_most',
|
||||
measured_from: 'engine_configuration',
|
||||
min_samples: 1,
|
||||
rationale: 'The configured ceiling (BI-P012 settings profile), not an observation.',
|
||||
},
|
||||
// ── reports ──────────────────────────────────────────────────────────────
|
||||
{
|
||||
objective_id: 'report.completion_sla',
|
||||
workload: 'report',
|
||||
kind: 'completion_sla',
|
||||
target: 900_000,
|
||||
unit: 'ms',
|
||||
direction: 'at_most',
|
||||
measured_from: 'aud0_bi_usage_events',
|
||||
min_samples: 20,
|
||||
rationale: 'aud0.report_generated duration_ms: submitted to produced.',
|
||||
},
|
||||
{
|
||||
objective_id: 'report.delivery_sla',
|
||||
workload: 'report',
|
||||
kind: 'delivery_sla',
|
||||
target: 300_000,
|
||||
unit: 'ms',
|
||||
direction: 'at_most',
|
||||
measured_from: 'n0tify_delivery_receipt',
|
||||
min_samples: 20,
|
||||
rationale:
|
||||
"Produced is not delivered. aud0 has no delivery event; the receipt is N0TIFY's, so this " +
|
||||
'objective is declared and reported unmeasured rather than proxied by completion.',
|
||||
},
|
||||
];
|
||||
|
||||
export const SLO_STATUSES = ['met', 'breached', 'insufficient_data', 'not_measurable_here'] as const;
|
||||
export type SloStatus = (typeof SLO_STATUSES)[number];
|
||||
|
||||
export interface SloResult {
|
||||
objective_id: string;
|
||||
workload: BiWorkloadClass;
|
||||
kind: SloObjectiveKind;
|
||||
status: SloStatus;
|
||||
target: number;
|
||||
unit: SloObjective['unit'];
|
||||
/** The measured value, or undefined when nothing could be measured. */
|
||||
observed?: number;
|
||||
sample_size: number;
|
||||
/** True when a percentile was computed from fewer samples than min_samples. */
|
||||
low_confidence?: boolean;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface SloReport {
|
||||
window_from: string;
|
||||
window_to: string;
|
||||
results: SloResult[];
|
||||
/** Queries that started and never reached a terminal event in the window. */
|
||||
unresolved_queries: number;
|
||||
/** True only when EVERY objective produced a real number. */
|
||||
fully_measured: boolean;
|
||||
}
|
||||
|
||||
interface QueryFact {
|
||||
query_id: string;
|
||||
surface?: string;
|
||||
started_at: number;
|
||||
outcome?: 'completed' | 'failed';
|
||||
duration_ms?: number;
|
||||
row_count?: number;
|
||||
cache_hit?: boolean;
|
||||
}
|
||||
|
||||
function num(v: unknown): number | undefined {
|
||||
return typeof v === 'number' && Number.isFinite(v) ? v : undefined;
|
||||
}
|
||||
|
||||
function str(v: unknown): string | undefined {
|
||||
return typeof v === 'string' ? v : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Nearest-rank percentile. No interpolation: with small samples an
|
||||
* interpolated p99 invents a value that no request ever experienced.
|
||||
*/
|
||||
export function percentile(sorted: number[], q: number): number | undefined {
|
||||
if (sorted.length === 0) return undefined;
|
||||
const rank = Math.ceil(q * sorted.length);
|
||||
return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))];
|
||||
}
|
||||
|
||||
/** Fold the seven event types into per-query facts. */
|
||||
function foldQueries(events: Aud0BiEvent[]): Map<string, QueryFact> {
|
||||
const facts = new Map<string, QueryFact>();
|
||||
for (const e of events) {
|
||||
const qid = str(e.payload.query_id);
|
||||
if (!qid) continue;
|
||||
const at = Date.parse(e.occurred_at);
|
||||
const existing = facts.get(qid) ?? { query_id: qid, started_at: Number.isNaN(at) ? 0 : at };
|
||||
if (e.payload.surface !== undefined) existing.surface = str(e.payload.surface);
|
||||
if (e.event_type === 'aud0.analytics_query_started' && !Number.isNaN(at)) existing.started_at = at;
|
||||
if (e.event_type === 'aud0.analytics_query_completed') {
|
||||
existing.outcome = 'completed';
|
||||
existing.duration_ms = num(e.payload.duration_ms);
|
||||
existing.row_count = num(e.payload.row_count);
|
||||
existing.cache_hit = e.payload.cache_hit === true;
|
||||
}
|
||||
if (e.event_type === 'aud0.analytics_query_failed') {
|
||||
existing.outcome = 'failed';
|
||||
existing.duration_ms = num(e.payload.duration_ms);
|
||||
}
|
||||
facts.set(qid, existing);
|
||||
}
|
||||
return facts;
|
||||
}
|
||||
|
||||
function result(
|
||||
o: SloObjective,
|
||||
status: SloStatus,
|
||||
sample_size: number,
|
||||
detail: string,
|
||||
observed?: number,
|
||||
low_confidence?: boolean,
|
||||
): SloResult {
|
||||
const r: SloResult = {
|
||||
objective_id: o.objective_id,
|
||||
workload: o.workload,
|
||||
kind: o.kind,
|
||||
status,
|
||||
target: o.target,
|
||||
unit: o.unit,
|
||||
sample_size,
|
||||
detail,
|
||||
};
|
||||
if (observed !== undefined) r.observed = observed;
|
||||
if (low_confidence) r.low_confidence = true;
|
||||
return r;
|
||||
}
|
||||
|
||||
function compare(o: SloObjective, observed: number): SloStatus {
|
||||
return o.direction === 'at_least'
|
||||
? observed >= o.target
|
||||
? 'met'
|
||||
: 'breached'
|
||||
: observed <= o.target
|
||||
? 'met'
|
||||
: 'breached';
|
||||
}
|
||||
|
||||
/**
|
||||
* Evaluate the catalogue over a window of AUD0 BI usage events.
|
||||
*
|
||||
* Events are DATA: they arrive from aud0's read model, they were written by
|
||||
* other services, and nothing in them is treated as an instruction or as a
|
||||
* schema. Unknown event types and unknown payload keys are ignored, never
|
||||
* spread.
|
||||
*/
|
||||
export function evaluateBiSlos(args: {
|
||||
events: Aud0BiEvent[];
|
||||
from: Date;
|
||||
to: Date;
|
||||
objectives?: readonly SloObjective[];
|
||||
}): SloReport {
|
||||
const objectives = args.objectives ?? BI_SLO_CATALOGUE;
|
||||
const fromMs = args.from.getTime();
|
||||
const toMs = args.to.getTime();
|
||||
const inWindow = args.events.filter((e) => {
|
||||
const t = Date.parse(e.occurred_at);
|
||||
return !Number.isNaN(t) && t >= fromMs && t <= toMs;
|
||||
});
|
||||
|
||||
const queries = [...foldQueries(inWindow).values()];
|
||||
const dashboardQueries = queries.filter((q) => q.surface === 'dashboard');
|
||||
const exploreQueries = queries.filter((q) => q.surface === 'explore');
|
||||
const terminal = (qs: QueryFact[]) => qs.filter((q) => q.outcome !== undefined);
|
||||
const durations = (qs: QueryFact[]) =>
|
||||
qs
|
||||
.map((q) => q.duration_ms)
|
||||
.filter((d): d is number => d !== undefined)
|
||||
.sort((a, b) => a - b);
|
||||
|
||||
const reportDurations = inWindow
|
||||
.filter((e) => e.event_type === 'aud0.report_generated')
|
||||
.map((e) => num(e.payload.duration_ms))
|
||||
.filter((d): d is number => d !== undefined)
|
||||
.sort((a, b) => a - b);
|
||||
|
||||
const results = objectives.map((o) => {
|
||||
if (o.measured_from !== 'aud0_bi_usage_events') {
|
||||
return result(
|
||||
o,
|
||||
'not_measurable_here',
|
||||
0,
|
||||
`measured from ${o.measured_from}; BI usage events cannot answer it — ${o.rationale}`,
|
||||
);
|
||||
}
|
||||
|
||||
switch (o.kind) {
|
||||
case 'availability':
|
||||
case 'error_rate': {
|
||||
const t = terminal(dashboardQueries);
|
||||
if (t.length === 0) {
|
||||
return result(o, 'insufficient_data', 0, 'no terminal dashboard query events in the window');
|
||||
}
|
||||
const failed = t.filter((q) => q.outcome === 'failed').length;
|
||||
const observed = o.kind === 'availability' ? (t.length - failed) / t.length : failed / t.length;
|
||||
const status = compare(o, observed);
|
||||
return result(
|
||||
o,
|
||||
t.length < o.min_samples ? 'insufficient_data' : status,
|
||||
t.length,
|
||||
t.length < o.min_samples
|
||||
? `${t.length} terminal queries is below the ${o.min_samples} this objective requires; ` +
|
||||
`the ratio ${observed.toFixed(4)} is reported but not judged`
|
||||
: `${failed} failed of ${t.length} terminal queries`,
|
||||
observed,
|
||||
t.length < o.min_samples,
|
||||
);
|
||||
}
|
||||
case 'latency_p95_first':
|
||||
case 'latency_p95_cached': {
|
||||
const wanted = o.kind === 'latency_p95_cached';
|
||||
const sample = durations(dashboardQueries.filter((q) => (q.cache_hit === true) === wanted));
|
||||
const p95 = percentile(sample, 0.95);
|
||||
if (p95 === undefined) {
|
||||
return result(
|
||||
o,
|
||||
'insufficient_data',
|
||||
0,
|
||||
`no ${wanted ? 'cached' : 'uncached'} dashboard durations`,
|
||||
);
|
||||
}
|
||||
const low = sample.length < o.min_samples;
|
||||
return result(
|
||||
o,
|
||||
low ? 'insufficient_data' : compare(o, p95),
|
||||
sample.length,
|
||||
low
|
||||
? `p95 over ${sample.length} samples (needs ${o.min_samples}) — reported, not judged`
|
||||
: `p95 over ${sample.length} samples`,
|
||||
p95,
|
||||
low,
|
||||
);
|
||||
}
|
||||
case 'latency_p95':
|
||||
case 'latency_p99': {
|
||||
const sample = durations(exploreQueries);
|
||||
const q = o.kind === 'latency_p99' ? 0.99 : 0.95;
|
||||
const value = percentile(sample, q);
|
||||
if (value === undefined) return result(o, 'insufficient_data', 0, 'no exploration durations');
|
||||
const low = sample.length < o.min_samples;
|
||||
return result(
|
||||
o,
|
||||
low ? 'insufficient_data' : compare(o, value),
|
||||
sample.length,
|
||||
low
|
||||
? `p${q * 100} over ${sample.length} samples (needs ${o.min_samples}); with this few samples ` +
|
||||
`it is the ${Math.ceil((1 - q) * sample.length) || 1}-slowest request, not a percentile`
|
||||
: `p${q * 100} over ${sample.length} samples`,
|
||||
value,
|
||||
low,
|
||||
);
|
||||
}
|
||||
case 'max_result_rows': {
|
||||
const rows = exploreQueries
|
||||
.map((q) => q.row_count)
|
||||
.filter((r): r is number => r !== undefined)
|
||||
.sort((a, b) => b - a);
|
||||
if (rows.length === 0) return result(o, 'insufficient_data', 0, 'no exploration row counts');
|
||||
return result(o, compare(o, rows[0]), rows.length, `largest result seen in the window`, rows[0]);
|
||||
}
|
||||
case 'completion_sla': {
|
||||
const p95 = percentile(reportDurations, 0.95);
|
||||
if (p95 === undefined) return result(o, 'insufficient_data', 0, 'no report_generated events');
|
||||
const low = reportDurations.length < o.min_samples;
|
||||
return result(
|
||||
o,
|
||||
low ? 'insufficient_data' : compare(o, p95),
|
||||
reportDurations.length,
|
||||
low ? `p95 over ${reportDurations.length} reports (needs ${o.min_samples})` : 'p95 report duration',
|
||||
p95,
|
||||
low,
|
||||
);
|
||||
}
|
||||
default:
|
||||
return result(o, 'not_measurable_here', 0, `no evaluator for kind "${o.kind}"`);
|
||||
}
|
||||
});
|
||||
|
||||
const unresolved = queries.filter((q) => q.outcome === undefined).length;
|
||||
return {
|
||||
window_from: args.from.toISOString(),
|
||||
window_to: args.to.toISOString(),
|
||||
results,
|
||||
unresolved_queries: unresolved,
|
||||
fully_measured: results.every((r) => r.status === 'met' || r.status === 'breached'),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* One line an operator can read: how many objectives are actually being
|
||||
* measured. "9 of 11 met" is meaningless if the other two were never
|
||||
* measured, so the summary counts them separately and never rolls them in.
|
||||
*/
|
||||
export function summariseSloReport(report: SloReport): {
|
||||
met: number;
|
||||
breached: number;
|
||||
insufficient_data: number;
|
||||
not_measurable_here: number;
|
||||
measured_fraction: number;
|
||||
} {
|
||||
const count = (s: SloStatus) => report.results.filter((r) => r.status === s).length;
|
||||
const measured = count('met') + count('breached');
|
||||
return {
|
||||
met: count('met'),
|
||||
breached: count('breached'),
|
||||
insufficient_data: count('insufficient_data'),
|
||||
not_measurable_here: count('not_measurable_here'),
|
||||
measured_fraction: report.results.length === 0 ? 0 : measured / report.results.length,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
/**
|
||||
* src/analytics/slo/usage-events.ts
|
||||
* ==================================
|
||||
* BI-P103's measurement path: 0pt emits the AUD0 BI usage events (aud0#34,
|
||||
* BI-P076) that the SLO evaluator then reads.
|
||||
*
|
||||
* aud0 says it plainly in docs/BI_ANALYTICS_USAGE.md: "Zero real producers
|
||||
* emit these seven types today." An SLO computed from a second, private
|
||||
* telemetry path would be a number nobody else can audit, so 0pt becomes a
|
||||
* producer instead of inventing one.
|
||||
*
|
||||
* THE PAYLOAD IS AN ALLOWLIST, BUILT FIELD BY NAMED FIELD. AUD0's ingest
|
||||
* refuses result-bearing payloads with a 400 and its schemas are `.strict()`,
|
||||
* so a leak here is a 400 rather than a breach — but a producer that has to be
|
||||
* refused is a producer that is broken. `biUsageEventsForExecution` therefore
|
||||
* never spreads an execution record; it names each field, and
|
||||
* tests/analytics/bi-slo.test.ts feeds it a record carrying SQL text, an error
|
||||
* message, a tenant id and a result sample, then asserts none of it appears in
|
||||
* the output.
|
||||
*
|
||||
* WHAT IS DELIBERATELY NOT SENT: `compiled_query_hash` goes out as
|
||||
* `query_hash` (aud0 requires a 64-hex digest and nothing else), the statement
|
||||
* never leaves 0pt at all, `error_code` travels but `error_message` does not,
|
||||
* and `row_count` travels but no row does.
|
||||
*/
|
||||
|
||||
/** The seven event types, named exactly as aud0 registers them. */
|
||||
export const AUD0_BI_EVENT_TYPES = [
|
||||
'aud0.analytics_query_started',
|
||||
'aud0.analytics_query_completed',
|
||||
'aud0.analytics_query_failed',
|
||||
'aud0.dashboard_viewed',
|
||||
'aud0.report_generated',
|
||||
'aud0.dataset_exported',
|
||||
'aud0.metric_queried',
|
||||
] as const;
|
||||
export type Aud0BiEventType = (typeof AUD0_BI_EVENT_TYPES)[number];
|
||||
|
||||
/** aud0's closed surface list (BI_SURFACES). */
|
||||
export const BI_SURFACES = ['dashboard', 'report', 'explore', 'api', 'scheduled', 'notebook'] as const;
|
||||
export type BiSurface = (typeof BI_SURFACES)[number];
|
||||
|
||||
export interface Aud0BiEvent {
|
||||
event_type: Aud0BiEventType;
|
||||
/** Bounded, allowlisted metadata. Never a result, a statement or a message. */
|
||||
payload: Record<string, string | number | boolean | string[]>;
|
||||
/** Occurrence time, used by the SLO window. */
|
||||
occurred_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The 0pt-side view of one execution. Intentionally a SUPERSET of what may be
|
||||
* emitted — it carries the fields 0pt genuinely has, including the ones that
|
||||
* must never leave — so that the mapping below is a real filter rather than a
|
||||
* rename.
|
||||
*/
|
||||
export interface ExecutionTelemetry {
|
||||
query_id: string;
|
||||
/** 64-hex; anything else is dropped rather than reshaped. */
|
||||
compiled_query_hash?: string;
|
||||
/** NEVER emitted. Present here because 0pt has it. */
|
||||
compiled_sql?: string;
|
||||
/** NEVER emitted. */
|
||||
error_message?: string;
|
||||
error_code?: string;
|
||||
/** NEVER emitted — a consumer must not be able to name a tenant, BI-P016. */
|
||||
tenant_id?: string;
|
||||
dataset_id?: string;
|
||||
engine?: string;
|
||||
surface?: BiSurface;
|
||||
state: 'succeeded' | 'failed' | 'cancelled' | 'running';
|
||||
started_at: string;
|
||||
finished_at?: string;
|
||||
duration_ms?: number;
|
||||
row_count?: number;
|
||||
bytes_scanned?: number;
|
||||
cache_hit?: boolean;
|
||||
filter_keys?: string[];
|
||||
/** NEVER emitted: filter VALUES are facts about a subject. */
|
||||
filter_values?: Record<string, string[]>;
|
||||
/** NEVER emitted. */
|
||||
result_sample?: unknown;
|
||||
metric_name?: string;
|
||||
dimension_keys?: string[];
|
||||
granularity?: string;
|
||||
dashboard_id?: string;
|
||||
widget_count?: number;
|
||||
}
|
||||
|
||||
const SHA256 = /^[0-9a-f]{64}$/;
|
||||
const MAX_KEY_NAMES = 50;
|
||||
const MAX_SHORT_TEXT = 120;
|
||||
|
||||
function shortText(v: string | undefined): string | undefined {
|
||||
if (!v) return undefined;
|
||||
return v.slice(0, MAX_SHORT_TEXT);
|
||||
}
|
||||
|
||||
function keyNames(v: string[] | undefined): string[] | undefined {
|
||||
if (!v || v.length === 0) return undefined;
|
||||
return v.slice(0, MAX_KEY_NAMES).map((k) => k.slice(0, MAX_SHORT_TEXT));
|
||||
}
|
||||
|
||||
function put(
|
||||
payload: Record<string, string | number | boolean | string[]>,
|
||||
key: string,
|
||||
value: string | number | boolean | string[] | undefined,
|
||||
): void {
|
||||
if (value === undefined) return;
|
||||
payload[key] = value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map one execution to its AUD0 usage events.
|
||||
*
|
||||
* A running execution emits only `started`: a query that has not finished is
|
||||
* neither a success nor a failure, and the SLO evaluator counts it as
|
||||
* unresolved rather than quietly as one or the other.
|
||||
*/
|
||||
export function biUsageEventsForExecution(t: ExecutionTelemetry): Aud0BiEvent[] {
|
||||
const base: Record<string, string | number | boolean | string[]> = {};
|
||||
put(base, 'query_id', t.query_id);
|
||||
if (t.compiled_query_hash && SHA256.test(t.compiled_query_hash)) {
|
||||
put(base, 'query_hash', t.compiled_query_hash);
|
||||
}
|
||||
put(base, 'dataset_id', t.dataset_id);
|
||||
put(base, 'engine', shortText(t.engine));
|
||||
put(base, 'surface', t.surface);
|
||||
put(base, 'filter_keys', keyNames(t.filter_keys));
|
||||
|
||||
const events: Aud0BiEvent[] = [
|
||||
{ event_type: 'aud0.analytics_query_started', payload: { ...base }, occurred_at: t.started_at },
|
||||
];
|
||||
|
||||
if (t.state === 'succeeded') {
|
||||
const payload = { ...base };
|
||||
put(payload, 'duration_ms', t.duration_ms);
|
||||
put(payload, 'row_count', t.row_count);
|
||||
put(payload, 'bytes_scanned', t.bytes_scanned);
|
||||
put(payload, 'cache_hit', t.cache_hit);
|
||||
events.push({
|
||||
event_type: 'aud0.analytics_query_completed',
|
||||
payload,
|
||||
occurred_at: t.finished_at ?? t.started_at,
|
||||
});
|
||||
} else if (t.state === 'failed' || t.state === 'cancelled') {
|
||||
const payload = { ...base };
|
||||
put(payload, 'duration_ms', t.duration_ms);
|
||||
// A CODE. `error_message` is never mapped, at any length.
|
||||
put(
|
||||
payload,
|
||||
'error_code',
|
||||
shortText(t.error_code ?? (t.state === 'cancelled' ? 'CANCELLED' : undefined)),
|
||||
);
|
||||
events.push({
|
||||
event_type: 'aud0.analytics_query_failed',
|
||||
payload,
|
||||
occurred_at: t.finished_at ?? t.started_at,
|
||||
});
|
||||
}
|
||||
|
||||
if (t.metric_name) {
|
||||
const payload: Record<string, string | number | boolean | string[]> = {};
|
||||
put(payload, 'metric_name', shortText(t.metric_name));
|
||||
put(payload, 'dataset_id', t.dataset_id);
|
||||
put(payload, 'surface', t.surface);
|
||||
put(payload, 'engine', shortText(t.engine));
|
||||
put(payload, 'dimension_keys', keyNames(t.dimension_keys));
|
||||
put(payload, 'granularity', shortText(t.granularity));
|
||||
put(payload, 'row_count', t.row_count);
|
||||
put(payload, 'duration_ms', t.duration_ms);
|
||||
events.push({
|
||||
event_type: 'aud0.metric_queried',
|
||||
payload,
|
||||
occurred_at: t.finished_at ?? t.started_at,
|
||||
});
|
||||
}
|
||||
|
||||
if (t.dashboard_id) {
|
||||
const payload: Record<string, string | number | boolean | string[]> = {};
|
||||
put(payload, 'dashboard_id', t.dashboard_id);
|
||||
put(payload, 'widget_count', t.widget_count);
|
||||
put(payload, 'duration_ms', t.duration_ms);
|
||||
put(payload, 'surface', t.surface ?? 'dashboard');
|
||||
events.push({ event_type: 'aud0.dashboard_viewed', payload, occurred_at: t.started_at });
|
||||
}
|
||||
|
||||
return events;
|
||||
}
|
||||
|
||||
/** Every payload key this producer can ever emit. Pinned by test. */
|
||||
export const EMITTABLE_PAYLOAD_KEYS = [
|
||||
'query_id',
|
||||
'query_hash',
|
||||
'dataset_id',
|
||||
'engine',
|
||||
'surface',
|
||||
'filter_keys',
|
||||
'duration_ms',
|
||||
'row_count',
|
||||
'bytes_scanned',
|
||||
'cache_hit',
|
||||
'error_code',
|
||||
'metric_name',
|
||||
'dimension_keys',
|
||||
'granularity',
|
||||
'dashboard_id',
|
||||
'widget_count',
|
||||
] as const;
|
||||
@@ -0,0 +1,194 @@
|
||||
/**
|
||||
* tests/analytics/benchmark-harness.test.ts
|
||||
* ==========================================
|
||||
* BI-P104 — the harness's job is to stop a benchmark from lying.
|
||||
*
|
||||
* The tests are therefore attacks on the reporting, not on the timing:
|
||||
*
|
||||
* - two overlapping distributions whose MEDIANS differ 2x must come back
|
||||
* `indistinguishable`, because that is the claim a noisy host manufactures;
|
||||
* - the real 150x/130x pair from yesterday's budget-enforcement lane is fed
|
||||
* in as data, and the harness must describe a RANGE rather than pick one;
|
||||
* - a three-sample run must refuse to be compared at all;
|
||||
* - every sentence `describeComparison` produces must carry the range, the
|
||||
* sample size and the stability, so it cannot be quoted as a bare figure;
|
||||
* - an engine that was not run must appear in the evidence with a reason.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
BENCH_ENGINES,
|
||||
buildEvidence,
|
||||
compareDistributions,
|
||||
describeComparison,
|
||||
measure,
|
||||
MIN_SAMPLES,
|
||||
summarizeSamples,
|
||||
} from '@/src/analytics/bench/benchmark-harness';
|
||||
|
||||
describe('BI-P104 distribution summary', () => {
|
||||
it('reports the shape, and never a single number', () => {
|
||||
const d = summarizeSamples([10, 11, 12, 13, 14, 15, 16, 40]);
|
||||
expect(d.n).toBe(8);
|
||||
expect(d.min).toBe(10);
|
||||
expect(d.max).toBe(40);
|
||||
expect(d.p50).toBe(13);
|
||||
expect(d.spread_ratio).toBe(4);
|
||||
expect(d.cv).toBeGreaterThan(0);
|
||||
// Every field a reader needs to judge the p50 is present next to it.
|
||||
for (const key of [
|
||||
'min',
|
||||
'p50',
|
||||
'p95',
|
||||
'p99',
|
||||
'max',
|
||||
'stdev',
|
||||
'cv',
|
||||
'iqr',
|
||||
'spread_ratio',
|
||||
'stability',
|
||||
]) {
|
||||
expect(Object.keys(d)).toContain(key);
|
||||
}
|
||||
});
|
||||
|
||||
it('calls a tight run stable and a spread run unstable', () => {
|
||||
expect(summarizeSamples([100, 101, 99, 100, 102, 98, 100]).stability).toBe('stable');
|
||||
expect(summarizeSamples([10, 200, 15, 400, 12, 350, 11]).stability).toBe('unstable');
|
||||
});
|
||||
|
||||
it('refuses to characterise fewer than MIN_SAMPLES', () => {
|
||||
const d = summarizeSamples([10, 11, 12]);
|
||||
expect(d.n).toBe(3);
|
||||
expect(d.stability).toBe('insufficient_samples');
|
||||
expect(MIN_SAMPLES).toBeGreaterThan(3);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P104 comparison — variance beats a flattering median', () => {
|
||||
it('overlapping ranges are INDISTINGUISHABLE even when medians differ 2x', () => {
|
||||
const baseline = summarizeSamples([50, 60, 200, 70, 300, 55, 65, 280]);
|
||||
const candidate = summarizeSamples([25, 30, 190, 35, 260, 28, 33, 240]);
|
||||
const c = compareDistributions(baseline, candidate);
|
||||
|
||||
expect(c.p50_ratio).toBeGreaterThan(1.5); // the tempting headline
|
||||
expect(c.ranges_overlap).toBe(true);
|
||||
expect(c.verdict).toBe('indistinguishable'); // the honest verdict
|
||||
expect(describeComparison(c)).toMatch(/INDISTINGUISHABLE/);
|
||||
});
|
||||
|
||||
it('clean separation is reported as faster, WITH the range', () => {
|
||||
const baseline = summarizeSamples([500, 520, 510, 530, 505, 515, 525]);
|
||||
const candidate = summarizeSamples([50, 55, 52, 58, 51, 53, 54]);
|
||||
const c = compareDistributions(baseline, candidate);
|
||||
|
||||
expect(c.verdict).toBe('faster');
|
||||
const sentence = describeComparison(c, 'pre-aggregate');
|
||||
expect(sentence).toMatch(/FASTER/);
|
||||
expect(sentence).toMatch(/x to /); // a range, not a number
|
||||
expect(sentence).toMatch(/n=7\/7/); // sample sizes
|
||||
expect(sentence).toMatch(/stability/);
|
||||
});
|
||||
|
||||
it('the real 150x / 130x overshoot pair is described as a range, not a figure', () => {
|
||||
// Two runs of the same measurement on the same machine, from the BI-P021
|
||||
// budget-enforcement lane. A harness that picks one of these is lying.
|
||||
const overshoot = summarizeSamples([153, 131, 149, 128, 144, 137, 152, 126, 141]);
|
||||
|
||||
// Worth being precise about: by the CV test this run is STABLE (0.07). The
|
||||
// problem was never instability — it is that a stable run still spans
|
||||
// 126x-153x, so any single figure quoted from it is ~20% wrong. The
|
||||
// harness's answer is to keep the ends attached to the middle.
|
||||
expect(overshoot.stability).toBe('stable');
|
||||
expect(overshoot.min).toBe(126);
|
||||
expect(overshoot.p50).toBe(141);
|
||||
expect(overshoot.max).toBe(153);
|
||||
expect(overshoot.spread_ratio).toBeCloseTo(1.21, 2);
|
||||
|
||||
const sentence = describeComparison(
|
||||
compareDistributions(summarizeSamples([1, 1, 1, 1, 1, 1, 1, 1, 1]), overshoot),
|
||||
'budget overshoot factor',
|
||||
);
|
||||
expect(sentence).toMatch(/x to /); // a range survives into the prose
|
||||
expect(sentence).toMatch(/n=9\/9/);
|
||||
});
|
||||
|
||||
it('a three-sample run cannot be compared at all', () => {
|
||||
const c = compareDistributions(summarizeSamples([10, 11, 12]), summarizeSamples([1, 2, 3]));
|
||||
expect(c.verdict).toBe('not_comparable');
|
||||
expect(describeComparison(c)).toMatch(/anecdote/);
|
||||
});
|
||||
|
||||
it('slower is reported as slower — the harness has no favourite', () => {
|
||||
const baseline = summarizeSamples([50, 51, 52, 53, 54, 55, 56]);
|
||||
const candidate = summarizeSamples([500, 510, 520, 530, 540, 550, 560]);
|
||||
expect(compareDistributions(baseline, candidate).verdict).toBe('slower');
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P104 measurement', () => {
|
||||
it('records warmups but excludes them from the statistics', async () => {
|
||||
let call = 0;
|
||||
let clock = 0;
|
||||
const run = await measure({
|
||||
engine: 'duckdb',
|
||||
dataset: 'small_saas',
|
||||
query_label: 'group by day',
|
||||
warmups: 2,
|
||||
repetitions: 6,
|
||||
// A cold first pair (1000ms) then a warm steady state (10ms).
|
||||
now: () => {
|
||||
const t = clock;
|
||||
clock += call <= 2 ? 1000 : 10;
|
||||
call += 1;
|
||||
return t;
|
||||
},
|
||||
run: async () => ({ rows: 42 }),
|
||||
});
|
||||
|
||||
expect(run.warmup_ms).toHaveLength(2);
|
||||
expect(run.samples_ms).toHaveLength(6);
|
||||
expect(run.distribution.n).toBe(6);
|
||||
expect(run.distribution.max).toBeLessThan(1000); // the cold pair is not in it
|
||||
expect(run.rows_returned).toBe(42);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P104 evidence', () => {
|
||||
it('lists every engine BI-P104 names, and says why the unrun ones were not run', () => {
|
||||
const evidence = buildEvidence({
|
||||
host_note: 'one idle host, warm cache',
|
||||
engines: [
|
||||
{ engine: 'clickhouse', available: true, version: '26.3' },
|
||||
{ engine: 'duckdb', available: true, version: '1.5.5' },
|
||||
{ engine: 'trino', available: false, reason: 'the running Trino belongs to another lane' },
|
||||
],
|
||||
runs: [
|
||||
{
|
||||
engine: 'clickhouse',
|
||||
dataset: 'small_saas',
|
||||
query_label: 'q',
|
||||
warmup_ms: [1],
|
||||
samples_ms: [1, 2, 3, 4, 5],
|
||||
distribution: summarizeSamples([1, 2, 3, 4, 5]),
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(evidence.engines.map((e) => e.engine)).toEqual([...BENCH_ENGINES]);
|
||||
const notRun = evidence.not_exercised.map((e) => e.engine);
|
||||
expect(notRun).toContain('superset');
|
||||
expect(notRun).toContain('perspective');
|
||||
expect(notRun).toContain('duckdb'); // declared available but produced no run
|
||||
for (const e of evidence.not_exercised) {
|
||||
expect(e.reason, e.engine).toBeTruthy();
|
||||
}
|
||||
expect(evidence.datasets.map((d) => d.id)).toEqual([
|
||||
'small_saas',
|
||||
'medium_business',
|
||||
'vehiq_market',
|
||||
'large_event',
|
||||
'wide_table',
|
||||
'geospatial',
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,293 @@
|
||||
/**
|
||||
* tests/analytics/bi-slo.test.ts
|
||||
* ===============================
|
||||
* BI-P103 — BI SLO, and the AUD0 usage events it is measured from.
|
||||
*
|
||||
* Two failure modes are attacked directly, because both look like a healthy
|
||||
* dashboard:
|
||||
*
|
||||
* 1. AN SLO REPORT OVER SILENCE. Zero events must not read as 100%
|
||||
* availability. Every objective in an empty window must come back
|
||||
* `insufficient_data`, and `fully_measured` must be false.
|
||||
* 2. A PRODUCER THAT LEAKS. The emitter is handed an execution record
|
||||
* carrying SQL text, a database error message quoting a personnummer,
|
||||
* filter VALUES, a tenant id and a result sample — everything AUD0's
|
||||
* BI-P076 ingest exists to refuse — and none of it may appear in any
|
||||
* emitted payload. The test greps the serialized events, and first
|
||||
* proves the grep finds those strings in the INPUT so a clean result
|
||||
* cannot be a broken detector.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
BI_SLO_CATALOGUE,
|
||||
evaluateBiSlos,
|
||||
percentile,
|
||||
summariseSloReport,
|
||||
type SloResult,
|
||||
} from '@/src/analytics/slo/bi-slo';
|
||||
import {
|
||||
biUsageEventsForExecution,
|
||||
EMITTABLE_PAYLOAD_KEYS,
|
||||
type Aud0BiEvent,
|
||||
type ExecutionTelemetry,
|
||||
} from '@/src/analytics/slo/usage-events';
|
||||
|
||||
const FROM = new Date('2026-09-07T00:00:00.000Z');
|
||||
const TO = new Date('2026-09-07T23:59:59.000Z');
|
||||
|
||||
function queryEvents(args: {
|
||||
n: number;
|
||||
surface: 'dashboard' | 'explore';
|
||||
outcome: 'completed' | 'failed' | 'running';
|
||||
durationMs: (i: number) => number;
|
||||
cacheHit?: boolean;
|
||||
idPrefix?: string;
|
||||
}): Aud0BiEvent[] {
|
||||
const out: Aud0BiEvent[] = [];
|
||||
for (let i = 0; i < args.n; i += 1) {
|
||||
const t: ExecutionTelemetry = {
|
||||
query_id: `${args.idPrefix ?? args.surface}_${i}`,
|
||||
surface: args.surface,
|
||||
state: args.outcome === 'completed' ? 'succeeded' : args.outcome === 'failed' ? 'failed' : 'running',
|
||||
started_at: new Date(FROM.getTime() + i * 1000).toISOString(),
|
||||
finished_at: new Date(FROM.getTime() + i * 1000 + args.durationMs(i)).toISOString(),
|
||||
duration_ms: args.durationMs(i),
|
||||
...(args.cacheHit !== undefined ? { cache_hit: args.cacheHit } : {}),
|
||||
...(args.outcome === 'failed' ? { error_code: 'TIMEOUT' } : {}),
|
||||
};
|
||||
out.push(...biUsageEventsForExecution(t));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function byId(results: SloResult[], id: string): SloResult {
|
||||
const r = results.find((x) => x.objective_id === id);
|
||||
if (!r) throw new Error(`no result for ${id}`);
|
||||
return r;
|
||||
}
|
||||
|
||||
describe('BI-P103 — an SLO over silence is not a met SLO', () => {
|
||||
it('an empty window makes every measurable objective insufficient_data, never 100%', () => {
|
||||
const report = evaluateBiSlos({ events: [], from: FROM, to: TO });
|
||||
expect(report.fully_measured).toBe(false);
|
||||
for (const r of report.results) {
|
||||
expect(['insufficient_data', 'not_measurable_here']).toContain(r.status);
|
||||
expect(r.status).not.toBe('met');
|
||||
}
|
||||
const summary = summariseSloReport(report);
|
||||
expect(summary.met).toBe(0);
|
||||
expect(summary.measured_fraction).toBe(0);
|
||||
});
|
||||
|
||||
it('a started query that never finished is unresolved, not a success', () => {
|
||||
const events = queryEvents({ n: 200, surface: 'dashboard', outcome: 'running', durationMs: () => 100 });
|
||||
const report = evaluateBiSlos({ events, from: FROM, to: TO });
|
||||
expect(report.unresolved_queries).toBe(200);
|
||||
expect(byId(report.results, 'dash.availability').status).toBe('insufficient_data');
|
||||
expect(byId(report.results, 'dash.availability').observed).toBeUndefined();
|
||||
});
|
||||
|
||||
it('a percentile below min_samples is reported WITH its sample size and not judged', () => {
|
||||
const events = queryEvents({
|
||||
n: 12,
|
||||
surface: 'explore',
|
||||
outcome: 'completed',
|
||||
durationMs: (i) => 1000 + i * 10,
|
||||
});
|
||||
const p99 = byId(evaluateBiSlos({ events, from: FROM, to: TO }).results, 'explore.p99');
|
||||
expect(p99.status).toBe('insufficient_data');
|
||||
expect(p99.low_confidence).toBe(true);
|
||||
expect(p99.sample_size).toBe(12);
|
||||
expect(p99.observed).toBeDefined(); // reported…
|
||||
expect(p99.detail).toMatch(/not a percentile/); // …with the caveat attached
|
||||
});
|
||||
|
||||
it('freshness and delivery are declared not_measurable_here rather than proxied', () => {
|
||||
const report = evaluateBiSlos({ events: [], from: FROM, to: TO });
|
||||
const freshness = byId(report.results, 'dash.freshness');
|
||||
const delivery = byId(report.results, 'report.delivery_sla');
|
||||
expect(freshness.status).toBe('not_measurable_here');
|
||||
expect(freshness.detail).toMatch(/analytics_freshness_watermark/);
|
||||
expect(delivery.status).toBe('not_measurable_here');
|
||||
expect(delivery.detail).toMatch(/n0tify_delivery_receipt/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P103 — real measurement from real events', () => {
|
||||
it('computes availability, error rate and both dashboard latencies', () => {
|
||||
const events = [
|
||||
...queryEvents({
|
||||
n: 180,
|
||||
surface: 'dashboard',
|
||||
outcome: 'completed',
|
||||
cacheHit: false,
|
||||
durationMs: (i) => 500 + i * 10,
|
||||
idPrefix: 'cold',
|
||||
}),
|
||||
...queryEvents({
|
||||
n: 120,
|
||||
surface: 'dashboard',
|
||||
outcome: 'completed',
|
||||
cacheHit: true,
|
||||
durationMs: () => 120,
|
||||
idPrefix: 'warm',
|
||||
}),
|
||||
...queryEvents({
|
||||
n: 3,
|
||||
surface: 'dashboard',
|
||||
outcome: 'failed',
|
||||
durationMs: () => 900,
|
||||
idPrefix: 'bad',
|
||||
}),
|
||||
];
|
||||
const { results } = evaluateBiSlos({ events, from: FROM, to: TO });
|
||||
|
||||
const availability = byId(results, 'dash.availability');
|
||||
expect(availability.sample_size).toBe(303);
|
||||
expect(availability.observed).toBeCloseTo(300 / 303, 6);
|
||||
expect(availability.status).toBe('breached'); // 0.9901 < 0.995 target
|
||||
|
||||
const errorRate = byId(results, 'dash.error_rate');
|
||||
expect(errorRate.observed).toBeCloseTo(3 / 303, 6);
|
||||
|
||||
const cold = byId(results, 'dash.p95_first_query');
|
||||
expect(cold.sample_size).toBe(183); // 180 cold + 3 failures, which had no cache hit
|
||||
expect(cold.status).toBe('met');
|
||||
|
||||
const cached = byId(results, 'dash.p95_cached');
|
||||
expect(cached.sample_size).toBe(120);
|
||||
expect(cached.observed).toBe(120);
|
||||
expect(cached.status).toBe('met');
|
||||
});
|
||||
|
||||
it('a breach is reported as a breach', () => {
|
||||
const events = queryEvents({
|
||||
n: 60,
|
||||
surface: 'dashboard',
|
||||
outcome: 'completed',
|
||||
cacheHit: false,
|
||||
durationMs: () => 9000,
|
||||
});
|
||||
expect(
|
||||
byId(evaluateBiSlos({ events, from: FROM, to: TO }).results, 'dash.p95_first_query'),
|
||||
).toMatchObject({ status: 'breached', observed: 9000 });
|
||||
});
|
||||
|
||||
it('events outside the window do not count', () => {
|
||||
const events = queryEvents({ n: 200, surface: 'dashboard', outcome: 'completed', durationMs: () => 100 });
|
||||
const report = evaluateBiSlos({
|
||||
events,
|
||||
from: new Date('2026-09-08T00:00:00.000Z'),
|
||||
to: new Date('2026-09-08T23:59:59.000Z'),
|
||||
});
|
||||
expect(byId(report.results, 'dash.availability').status).toBe('insufficient_data');
|
||||
});
|
||||
|
||||
it('percentile is nearest-rank — it never invents a value nobody experienced', () => {
|
||||
const sample = [10, 20, 30, 40];
|
||||
expect(percentile(sample, 0.95)).toBe(40);
|
||||
expect(percentile(sample, 0.5)).toBe(20);
|
||||
expect(percentile([], 0.95)).toBeUndefined();
|
||||
});
|
||||
|
||||
it('the catalogue covers all three workload classes and every field BI-P103 names', () => {
|
||||
const kinds = new Set(BI_SLO_CATALOGUE.map((o) => `${o.workload}:${o.kind}`));
|
||||
for (const expected of [
|
||||
'interactive_dashboard:availability',
|
||||
'interactive_dashboard:latency_p95_first',
|
||||
'interactive_dashboard:latency_p95_cached',
|
||||
'interactive_dashboard:error_rate',
|
||||
'interactive_dashboard:freshness',
|
||||
'exploration:latency_p95',
|
||||
'exploration:latency_p99',
|
||||
'exploration:max_result_rows',
|
||||
'exploration:timeout',
|
||||
'report:completion_sla',
|
||||
'report:delivery_sla',
|
||||
]) {
|
||||
expect(kinds.has(expected), expected).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P103 — the producer cannot leak a result into AUD0', () => {
|
||||
const hostile: ExecutionTelemetry = {
|
||||
query_id: 'q_1',
|
||||
compiled_query_hash: 'f'.repeat(64),
|
||||
compiled_sql: "SELECT * FROM customers WHERE personnummer = '19800101-1234'",
|
||||
error_message: 'duplicate key value violates unique constraint (personnummer)=(19800101-1234)',
|
||||
error_code: 'CONFLICT',
|
||||
tenant_id: 't_alpha',
|
||||
dataset_id: 'ds_sales',
|
||||
engine: 'clickhouse',
|
||||
surface: 'dashboard',
|
||||
state: 'failed',
|
||||
started_at: '2026-09-07T10:00:00.000Z',
|
||||
finished_at: '2026-09-07T10:00:02.000Z',
|
||||
duration_ms: 2000,
|
||||
filter_keys: ['region'],
|
||||
filter_values: { region: ['Gotland'] },
|
||||
result_sample: [{ customer: 'Anna Andersson', revenue: 42 }],
|
||||
metric_name: 'revenue',
|
||||
dashboard_id: 'dash_sales',
|
||||
widget_count: 4,
|
||||
};
|
||||
|
||||
it('emits none of the forbidden material', () => {
|
||||
const events = biUsageEventsForExecution(hostile);
|
||||
const serialized = JSON.stringify(events);
|
||||
|
||||
// The detector must be able to find these strings — they are in the input.
|
||||
const input = JSON.stringify(hostile);
|
||||
for (const secret of ['19800101-1234', 'Anna Andersson', 'Gotland', 't_alpha', 'SELECT *']) {
|
||||
expect(input, `detector check for ${secret}`).toContain(secret);
|
||||
}
|
||||
for (const secret of [
|
||||
'19800101-1234',
|
||||
'Anna Andersson',
|
||||
'Gotland',
|
||||
't_alpha',
|
||||
'SELECT *',
|
||||
'duplicate key',
|
||||
]) {
|
||||
expect(serialized, secret).not.toContain(secret);
|
||||
}
|
||||
});
|
||||
|
||||
it('every emitted key is on the allowlist AUD0 accepts', () => {
|
||||
const events = biUsageEventsForExecution(hostile);
|
||||
for (const e of events) {
|
||||
for (const key of Object.keys(e.payload)) {
|
||||
expect(EMITTABLE_PAYLOAD_KEYS, `${e.event_type}.${key}`).toContain(key);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('a query_hash that is not a sha256 digest is dropped, not reshaped', () => {
|
||||
const events = biUsageEventsForExecution({ ...hostile, compiled_query_hash: 'SELECT 1' });
|
||||
for (const e of events) expect(e.payload.query_hash).toBeUndefined();
|
||||
});
|
||||
|
||||
it('a running query emits only "started" — it is neither a success nor a failure', () => {
|
||||
const events = biUsageEventsForExecution({
|
||||
...hostile,
|
||||
state: 'running',
|
||||
metric_name: undefined,
|
||||
dashboard_id: undefined,
|
||||
});
|
||||
expect(events.map((e) => e.event_type)).toEqual(['aud0.analytics_query_started']);
|
||||
});
|
||||
|
||||
it('a cancelled query is reported as failed with a CODE', () => {
|
||||
const events = biUsageEventsForExecution({
|
||||
...hostile,
|
||||
state: 'cancelled',
|
||||
error_code: undefined,
|
||||
metric_name: undefined,
|
||||
dashboard_id: undefined,
|
||||
});
|
||||
const failed = events.find((e) => e.event_type === 'aud0.analytics_query_failed');
|
||||
expect(failed?.payload.error_code).toBe('CANCELLED');
|
||||
expect(failed?.payload.error_message).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,330 @@
|
||||
/**
|
||||
* tests/analytics/dashboard-model.test.ts
|
||||
* ========================================
|
||||
* BI-P060 — ANALYTICS DASHBOARD MODEL.
|
||||
*
|
||||
* The DoD is "a native SIAX dashboard can be reconstructed without Superset",
|
||||
* so the central test takes a document that CAME from Superset (it still
|
||||
* carries the slice ids in `vendor_ref`), plans it, and then greps the whole
|
||||
* plan for any trace of the vendor. A plan that still needs Superset to be
|
||||
* understood fails here.
|
||||
*
|
||||
* The counterfactual matters as much: `vendor-only-widget-is-refused` builds
|
||||
* the document that CANNOT be reconstructed — a widget whose only content is a
|
||||
* Superset id — and requires the model to refuse it. Without that test the
|
||||
* vendor-free assertion would pass trivially for documents nobody could
|
||||
* render.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { SiaxError } from '@/src/platform/errors';
|
||||
import {
|
||||
dashboardDocumentHash,
|
||||
parseDashboardDocument,
|
||||
planDashboard,
|
||||
VISUALIZATION_TYPES,
|
||||
type DashboardDocument,
|
||||
} from '@/src/analytics/dashboards/dashboard-document';
|
||||
|
||||
/** A dashboard exported from Superset and rewritten into the canonical model. */
|
||||
function supersetImportedDocument(): unknown {
|
||||
return {
|
||||
id: 'dash_sales_overview',
|
||||
title: 'Sales overview',
|
||||
description: 'Migrated from Superset dashboard 41.',
|
||||
version: 1,
|
||||
layout: { grid_columns: 12, row_height_px: 40 },
|
||||
parameters: [
|
||||
{
|
||||
name: 'region',
|
||||
type: 'enum',
|
||||
required: false,
|
||||
default_value: 'SE',
|
||||
allowed_values: ['SE', 'NO', 'DK'],
|
||||
},
|
||||
{ name: 'since', type: 'date', required: true },
|
||||
],
|
||||
filters: [
|
||||
{
|
||||
filter_id: 'f_region',
|
||||
label: 'Region',
|
||||
dimension: 'region',
|
||||
operator: 'eq',
|
||||
default_values: ['{{region}}'],
|
||||
},
|
||||
{
|
||||
filter_id: 'f_channel',
|
||||
label: 'Channel',
|
||||
dimension: 'channel',
|
||||
operator: 'in',
|
||||
default_values: ['web'],
|
||||
},
|
||||
{
|
||||
filter_id: 'f_tenant_scope',
|
||||
label: 'Business unit',
|
||||
dimension: 'business_unit',
|
||||
operator: 'eq',
|
||||
default_values: ['bu_north'],
|
||||
locked: true,
|
||||
},
|
||||
],
|
||||
widgets: [
|
||||
{
|
||||
widget_id: 'w_revenue_total',
|
||||
title: 'Revenue',
|
||||
source: { kind: 'metric', dataset: 'ds_sales', metric: 'revenue' },
|
||||
visualization: { type: 'big_number', encoding: { value: 'revenue' } },
|
||||
layout: { x: 0, y: 0, w: 3, h: 2 },
|
||||
vendor_ref: { system: 'superset', id: 'slice-8812' },
|
||||
},
|
||||
{
|
||||
widget_id: 'w_revenue_by_day',
|
||||
title: 'Revenue by day',
|
||||
source: {
|
||||
kind: 'metric',
|
||||
dataset: 'ds_sales',
|
||||
metric: 'revenue',
|
||||
dimensions: ['order_date'],
|
||||
grain: 'day',
|
||||
},
|
||||
visualization: {
|
||||
type: 'line',
|
||||
encoding: { x: 'order_date', y: ['revenue'] },
|
||||
options: { stacked: false },
|
||||
},
|
||||
layout: { x: 3, y: 0, w: 9, h: 4 },
|
||||
filter_bindings: ['f_region', 'f_tenant_scope'],
|
||||
vendor_ref: { system: 'superset', id: 'slice-8813' },
|
||||
},
|
||||
{
|
||||
widget_id: 'w_top_products',
|
||||
title: 'Top products',
|
||||
source: {
|
||||
kind: 'query',
|
||||
request: {
|
||||
dataset: 'ds_sales',
|
||||
metrics: ['revenue', 'orders'],
|
||||
dimensions: ['product'],
|
||||
filters: [{ dimension: 'status', operator: 'eq', values: ['completed'] }],
|
||||
orderBy: [{ ref: 'revenue', direction: 'desc' }],
|
||||
limit: 20,
|
||||
},
|
||||
},
|
||||
visualization: { type: 'table', encoding: { x: 'product', y: ['revenue', 'orders'] } },
|
||||
layout: { x: 0, y: 4, w: 12, h: 6 },
|
||||
},
|
||||
],
|
||||
refresh: { mode: 'interval', interval_seconds: 300, max_staleness_seconds: 900 },
|
||||
permissions: {
|
||||
visibility: 'roles',
|
||||
roles: ['analyst', 'sales_lead'],
|
||||
required_entitlement: 'bi_dashboards',
|
||||
},
|
||||
vendor_ref: { system: 'superset', id: 'dashboard-41' },
|
||||
};
|
||||
}
|
||||
|
||||
describe('BI-P060 dashboard model — reconstruction without Superset', () => {
|
||||
it('plans every widget into a runnable analytics request', () => {
|
||||
const doc = parseDashboardDocument(supersetImportedDocument());
|
||||
const plan = planDashboard(doc, { parameters: { since: '2026-01-01', region: 'NO' } });
|
||||
|
||||
expect(plan.widgets).toHaveLength(3);
|
||||
for (const w of plan.widgets) {
|
||||
expect(w.request.dataset).toBe('ds_sales');
|
||||
expect(w.request.metrics.length).toBeGreaterThan(0);
|
||||
expect(VISUALIZATION_TYPES).toContain(w.visualization.type);
|
||||
}
|
||||
// The parameter reference was BOUND, not left as a template string.
|
||||
const byDay = plan.widgets.find((w) => w.widget_id === 'w_revenue_by_day');
|
||||
expect(byDay?.request.filters.find((f) => f.dimension === 'region')?.values).toEqual(['NO']);
|
||||
expect(plan.bound_parameters).toEqual({ region: 'NO', since: '2026-01-01' });
|
||||
});
|
||||
|
||||
it('the plan contains no vendor identifier anywhere', () => {
|
||||
const doc = parseDashboardDocument(supersetImportedDocument());
|
||||
const plan = planDashboard(doc, { parameters: { since: '2026-01-01' } });
|
||||
const serialized = JSON.stringify(plan).toLowerCase();
|
||||
|
||||
// The detector must be capable of finding one: the document DOES carry
|
||||
// these ids, so a grep that finds nothing in the plan means the plan is
|
||||
// clean, not that the grep is broken.
|
||||
expect(JSON.stringify(doc).toLowerCase()).toContain('slice-8812');
|
||||
|
||||
for (const trace of ['superset', 'slice-8812', 'slice-8813', 'dashboard-41', 'viz_type', 'form_data']) {
|
||||
expect(serialized).not.toContain(trace);
|
||||
}
|
||||
});
|
||||
|
||||
it('vendor-only widget is refused — it is the document that cannot be reconstructed', () => {
|
||||
const raw = supersetImportedDocument() as { widgets: unknown[] };
|
||||
raw.widgets = [
|
||||
{
|
||||
widget_id: 'w_mystery',
|
||||
title: 'Whatever Superset had',
|
||||
source: { kind: 'vendor', system: 'superset', id: 'slice-9001' },
|
||||
visualization: { type: 'line', encoding: {} },
|
||||
layout: { x: 0, y: 0, w: 4, h: 3 },
|
||||
},
|
||||
];
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(SiaxError);
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/reconstruct/i);
|
||||
});
|
||||
|
||||
it('vendor-native fields are refused at the top level and inside a widget', () => {
|
||||
const top = { ...(supersetImportedDocument() as object), position_json: '{}' };
|
||||
expect(() => parseDashboardDocument(top)).toThrow(/vendor-native/);
|
||||
|
||||
const raw = supersetImportedDocument() as { widgets: Record<string, unknown>[] };
|
||||
raw.widgets[0].viz_type = 'echarts_timeseries';
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/vendor-native/);
|
||||
});
|
||||
|
||||
it('a Superset viz_type value is not a SIAX visualization type', () => {
|
||||
const raw = supersetImportedDocument() as { widgets: Record<string, unknown>[] };
|
||||
raw.widgets[0].visualization = { type: 'echarts_timeseries', encoding: {} };
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/not a SIAX visualization type/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P060 dashboard model — no SQL surface', () => {
|
||||
it('a widget cannot carry SQL, however it is spelled', () => {
|
||||
for (const key of ['sql', 'query_text', 'statement', 'where', 'expression']) {
|
||||
const raw = supersetImportedDocument() as { widgets: Record<string, unknown>[] };
|
||||
raw.widgets[0][key] = 'SELECT 1';
|
||||
expect(() => parseDashboardDocument(raw), key).toThrow(SiaxError);
|
||||
}
|
||||
});
|
||||
|
||||
it('a query widget goes through the BI-P016 contract, so tenant selection is impossible', () => {
|
||||
const raw = supersetImportedDocument() as { widgets: Record<string, unknown>[] };
|
||||
raw.widgets[2].source = {
|
||||
kind: 'query',
|
||||
request: {
|
||||
dataset: 'ds_sales',
|
||||
metrics: ['revenue'],
|
||||
dimensions: [],
|
||||
filters: [],
|
||||
tenant_id: 't_victim',
|
||||
},
|
||||
};
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/does not accept SQL, catalogs, or tenant selection/);
|
||||
});
|
||||
|
||||
it('the document itself cannot name a tenant', () => {
|
||||
const raw = { ...(supersetImportedDocument() as object), tenant_id: 't_victim' };
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/dashboards do not carry SQL|unknown field/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P060 dashboard model — validation that would otherwise bite at render time', () => {
|
||||
it('a widget binding an undeclared filter is refused', () => {
|
||||
const raw = supersetImportedDocument() as { widgets: Record<string, unknown>[] };
|
||||
raw.widgets[1].filter_bindings = ['f_nonexistent'];
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/undeclared filter/);
|
||||
});
|
||||
|
||||
it('a filter defaulting to an undeclared parameter is refused', () => {
|
||||
const raw = supersetImportedDocument() as { filters: Record<string, unknown>[] };
|
||||
raw.filters[0].default_values = ['{{not_declared}}'];
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/undeclared parameter/);
|
||||
});
|
||||
|
||||
it('a widget overflowing the grid is refused', () => {
|
||||
const raw = supersetImportedDocument() as { widgets: Record<string, unknown>[] };
|
||||
raw.widgets[1].layout = { x: 9, y: 0, w: 6, h: 3 };
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/overflows the 12-column grid/);
|
||||
});
|
||||
|
||||
it('an interval refresh with a cron, and a manual refresh with a schedule, are both refused', () => {
|
||||
const a = supersetImportedDocument() as Record<string, unknown>;
|
||||
a.refresh = { mode: 'interval', interval_seconds: 300, cron: '0 * * * *', max_staleness_seconds: 900 };
|
||||
expect(() => parseDashboardDocument(a)).toThrow(/meaningless/);
|
||||
|
||||
const b = supersetImportedDocument() as Record<string, unknown>;
|
||||
b.refresh = { mode: 'manual', interval_seconds: 300, max_staleness_seconds: 900 };
|
||||
expect(() => parseDashboardDocument(b)).toThrow(/manual/);
|
||||
});
|
||||
|
||||
it('visibility "roles" with no roles is refused rather than guessed', () => {
|
||||
const raw = supersetImportedDocument() as Record<string, unknown>;
|
||||
raw.permissions = { visibility: 'roles', roles: [] };
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/deny all/);
|
||||
});
|
||||
|
||||
it('a required parameter that was not supplied fails the plan, not the render', () => {
|
||||
const doc = parseDashboardDocument(supersetImportedDocument());
|
||||
expect(() => planDashboard(doc, { parameters: {} })).toThrow(/required parameter "since"/);
|
||||
});
|
||||
|
||||
it('a locked filter cannot be switched off by a viewer', () => {
|
||||
const doc = parseDashboardDocument(supersetImportedDocument());
|
||||
expect(() =>
|
||||
planDashboard(doc, { parameters: { since: '2026-01-01' }, disabled_filter_ids: ['f_tenant_scope'] }),
|
||||
).toThrow(/locked/);
|
||||
// An unlocked one can be, and disappears from the compiled request.
|
||||
const plan = planDashboard(doc, {
|
||||
parameters: { since: '2026-01-01' },
|
||||
disabled_filter_ids: ['f_channel'],
|
||||
});
|
||||
const table = plan.widgets.find((w) => w.widget_id === 'w_top_products');
|
||||
expect(table?.request.filters.map((f) => f.dimension)).not.toContain('channel');
|
||||
});
|
||||
|
||||
it('an enum parameter without allowed_values is refused', () => {
|
||||
const raw = supersetImportedDocument() as { parameters: Record<string, unknown>[] };
|
||||
raw.parameters[0] = { name: 'region', type: 'enum', required: false, default_value: 'SE' };
|
||||
expect(() => parseDashboardDocument(raw)).toThrow(/unbounded enum/);
|
||||
});
|
||||
});
|
||||
|
||||
/** Rebuild an object graph with every object's keys in reverse order. */
|
||||
function shuffleKeys(value: unknown): unknown {
|
||||
if (Array.isArray(value)) return value.map(shuffleKeys);
|
||||
if (value !== null && typeof value === 'object') {
|
||||
const out: Record<string, unknown> = {};
|
||||
for (const k of Object.keys(value as object).reverse()) {
|
||||
out[k] = shuffleKeys((value as Record<string, unknown>)[k]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
describe('BI-P060 dashboard model — canonical hash', () => {
|
||||
function parsed(): DashboardDocument {
|
||||
return parseDashboardDocument(supersetImportedDocument());
|
||||
}
|
||||
|
||||
it('is stable across key order and vendor re-export', () => {
|
||||
const a = parsed();
|
||||
const b = parsed();
|
||||
b.vendor_ref = { system: 'superset', id: 'dashboard-999-after-migration' };
|
||||
b.widgets[0].vendor_ref = { system: 'metabase', id: 'card-3' };
|
||||
expect(dashboardDocumentHash(b)).toBe(dashboardDocumentHash(a));
|
||||
|
||||
// Key order in the source JSON must not change identity either.
|
||||
expect(dashboardDocumentHash(parseDashboardDocument(shuffleKeys(supersetImportedDocument())))).toBe(
|
||||
dashboardDocumentHash(a),
|
||||
);
|
||||
});
|
||||
|
||||
it('changes when meaning changes', () => {
|
||||
const base = dashboardDocumentHash(parsed());
|
||||
|
||||
const retitled = parsed();
|
||||
retitled.widgets[0].title = 'Net revenue';
|
||||
expect(dashboardDocumentHash(retitled)).not.toBe(base);
|
||||
|
||||
const remetric = parsed();
|
||||
(remetric.widgets[0].source as { metric: string }).metric = 'revenue_net';
|
||||
expect(dashboardDocumentHash(remetric)).not.toBe(base);
|
||||
|
||||
const moved = parsed();
|
||||
moved.widgets[1].layout = { x: 3, y: 1, w: 9, h: 4 };
|
||||
expect(dashboardDocumentHash(moved)).not.toBe(base);
|
||||
|
||||
const refiltered = parsed();
|
||||
refiltered.filters[1].default_values = ['store'];
|
||||
expect(dashboardDocumentHash(refiltered)).not.toBe(base);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,346 @@
|
||||
/**
|
||||
* tests/analytics/dashboard-release-postgres.realinfra.test.ts
|
||||
* =============================================================
|
||||
* sql/010 (BI-P061) against a REAL Postgres.
|
||||
*
|
||||
* The interesting claim in sql/010 is not "there is a state machine" — it is
|
||||
* "TWO PEOPLE CANNOT PUBLISH AT ONCE". That claim lives in a partial unique
|
||||
* index, and an index nobody raced is an index nobody has tested. So the
|
||||
* concurrency test runs BOTH directions, in two separate schemas so neither
|
||||
* can contaminate the other:
|
||||
*
|
||||
* 1. WITHOUT the index (a schema where it is dropped): two sessions publish
|
||||
* a release of the same dashboard concurrently, each having first read
|
||||
* "nothing is live" — the exact interleaving an application-level check
|
||||
* cannot see. The database ends up with TWO published releases. This is
|
||||
* the counterfactual: it proves the read-then-write check in
|
||||
* DashboardReleaseService.publish() is NOT the guarantee, and that this
|
||||
* test can detect the failure it claims to prevent.
|
||||
* 2. WITH the index: the same race, and the loser fails with
|
||||
* unique_violation (23505). Exactly one published release remains.
|
||||
*
|
||||
* The transition trigger is attacked the same way: every illegal transition is
|
||||
* attempted against the database, not against the TypeScript.
|
||||
*
|
||||
* Gated on ANALYTICS_TEST_DATABASE_URL, matching the convention in
|
||||
* analytics-governance-postgres.realinfra.test.ts. With it set and Postgres
|
||||
* down this FAILS rather than skipping.
|
||||
*
|
||||
* docker run -d --name pg -p 25441:5432 -e POSTGRES_PASSWORD=test \
|
||||
* -e POSTGRES_DB=opt postgres:16-alpine
|
||||
* ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25441/opt \
|
||||
* pnpm vitest run tests/analytics/dashboard-release-postgres.realinfra.test.ts
|
||||
*/
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
|
||||
import postgres from 'postgres';
|
||||
|
||||
const TEST_DATABASE_URL = process.env.ANALYTICS_TEST_DATABASE_URL;
|
||||
const run = TEST_DATABASE_URL ? describe : describe.skip;
|
||||
const STAMP = Date.now().toString(36);
|
||||
const SCHEMA = `bi_p061_${STAMP}`;
|
||||
/** The counterfactual lives in its own schema: no cleanup, no contamination. */
|
||||
const SCHEMA_NOGUARD = `bi_p061_noguard_${STAMP}`;
|
||||
const HASH_A = 'a'.repeat(64);
|
||||
const HASH_B = 'b'.repeat(64);
|
||||
const MIGRATIONS = ['sql/008_analytics_domain_model.sql', 'sql/010_analytics_dashboard_releases.sql'];
|
||||
|
||||
let sql: ReturnType<typeof postgres>;
|
||||
|
||||
function connect(schema: string, max = 1): ReturnType<typeof postgres> {
|
||||
return postgres(TEST_DATABASE_URL!, {
|
||||
max,
|
||||
onnotice: () => undefined,
|
||||
connection: { search_path: schema },
|
||||
});
|
||||
}
|
||||
|
||||
async function refusalOf(fn: () => Promise<unknown>): Promise<string | undefined> {
|
||||
try {
|
||||
await fn();
|
||||
return undefined;
|
||||
} catch (e) {
|
||||
return String((e as Error).message);
|
||||
}
|
||||
}
|
||||
|
||||
async function migrate(conn: ReturnType<typeof postgres>): Promise<void> {
|
||||
for (const file of MIGRATIONS) {
|
||||
await conn.unsafe(fs.readFileSync(path.join(process.cwd(), file), 'utf8'));
|
||||
}
|
||||
}
|
||||
|
||||
async function seedDashboard(conn: ReturnType<typeof postgres>, id: string): Promise<void> {
|
||||
await conn.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_definition
|
||||
(dashboard_id, tenant_id, project_id, name, current_version, owner_id)
|
||||
VALUES ('${id}', 't_alpha', 'p_main', '${id}', 1, 'u_1')
|
||||
ON CONFLICT DO NOTHING`);
|
||||
}
|
||||
|
||||
async function seedVersion(
|
||||
conn: ReturnType<typeof postgres>,
|
||||
versionId: string,
|
||||
dashboardId: string,
|
||||
version: number,
|
||||
hash = HASH_A,
|
||||
): Promise<void> {
|
||||
await conn.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_version
|
||||
(dashboard_version_id, dashboard_id, tenant_id, project_id, version, layout_json, layout_hash)
|
||||
VALUES ('${versionId}', '${dashboardId}', 't_alpha', 'p_main', ${version}, '{}', '${hash}')`);
|
||||
}
|
||||
|
||||
async function seedDraftRelease(
|
||||
conn: ReturnType<typeof postgres>,
|
||||
releaseId: string,
|
||||
versionId: string,
|
||||
dashboardId: string,
|
||||
version: number,
|
||||
hash = HASH_A,
|
||||
): Promise<void> {
|
||||
await conn.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_release
|
||||
(release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, document_hash)
|
||||
VALUES ('${releaseId}', '${dashboardId}', '${versionId}', 't_alpha', 'p_main', ${version}, 'draft', '${hash}')`);
|
||||
}
|
||||
|
||||
async function publishedCount(conn: ReturnType<typeof postgres>, dashboardId: string): Promise<number> {
|
||||
const rows = (await conn.unsafe(
|
||||
`SELECT count(*)::int AS n FROM opt_analytics_dashboard_release
|
||||
WHERE dashboard_id = '${dashboardId}' AND state = 'published'`,
|
||||
)) as unknown as Array<{ n: number }>;
|
||||
return rows[0].n;
|
||||
}
|
||||
|
||||
/**
|
||||
* Two sessions publish a release of the SAME dashboard at the same time, each
|
||||
* having first read "nothing is live". This is the create-and-publish path:
|
||||
* a new version is minted and released in one transaction, which is what a
|
||||
* "save and publish" button does.
|
||||
*/
|
||||
async function racePublish(schema: string, dashboardId: string): Promise<{ ok: number; codes: string[] }> {
|
||||
const a = connect(schema);
|
||||
const b = connect(schema);
|
||||
const publish = (conn: ReturnType<typeof postgres>, n: number) =>
|
||||
conn.begin(async (tx) => {
|
||||
await tx.unsafe(
|
||||
`SELECT release_id FROM opt_analytics_dashboard_release
|
||||
WHERE dashboard_id = '${dashboardId}' AND state = 'published'`,
|
||||
);
|
||||
// A tiny pause so both transactions have certainly done their read
|
||||
// before either writes. Without it the race may not interleave and the
|
||||
// counterfactual could pass for the wrong reason.
|
||||
await new Promise((r) => setTimeout(r, 60));
|
||||
await tx.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_release
|
||||
(release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state,
|
||||
document_hash, published_at)
|
||||
VALUES ('rel_race_${n}', '${dashboardId}', 'dv_race_${n}', 't_alpha', 'p_main', ${400 + n},
|
||||
'published', '${n === 1 ? HASH_A : HASH_B}', now())`);
|
||||
});
|
||||
const results = await Promise.allSettled([publish(a, 1), publish(b, 2)]);
|
||||
await a.end({ timeout: 5 });
|
||||
await b.end({ timeout: 5 });
|
||||
return {
|
||||
ok: results.filter((r) => r.status === 'fulfilled').length,
|
||||
codes: results.flatMap((r) =>
|
||||
r.status === 'rejected' ? [String((r.reason as { code?: string })?.code ?? r.reason)] : [],
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!TEST_DATABASE_URL) return;
|
||||
const admin = postgres(TEST_DATABASE_URL, { max: 1, onnotice: () => undefined });
|
||||
await admin.unsafe(`CREATE SCHEMA IF NOT EXISTS ${SCHEMA}`);
|
||||
await admin.unsafe(`CREATE SCHEMA IF NOT EXISTS ${SCHEMA_NOGUARD}`);
|
||||
await admin.end({ timeout: 5 });
|
||||
|
||||
sql = connect(SCHEMA, 3);
|
||||
await migrate(sql);
|
||||
for (const d of ['dash_txn', 'dash_race', 'dash_draft']) await seedDashboard(sql, d);
|
||||
}, 120_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (!TEST_DATABASE_URL || !sql) return;
|
||||
await sql.unsafe(`DROP SCHEMA IF EXISTS ${SCHEMA} CASCADE`);
|
||||
await sql.unsafe(`DROP SCHEMA IF EXISTS ${SCHEMA_NOGUARD} CASCADE`);
|
||||
await sql.end({ timeout: 5 });
|
||||
});
|
||||
|
||||
run('sql/010 applies and replays', () => {
|
||||
it('is idempotent — a replay onto an already-migrated schema succeeds', async () => {
|
||||
const text = fs.readFileSync(
|
||||
path.join(process.cwd(), 'sql/010_analytics_dashboard_releases.sql'),
|
||||
'utf8',
|
||||
);
|
||||
await expect(sql.unsafe(text)).resolves.toBeDefined();
|
||||
});
|
||||
|
||||
it('has no column that could hold a document body — the document lives in the version row', async () => {
|
||||
const rows = (await sql.unsafe(
|
||||
`SELECT column_name FROM information_schema.columns
|
||||
WHERE table_schema = '${SCHEMA}' AND table_name = 'opt_analytics_dashboard_release'`,
|
||||
)) as unknown as Array<{ column_name: string }>;
|
||||
const names = new Set(rows.map((r) => r.column_name));
|
||||
for (const forbidden of ['layout_json', 'document', 'definition', 'payload', 'sql']) {
|
||||
expect(names.has(forbidden), forbidden).toBe(false);
|
||||
}
|
||||
expect(names.has('document_hash')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
run('BI-P061 transitions are enforced by the database', () => {
|
||||
beforeAll(async () => {
|
||||
await seedVersion(sql, 'dv_t1', 'dash_txn', 101);
|
||||
await seedVersion(sql, 'dv_t3', 'dash_txn', 103, HASH_B);
|
||||
await seedDraftRelease(sql, 'rel_t1', 'dv_t1', 'dash_txn', 101);
|
||||
});
|
||||
|
||||
it('draft -> published is allowed', async () => {
|
||||
await sql.unsafe(
|
||||
`UPDATE opt_analytics_dashboard_release SET state='published', published_at=now() WHERE release_id='rel_t1'`,
|
||||
);
|
||||
const rows = (await sql.unsafe(
|
||||
`SELECT state FROM opt_analytics_dashboard_release WHERE release_id='rel_t1'`,
|
||||
)) as unknown as Array<{ state: string }>;
|
||||
expect(rows[0].state).toBe('published');
|
||||
});
|
||||
|
||||
it('published -> draft (un-publishing) is REFUSED', async () => {
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(
|
||||
`UPDATE opt_analytics_dashboard_release SET state='draft', published_at=NULL WHERE release_id='rel_t1'`,
|
||||
),
|
||||
);
|
||||
expect(msg).toMatch(/illegal dashboard release transition published -> draft/);
|
||||
});
|
||||
|
||||
it('repointing a release at a different version is REFUSED', async () => {
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(
|
||||
`UPDATE opt_analytics_dashboard_release SET dashboard_version_id='dv_t3' WHERE release_id='rel_t1'`,
|
||||
),
|
||||
);
|
||||
expect(msg).toMatch(/immutable apart from its state/);
|
||||
});
|
||||
|
||||
it('rewriting document_hash on a release is REFUSED', async () => {
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(
|
||||
`UPDATE opt_analytics_dashboard_release SET document_hash='${HASH_B}' WHERE release_id='rel_t1'`,
|
||||
),
|
||||
);
|
||||
expect(msg).toMatch(/immutable apart from its state/);
|
||||
});
|
||||
|
||||
it('rewriting published_at is REFUSED — a published timestamp is history', async () => {
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(
|
||||
`UPDATE opt_analytics_dashboard_release SET published_at = now() - interval '10 days'
|
||||
WHERE release_id='rel_t1'`,
|
||||
),
|
||||
);
|
||||
expect(msg).toMatch(/published_at is history/);
|
||||
});
|
||||
|
||||
it('published -> superseded is allowed, and superseded -> published is REFUSED', async () => {
|
||||
await sql.unsafe(
|
||||
`UPDATE opt_analytics_dashboard_release SET state='superseded', superseded_at=now() WHERE release_id='rel_t1'`,
|
||||
);
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(`UPDATE opt_analytics_dashboard_release SET state='published' WHERE release_id='rel_t1'`),
|
||||
);
|
||||
expect(msg).toMatch(/illegal dashboard release transition superseded -> published/);
|
||||
});
|
||||
|
||||
it('deleting a release is REFUSED — the ledger has no eraser', async () => {
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(`DELETE FROM opt_analytics_dashboard_release WHERE release_id='rel_t1'`),
|
||||
);
|
||||
expect(msg).toMatch(/ledger; delete is refused/);
|
||||
});
|
||||
|
||||
it('a published release with no published_at is REFUSED by the CHECK', async () => {
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_release
|
||||
(release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, document_hash)
|
||||
VALUES ('rel_bad', 'dash_txn', 'dv_t3', 't_alpha', 'p_main', 900, 'published', '${HASH_A}')`),
|
||||
);
|
||||
expect(msg).toMatch(/opt_analytics_dashboard_release_timestamps/);
|
||||
});
|
||||
|
||||
it('a document_hash that is not a sha256 digest is REFUSED', async () => {
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(`
|
||||
INSERT INTO opt_analytics_dashboard_release
|
||||
(release_id, dashboard_id, dashboard_version_id, tenant_id, project_id, version, state, document_hash)
|
||||
VALUES ('rel_bad2', 'dash_txn', 'dv_t3', 't_alpha', 'p_main', 901, 'draft', 'not-a-hash')`),
|
||||
);
|
||||
expect(msg).toMatch(/document_hash/);
|
||||
});
|
||||
|
||||
it('the sql/008 immutability trigger still forbids editing the frozen document', async () => {
|
||||
// Assert the row exists FIRST: an UPDATE that matches nothing raises
|
||||
// nothing, which would read as "refused" to a careless test.
|
||||
const before = (await sql.unsafe(
|
||||
`SELECT count(*)::int AS n FROM opt_analytics_dashboard_version WHERE dashboard_version_id='dv_t3'`,
|
||||
)) as unknown as Array<{ n: number }>;
|
||||
expect(before[0].n).toBe(1);
|
||||
|
||||
const msg = await refusalOf(() =>
|
||||
sql.unsafe(
|
||||
`UPDATE opt_analytics_dashboard_version SET layout_json='{"tampered":true}'
|
||||
WHERE dashboard_version_id='dv_t3'`,
|
||||
),
|
||||
);
|
||||
expect(msg).toMatch(/rows are immutable/);
|
||||
});
|
||||
|
||||
it('a second open draft for one dashboard is REFUSED', async () => {
|
||||
await seedVersion(sql, 'dv_d1', 'dash_draft', 301);
|
||||
await seedVersion(sql, 'dv_d2', 'dash_draft', 302);
|
||||
await seedDraftRelease(sql, 'rel_d1', 'dv_d1', 'dash_draft', 301);
|
||||
const msg = await refusalOf(() => seedDraftRelease(sql, 'rel_d2', 'dv_d2', 'dash_draft', 302));
|
||||
expect(msg).toMatch(/one_draft/);
|
||||
});
|
||||
});
|
||||
|
||||
run('BI-P061 concurrency: the partial unique index is the guarantee', () => {
|
||||
let noguard: ReturnType<typeof postgres>;
|
||||
|
||||
beforeAll(async () => {
|
||||
noguard = connect(SCHEMA_NOGUARD, 3);
|
||||
await migrate(noguard);
|
||||
await noguard.unsafe(`DROP INDEX opt_analytics_dashboard_release_one_published`);
|
||||
await seedDashboard(noguard, 'dash_race');
|
||||
await seedVersion(noguard, 'dv_race_1', 'dash_race', 401);
|
||||
await seedVersion(noguard, 'dv_race_2', 'dash_race', 402, HASH_B);
|
||||
|
||||
await seedVersion(sql, 'dv_race_1', 'dash_race', 401);
|
||||
await seedVersion(sql, 'dv_race_2', 'dash_race', 402, HASH_B);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (noguard) await noguard.end({ timeout: 5 });
|
||||
});
|
||||
|
||||
it('WITHOUT the index, two concurrent publishes BOTH succeed — two live dashboards', async () => {
|
||||
const race = await racePublish(SCHEMA_NOGUARD, 'dash_race');
|
||||
expect(race.ok).toBe(2);
|
||||
expect(await publishedCount(noguard, 'dash_race')).toBe(2);
|
||||
}, 30_000);
|
||||
|
||||
it('WITH the index, the same race leaves exactly one published release', async () => {
|
||||
expect(await publishedCount(sql, 'dash_race')).toBe(0);
|
||||
|
||||
const race = await racePublish(SCHEMA, 'dash_race');
|
||||
|
||||
expect(race.ok).toBe(1);
|
||||
expect(race.codes.join(' ')).toContain('23505');
|
||||
expect(await publishedCount(sql, 'dash_race')).toBe(1);
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,212 @@
|
||||
/**
|
||||
* tests/analytics/dashboard-release.test.ts
|
||||
* ==========================================
|
||||
* BI-P061 — DASHBOARD VERSIONING. DoD: "rollback to previous dashboard
|
||||
* definition".
|
||||
*
|
||||
* A rollback test that only asserts "the service returned version 4" proves
|
||||
* nothing: the interesting failure is a rollback that publishes a DIFFERENT
|
||||
* document while reporting success. So every rollback assertion here compares
|
||||
* BI-P060 canonical hashes and re-plans the restored dashboard, and one test
|
||||
* corrupts the stored document underneath a release to show that serving it
|
||||
* is refused rather than silently rendered.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { SiaxError } from '@/src/platform/errors';
|
||||
import {
|
||||
dashboardDocumentHash,
|
||||
parseDashboardDocument,
|
||||
planDashboard,
|
||||
} from '@/src/analytics/dashboards/dashboard-document';
|
||||
import {
|
||||
DashboardReleaseService,
|
||||
InMemoryDashboardReleaseStore,
|
||||
isLegalTransition,
|
||||
DASHBOARD_RELEASE_STATES,
|
||||
type DashboardReleaseState,
|
||||
} from '@/src/analytics/dashboards/dashboard-release';
|
||||
|
||||
const SCOPE = { tenant_id: 't_alpha', project_id: 'p_main' };
|
||||
|
||||
function doc(title: string, metric = 'revenue') {
|
||||
return {
|
||||
id: 'dash_sales',
|
||||
title,
|
||||
version: 1,
|
||||
layout: { grid_columns: 12, row_height_px: 40 },
|
||||
parameters: [],
|
||||
filters: [],
|
||||
widgets: [
|
||||
{
|
||||
widget_id: 'w_1',
|
||||
title,
|
||||
source: { kind: 'metric', dataset: 'ds_sales', metric },
|
||||
visualization: { type: 'big_number', encoding: { value: metric } },
|
||||
layout: { x: 0, y: 0, w: 4, h: 2 },
|
||||
},
|
||||
],
|
||||
refresh: { mode: 'manual', max_staleness_seconds: 3600 },
|
||||
permissions: { visibility: 'tenant' },
|
||||
};
|
||||
}
|
||||
|
||||
function service() {
|
||||
let n = 0;
|
||||
let t = Date.parse('2026-09-07T10:00:00.000Z');
|
||||
return new DashboardReleaseService(
|
||||
new InMemoryDashboardReleaseStore(),
|
||||
() => new Date((t += 1000)),
|
||||
() => `id_${++n}`,
|
||||
);
|
||||
}
|
||||
|
||||
describe('BI-P061 release lifecycle', () => {
|
||||
it('the transition table is exactly DRAFT -> PUBLISHED -> SUPERSEDED', () => {
|
||||
const legal: Array<[DashboardReleaseState, DashboardReleaseState]> = [
|
||||
['draft', 'published'],
|
||||
['published', 'superseded'],
|
||||
];
|
||||
for (const from of DASHBOARD_RELEASE_STATES) {
|
||||
for (const to of DASHBOARD_RELEASE_STATES) {
|
||||
const expected = legal.some(([f, t]) => f === from && t === to);
|
||||
expect(isLegalTransition(from, to), `${from} -> ${to}`).toBe(expected);
|
||||
}
|
||||
}
|
||||
// Named explicitly because each is a real request someone will make.
|
||||
expect(isLegalTransition('published', 'draft')).toBe(false);
|
||||
expect(isLegalTransition('superseded', 'published')).toBe(false);
|
||||
expect(isLegalTransition('draft', 'superseded')).toBe(false);
|
||||
});
|
||||
|
||||
it('publishing supersedes the previous release, and only one is ever live', async () => {
|
||||
const svc = service();
|
||||
const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') });
|
||||
await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id });
|
||||
const v2 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v2') });
|
||||
const result = await svc.publish({ scope: SCOPE, releaseId: v2.release.release_id });
|
||||
|
||||
expect(result.published.version).toBe(2);
|
||||
expect(result.superseded?.version).toBe(1);
|
||||
expect(result.superseded?.superseded_by_release_id).toBe(v2.release.release_id);
|
||||
|
||||
const history = await svc.history(SCOPE, 'dash_sales');
|
||||
expect(history.map((r) => `${r.version}:${r.state}`)).toEqual(['1:superseded', '2:published']);
|
||||
expect((await svc.live(SCOPE, 'dash_sales'))?.version).toBe(2);
|
||||
});
|
||||
|
||||
it('a second open draft is refused rather than racing the first', async () => {
|
||||
const svc = service();
|
||||
await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') });
|
||||
await expect(
|
||||
svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1-again') }),
|
||||
).rejects.toThrow(/already has an open draft/);
|
||||
});
|
||||
|
||||
it('a superseded release cannot be re-published', async () => {
|
||||
const svc = service();
|
||||
const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') });
|
||||
await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id });
|
||||
const v2 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v2') });
|
||||
await svc.publish({ scope: SCOPE, releaseId: v2.release.release_id });
|
||||
|
||||
await expect(svc.publish({ scope: SCOPE, releaseId: v1.release.release_id })).rejects.toThrow(
|
||||
/is superseded, not draft/,
|
||||
);
|
||||
});
|
||||
|
||||
it('another tenant cannot see, publish or roll back this tenant’s releases', async () => {
|
||||
const svc = service();
|
||||
const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') });
|
||||
const other = { tenant_id: 't_beta', project_id: 'p_main' };
|
||||
|
||||
await expect(svc.publish({ scope: other, releaseId: v1.release.release_id })).rejects.toThrow(
|
||||
/does not exist in this tenant/,
|
||||
);
|
||||
expect(await svc.live(other, 'dash_sales')).toBeUndefined();
|
||||
await expect(svc.rollback({ scope: other, dashboardId: 'dash_sales', toVersion: 1 })).rejects.toThrow(
|
||||
/no version 1 in this tenant/,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P061 rollback — the DoD', () => {
|
||||
it('restores the earlier definition byte-for-byte, as a NEW version', async () => {
|
||||
const svc = service();
|
||||
const original = doc('Sales v1', 'revenue');
|
||||
const originalHash = dashboardDocumentHash(parseDashboardDocument(original));
|
||||
|
||||
const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: original });
|
||||
await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id });
|
||||
const v2 = await svc.draft({
|
||||
scope: SCOPE,
|
||||
dashboardId: 'dash_sales',
|
||||
document: doc('Sales v2 (broken)', 'revenue_net'),
|
||||
});
|
||||
await svc.publish({ scope: SCOPE, releaseId: v2.release.release_id });
|
||||
expect((await svc.live(SCOPE, 'dash_sales'))?.document_hash).not.toBe(originalHash);
|
||||
|
||||
const rolled = await svc.rollback({ scope: SCOPE, dashboardId: 'dash_sales', toVersion: 1 });
|
||||
|
||||
// Faithful: same canonical document.
|
||||
expect(rolled.document_hash).toBe(originalHash);
|
||||
// New version, not a resurrected row — the history stays linear.
|
||||
expect(rolled.release.version).toBe(3);
|
||||
expect(rolled.release.rollback_of_version).toBe(1);
|
||||
|
||||
const history = await svc.history(SCOPE, 'dash_sales');
|
||||
expect(history.map((r) => `${r.version}:${r.state}`)).toEqual([
|
||||
'1:superseded',
|
||||
'2:superseded',
|
||||
'3:published',
|
||||
]);
|
||||
|
||||
// And the restored dashboard actually renders as the original did.
|
||||
const live = await svc.live(SCOPE, 'dash_sales');
|
||||
const restored = await svc.documentOf(SCOPE, live!);
|
||||
expect(planDashboard(restored)).toEqual(planDashboard(parseDashboardDocument(original)));
|
||||
});
|
||||
|
||||
it('rolling back to the live version is refused as a no-op', async () => {
|
||||
const svc = service();
|
||||
const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') });
|
||||
await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id });
|
||||
await expect(svc.rollback({ scope: SCOPE, dashboardId: 'dash_sales', toVersion: 1 })).rejects.toThrow(
|
||||
/already live/,
|
||||
);
|
||||
});
|
||||
|
||||
it('a stored document that no longer matches its released hash is refused, not rendered', async () => {
|
||||
const store = new InMemoryDashboardReleaseStore();
|
||||
const svc = new DashboardReleaseService(
|
||||
store,
|
||||
() => new Date(),
|
||||
() => crypto.randomUUID(),
|
||||
);
|
||||
const v1 = await svc.draft({ scope: SCOPE, dashboardId: 'dash_sales', document: doc('v1') });
|
||||
const { published } = await svc.publish({ scope: SCOPE, releaseId: v1.release.release_id });
|
||||
|
||||
// Tamper with storage the way a bad migration or a manual UPDATE would.
|
||||
const stored = await store.getVersion(SCOPE, published.dashboard_version_id);
|
||||
const tampered = JSON.parse(stored!.layout_json);
|
||||
tampered.widgets[0].source.metric = 'revenue_including_vat';
|
||||
await store.insertVersionAndRelease(
|
||||
{ ...stored!, dashboard_version_id: 'tampered', version: 99, layout_json: JSON.stringify(tampered) },
|
||||
{
|
||||
...published,
|
||||
release_id: 'r_tampered',
|
||||
dashboard_version_id: 'tampered',
|
||||
version: 99,
|
||||
state: 'draft',
|
||||
published_at: undefined,
|
||||
},
|
||||
);
|
||||
|
||||
// documentOf must compare hashes, not trust the row.
|
||||
await expect(svc.documentOf(SCOPE, { ...published, dashboard_version_id: 'tampered' })).rejects.toThrow(
|
||||
/refusing to serve an unapproved definition/,
|
||||
);
|
||||
await expect(svc.documentOf(SCOPE, { ...published, dashboard_version_id: 'tampered' })).rejects.toThrow(
|
||||
SiaxError,
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,327 @@
|
||||
/**
|
||||
* tests/analytics/materialization-drift.test.ts
|
||||
* ==============================================
|
||||
* BI-P105 — MATERIALIZATION / PRE-AGGREGATION.
|
||||
*
|
||||
* The DoD asks for measured improvement and documented freshness semantics.
|
||||
* The failure mode it does not name, and the one that actually costs money, is
|
||||
* a pre-aggregate that returns a confident wrong number. So every test here is
|
||||
* written as an attack on the router:
|
||||
*
|
||||
* - a drifted aggregate must NOT serve, and the counterfactual (a naive
|
||||
* router that only checks staleness) must serve it — otherwise the guard
|
||||
* is not known to be doing anything;
|
||||
* - a MISSING bucket must count as drift even though every value present is
|
||||
* correct;
|
||||
* - a comparison over an empty window must be `insufficient_data`, never
|
||||
* `match`;
|
||||
* - an aggregate that has never been verified must not serve, because on
|
||||
* launch day it is exactly as unproven as a drifted one.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
applyDriftReport,
|
||||
detectDrift,
|
||||
type AggregateBucket,
|
||||
} from '@/src/analytics/materialization/drift-detector';
|
||||
import {
|
||||
assertSpec,
|
||||
describeRouting,
|
||||
materializationStaleness,
|
||||
selectQuerySource,
|
||||
type MaterializationSpec,
|
||||
type MaterializationStatus,
|
||||
} from '@/src/analytics/materialization/materialization';
|
||||
import type { AnalyticsQueryRequest } from '@/src/analytics/contract/query-request';
|
||||
|
||||
const NOW = new Date('2026-09-07T12:00:00.000Z');
|
||||
const TOLERANCE = { absolute: 0.5, relative: 0.001 };
|
||||
|
||||
const SPEC: MaterializationSpec = {
|
||||
materialization_id: 'mv_sales_daily',
|
||||
tenant_id: 't_alpha',
|
||||
project_id: 'p_main',
|
||||
dataset: 'ds_sales',
|
||||
strategy: 'clickhouse_materialized_view',
|
||||
target_ref: 'opt_sales_daily',
|
||||
metrics: ['revenue', 'orders'],
|
||||
dimensions: ['order_date', 'region'],
|
||||
time_dimension: 'order_date',
|
||||
grain: 'day',
|
||||
max_staleness_seconds: 3600,
|
||||
drift_check_interval_seconds: 86_400,
|
||||
tolerance: TOLERANCE,
|
||||
};
|
||||
|
||||
/** Fresh, recently verified, matching. The only state that may serve. */
|
||||
function healthyStatus(): MaterializationStatus {
|
||||
return {
|
||||
materialization_id: SPEC.materialization_id,
|
||||
state: 'serving',
|
||||
data_watermark: '2026-09-07T11:45:00.000Z',
|
||||
last_refresh_at: '2026-09-07T11:46:00.000Z',
|
||||
last_drift_check_at: '2026-09-07T06:00:00.000Z',
|
||||
last_drift_verdict: 'match',
|
||||
};
|
||||
}
|
||||
|
||||
const REQUEST: AnalyticsQueryRequest = {
|
||||
dataset: 'ds_sales',
|
||||
metrics: ['revenue'],
|
||||
dimensions: ['order_date'],
|
||||
filters: [{ dimension: 'region', operator: 'eq', values: ['SE'] }],
|
||||
timeRange: { from: '2026-09-01T00:00:00.000Z', to: '2026-09-07T00:00:00.000Z', grain: 'day' },
|
||||
};
|
||||
|
||||
/**
|
||||
* The router someone writes when they have thought about staleness but not
|
||||
* about correctness. It exists here as the counterfactual: it must SERVE the
|
||||
* drifted aggregate that `selectQuerySource` refuses.
|
||||
*/
|
||||
function naiveRouter(status: MaterializationStatus, now: Date): 'materialization' | 'base_table' {
|
||||
const staleness = materializationStaleness(status, now) ?? 0;
|
||||
return staleness <= SPEC.max_staleness_seconds ? 'materialization' : 'base_table';
|
||||
}
|
||||
|
||||
function buckets(spec: Record<string, [number, number]>): AggregateBucket[] {
|
||||
return Object.entries(spec).map(([key, [revenue, orders]]) => ({ key, values: { revenue, orders } }));
|
||||
}
|
||||
|
||||
const SOURCE = buckets({
|
||||
'2026-09-01|SE': [120_000, 410],
|
||||
'2026-09-02|SE': [131_500, 442],
|
||||
'2026-09-03|SE': [98_250, 305],
|
||||
'2026-09-04|SE': [140_000, 480],
|
||||
});
|
||||
|
||||
describe('BI-P105 drift detection', () => {
|
||||
it('an identical aggregate matches', () => {
|
||||
const report = detectDrift({
|
||||
fromSource: SOURCE,
|
||||
fromAggregate: SOURCE,
|
||||
tolerance: TOLERANCE,
|
||||
metrics: ['revenue', 'orders'],
|
||||
now: NOW,
|
||||
});
|
||||
expect(report.verdict).toBe('match');
|
||||
expect(report.buckets_compared).toBe(4);
|
||||
expect(report.totals.revenue.delta).toBe(0);
|
||||
});
|
||||
|
||||
it('a value beyond tolerance is drift, and the report says what a caller would have been told', () => {
|
||||
const drifted = buckets({
|
||||
'2026-09-01|SE': [120_000, 410],
|
||||
'2026-09-02|SE': [131_500, 442],
|
||||
'2026-09-03|SE': [98_250, 305],
|
||||
// A source row was deleted (GDPR erasure) but the aggregate still counts it.
|
||||
'2026-09-04|SE': [152_400, 495],
|
||||
});
|
||||
const report = detectDrift({
|
||||
fromSource: SOURCE,
|
||||
fromAggregate: drifted,
|
||||
tolerance: TOLERANCE,
|
||||
metrics: ['revenue', 'orders'],
|
||||
now: NOW,
|
||||
});
|
||||
expect(report.verdict).toBe('drifted');
|
||||
expect(report.worst?.key).toBe('2026-09-04|SE');
|
||||
expect(report.worst?.absolute_delta).toBe(12_400);
|
||||
expect(report.totals.revenue.aggregate - report.totals.revenue.source).toBe(12_400);
|
||||
});
|
||||
|
||||
it('a MISSING bucket is drift even though every value present is correct', () => {
|
||||
const lateArriving = SOURCE.filter((b) => b.key !== '2026-09-04|SE');
|
||||
const report = detectDrift({
|
||||
fromSource: SOURCE,
|
||||
fromAggregate: lateArriving,
|
||||
tolerance: TOLERANCE,
|
||||
metrics: ['revenue', 'orders'],
|
||||
now: NOW,
|
||||
});
|
||||
expect(report.discrepancies).toHaveLength(0); // every compared value agrees
|
||||
expect(report.verdict).toBe('drifted'); // and it is still wrong
|
||||
expect(report.missing_in_aggregate).toEqual(['2026-09-04|SE']);
|
||||
expect(report.totals.revenue.delta).toBe(-140_000);
|
||||
});
|
||||
|
||||
it('an EXTRA bucket is drift — the source rows are gone, the aggregate is not', () => {
|
||||
const stale = [...SOURCE, { key: '2026-08-31|SE', values: { revenue: 90_000, orders: 300 } }];
|
||||
const report = detectDrift({
|
||||
fromSource: SOURCE,
|
||||
fromAggregate: stale,
|
||||
tolerance: TOLERANCE,
|
||||
metrics: ['revenue', 'orders'],
|
||||
now: NOW,
|
||||
});
|
||||
expect(report.verdict).toBe('drifted');
|
||||
expect(report.extra_in_aggregate).toEqual(['2026-08-31|SE']);
|
||||
});
|
||||
|
||||
it('an EMPTY comparison is insufficient_data, not match', () => {
|
||||
const report = detectDrift({
|
||||
fromSource: [],
|
||||
fromAggregate: [],
|
||||
tolerance: TOLERANCE,
|
||||
metrics: ['revenue'],
|
||||
now: NOW,
|
||||
});
|
||||
expect(report.verdict).toBe('insufficient_data');
|
||||
expect(report.buckets_compared).toBe(0);
|
||||
// And it must not be allowed to serve on the strength of that.
|
||||
const status = applyDriftReport(healthyStatus(), report);
|
||||
expect(status.state).toBe('drifted');
|
||||
});
|
||||
|
||||
it('both tolerances must hold — a large bucket cannot hide behind a relative bound', () => {
|
||||
const source = buckets({ 'b|SE': [40_000_000, 1] });
|
||||
const off = buckets({ 'b|SE': [40_020_000, 1] }); // 0.05%: inside relative, far outside absolute
|
||||
const report = detectDrift({
|
||||
fromSource: source,
|
||||
fromAggregate: off,
|
||||
tolerance: { absolute: 1000, relative: 0.001 },
|
||||
metrics: ['revenue'],
|
||||
now: NOW,
|
||||
});
|
||||
expect(report.verdict).toBe('drifted');
|
||||
expect(report.worst?.relative_delta).toBeCloseTo(0.0005, 6);
|
||||
});
|
||||
|
||||
it('a bucket that should be zero and is not is drift, not a division error', () => {
|
||||
const report = detectDrift({
|
||||
fromSource: buckets({ 'b|SE': [0, 0] }),
|
||||
fromAggregate: buckets({ 'b|SE': [12, 1] }),
|
||||
tolerance: TOLERANCE,
|
||||
metrics: ['revenue'],
|
||||
now: NOW,
|
||||
});
|
||||
expect(report.verdict).toBe('drifted');
|
||||
expect(report.worst?.relative_delta).toBe(Number.POSITIVE_INFINITY);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P105 routing — a pre-aggregate serves only while it is provably current and undrifted', () => {
|
||||
it('serves when everything is proved', () => {
|
||||
const decision = selectQuerySource({ request: REQUEST, spec: SPEC, status: healthyStatus(), now: NOW });
|
||||
expect(decision.source).toBe('materialization');
|
||||
expect(describeRouting(decision)).toMatchObject({
|
||||
served_from: 'materialization',
|
||||
materialization_id: 'mv_sales_daily',
|
||||
staleness_seconds: 900,
|
||||
});
|
||||
});
|
||||
|
||||
it('a DRIFTED aggregate is refused — and the naive router serves it', () => {
|
||||
const drifted = applyDriftReport(healthyStatus(), {
|
||||
verdict: 'drifted',
|
||||
checked_at: '2026-09-07T11:00:00.000Z',
|
||||
buckets_compared: 4,
|
||||
missing_in_aggregate: [],
|
||||
extra_in_aggregate: [],
|
||||
discrepancies: [],
|
||||
tolerance: TOLERANCE,
|
||||
totals: {},
|
||||
});
|
||||
|
||||
const guarded = selectQuerySource({ request: REQUEST, spec: SPEC, status: drifted, now: NOW });
|
||||
expect(guarded.source).toBe('base_table');
|
||||
expect(guarded).toMatchObject({ refusal: 'state_not_serving' });
|
||||
|
||||
// The counterfactual: the same state, routed by staleness alone.
|
||||
expect(naiveRouter(drifted, NOW)).toBe('materialization');
|
||||
});
|
||||
|
||||
it('a NEVER-VERIFIED aggregate is refused — and the naive router serves it', () => {
|
||||
const brandNew: MaterializationStatus = {
|
||||
materialization_id: SPEC.materialization_id,
|
||||
state: 'serving',
|
||||
data_watermark: '2026-09-07T11:59:00.000Z',
|
||||
last_refresh_at: '2026-09-07T11:59:00.000Z',
|
||||
};
|
||||
const guarded = selectQuerySource({ request: REQUEST, spec: SPEC, status: brandNew, now: NOW });
|
||||
expect(guarded).toMatchObject({ source: 'base_table', refusal: 'drift_never_verified' });
|
||||
expect(naiveRouter(brandNew, NOW)).toBe('materialization');
|
||||
});
|
||||
|
||||
it('a verification older than the required interval is refused', () => {
|
||||
const status = { ...healthyStatus(), last_drift_check_at: '2026-09-05T06:00:00.000Z' };
|
||||
expect(selectQuerySource({ request: REQUEST, spec: SPEC, status, now: NOW })).toMatchObject({
|
||||
refusal: 'drift_verification_stale',
|
||||
});
|
||||
});
|
||||
|
||||
it('an UNKNOWN watermark is refused rather than treated as current', () => {
|
||||
const status = { ...healthyStatus(), data_watermark: undefined };
|
||||
expect(materializationStaleness(status, NOW)).toBeUndefined();
|
||||
expect(selectQuerySource({ request: REQUEST, spec: SPEC, status, now: NOW })).toMatchObject({
|
||||
refusal: 'watermark_unknown',
|
||||
});
|
||||
});
|
||||
|
||||
it('an aggregate older than its freshness contract is refused', () => {
|
||||
const status = { ...healthyStatus(), data_watermark: '2026-09-07T09:00:00.000Z' };
|
||||
expect(selectQuerySource({ request: REQUEST, spec: SPEC, status, now: NOW })).toMatchObject({
|
||||
refusal: 'staleness_exceeded',
|
||||
});
|
||||
});
|
||||
|
||||
it('a question that reaches past the watermark is refused, not silently shortened', () => {
|
||||
const request = { ...REQUEST, timeRange: { ...REQUEST.timeRange, to: '2026-09-07T12:00:00.000Z' } };
|
||||
expect(selectQuerySource({ request, spec: SPEC, status: healthyStatus(), now: NOW })).toMatchObject({
|
||||
refusal: 'window_beyond_watermark',
|
||||
});
|
||||
});
|
||||
|
||||
it('a filter on a dimension the aggregate summed across is refused', () => {
|
||||
const request = {
|
||||
...REQUEST,
|
||||
filters: [{ dimension: 'sales_channel', operator: 'eq' as const, values: ['web'] }],
|
||||
};
|
||||
const decision = selectQuerySource({ request, spec: SPEC, status: healthyStatus(), now: NOW });
|
||||
expect(decision).toMatchObject({ source: 'base_table', refusal: 'coverage_miss' });
|
||||
expect((decision as { detail: string }).detail).toMatch(/already summed across it/);
|
||||
});
|
||||
|
||||
it('a metric or dimension the aggregate does not carry is refused', () => {
|
||||
expect(
|
||||
selectQuerySource({
|
||||
request: { ...REQUEST, metrics: ['margin'] },
|
||||
spec: SPEC,
|
||||
status: healthyStatus(),
|
||||
now: NOW,
|
||||
}),
|
||||
).toMatchObject({ refusal: 'coverage_miss' });
|
||||
expect(
|
||||
selectQuerySource({
|
||||
request: { ...REQUEST, dimensions: ['product'] },
|
||||
spec: SPEC,
|
||||
status: healthyStatus(),
|
||||
now: NOW,
|
||||
}),
|
||||
).toMatchObject({ refusal: 'coverage_miss' });
|
||||
});
|
||||
|
||||
it('a daily aggregate cannot answer an hourly question, but can answer a monthly one', () => {
|
||||
const hourly = { ...REQUEST, timeRange: { ...REQUEST.timeRange, grain: 'hour' as const } };
|
||||
expect(
|
||||
selectQuerySource({ request: hourly, spec: SPEC, status: healthyStatus(), now: NOW }),
|
||||
).toMatchObject({ refusal: 'coverage_miss' });
|
||||
|
||||
const monthly = { ...REQUEST, timeRange: { ...REQUEST.timeRange, grain: 'month' as const } };
|
||||
expect(
|
||||
selectQuerySource({ request: monthly, spec: SPEC, status: healthyStatus(), now: NOW }).source,
|
||||
).toBe('materialization');
|
||||
});
|
||||
|
||||
it('a WEEKLY aggregate cannot answer a monthly question — weeks do not compose into months', () => {
|
||||
const weekly = { ...SPEC, grain: 'week' as const };
|
||||
const monthly = { ...REQUEST, timeRange: { ...REQUEST.timeRange, grain: 'month' as const } };
|
||||
expect(
|
||||
selectQuerySource({ request: monthly, spec: weekly, status: healthyStatus(), now: NOW }),
|
||||
).toMatchObject({ refusal: 'coverage_miss' });
|
||||
});
|
||||
|
||||
it('a spec with no drift check interval is refused at declaration time', () => {
|
||||
expect(() => assertSpec({ ...SPEC, drift_check_interval_seconds: 0 })).toThrow(/never re-verified/);
|
||||
expect(() => assertSpec(SPEC)).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,344 @@
|
||||
/**
|
||||
* tests/integration/bi-p104-benchmark.integration.test.ts
|
||||
* ========================================================
|
||||
* BI-P104 — the benchmark itself, against real engines. Skips honestly when
|
||||
* CLICKHOUSE_URL is unset; DuckDB additionally needs DUCKDB_PYTHON (an
|
||||
* interpreter with the `duckdb` package), matching
|
||||
* tests/duckdb/duckdb-provider.live.test.ts.
|
||||
*
|
||||
* Six dataset profiles, one representative question each, the same logical
|
||||
* question on both engines. Deliberately NO engine-specific functions in the
|
||||
* geospatial case (bounding box + grid arithmetic rather than each engine's
|
||||
* GIS library) so the comparison is about the storage and execution engine
|
||||
* rather than about who ships the nicer geo function.
|
||||
*
|
||||
* Statistics come from src/analytics/bench/benchmark-harness.ts — the shipped
|
||||
* code, not a copy — so the evidence file is produced by the thing under test.
|
||||
*
|
||||
* Trino, Superset, native charts and Perspective are NOT exercised, and each
|
||||
* appears in the evidence with its reason. An engine omitted from a benchmark
|
||||
* table reads as a choice, so nothing is omitted.
|
||||
*
|
||||
* CLICKHOUSE_URL=http://127.0.0.1:28123 CLICKHOUSE_AUTH=default:test \
|
||||
* DUCKDB_PYTHON=/path/to/venv/bin/python \
|
||||
* pnpm vitest run tests/integration/bi-p104-benchmark.integration.test.ts
|
||||
*
|
||||
* It CREATEs and DROPs its own ClickHouse database and uses an in-memory
|
||||
* DuckDB. Never point it at the b0ts ClickHouse or any quarantined data.
|
||||
*/
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
|
||||
import {
|
||||
buildEvidence,
|
||||
compareDistributions,
|
||||
describeComparison,
|
||||
summarizeSamples,
|
||||
type BenchDatasetId,
|
||||
type BenchmarkRun,
|
||||
type EngineAvailability,
|
||||
} from '@/src/analytics/bench/benchmark-harness';
|
||||
|
||||
const CLICKHOUSE_URL = process.env.CLICKHOUSE_URL;
|
||||
const CLICKHOUSE_AUTH = process.env.CLICKHOUSE_AUTH;
|
||||
const DUCKDB_PYTHON = process.env.DUCKDB_PYTHON;
|
||||
const run = CLICKHOUSE_URL ? describe : describe.skip;
|
||||
const DB = `opt_p104_${Date.now().toString(36)}`;
|
||||
const WARMUPS = 2;
|
||||
const REPETITIONS = 9;
|
||||
const WIDE_COLUMNS = 50;
|
||||
|
||||
const headers: Record<string, string> = CLICKHOUSE_AUTH
|
||||
? { Authorization: `Basic ${Buffer.from(CLICKHOUSE_AUTH).toString('base64')}` }
|
||||
: {};
|
||||
|
||||
async function ch(query: string): Promise<string> {
|
||||
const res = await fetch(CLICKHOUSE_URL ?? 'http://skipped.invalid', {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: query,
|
||||
});
|
||||
const text = await res.text();
|
||||
if (!res.ok) throw new Error(`ClickHouse ${res.status}: ${text.slice(0, 300)}`);
|
||||
return text;
|
||||
}
|
||||
|
||||
/** The six questions, one per profile, phrased for each engine. */
|
||||
const QUERIES: Record<BenchDatasetId, { label: string; clickhouse: string; duckdb: string }> = {
|
||||
small_saas: {
|
||||
label: 'revenue by day',
|
||||
clickhouse: `SELECT d, sum(amount) AS revenue, count() AS n FROM ${DB}.small_saas WHERE tenant_id='t_1' GROUP BY d ORDER BY d`,
|
||||
duckdb: `SELECT d, sum(amount) AS revenue, count(*) AS n FROM small_saas WHERE tenant_id='t_1' GROUP BY d ORDER BY d`,
|
||||
},
|
||||
medium_business: {
|
||||
label: 'revenue by month and region for one tenant',
|
||||
clickhouse: `SELECT toStartOfMonth(d) AS m, region, sum(amount) AS revenue FROM ${DB}.medium_business WHERE tenant_id='t_2' GROUP BY m, region ORDER BY m, region`,
|
||||
duckdb: `SELECT date_trunc('month', d) AS m, region, sum(amount) AS revenue FROM medium_business WHERE tenant_id='t_2' GROUP BY m, region ORDER BY m, region`,
|
||||
},
|
||||
vehiq_market: {
|
||||
label: 'average asking price by make and model year',
|
||||
clickhouse: `SELECT make, model_year, avg(price) AS avg_price, count() AS listings FROM ${DB}.vehiq_market WHERE mileage < 150000 GROUP BY make, model_year ORDER BY make, model_year`,
|
||||
duckdb: `SELECT make, model_year, avg(price) AS avg_price, count(*) AS listings FROM vehiq_market WHERE mileage < 150000 GROUP BY make, model_year ORDER BY make, model_year`,
|
||||
},
|
||||
large_event: {
|
||||
label: 'hourly event volume for one event type',
|
||||
clickhouse: `SELECT toStartOfHour(ts) AS h, count() AS n, uniqExact(user_id) AS users FROM ${DB}.large_event WHERE event_type='checkout' GROUP BY h ORDER BY h`,
|
||||
duckdb: `SELECT date_trunc('hour', ts) AS h, count(*) AS n, count(DISTINCT user_id) AS users FROM large_event WHERE event_type='checkout' GROUP BY h ORDER BY h`,
|
||||
},
|
||||
wide_table: {
|
||||
label: 'three columns of fifty, grouped',
|
||||
clickhouse: `SELECT grp, sum(c7) AS a, sum(c23) AS b, sum(c41) AS c FROM ${DB}.wide_table GROUP BY grp ORDER BY grp`,
|
||||
duckdb: `SELECT grp, sum(c7) AS a, sum(c23) AS b, sum(c41) AS c FROM wide_table GROUP BY grp ORDER BY grp`,
|
||||
},
|
||||
geospatial: {
|
||||
label: 'bounding box, aggregated to a grid cell',
|
||||
clickhouse: `SELECT floor(lat*10)/10 AS cell_lat, floor(lon*10)/10 AS cell_lon, count() AS n FROM ${DB}.geospatial WHERE lat BETWEEN 57.5 AND 59.5 AND lon BETWEEN 11.5 AND 18.5 GROUP BY cell_lat, cell_lon ORDER BY cell_lat, cell_lon`,
|
||||
duckdb: `SELECT floor(lat*10)/10 AS cell_lat, floor(lon*10)/10 AS cell_lon, count(*) AS n FROM geospatial WHERE lat BETWEEN 57.5 AND 59.5 AND lon BETWEEN 11.5 AND 18.5 GROUP BY cell_lat, cell_lon ORDER BY cell_lat, cell_lon`,
|
||||
},
|
||||
};
|
||||
|
||||
const CH_DDL: Array<[BenchDatasetId, string]> = [
|
||||
[
|
||||
'small_saas',
|
||||
`CREATE TABLE ${DB}.small_saas ENGINE = MergeTree ORDER BY (tenant_id, d) AS
|
||||
SELECT concat('t_', toString(number % 3)) AS tenant_id,
|
||||
toDate('2026-01-01') + toIntervalDay(number % 365) AS d,
|
||||
['free','pro','enterprise'][(number % 3) + 1] AS plan,
|
||||
toDecimal64((number % 499) + 9, 2) AS amount
|
||||
FROM numbers(100000)`,
|
||||
],
|
||||
[
|
||||
'medium_business',
|
||||
`CREATE TABLE ${DB}.medium_business ENGINE = MergeTree ORDER BY (tenant_id, d) AS
|
||||
SELECT concat('t_', toString(number % 8)) AS tenant_id,
|
||||
toDate('2025-01-01') + toIntervalDay(number % 730) AS d,
|
||||
['SE','NO','DK','FI','DE'][(number % 5) + 1] AS region,
|
||||
toDecimal64((number % 977) + 13, 2) AS amount
|
||||
FROM numbers(2000000)`,
|
||||
],
|
||||
[
|
||||
'vehiq_market',
|
||||
`CREATE TABLE ${DB}.vehiq_market ENGINE = MergeTree ORDER BY (make, model_year) AS
|
||||
SELECT ['Volvo','VW','Toyota','BMW','Kia','Tesla'][(number % 6) + 1] AS make,
|
||||
toUInt16(2005 + (number % 20)) AS model_year,
|
||||
toUInt32(30000 + (number % 900000)) AS price,
|
||||
toUInt32((number * 7) % 400000) AS mileage,
|
||||
['diesel','petrol','hybrid','electric'][(number % 4) + 1] AS fuel
|
||||
FROM numbers(1000000)`,
|
||||
],
|
||||
[
|
||||
'large_event',
|
||||
`CREATE TABLE ${DB}.large_event ENGINE = MergeTree ORDER BY (event_type, ts) AS
|
||||
SELECT toDateTime('2026-08-01 00:00:00') + toIntervalSecond(number % 2592000) AS ts,
|
||||
toUInt32(number % 250000) AS user_id,
|
||||
['view','click','checkout','signup'][(number % 4) + 1] AS event_type,
|
||||
toFloat64(number % 1000) AS value
|
||||
FROM numbers(5000000)`,
|
||||
],
|
||||
[
|
||||
'wide_table',
|
||||
`CREATE TABLE ${DB}.wide_table ENGINE = MergeTree ORDER BY grp AS
|
||||
SELECT toUInt16(number % 500) AS grp,
|
||||
${Array.from({ length: WIDE_COLUMNS }, (_, i) => `toFloat64((number + ${i}) % 1000) AS c${i}`).join(',\n ')}
|
||||
FROM numbers(200000)`,
|
||||
],
|
||||
[
|
||||
'geospatial',
|
||||
`CREATE TABLE ${DB}.geospatial ENGINE = MergeTree ORDER BY (lat, lon) AS
|
||||
SELECT 55.0 + (number % 10000) / 1000.0 AS lat,
|
||||
10.0 + ((number * 13) % 10000) / 1000.0 AS lon,
|
||||
toUInt32(number) AS point_id
|
||||
FROM numbers(500000)`,
|
||||
],
|
||||
];
|
||||
|
||||
/** Runs all six DuckDB measurements inside ONE python process. */
|
||||
const DUCKDB_PROGRAM = `
|
||||
import duckdb, json, time, sys
|
||||
con = duckdb.connect()
|
||||
# The progress bar writes to STDOUT and corrupts the JSON transcript. It cost
|
||||
# one run to find, so it is turned off explicitly rather than filtered out.
|
||||
con.execute("SET enable_progress_bar=false")
|
||||
WIDE = ${WIDE_COLUMNS}
|
||||
wide_cols = ", ".join(f"(i + {j}) % 1000.0 AS c{j}" for j in range(WIDE))
|
||||
ddl = {
|
||||
"small_saas": "CREATE TABLE small_saas AS SELECT 't_' || (i % 3) AS tenant_id, DATE '2026-01-01' + INTERVAL (i % 365) DAY AS d, ['free','pro','enterprise'][(i % 3) + 1] AS plan, ((i % 499) + 9)::DECIMAL(18,2) AS amount FROM range(100000) t(i)",
|
||||
"medium_business": "CREATE TABLE medium_business AS SELECT 't_' || (i % 8) AS tenant_id, DATE '2025-01-01' + INTERVAL (i % 730) DAY AS d, ['SE','NO','DK','FI','DE'][(i % 5) + 1] AS region, ((i % 977) + 13)::DECIMAL(18,2) AS amount FROM range(2000000) t(i)",
|
||||
"vehiq_market": "CREATE TABLE vehiq_market AS SELECT ['Volvo','VW','Toyota','BMW','Kia','Tesla'][(i % 6) + 1] AS make, (2005 + (i % 20))::USMALLINT AS model_year, (30000 + (i % 900000))::UINTEGER AS price, ((i * 7) % 400000)::UINTEGER AS mileage, ['diesel','petrol','hybrid','electric'][(i % 4) + 1] AS fuel FROM range(1000000) t(i)",
|
||||
"large_event": "CREATE TABLE large_event AS SELECT TIMESTAMP '2026-08-01 00:00:00' + INTERVAL (i % 2592000) SECOND AS ts, (i % 250000)::UINTEGER AS user_id, ['view','click','checkout','signup'][(i % 4) + 1] AS event_type, (i % 1000)::DOUBLE AS value FROM range(5000000) t(i)",
|
||||
"wide_table": f"CREATE TABLE wide_table AS SELECT (i % 500)::USMALLINT AS grp, {wide_cols} FROM range(200000) t(i)",
|
||||
"geospatial": "CREATE TABLE geospatial AS SELECT 55.0 + (i % 10000) / 1000.0 AS lat, 10.0 + ((i * 13) % 10000) / 1000.0 AS lon, i::UINTEGER AS point_id FROM range(500000) t(i)",
|
||||
}
|
||||
queries = json.loads(sys.stdin.read())
|
||||
out = {"version": duckdb.__version__, "runs": {}}
|
||||
for name, sql in ddl.items():
|
||||
con.execute(sql)
|
||||
for name, q in queries.items():
|
||||
warm = []
|
||||
for _ in range(${WARMUPS}):
|
||||
t0 = time.perf_counter(); rows = con.execute(q).fetchall(); warm.append((time.perf_counter() - t0) * 1000)
|
||||
samples = []
|
||||
for _ in range(${REPETITIONS}):
|
||||
t0 = time.perf_counter(); rows = con.execute(q).fetchall(); samples.append((time.perf_counter() - t0) * 1000)
|
||||
out["runs"][name] = {"warmup_ms": warm, "samples_ms": samples, "rows": len(rows)}
|
||||
sys.stdout.write("<<<BENCH>>>" + json.dumps(out))
|
||||
`;
|
||||
|
||||
const runs: BenchmarkRun[] = [];
|
||||
const engines: EngineAvailability[] = [];
|
||||
let duckdbResult:
|
||||
| { version: string; runs: Record<string, { warmup_ms: number[]; samples_ms: number[]; rows: number }> }
|
||||
| undefined;
|
||||
|
||||
run('BI-P104 benchmark harness against real engines', () => {
|
||||
beforeAll(async () => {
|
||||
const version = (await ch('SELECT version()')).trim();
|
||||
engines.push({ engine: 'clickhouse', available: true, version });
|
||||
await ch(`CREATE DATABASE IF NOT EXISTS ${DB}`);
|
||||
for (const [, ddl] of CH_DDL) await ch(ddl);
|
||||
}, 900_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (!CLICKHOUSE_URL) return;
|
||||
try {
|
||||
await ch(`DROP DATABASE IF EXISTS ${DB}`);
|
||||
} finally {
|
||||
const comparisons = duckdbResult
|
||||
? (Object.keys(QUERIES) as BenchDatasetId[]).flatMap((dataset) => {
|
||||
const chRun = runs.find((r) => r.engine === 'clickhouse' && r.dataset === dataset);
|
||||
const duckRun = runs.find((r) => r.engine === 'duckdb' && r.dataset === dataset);
|
||||
if (!chRun || !duckRun) return [];
|
||||
const c = compareDistributions(chRun.distribution, duckRun.distribution);
|
||||
return [
|
||||
{
|
||||
dataset,
|
||||
query_label: QUERIES[dataset].label,
|
||||
description: describeComparison(c, `duckdb vs clickhouse (${dataset})`),
|
||||
...c,
|
||||
},
|
||||
];
|
||||
})
|
||||
: [];
|
||||
const evidence = buildEvidence({
|
||||
host_note:
|
||||
'One idle host, warm caches, both engines local, single-threaded client. These numbers ' +
|
||||
'describe these query shapes on this data on this machine. They are not engine rankings.',
|
||||
engines,
|
||||
runs,
|
||||
comparisons,
|
||||
});
|
||||
const dir = path.join(process.cwd(), 'evidence/bi-p104');
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, 'benchmark.json'), `${JSON.stringify(evidence, null, 2)}\n`);
|
||||
}
|
||||
}, 120_000);
|
||||
|
||||
it('measures all six dataset profiles on ClickHouse', async () => {
|
||||
for (const dataset of Object.keys(QUERIES) as BenchDatasetId[]) {
|
||||
const q = QUERIES[dataset];
|
||||
const warmup_ms: number[] = [];
|
||||
const samples_ms: number[] = [];
|
||||
let rows = 0;
|
||||
for (let i = 0; i < WARMUPS; i += 1) {
|
||||
const t0 = performance.now();
|
||||
await ch(`${q.clickhouse} FORMAT JSONCompact`);
|
||||
warmup_ms.push(performance.now() - t0);
|
||||
}
|
||||
for (let i = 0; i < REPETITIONS; i += 1) {
|
||||
const t0 = performance.now();
|
||||
const text = await ch(`${q.clickhouse} FORMAT JSONCompact`);
|
||||
samples_ms.push(performance.now() - t0);
|
||||
rows = (JSON.parse(text) as { rows: number }).rows;
|
||||
}
|
||||
runs.push({
|
||||
engine: 'clickhouse',
|
||||
dataset,
|
||||
query_label: q.label,
|
||||
warmup_ms,
|
||||
samples_ms,
|
||||
distribution: summarizeSamples(samples_ms),
|
||||
rows_returned: rows,
|
||||
});
|
||||
}
|
||||
expect(runs.filter((r) => r.engine === 'clickhouse')).toHaveLength(6);
|
||||
for (const r of runs) {
|
||||
expect(r.distribution.n).toBe(REPETITIONS);
|
||||
expect(r.distribution.stability).not.toBe('insufficient_samples');
|
||||
}
|
||||
}, 900_000);
|
||||
|
||||
it('measures the same six on DuckDB, or records why it did not', async () => {
|
||||
if (!DUCKDB_PYTHON) {
|
||||
engines.push({
|
||||
engine: 'duckdb',
|
||||
available: false,
|
||||
reason: 'DUCKDB_PYTHON not set on this runner',
|
||||
});
|
||||
expect(engines.find((e) => e.engine === 'duckdb')?.reason).toBeTruthy();
|
||||
return;
|
||||
}
|
||||
const payload = Object.fromEntries(
|
||||
(Object.keys(QUERIES) as BenchDatasetId[]).map((d) => [d, QUERIES[d].duckdb]),
|
||||
);
|
||||
const stdout = execFileSync(DUCKDB_PYTHON, ['-c', DUCKDB_PROGRAM], {
|
||||
input: JSON.stringify(payload),
|
||||
encoding: 'utf8',
|
||||
maxBuffer: 32 * 1024 * 1024,
|
||||
timeout: 900_000,
|
||||
});
|
||||
// Parse from the sentinel: anything DuckDB writes to stdout before it
|
||||
// (a progress bar, a warning) must not be able to corrupt the transcript.
|
||||
const marker = stdout.lastIndexOf('<<<BENCH>>>');
|
||||
expect(marker, 'duckdb transcript marker missing').toBeGreaterThanOrEqual(0);
|
||||
duckdbResult = JSON.parse(stdout.slice(marker + '<<<BENCH>>>'.length));
|
||||
engines.push({ engine: 'duckdb', available: true, version: duckdbResult!.version });
|
||||
|
||||
for (const [dataset, r] of Object.entries(duckdbResult!.runs)) {
|
||||
runs.push({
|
||||
engine: 'duckdb',
|
||||
dataset: dataset as BenchDatasetId,
|
||||
query_label: QUERIES[dataset as BenchDatasetId].label,
|
||||
warmup_ms: r.warmup_ms,
|
||||
samples_ms: r.samples_ms,
|
||||
distribution: summarizeSamples(r.samples_ms),
|
||||
rows_returned: r.rows,
|
||||
});
|
||||
}
|
||||
expect(runs.filter((r) => r.engine === 'duckdb')).toHaveLength(6);
|
||||
}, 900_000);
|
||||
|
||||
it('records the engines BI-P104 names that nothing measured, with reasons', () => {
|
||||
engines.push(
|
||||
{
|
||||
engine: 'trino',
|
||||
available: false,
|
||||
reason:
|
||||
'a Trino is running on this host but belongs to another lane (st0re-verify-trino); ' +
|
||||
'benchmarking through someone else’s container would measure their configuration',
|
||||
},
|
||||
{
|
||||
engine: 'semantic_query',
|
||||
available: false,
|
||||
reason: 'MetricFlow has never been executed in 0pt (BI-P029 honest status)',
|
||||
},
|
||||
{ engine: 'superset', available: false, reason: 'no Superset deployment in the estate yet' },
|
||||
{ engine: 'native_charts', available: false, reason: 'render-side; DES1GN owns it (BI-P051..P057)' },
|
||||
{ engine: 'perspective', available: false, reason: 'not deployed' },
|
||||
);
|
||||
const evidence = buildEvidence({ host_note: 'x', engines, runs });
|
||||
for (const e of evidence.not_exercised) expect(e.reason, e.engine).toBeTruthy();
|
||||
expect(evidence.engines).toHaveLength(7);
|
||||
});
|
||||
|
||||
it('reports every comparison as a range with its stability, never a bare figure', () => {
|
||||
if (!duckdbResult) return;
|
||||
for (const dataset of Object.keys(QUERIES) as BenchDatasetId[]) {
|
||||
const a = runs.find((r) => r.engine === 'clickhouse' && r.dataset === dataset)!;
|
||||
const b = runs.find((r) => r.engine === 'duckdb' && r.dataset === dataset)!;
|
||||
const sentence = describeComparison(compareDistributions(a.distribution, b.distribution), dataset);
|
||||
expect(sentence).toMatch(/n=9\/9/);
|
||||
expect(sentence).toMatch(/INDISTINGUISHABLE|x to /);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,348 @@
|
||||
/**
|
||||
* tests/integration/materialization-drift-clickhouse.integration.test.ts
|
||||
* =======================================================================
|
||||
* BI-P105 against a LIVE ClickHouse. Skips honestly when CLICKHOUSE_URL is
|
||||
* unset; with it set and the server down, it FAILS rather than skipping.
|
||||
*
|
||||
* BI-P105's DoD is "performance improvement measured and freshness semantics
|
||||
* documented", and both halves are done here against real data rather than
|
||||
* asserted. What the DoD does not say, and what this file is really about, is
|
||||
* that a pre-aggregate fails SILENTLY: it answers instantly and wrongly.
|
||||
*
|
||||
* The drill drives a real ClickHouse materialized view through the exact
|
||||
* sequence that produces a confident wrong number in production:
|
||||
*
|
||||
* 1. 2M source rows, a SummingMergeTree target and a materialized view.
|
||||
* 2. Backfill, verify with the SHIPPED detector: verdict `match`.
|
||||
* 3. Measure both paths repeatedly, and report the DISTRIBUTION — a single
|
||||
* speed-up figure quoted as a system property is misleading (BI-P104).
|
||||
* 4. `ALTER TABLE ... DELETE` on the source: a GDPR erasure, the single most
|
||||
* common real cause of pre-aggregate drift. A ClickHouse materialized
|
||||
* view is an INSERT trigger; it does not see mutations, so the target
|
||||
* keeps the deleted revenue forever and nothing errors.
|
||||
* 5. The detector must catch it, and `selectQuerySource` must refuse to
|
||||
* serve. The counterfactual — a router that checks only staleness — is
|
||||
* run on the same state and MUST serve, with the wrong number recorded
|
||||
* in kronor.
|
||||
*
|
||||
* Everything measured lands in evidence/bi-p105/materialization-drift.json.
|
||||
*
|
||||
* docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_PASSWORD=test \
|
||||
* clickhouse/clickhouse-server:26.3
|
||||
* CLICKHOUSE_URL=http://127.0.0.1:28123 CLICKHOUSE_AUTH=default:test \
|
||||
* pnpm vitest run tests/integration/materialization-drift-clickhouse.integration.test.ts
|
||||
*
|
||||
* It CREATEs and DROPs its own database. Never point it at the b0ts
|
||||
* ClickHouse or any quarantined market data.
|
||||
*/
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
|
||||
import {
|
||||
applyDriftReport,
|
||||
detectDrift,
|
||||
type AggregateBucket,
|
||||
type DriftReport,
|
||||
} from '@/src/analytics/materialization/drift-detector';
|
||||
import {
|
||||
materializationStaleness,
|
||||
selectQuerySource,
|
||||
type MaterializationSpec,
|
||||
type MaterializationStatus,
|
||||
} from '@/src/analytics/materialization/materialization';
|
||||
import type { AnalyticsQueryRequest } from '@/src/analytics/contract/query-request';
|
||||
|
||||
const CLICKHOUSE_URL = process.env.CLICKHOUSE_URL;
|
||||
const CLICKHOUSE_AUTH = process.env.CLICKHOUSE_AUTH;
|
||||
const run = CLICKHOUSE_URL ? describe : describe.skip;
|
||||
const DB = `opt_p105_${Date.now().toString(36)}`;
|
||||
const ROWS = 2_000_000;
|
||||
const REPETITIONS = 9;
|
||||
|
||||
const headers: Record<string, string> = CLICKHOUSE_AUTH
|
||||
? { Authorization: `Basic ${Buffer.from(CLICKHOUSE_AUTH).toString('base64')}` }
|
||||
: {};
|
||||
|
||||
async function ch(query: string): Promise<string> {
|
||||
const res = await fetch(CLICKHOUSE_URL ?? 'http://skipped.invalid', {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: query,
|
||||
});
|
||||
const text = await res.text();
|
||||
if (!res.ok) throw new Error(`ClickHouse ${res.status}: ${text.slice(0, 400)}`);
|
||||
return text;
|
||||
}
|
||||
|
||||
/** Rows as buckets keyed "date|region", which is what the detector compares. */
|
||||
async function bucketsFrom(query: string): Promise<AggregateBucket[]> {
|
||||
const text = await ch(`${query} FORMAT JSONEachRow`);
|
||||
return text
|
||||
.trim()
|
||||
.split('\n')
|
||||
.filter(Boolean)
|
||||
.map((line) => {
|
||||
const r = JSON.parse(line) as { d: string; region: string; revenue: number; orders: number };
|
||||
return { key: `${r.d}|${r.region}`, values: { revenue: Number(r.revenue), orders: Number(r.orders) } };
|
||||
});
|
||||
}
|
||||
|
||||
/** Median and the spread around it. A single number is not a system property. */
|
||||
function distribution(samples: number[]): {
|
||||
n: number;
|
||||
min: number;
|
||||
p50: number;
|
||||
p95: number;
|
||||
max: number;
|
||||
spread_ratio: number;
|
||||
} {
|
||||
const s = [...samples].sort((a, b) => a - b);
|
||||
const at = (q: number) => s[Math.min(s.length - 1, Math.floor(q * s.length))];
|
||||
return {
|
||||
n: s.length,
|
||||
min: s[0],
|
||||
p50: at(0.5),
|
||||
p95: at(0.95),
|
||||
max: s[s.length - 1],
|
||||
spread_ratio: Number((s[s.length - 1] / Math.max(s[0], 1e-9)).toFixed(2)),
|
||||
};
|
||||
}
|
||||
|
||||
async function timeQuery(query: string): Promise<number> {
|
||||
const t0 = performance.now();
|
||||
await ch(`${query} FORMAT JSONEachRow`);
|
||||
return performance.now() - t0;
|
||||
}
|
||||
|
||||
const SOURCE_AGG = `
|
||||
SELECT toString(order_date) AS d, region, sum(amount) AS revenue, count() AS orders
|
||||
FROM ${DB}.sales WHERE tenant_id = 't_alpha' GROUP BY d, region ORDER BY d, region`;
|
||||
const TARGET_AGG = `
|
||||
SELECT toString(order_date) AS d, region, sum(revenue) AS revenue, sum(orders) AS orders
|
||||
FROM ${DB}.sales_daily WHERE tenant_id = 't_alpha' GROUP BY d, region ORDER BY d, region`;
|
||||
|
||||
const SPEC: MaterializationSpec = {
|
||||
materialization_id: 'mv_sales_daily',
|
||||
tenant_id: 't_alpha',
|
||||
project_id: 'p_main',
|
||||
dataset: 'ds_sales',
|
||||
strategy: 'clickhouse_materialized_view',
|
||||
target_ref: `${DB}.sales_daily`,
|
||||
metrics: ['revenue', 'orders'],
|
||||
dimensions: ['order_date', 'region'],
|
||||
time_dimension: 'order_date',
|
||||
grain: 'day',
|
||||
max_staleness_seconds: 3600,
|
||||
drift_check_interval_seconds: 86_400,
|
||||
tolerance: { absolute: 0.5, relative: 0.0001 },
|
||||
};
|
||||
|
||||
const REQUEST: AnalyticsQueryRequest = {
|
||||
dataset: 'ds_sales',
|
||||
metrics: ['revenue'],
|
||||
dimensions: ['order_date'],
|
||||
filters: [],
|
||||
timeRange: { grain: 'day' },
|
||||
};
|
||||
|
||||
const evidence: Record<string, unknown> = {};
|
||||
let status: MaterializationStatus;
|
||||
|
||||
run('BI-P105 materialization against a LIVE ClickHouse', () => {
|
||||
beforeAll(async () => {
|
||||
evidence.clickhouse_version = (await ch('SELECT version()')).trim();
|
||||
evidence.recorded_at = new Date().toISOString();
|
||||
evidence.source_rows = ROWS;
|
||||
|
||||
await ch(`CREATE DATABASE IF NOT EXISTS ${DB}`);
|
||||
await ch(`
|
||||
CREATE TABLE ${DB}.sales (
|
||||
tenant_id LowCardinality(String),
|
||||
order_date Date,
|
||||
region LowCardinality(String),
|
||||
customer_id UInt32,
|
||||
amount Decimal(18, 2)
|
||||
) ENGINE = MergeTree ORDER BY (tenant_id, order_date, region)`);
|
||||
await ch(`
|
||||
INSERT INTO ${DB}.sales
|
||||
SELECT
|
||||
if(number % 5 = 0, 't_beta', 't_alpha') AS tenant_id,
|
||||
toDate('2026-08-01') + toIntervalDay(number % 30) AS order_date,
|
||||
['SE','NO','DK','FI'][(number % 4) + 1] AS region,
|
||||
toUInt32(number % 5000) AS customer_id,
|
||||
toDecimal64((number % 977) + 13, 2) AS amount
|
||||
FROM numbers(${ROWS})`);
|
||||
|
||||
await ch(`
|
||||
CREATE TABLE ${DB}.sales_daily (
|
||||
tenant_id LowCardinality(String),
|
||||
order_date Date,
|
||||
region LowCardinality(String),
|
||||
revenue Decimal(38, 2),
|
||||
orders UInt64
|
||||
) ENGINE = SummingMergeTree ORDER BY (tenant_id, order_date, region)`);
|
||||
await ch(`
|
||||
CREATE MATERIALIZED VIEW ${DB}.sales_daily_mv TO ${DB}.sales_daily AS
|
||||
SELECT tenant_id, order_date, region, sum(amount) AS revenue, count() AS orders
|
||||
FROM ${DB}.sales GROUP BY tenant_id, order_date, region`);
|
||||
// The MV is an INSERT trigger: it saw nothing that was already there, so
|
||||
// the backfill is explicit. (This is drift cause #1, and it is why the
|
||||
// first check below has to be able to fail.)
|
||||
await ch(`
|
||||
INSERT INTO ${DB}.sales_daily
|
||||
SELECT tenant_id, order_date, region, sum(amount) AS revenue, count() AS orders
|
||||
FROM ${DB}.sales GROUP BY tenant_id, order_date, region`);
|
||||
await ch(`OPTIMIZE TABLE ${DB}.sales_daily FINAL`);
|
||||
|
||||
status = {
|
||||
materialization_id: SPEC.materialization_id,
|
||||
state: 'serving',
|
||||
data_watermark: new Date().toISOString(),
|
||||
last_refresh_at: new Date().toISOString(),
|
||||
};
|
||||
}, 300_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (!CLICKHOUSE_URL) return;
|
||||
try {
|
||||
await ch(`DROP DATABASE IF EXISTS ${DB}`);
|
||||
} finally {
|
||||
const dir = path.join(process.cwd(), 'evidence/bi-p105');
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'materialization-drift.json'),
|
||||
`${JSON.stringify(evidence, null, 2)}\n`,
|
||||
);
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it('the freshly backfilled aggregate MATCHES its source', async () => {
|
||||
const report = detectDrift({
|
||||
fromSource: await bucketsFrom(SOURCE_AGG),
|
||||
fromAggregate: await bucketsFrom(TARGET_AGG),
|
||||
tolerance: SPEC.tolerance,
|
||||
metrics: ['revenue', 'orders'],
|
||||
});
|
||||
expect(report.buckets_compared).toBeGreaterThan(0); // an empty compare proves nothing
|
||||
expect(report.verdict).toBe('match');
|
||||
|
||||
status = applyDriftReport(status, report);
|
||||
evidence.after_backfill = {
|
||||
verdict: report.verdict,
|
||||
buckets_compared: report.buckets_compared,
|
||||
revenue_total: report.totals.revenue,
|
||||
};
|
||||
}, 120_000);
|
||||
|
||||
it('measures the improvement as a DISTRIBUTION, not a single number', async () => {
|
||||
const base: number[] = [];
|
||||
const agg: number[] = [];
|
||||
for (let i = 0; i < REPETITIONS; i += 1) {
|
||||
// Interleaved so a machine that gets busy halfway through penalises both.
|
||||
base.push(await timeQuery(SOURCE_AGG));
|
||||
agg.push(await timeQuery(TARGET_AGG));
|
||||
}
|
||||
const baseD = distribution(base);
|
||||
const aggD = distribution(agg);
|
||||
evidence.performance = {
|
||||
note:
|
||||
'ClickHouse caches marks and pages between repetitions, so these are WARM numbers on one host ' +
|
||||
'with nothing else running. They describe this shape of query on this data, not a system property.',
|
||||
repetitions: REPETITIONS,
|
||||
base_table_ms: baseD,
|
||||
pre_aggregate_ms: aggD,
|
||||
speedup_p50: Number((baseD.p50 / Math.max(aggD.p50, 1e-9)).toFixed(1)),
|
||||
speedup_range_min_max: [
|
||||
Number((baseD.min / Math.max(aggD.max, 1e-9)).toFixed(1)),
|
||||
Number((baseD.max / Math.max(aggD.min, 1e-9)).toFixed(1)),
|
||||
],
|
||||
};
|
||||
// The claim under test is only that the pre-aggregate is not SLOWER; the
|
||||
// magnitude is reported, never asserted as a threshold.
|
||||
expect(aggD.p50).toBeLessThanOrEqual(baseD.p50);
|
||||
}, 300_000);
|
||||
|
||||
it('a GDPR erasure on the source DRIFTS the aggregate, and the drift is caught', async () => {
|
||||
const before = await bucketsFrom(TARGET_AGG);
|
||||
|
||||
// The mutation a right-to-erasure request produces. A materialized view is
|
||||
// an insert trigger — it never sees this.
|
||||
await ch(
|
||||
`ALTER TABLE ${DB}.sales DELETE WHERE tenant_id = 't_alpha' AND customer_id < 250 SETTINGS mutations_sync = 2`,
|
||||
);
|
||||
|
||||
const fromSource = await bucketsFrom(SOURCE_AGG);
|
||||
const fromAggregate = await bucketsFrom(TARGET_AGG);
|
||||
expect(fromAggregate).toEqual(before); // the aggregate genuinely did not move
|
||||
|
||||
const report: DriftReport = detectDrift({
|
||||
fromSource,
|
||||
fromAggregate,
|
||||
tolerance: SPEC.tolerance,
|
||||
metrics: ['revenue', 'orders'],
|
||||
});
|
||||
expect(report.verdict).toBe('drifted');
|
||||
expect(report.worst).toBeDefined();
|
||||
|
||||
const drifted = applyDriftReport(status, report);
|
||||
const guarded = selectQuerySource({ request: REQUEST, spec: SPEC, status: drifted, now: new Date() });
|
||||
expect(guarded.source).toBe('base_table');
|
||||
|
||||
// COUNTERFACTUAL: a router that checks only staleness serves it, and this
|
||||
// is the number a dashboard would have shown.
|
||||
const stalenessOnlyServes =
|
||||
(materializationStaleness(drifted, new Date()) ?? 0) <= SPEC.max_staleness_seconds;
|
||||
expect(stalenessOnlyServes).toBe(true);
|
||||
|
||||
evidence.gdpr_erasure_drill = {
|
||||
mutation: "ALTER TABLE sales DELETE WHERE tenant_id='t_alpha' AND customer_id < 250",
|
||||
verdict: report.verdict,
|
||||
buckets_compared: report.buckets_compared,
|
||||
revenue_true_after_erasure: report.totals.revenue.source,
|
||||
revenue_pre_aggregate_would_report: report.totals.revenue.aggregate,
|
||||
revenue_overstatement: report.totals.revenue.delta,
|
||||
revenue_overstatement_pct: Number(
|
||||
((report.totals.revenue.delta / report.totals.revenue.source) * 100).toFixed(3),
|
||||
),
|
||||
worst_bucket: report.worst,
|
||||
guarded_router_decision: guarded,
|
||||
staleness_only_router_would_serve: stalenessOnlyServes,
|
||||
staleness_seconds_at_decision: materializationStaleness(drifted, new Date()),
|
||||
note:
|
||||
'The aggregate was NOT stale by any clock: it had just been refreshed, and every freshness check ' +
|
||||
'passed. Only the source comparison caught it. This is why staleness alone is not a correctness ' +
|
||||
'guarantee.',
|
||||
};
|
||||
status = drifted;
|
||||
}, 300_000);
|
||||
|
||||
it('and a refresh that re-derives the aggregate clears the drift', async () => {
|
||||
await ch(`TRUNCATE TABLE ${DB}.sales_daily`);
|
||||
await ch(`
|
||||
INSERT INTO ${DB}.sales_daily
|
||||
SELECT tenant_id, order_date, region, sum(amount) AS revenue, count() AS orders
|
||||
FROM ${DB}.sales GROUP BY tenant_id, order_date, region`);
|
||||
await ch(`OPTIMIZE TABLE ${DB}.sales_daily FINAL`);
|
||||
|
||||
const report = detectDrift({
|
||||
fromSource: await bucketsFrom(SOURCE_AGG),
|
||||
fromAggregate: await bucketsFrom(TARGET_AGG),
|
||||
tolerance: SPEC.tolerance,
|
||||
metrics: ['revenue', 'orders'],
|
||||
});
|
||||
expect(report.verdict).toBe('match');
|
||||
|
||||
const healed: MaterializationStatus = applyDriftReport(
|
||||
{ ...status, state: 'serving', data_watermark: new Date().toISOString() },
|
||||
report,
|
||||
);
|
||||
const decision = selectQuerySource({ request: REQUEST, spec: SPEC, status: healed, now: new Date() });
|
||||
expect(decision.source).toBe('materialization');
|
||||
|
||||
evidence.after_rebuild = {
|
||||
verdict: report.verdict,
|
||||
revenue_total: report.totals.revenue,
|
||||
routing: decision,
|
||||
};
|
||||
}, 300_000);
|
||||
});
|
||||
Reference in New Issue
Block a user