BI-P029 + BI-P103: two prompts cited as done were not — writing the cited test found a metric-definition filter dropped at execution #38

Merged

admin merged 2 commits from feat/bi-p029-semantic-parity into main 2026-09-07 06:32:14 +00:00

Conversation 0 Commits 2 Files Changed 9

+1178 -15

2 Commits

Author	SHA1	Message	Date
Claude Opus 5	7441de5429	BI-P103: the cited evidence/bi-p103 did not exist — measure it ... Same shape as the BI-P029 finding in the previous commit. `bi-slo.ts` said "evidence/bi-p103 records the first baselines". There was no evidence/bi-p103. The catalogue and the evaluator were both real and both well covered — against SYNTHETIC events. A synthetic event carries whatever duration the test author typed, so that suite proves the arithmetic and nothing about the system: it cannot tell you whether a dashboard query takes 8 ms or 8 s. BI-P103's DoD is one sentence, "measured baselines exist", and it was the part not done. tests/integration/bi-p103-slo-baseline.integration.test.ts runs 460 real queries against a live ClickHouse over a 400 000-row fact table -- 120 cold dashboard queries (each a different bucket so the BI-P020 cache genuinely misses), the same 120 again through the same shipped cache now hitting, and 220 exploration queries, that last number chosen so explore.p99's min_samples of 200 is MET rather than reported low-confidence. Every duration is wall-clock around the real call, and the telemetry goes through the shipped producer and the shipped evaluator, so the evidence is written by the code under test. Measured: 7 of 11 objectives produced a number, `fully_measured` false, measured_fraction 0.64. The four that did not each say why -- dashboard freshness is a property of the dataset, the exploration timeout is a configured ceiling, report completion has no producer in 0pt at all, and report delivery is N0TIFY's receipt. A baseline table listing only the rows that worked is how an estate comes to believe it measures more than it does. The evidence also records what is NARROWER than production, because these numbers will be quoted: the cached path is the in-process reference cache and not Valkey, every latency is the engine round trip with no HTTP or authorization layer, and there is no concurrency. The suite is adversarial about its own inputs: a "cold" sample that came from the cache fails, a cache "hit" that missed fails, a cached p95 that is not faster than the cold p95 fails, and dropping the exploration sample to 60 fails on explore.p99 being low-confidence rather than quietly recording a p99 over 60 requests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 08:27:37 +02:00
Claude Opus 5	468a87c00c	BI-P029: the cited parity test did not exist, and it caught a real bug ... `semantic-provider.ts` cited tests/integration/semantic-parity.integration.test.ts as the proof of the one BI-P029 DoD surface it claimed was closed, and `providerReadiness().proven` asserted the comparison had been made against live ClickHouse. The file did not exist. The citation was the evidence. Writing it exposed a divergence that had been shipping: `compileRequestToClickHouse` had no notion of a metric's OWN filters (BI-P028 `MetricSpec.filters`). `ResolvedMetric` did not carry them and the compiler never asked, so a metric defined as "invoiced revenue EXCLUDING internal transfers" executed as "invoiced revenue". On the committed fixture that is 11 249 instead of 1 250 for one country -- a plausible-looking number, 9x too large, under the correct canonical metric id. The registry, the governance checks and the MetricFlow manifest all carried the filter; only the thing that actually ran did not. * `ResolvedMetric` gains `definition_filters`, already resolved onto the physical dim_names key so the compilers stay dumb. * The ClickHouse compiler folds them into THAT metric's conditional aggregate, never into the shared WHERE -- two metrics in one request may exclude different rows, and a shared predicate would apply one metric's meaning to the other. `renderFilter`'s uniquifier becomes a string so request filter `0` and metric filter `0_0m` cannot collide in the parameter map. * `compileRequestToSemantic` carries them through, and `MetricFlowProvider.compile` emits them per metric. The test runs both compile paths against one live ClickHouse: 0pt's `sumIf` compiler, and a renderer driven only by the provider's structured query that shares no SQL-generation code and emits `sum(if(...))` with inlined literals. It asserts the two SQL strings differ, so parity is not vacuous. Backing the fix out reproduces the pre-fix numbers, and the suite records them itself rather than quoting a console: evidence/bi-p029/semantic-parity.json. Three of BI-P029's four DoD surfaces stay OPEN and still say so: MetricFlow itself is not executed (no Python runtime is provisioned here), Superset is not implemented, and the AI Analyst path has not been compared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 08:23:43 +02:00