Files
0pt/docs/P0_DATASET_QUALITY_FLOW.md
T
adminandClaude Sonnet 5 2902a26bac docs: full accuracy overhaul, grounded claim-by-claim in real code (#4)
* docs: full accuracy overhaul, grounded claim-by-claim in real code

Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was
audited and rewritten by a multi-pass process: independent rewrite per
cluster, adversarial re-verification of that rewrite against the actual
code, then a final cross-doc consistency pass — each stage re-checking
claims against the repo directly rather than trusting the previous stage.

Two corrections matter most:

1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and
   docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts,
   quality-report-service.ts, src/data-platform/services/catalog-service.ts,
   and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code.
   git log --all across every branch shows zero commits for any of these
   paths, ever. 0pt's data-quality/QA enforcement does not exist in code.
   Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added
   as an explicit target-spec (not-yet-built) replacement.

2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are
   two diverged remotes sharing part of the same history (services/api/
   workers/adaptive_router.py is byte-identical between them). Gitea's main
   independently accumulated a WAF-bypass/proxy-rotation scraper stack in
   three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main
   never had it. Which remote is canonical/deployed is unresolved — flagged
   consistently across the corpus, not decided here.

Also newly and consistently documented: the tenant_id/project_id-read-from-
client-body gap on the P0 lakehouse/rate-policy/source-query routes (no
auth/session system exists anywhere in this repo yet); that every CL0UD/
N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env
reads for any of their *_URL vars); that two P0 mutation routes
(lakehouse/mappings, sources/rate-policies) always fail over HTTP today
because their prerequisite entity can never be created via any route; a
schema bug in sql/004 (FK references to opt_datasets/opt_sources, which
don't exist); and that 0pt.app's live deployment was independently
confirmed (curled this pass) to be running this exact codebase with
DATABASE_URL/REDIS_URL unset in production.

New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript
capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously
undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files) — docs-only change, no source touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes

Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs
land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the
lakehouse mapping routes and added proper status-code mapping to the
document extraction route. This doc was written against the pre-fix
state in the same pass that added it, so it needs the same correction.

Does not claim the lakehouse write path fully works now -- the deeper
dataset-registration gap (no route ever calls DatasetService.registerDataset)
is unchanged and still blocks it end-to-end; only the shared-instance half
of the bug is fixed.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 18:49:01 +02:00

181 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 0pt Dataset Quality / QA Flow — TARGET SPEC ONLY, NOT YET BUILT
> **This document describes what does not exist yet.** Every file, service,
> type, and schema named below is a proposal, not a fact. Read this doc with
> "should"/"would need to"/"is intended to" — never "does" or "is." If any
> other doc in this repo states or implies that dataset-level quality
> enforcement is already built, real, and tested, that statement is wrong;
> see "Why this doc exists" below.
## Why this doc exists
Several docs in this repo — `docs/P0_QUERY_SOURCE_FLOW.md` (before this
pass's correction), `docs/ICEBERG_WRITE_BOUNDARY.md`, and
`docs/11-lakehouse-write-plan.md` — reference the following as if they are
real, already-built, already-tested files in this repo:
- `src/services/quality-check-service.ts`
- `src/services/quality-report-service.ts`
- `src/types/quality-check.ts`
- `src/types/quality-report.ts`
- `src/data-platform/services/catalog-service.ts`
- `sql/002_p0_dataset_quality.sql`
- `sql/003_*.sql` (unspecified third migration)
**Verified this pass:** `git log --all` for every one of those paths returns
**zero commits, ever, on any branch** (`main`, `feature/0pt-env-example`,
`ci/add-quality-gate` all checked). A full-repo `find` for each filename
also returns nothing. These files have never existed in this repository's
history. `docs/ICEBERG_WRITE_BOUNDARY.md` even states outright that "0pt's
QA stage... remains entirely 0pt-owned" and cites `quality-check-service.ts`
as the thing that wasn't touched by that pass — implying it exists to not
touch. It doesn't exist to begin with.
This means: **0pt's dataset-level data-quality/QA enforcement stage does
not exist in code at all**, despite being described elsewhere as complete.
This is a real, currently-open gap — not a doc typo. This document replaces
those false claims with an honest target specification: what this stage
*should* look like if/when it gets built, grounded in the parts of this
repo's design that *are* real today.
`docs/P0_QUERY_SOURCE_FLOW.md`'s "Known gaps" section has the fuller list of
exactly which files make the false claim and where; this doc does not
repeat that inventory, only the target design.
## What already exists that this would build on (real, verified)
- **The provenance/confidence model** — `docs/06-data-engine.md`'s pipeline
stage 5 ("QA") already specifies the *rules* this stage should enforce,
even though no code implements them yet:
- Per-field `bridge_provenance`: `source_id`, `freshness`, `confidence`
(0–1), `level` (0–3), `agreed_sources`.
- Level semantics: 0 = unavailable/licensed, 1 = single source, 2 =
partial corroboration, 3 = ≥2 independent sources agree.
- The QA rule: **drop rows below the order's `confidence_target`** —
deliver fewer rows, never fabricate a value to hit a target.
- `est_confidence` (order-level target) vs. `avg_confidence` (delivered,
actual) as the two numbers this stage would need to reconcile.
- **The Dataset registry** — `src/services/dataset-service.ts` /
`src/types/dataset.ts` are real, in-memory, tested code. `Dataset` already
has `dataset_type: "raw" | "structured" | "enriched" | "curated"` and
`status: "active" | "archived" | "deprecated"` — a quality stage would
need to decide whether a failed QA pass changes `status`, or whether
quality state lives entirely in a separate table/type instead (see
"Open questions" below).
- **The N0D job pattern** — `src/services/iceberg-write-service.ts` is real,
tested code showing the actual "sync path vs. N0D job" split this
codebase already uses elsewhere: an `input.heavy` flag routes to
`this.n0d.createJob(...)` instead of running inline. A quality-check
service would plausibly reuse this same shape (small samples checked
inline; a real run against a full dataset dispatched as an N0D job) —
but nothing today wires that up for quality checks specifically.
- **The CL0UD-policy-before-execution pattern** — `src/services/external-writeback-service.ts`
(`0pt.write_external_record`, real, tested) calls
`Cl0udClient.evaluatePolicy()` before performing its write. If a quality
service is meant to policy-gate before running (as
`docs/ICEBERG_WRITE_BOUNDARY.md` implies with its "CL0UD is always
re-consulted" framing), this is the real precedent to follow, not
something already done by the nonexistent `quality-check-service.ts`.
- **The AUD0 event pattern** — every real service in this repo
(`source-service.ts`, `dataset-service.ts`, `source-query-service.ts`,
`external-writeback-service.ts`, `iceberg-write-service.ts`) emits AUD0
events via `createAud0Client()` against an in-memory mock (no live AUD0
endpoint reachable from this repo). A quality service would follow the
same convention.
## What this stage should do (target design — none of this is built)
1. **Trigger.** A quality check would need to run after the enrichment/
fusion stage (`docs/06-data-engine.md` stage 4) and before a dataset is
marked ready for delivery or written to Iceberg via
`IcebergWriteService`. Whether this is a synchronous call, an
automatically-triggered N0D job, or an explicit caller action is
undecided.
2. **Checks to run**, per `docs/06-data-engine.md`'s stage 5 language
(originally scoped as "Great Expectations or equivalent, run inside the
worker"):
- **Validity** — field format/range checks against the schema for that
`dataset_type`.
- **Completeness** — required-field coverage.
- **Freshness** — age of each `bridge_provenance.freshness` against a
staleness threshold.
- **Uniqueness** — post-dedup uniqueness (RapidFuzz dedup already
happens upstream per `docs/06-data-engine.md` stage 3; this would be a
verification pass, not the dedup itself).
- **Corroboration** — cross-check `bridge_provenance.level`/`agreed_sources`
against the order's declared `confidence_target`; rows below target
get dropped, never backfilled with a guess.
3. **Output: a quality report.** Would need a `QualityReport` shape
(proposed home: `src/types/quality-report.ts`, not yet created) capturing
at minimum: `dataset_id`, `check_run_id`, per-check pass/fail counts,
`rows_dropped_below_confidence_target`, `avg_confidence` (delivered,
actual — per `docs/06-data-engine.md`), and a timestamp. This is the
thing `docs/06-data-engine.md` calls "the quality report stored on
`deliveries`" — that storage location (a `deliveries` table) does not
exist in this repo's `sql/` directory either; `sql/001_init.sql` would
need to be checked/extended for it, or a new migration written.
4. **Gate, not silent pass-through.** Per CLAUDE.md's non-negotiable
constraint #1 ("we never invent values") and #3 ("provenance per field"),
a dataset that fails its quality gate should not silently ship — the
order's delivery flow (`docs/01-product.md`'s order lifecycle) would need
to surface a failed/partial QA result to the customer, not swallow it.
## What would need to be created (none of this exists)
| Proposed file | Would contain |
|---|---|
| `src/types/quality-check.ts` | `QualityCheckRequest`/`QualityCheckResult` shapes: which checks to run, thresholds, dataset/source refs |
| `src/types/quality-report.ts` | `QualityReport` shape described above |
| `src/services/quality-check-service.ts` | Orchestrates the checks (validity/completeness/freshness/uniqueness/corroboration), CL0UD policy check, N0D job dispatch for heavy runs (same `input.heavy` shape as `iceberg-write-service.ts`), AUD0 emission |
| `src/services/quality-report-service.ts` | Persists/retrieves `QualityReport`s; registry pattern matching `DatasetService`/`SourceService` (in-memory today, same known "not shared across route modules" limitation until real persistence lands) |
| A new SQL migration | Schema for quality-check runs and reports. **Numbering note:** this repo's `sql/` directory today contains only `001_init.sql` and `004_iceberg_write_boundary.sql` — `002` and `003` were referenced in other docs but never created, so there is a numbering gap. Whether a new migration should fill `002`/`003` or continue from `005` is an open question — do not assume `002_p0_dataset_quality.sql` is the correct filename just because other docs guessed it. |
`src/data-platform/services/catalog-service.ts` is referenced elsewhere in
this repo's docs (as the source of an "`L4` = restricted sensitivity"
precedent) but was also confirmed this pass to not exist anywhere in this
repo — it is a separate fabrication from the quality-check one, not
something this quality stage specifically depends on. If a dataset-level
catalog/listing capability is needed as part of this work, the registered
`0pt.catalog_list` and `0pt.source_registry_list` capabilities
(`cl0ud`'s `packages/capability-registry/src/registry.ts`, both real
registry entries, `surfaces: ['api']`) are the closest existing hooks — but
whether either is meant to serve this purpose, or a new capability is
needed, is undecided.
## No capability registered for this yet
`cl0ud`'s `packages/capability-registry/src/registry.ts` was checked this
pass for anything quality-related: no `0pt.quality_check`,
`0pt.run_quality_checks`, or similarly-named capability exists. The
registry does have a `qualityGateRequired` boolean flag used by *other*
capabilities (e.g. one INF0 capability sets it `true`), but nothing wires
`0pt`'s own dataset QA into that flag or into any capability of its own.
Building this stage would need a new capability entry, decided surfaces
(`api` at minimum, per the six-surface model; `job` likely for a
full-dataset run), a `riskLevel`, and a `piiLevel`.
## Open questions for Simon
- Does quality-check execution belong entirely in 0pt (as
`docs/ICEBERG_WRITE_BOUNDARY.md` asserts should be true, even though the
code backing that assertion doesn't exist), or should the actual
check-running compute move to N0D as a job type, with 0pt only owning the
*rules*?
- Where does a `QualityReport` get persisted — a new `sql/00X_*.sql` table
in 0pt's own Postgres, or through ST0RE (matching the "ST0RE owns
metadata, 0pt owns production" boundary `docs/ICEBERG_WRITE_BOUNDARY.md`
draws for Iceberg tables)?
- Does a dataset that fails QA change `Dataset.status`
(`src/types/dataset.ts`), or does quality state live entirely outside the
`Dataset` record?
- What migration number should the new schema use, given the `002`/`003`
gap described above?
- Should this become its own registered `0pt.*` capability, and if so what
`riskLevel`/`piiLevel` — quality checks read dataset content, which per
the reasoning already used for `0pt.query_source` (`piiLevel: 'high'`
because "can return any column, including PII") would plausibly need the
same `piiLevel`.
None of the above is decided by this document. It exists to give a future
build pass an honest starting point instead of a fabricated one.