Files
0pt/docs/10-lakehouse-decision.md
T
adminandClaude Sonnet 5 2902a26bac docs: full accuracy overhaul, grounded claim-by-claim in real code (#4)
* docs: full accuracy overhaul, grounded claim-by-claim in real code

Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was
audited and rewritten by a multi-pass process: independent rewrite per
cluster, adversarial re-verification of that rewrite against the actual
code, then a final cross-doc consistency pass — each stage re-checking
claims against the repo directly rather than trusting the previous stage.

Two corrections matter most:

1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and
   docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts,
   quality-report-service.ts, src/data-platform/services/catalog-service.ts,
   and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code.
   git log --all across every branch shows zero commits for any of these
   paths, ever. 0pt's data-quality/QA enforcement does not exist in code.
   Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added
   as an explicit target-spec (not-yet-built) replacement.

2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are
   two diverged remotes sharing part of the same history (services/api/
   workers/adaptive_router.py is byte-identical between them). Gitea's main
   independently accumulated a WAF-bypass/proxy-rotation scraper stack in
   three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main
   never had it. Which remote is canonical/deployed is unresolved — flagged
   consistently across the corpus, not decided here.

Also newly and consistently documented: the tenant_id/project_id-read-from-
client-body gap on the P0 lakehouse/rate-policy/source-query routes (no
auth/session system exists anywhere in this repo yet); that every CL0UD/
N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env
reads for any of their *_URL vars); that two P0 mutation routes
(lakehouse/mappings, sources/rate-policies) always fail over HTTP today
because their prerequisite entity can never be created via any route; a
schema bug in sql/004 (FK references to opt_datasets/opt_sources, which
don't exist); and that 0pt.app's live deployment was independently
confirmed (curled this pass) to be running this exact codebase with
DATABASE_URL/REDIS_URL unset in production.

New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript
capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously
undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files) — docs-only change, no source touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes

Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs
land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the
lakehouse mapping routes and added proper status-code mapping to the
document extraction route. This doc was written against the pre-fix
state in the same pass that added it, so it needs the same correction.

Does not claim the lakehouse write path fully works now -- the deeper
dataset-registration gap (no route ever calls DatasetService.registerDataset)
is unchanged and still blocks it end-to-end; only the shared-instance half
of the bug is fixed.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 18:49:01 +02:00

105 lines
7.0 KiB
Markdown

# 10 · Lakehouse decision (pointer)
This doc is a **pointer**, not a new architecture. It records an ecosystem-wide decision made
outside this repo and states what it means for 0pt specifically. It does not change anything in
`docs/02-architecture.md` or `docs/03-data-model.md` today — Postgres remains the backbone for
0pt's platform tables and current data entities (`docs/03`). Read this alongside those docs, not
instead of them.
> **Status update.** Since this pointer doc was first written, a follow-up implementation pass
> turned most of 0pt's side of this plan into real TypeScript — see
> `docs/11-lakehouse-write-plan.md`'s "what the implementation pass closed" section and
> `docs/ICEBERG_WRITE_BOUNDARY.md` for the current, corrected status. Two things to know before
> reading the rest of this doc: (1) that write-path code is **real and unit-tested against a mock
> ST0RE client, not verified against a running ST0RE deployment**; (2) an earlier version of
> `docs/11` and `docs/ICEBERG_WRITE_BOUNDARY.md` also claimed 0pt's data-quality/QA stage had a
> dedicated, tested `quality-check-service.ts`/`quality-report-service.ts` implementation — that
> claim was **fabricated**. `git log --all` for those paths (and `sql/002_p0_dataset_quality.sql`,
> `sql/003_*.sql`) returns zero commits, on any branch, ever. Both docs have been corrected; see
> them for the full account. This doc's own references to "QA" below point only to the *spec* in
> `docs/06-data-engine.md`, not to any implemented service.
## The decision (made elsewhere — do not re-litigate here)
**Apache Iceberg is the canonical SIAX lakehouse table format**, per **ADR-0030**, authored in the
`n0d` (Compute OS) repo (a dated copy also lives in `cl0ud`'s `docs/architecture/`, since the
boundary audit that motivated it originated in n0d but the ADR itself is recorded as SIAX-wide;
both copies describe the same accepted decision — verified by reading both this pass). Delta Lake
is explicitly rejected as the canonical P0/P1 format across SIAX Technology; it may only ever be
used later as an import/export bridge, never as the format 0pt writes natively.
Rationale (from the ecosystem ADR, summarized): SIAX needs one vendor-neutral, multi-engine,
catalog-first open table format for the MinIO/ST0RE lakehouse, because several products need to
share the same tables — ST0RE owns catalog/metadata, 0pt owns dataset writes, N0D owns
query/compaction execution, AUD0 owns lineage, API0 owns docs — and because future reads must
work from many engines (Trino/Spark/Flink/DuckDB/etc.), not one.
## Cross-product ownership split (context, not 0pt's to change)
| Product | Owns |
|---|---|
| ST0RE | Table/catalog/artifact metadata (the Iceberg catalog + table registry) |
| **0pt** | **Dataset production and writes into Iceberg tables**, through ST0RE's table registry |
| N0D | Query/compaction/maintenance execution only |
| AUD0 | Lineage |
| CL0UD | Policy |
This table condenses ADR-0030 §2.1/§2.4's ownership split for 0pt's purposes — it is a paraphrase,
not a verbatim quote (corrected this verification pass; re-checked against `cl0ud`'s copy of the
ADR). The ADR's §2.4 table also names **INF0** (model registry/experiment tracking, MLflow as an
adapter) and its §2.1 also names **API0** (schema/docs exposure); both rows are omitted here
because neither changes anything for 0pt specifically. One paraphrase is deliberate, not
accidental: the ADR's own 0pt row reads "Dataset production, **quality**, data contracts, writes"
— the word "quality" is intentionally dropped from this table's 0pt row above, because 0pt's
quality/QA stage is not yet built (see the status callout at the top of this doc). See ADR-0030
§2.4 directly (`cl0ud`'s or `n0d`'s copy) for its own exact wording of every row.
## What this means for 0pt
0pt is the **write-path owner** for the datasets it produces: 0pt runs sourcing → extraction →
structuring → enrichment → QA (`docs/06-data-engine.md` — a design spec, not yet a built service;
see the status callout above) as it does today, and writes the resulting datasets as **Iceberg
tables through ST0RE's table registry**. 0pt does **not** talk to N0D's catalog or any lakehouse
storage directly; it goes through ST0RE.
ST0RE's table registry is real, committed code (`sax3l/st0re`, `/v1/st0re/lakehouse/*`), and 0pt
has a real client and services that target it — `src/clients/st0re-lakehouse-client.ts`,
`src/services/iceberg-write-service.ts`, `src/services/dataset-iceberg-mapping-service.ts`. None of
this has been exercised against a *running* ST0RE instance from this repo: every test to date runs
against that client's deterministic in-memory mock (no `ST0RE_URL` was reachable from this
environment). Treat "0pt writes Iceberg tables through ST0RE" as **real code, mock-tested, not
live-verified** — see `docs/ICEBERG_WRITE_BOUNDARY.md`'s "Verification status" section for the full
detail, including what is and isn't wired end-to-end.
This does not replace Postgres as the platform backbone (orders, users, jobs, billing — `docs/03`
— stay exactly as they are). It is additive: dataset **output** (the delivered data product, today
exported as CSV/Parquet/JSON per `docs/04-api.md`) gains an additional, canonical Iceberg
write-path. Concrete to-dos, current build status, and required schema/event names are in
`docs/11-lakehouse-write-plan.md`.
## Also relevant: the `sourceRatePolicy` deprecation in N0D
The same ecosystem audit that produced ADR-0030 found N0D's `ComputePolicy` type carrying a
`sourceRatePolicy` field (per-source-domain crawl rate limits, e.g.
`{'allabolag.se': 1, 'ratsit.se': 1}`) hardcoded into N0D's policy defaults. That is 0pt's own
crawl-domain knowledge — it has been deprecated in N0D (the field is still present on the type for
backward compatibility, but N0D's own boundary-repair tests assert it now has **zero effect** on
any allow/deny decision; verified by reading `n0d`'s `src/policy.ts` and
`src/__tests__/boundary-repair.test.ts` this pass). 0pt does not rely on N0D for source-rate/
crawl-compliance policy; 0pt already owns the sourcing chain and its rate limits today
(`sources.yaml`, `docs/06-data-engine.md` §`rate_limit`), and has since formalized its own model —
`src/services/source-rate-policy-service.ts`, real and tested, see `docs/11-lakehouse-write-plan.md`
and `docs/ICEBERG_WRITE_BOUNDARY.md`.
## Non-goals of this doc
- Does not itself implement Iceberg writes — see `docs/11-lakehouse-write-plan.md` /
`docs/ICEBERG_WRITE_BOUNDARY.md` for what a later pass actually built (real, mock-tested code;
not live-verified; data-quality/QA enforcement specifically is **not built** — see the status
callout above).
- Does not modify `docs/03-data-model.md`'s Postgres schema.
- Does not require ST0RE's table registry to exist — as of this pass it does exist as real,
committed code in `sax3l/st0re`, though this repo has not confirmed it live/reachable. This doc
remains the pointer; `docs/11` lists what's still blocked on a *live, reachable* ST0RE vs. what
0pt has already built against the mock.