From 4bc79c25d3c8a6c82803b66ed8b434ecc5709bee Mon Sep 17 00:00:00 2001 From: Claude Opus 5 Date: Sun, 6 Sep 2026 09:46:35 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20withdraw=20the=20"Superset=20/=20OpenMe?= =?UTF-8?q?tadata=20are=20OUT"=20rows=20=E2=80=94=20both=20are=20canonical?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/02-architecture.md` was written when 0pt was planned as a standalone SaaS product. Its "Stack decisions — OUT (do not add)" table rejects Apache Superset ("Metabase already covers BI") and DataHub/OpenMetadata ("enterprise-heavy"), and the "IN" table names Metabase as the BI choice. Under the SIAX Cloud analytics decision, 0pt is the canonical Analytics/Data/Semantic owner — there is no BI0/Analytics0 repo and no separate Superset platform. Superset is the canonical BI engine (Apache-2.0, official embed SDK + guest tokens) and OpenMetadata the canonical catalog, default ahead of DataHub. Metabase, DataHub, Lightdash, Rill and Grafana are optional App Catalog apps, never the platform default. This repository had already shipped both of the "rejected" tools, which is what makes the old rows self-contradicting rather than merely stale: * src/metadata/openmetadata-adapter.ts + tests/contracts/metadata-features.contract.test.ts (STATUS.md classifies the OpenMetadata projection IMPLEMENTED) * services/infra/docker-compose.data-stack.yml defines a `superset` service (apache/superset:latest), and docs/compliance/OSS-INVENTORY.md line 74 already inventories `superset | Apache-2.0 | services/infra/data-stack` Falsifiable check used (asserts the OUT table no longer rejects either tool and that the IN table's BI row names Superset): origin/main -> FAIL (4 findings) this branch -> PASS mutation: BI row back to Metabase -> FAIL mutation: OUT rows un-struck -> FAIL Docs only. No workflow files, no .gitignore, no source. `node scripts/siax-doctor.mjs` gives 39/40 on this branch and 39/40 on main — the one failure (P1-15 CODEOWNERS missing) is pre-existing and is what PR #19 supplies. Co-Authored-By: Claude Opus 5 --- docs/02-architecture.md | 34 ++++++++++++++++++++++++++++++---- docs/07-dependencies.md | 2 +- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/docs/02-architecture.md b/docs/02-architecture.md index 9488ddd..69a392e 100644 --- a/docs/02-architecture.md +++ b/docs/02-architecture.md @@ -37,7 +37,7 @@ This principle describes — and is largely realized by — the **original edge - **FastAPI orchestrator** (`services/api/orchestrator/main.py`) — REST contract in `docs/04-api.md`: estimates, orders, sources, vehicle lookup, sanctions screening, job run/status. **Not on Vercel** (serverless timeouts kill long jobs). *Correction: earlier drafts of this doc pointed at `src/orchestrator`; that was the pre-migration location. `docs/09-build-plan.md` Phase 0 called for moving it into `services/api`, and that move has happened — the code lives at `services/api/orchestrator` today.* - **Workers** (`services/api/workers/`) — adaptive router (`adaptive_router.py`, verified: chooses only among compliant methods — official API → bulk → authorized web → licensed, no bypass/stealth/proxy-rotation tier exists in it, by design) + pipeline stages (`pipeline.py`, `queue.py`, `resolve.py`, `run.py`). See `docs/06-data-engine.md`. - **Postgres** — backbone (`docs/03-data-model.md`) incl. `pgvector` for matching/embeddings, for the original edge + engine plane. Wired via `lib/db.ts` (env-guarded: `DATABASE_URL`/`POSTGRES_URL`; falls back to an in-memory, non-durable store when unset, e.g. `lib/orders-store.ts`). -- **NocoDB / Metabase** — both are defined as services in `infra/docker-compose.yml`; embedding them into the portal as "Min data" / customer dashboards is the target, not yet confirmed wired into a portal page in this pass. +- **NocoDB / Metabase** — both are defined as services in `infra/docker-compose.yml`; embedding them into the portal as "Min data" / customer dashboards was the original target, and neither is confirmed wired into a portal page. *Correction 2026-09-06: Metabase is no longer the canonical BI lens — Apache Superset is (see "Stack decisions" below). Metabase may stay as an optional App Catalog app; it must not be treated as the platform default.* - **Cloudflare R2** — file storage (uploads + deliverables) via presigned URLs. Env-guarded (`app/api/uploads/route.ts` checks `R2_ACCESS_KEY_ID`/`R2_BUCKET`/`R2_ENDPOINT` and returns `501` until set); the presign call itself is still a stub (`put_url: null`) even when the env vars are present — not wired end-to-end yet. - **Stripe** — deposit capture + escrow balance release. This one **is** wired: `app/api/payments/[order_id]/deposit/route.ts` and `.../release/route.ts` are real Stripe Checkout calls (env-guarded on `STRIPE_SECRET_KEY`, `501` until configured). See the correction in `docs/01-product.md`'s business model section for exactly which escrow mechanism is implemented. - **Resend** — documented as the transactional-email provider (`docs/08-infrastructure.md`) but has no call site anywhere in this repo yet. Treat "+ email" as not-yet-implemented, not merely unconfigured. @@ -157,7 +157,7 @@ Flagged so it is not repeated: `src/services/quality-check-service.ts`, `src/ser | Backend/jobs | FastAPI + workers (`services/api`) | already built; long-running jobs | | DB | Postgres (+ pgvector) | backbone; Neon free tier or on-VPS | | Airtable layer | **NocoDB** | sits on existing tables, no duplication | -| BI | **Metabase** | customer-facing embedding, simple | +| BI | **Apache Superset** | canonical generic BI for SIAX Cloud: Apache-2.0 with an official embed SDK + guest tokens, so embedding never needs a commercial tier. Already defined as a service in `services/infra/docker-compose.data-stack.yml` and inventoried in `docs/compliance/OSS-INVENTORY.md`. **Corrected 2026-09-06** — this row previously said Metabase; see the correction under "OUT" below | | Files | Cloudflare R2 + presigned URLs | cheapest S3-compatible | | Payments | Stripe Checkout + webhook → Next.js API | ~1 day, no SaaS boilerplate needed | | Auth | Auth.js + Postgres adapter | one system — **target, not yet built** (see Components above) | @@ -166,10 +166,36 @@ Flagged so it is not repeated: `src/services/quality-check-service.ts`, `src/ser ## Stack decisions — OUT (do not add) +> **CORRECTED 2026-09-06 — two rows in this table were wrong and are withdrawn below.** +> This section was written when 0pt was planned as a standalone SaaS product. Under the SIAX +> Cloud analytics decision, **0pt is the canonical Analytics/Data/Semantic owner** — there is +> no `BI0`/`Analytics0` repo, no separate Superset platform, and no product-specific BI +> backend acting as a parallel capability owner. Third-party BI tools are interchangeable +> providers or App Catalog apps. Two consequences contradict the table as it stood: +> +> - **Apache Superset is canonical**, not rejected (Apache-2.0, official embed SDK + guest +> tokens). Metabase moves to the optional App Catalog. +> - **OpenMetadata is the canonical catalog** (default *before* DataHub), not rejected. +> DataHub remains optional. +> +> This is not a paper change — **this repository already shipped both**, which is what makes +> the old rows self-contradicting rather than merely out of date: +> +> - `src/metadata/openmetadata-adapter.ts` + `tests/contracts/metadata-features.contract.test.ts` +> (OpenMetadata projection; `STATUS.md` classifies it `IMPLEMENTED`, projection-never-canonical), +> - `services/infra/docker-compose.data-stack.yml` defines a `superset` service +> (`apache/superset:latest`), and `docs/compliance/OSS-INVENTORY.md` already inventories +> `superset | Apache-2.0 | services/infra/data-stack`. +> +> The governing principle behind the change is **one definition of every metric**: the 0pt +> Metric Catalog feeds MetricFlow, which feeds Superset / Cube / the native API / INF0. Point a +> product at a BI engine directly and each surface invents its own "average price". + | Rejected | Reason | |---|---| -| Superset | Metabase already covers BI — one tool, not two | -| DataHub / OpenMetadata | enterprise-heavy; catalog = a Postgres table in Next.js + NocoDB. Add lineage only if an enterprise customer demands it | +| ~~Superset~~ | **WITHDRAWN 2026-09-06 — see the correction note above.** Superset is now the canonical BI engine, not a rejected one | +| ~~DataHub / OpenMetadata~~ | **WITHDRAWN 2026-09-06 — see the correction note above.** OpenMetadata is the canonical catalog; only DataHub stays non-canonical (optional App Catalog) | +| Metabase / Lightdash / Rill / Grafana / DataHub as *canonical* BI or catalog | optional App Catalog apps only, never the platform default. Metabase's OSS build is AGPL and its full app-/SSO-embedding sits behind Pro/Enterprise, which does not work for SIAX as a platform provider; Lightdash gates embedding/caching behind an enterprise self-host licence; Rill's documented deployment path is bound to Rill Cloud (breaks zero-SaaS); Grafana (AGPLv3) is the right tool for **operational** analysis and is AUD0-owned, not business BI | | Baserow / Teable | NocoDB chosen — not three Airtable clones | | Meilisearch (v1) | Postgres full-text suffices to a few thousand rows; add later | | Delta Sharing (v1) | unnecessary; customer pull via scoped role + exports | diff --git a/docs/07-dependencies.md b/docs/07-dependencies.md index 65b8841..325a705 100644 --- a/docs/07-dependencies.md +++ b/docs/07-dependencies.md @@ -74,6 +74,6 @@ sourcing and rapidfuzz-based dedup are real today; extraction/OCR, NER, and qual - **Bypass/stealth/evasion**: FlareSolverr, "CloakBrowser", undetected-chromedriver/undetected-playwright (as bypass), proxy/fingerprint-rotation engines, any "WAF bypass" repo. Prohibited by CLAUDE.md §1 — they destroy the honest-accuracy positioning and create legal exposure. **This is not hypothetical**: a real WAF-bypass/proxy-rotation/fake-identity scraper stack (`services/scrapers/*`) briefly existed on this repo's Gitea remote (`git.siax.io/sax3l/0pt`, three commits dated 2026-07-13) and has since been removed there via PR #1. It never touched `origin` (`github.com/sax3l/0pt`), which this doc corpus was verified against. See `docs/06-data-engine.md`'s Known Issue callout for the full, unresolved detail — which remote is canonical is Simon's call, not decided this pass. - **Fake-account automation** on Apollo/Cognism/etc. - **Unverified / likely-hallucinated** from earlier brainstorming: "CloakBrowser/CloakHQ", "Rota proxy engine", "OpenClaw". Do not adopt unless you confirm a real, maintained project — and even then, not if it falls in the bypass category. -- **Overlapping tools** already decided against (Superset, DataHub/OpenMetadata, Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`. +- **Overlapping tools** already decided against (Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`. *Corrected 2026-09-06: Superset and OpenMetadata were previously listed here as rejected. They are canonical (BI engine and catalog respectively) — see the correction note under "Stack decisions — OUT" in `docs/02-architecture.md`.* > Note on real-but-overhyped tools (Firecrawl, browser-use, etc.): they exist, but ignore inflated star-counts and only adopt if they fit a *compliant* role and don't duplicate the stack. -- 2.54.0