diff --git a/docs/02-architecture.md b/docs/02-architecture.md index 9488ddd..69a392e 100644 --- a/docs/02-architecture.md +++ b/docs/02-architecture.md @@ -37,7 +37,7 @@ This principle describes — and is largely realized by — the **original edge - **FastAPI orchestrator** (`services/api/orchestrator/main.py`) — REST contract in `docs/04-api.md`: estimates, orders, sources, vehicle lookup, sanctions screening, job run/status. **Not on Vercel** (serverless timeouts kill long jobs). *Correction: earlier drafts of this doc pointed at `src/orchestrator`; that was the pre-migration location. `docs/09-build-plan.md` Phase 0 called for moving it into `services/api`, and that move has happened — the code lives at `services/api/orchestrator` today.* - **Workers** (`services/api/workers/`) — adaptive router (`adaptive_router.py`, verified: chooses only among compliant methods — official API → bulk → authorized web → licensed, no bypass/stealth/proxy-rotation tier exists in it, by design) + pipeline stages (`pipeline.py`, `queue.py`, `resolve.py`, `run.py`). See `docs/06-data-engine.md`. - **Postgres** — backbone (`docs/03-data-model.md`) incl. `pgvector` for matching/embeddings, for the original edge + engine plane. Wired via `lib/db.ts` (env-guarded: `DATABASE_URL`/`POSTGRES_URL`; falls back to an in-memory, non-durable store when unset, e.g. `lib/orders-store.ts`). -- **NocoDB / Metabase** — both are defined as services in `infra/docker-compose.yml`; embedding them into the portal as "Min data" / customer dashboards is the target, not yet confirmed wired into a portal page in this pass. +- **NocoDB / Metabase** — both are defined as services in `infra/docker-compose.yml`; embedding them into the portal as "Min data" / customer dashboards was the original target, and neither is confirmed wired into a portal page. *Correction 2026-09-06: Metabase is no longer the canonical BI lens — Apache Superset is (see "Stack decisions" below). Metabase may stay as an optional App Catalog app; it must not be treated as the platform default.* - **Cloudflare R2** — file storage (uploads + deliverables) via presigned URLs. Env-guarded (`app/api/uploads/route.ts` checks `R2_ACCESS_KEY_ID`/`R2_BUCKET`/`R2_ENDPOINT` and returns `501` until set); the presign call itself is still a stub (`put_url: null`) even when the env vars are present — not wired end-to-end yet. - **Stripe** — deposit capture + escrow balance release. This one **is** wired: `app/api/payments/[order_id]/deposit/route.ts` and `.../release/route.ts` are real Stripe Checkout calls (env-guarded on `STRIPE_SECRET_KEY`, `501` until configured). See the correction in `docs/01-product.md`'s business model section for exactly which escrow mechanism is implemented. - **Resend** — documented as the transactional-email provider (`docs/08-infrastructure.md`) but has no call site anywhere in this repo yet. Treat "+ email" as not-yet-implemented, not merely unconfigured. @@ -157,7 +157,7 @@ Flagged so it is not repeated: `src/services/quality-check-service.ts`, `src/ser | Backend/jobs | FastAPI + workers (`services/api`) | already built; long-running jobs | | DB | Postgres (+ pgvector) | backbone; Neon free tier or on-VPS | | Airtable layer | **NocoDB** | sits on existing tables, no duplication | -| BI | **Metabase** | customer-facing embedding, simple | +| BI | **Apache Superset** | canonical generic BI for SIAX Cloud: Apache-2.0 with an official embed SDK + guest tokens, so embedding never needs a commercial tier. Already defined as a service in `services/infra/docker-compose.data-stack.yml` and inventoried in `docs/compliance/OSS-INVENTORY.md`. **Corrected 2026-09-06** — this row previously said Metabase; see the correction under "OUT" below | | Files | Cloudflare R2 + presigned URLs | cheapest S3-compatible | | Payments | Stripe Checkout + webhook → Next.js API | ~1 day, no SaaS boilerplate needed | | Auth | Auth.js + Postgres adapter | one system — **target, not yet built** (see Components above) | @@ -166,10 +166,36 @@ Flagged so it is not repeated: `src/services/quality-check-service.ts`, `src/ser ## Stack decisions — OUT (do not add) +> **CORRECTED 2026-09-06 — two rows in this table were wrong and are withdrawn below.** +> This section was written when 0pt was planned as a standalone SaaS product. Under the SIAX +> Cloud analytics decision, **0pt is the canonical Analytics/Data/Semantic owner** — there is +> no `BI0`/`Analytics0` repo, no separate Superset platform, and no product-specific BI +> backend acting as a parallel capability owner. Third-party BI tools are interchangeable +> providers or App Catalog apps. Two consequences contradict the table as it stood: +> +> - **Apache Superset is canonical**, not rejected (Apache-2.0, official embed SDK + guest +> tokens). Metabase moves to the optional App Catalog. +> - **OpenMetadata is the canonical catalog** (default *before* DataHub), not rejected. +> DataHub remains optional. +> +> This is not a paper change — **this repository already shipped both**, which is what makes +> the old rows self-contradicting rather than merely out of date: +> +> - `src/metadata/openmetadata-adapter.ts` + `tests/contracts/metadata-features.contract.test.ts` +> (OpenMetadata projection; `STATUS.md` classifies it `IMPLEMENTED`, projection-never-canonical), +> - `services/infra/docker-compose.data-stack.yml` defines a `superset` service +> (`apache/superset:latest`), and `docs/compliance/OSS-INVENTORY.md` already inventories +> `superset | Apache-2.0 | services/infra/data-stack`. +> +> The governing principle behind the change is **one definition of every metric**: the 0pt +> Metric Catalog feeds MetricFlow, which feeds Superset / Cube / the native API / INF0. Point a +> product at a BI engine directly and each surface invents its own "average price". + | Rejected | Reason | |---|---| -| Superset | Metabase already covers BI — one tool, not two | -| DataHub / OpenMetadata | enterprise-heavy; catalog = a Postgres table in Next.js + NocoDB. Add lineage only if an enterprise customer demands it | +| ~~Superset~~ | **WITHDRAWN 2026-09-06 — see the correction note above.** Superset is now the canonical BI engine, not a rejected one | +| ~~DataHub / OpenMetadata~~ | **WITHDRAWN 2026-09-06 — see the correction note above.** OpenMetadata is the canonical catalog; only DataHub stays non-canonical (optional App Catalog) | +| Metabase / Lightdash / Rill / Grafana / DataHub as *canonical* BI or catalog | optional App Catalog apps only, never the platform default. Metabase's OSS build is AGPL and its full app-/SSO-embedding sits behind Pro/Enterprise, which does not work for SIAX as a platform provider; Lightdash gates embedding/caching behind an enterprise self-host licence; Rill's documented deployment path is bound to Rill Cloud (breaks zero-SaaS); Grafana (AGPLv3) is the right tool for **operational** analysis and is AUD0-owned, not business BI | | Baserow / Teable | NocoDB chosen — not three Airtable clones | | Meilisearch (v1) | Postgres full-text suffices to a few thousand rows; add later | | Delta Sharing (v1) | unnecessary; customer pull via scoped role + exports | diff --git a/docs/07-dependencies.md b/docs/07-dependencies.md index 65b8841..325a705 100644 --- a/docs/07-dependencies.md +++ b/docs/07-dependencies.md @@ -74,6 +74,6 @@ sourcing and rapidfuzz-based dedup are real today; extraction/OCR, NER, and qual - **Bypass/stealth/evasion**: FlareSolverr, "CloakBrowser", undetected-chromedriver/undetected-playwright (as bypass), proxy/fingerprint-rotation engines, any "WAF bypass" repo. Prohibited by CLAUDE.md §1 — they destroy the honest-accuracy positioning and create legal exposure. **This is not hypothetical**: a real WAF-bypass/proxy-rotation/fake-identity scraper stack (`services/scrapers/*`) briefly existed on this repo's Gitea remote (`git.siax.io/sax3l/0pt`, three commits dated 2026-07-13) and has since been removed there via PR #1. It never touched `origin` (`github.com/sax3l/0pt`), which this doc corpus was verified against. See `docs/06-data-engine.md`'s Known Issue callout for the full, unresolved detail — which remote is canonical is Simon's call, not decided this pass. - **Fake-account automation** on Apollo/Cognism/etc. - **Unverified / likely-hallucinated** from earlier brainstorming: "CloakBrowser/CloakHQ", "Rota proxy engine", "OpenClaw". Do not adopt unless you confirm a real, maintained project — and even then, not if it falls in the bypass category. -- **Overlapping tools** already decided against (Superset, DataHub/OpenMetadata, Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`. +- **Overlapping tools** already decided against (Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`. *Corrected 2026-09-06: Superset and OpenMetadata were previously listed here as rejected. They are canonical (BI engine and catalog respectively) — see the correction note under "Stack decisions — OUT" in `docs/02-architecture.md`.* > Note on real-but-overhyped tools (Firecrawl, browser-use, etc.): they exist, but ignore inflated star-counts and only adopt if they fit a *compliant* role and don't duplicate the stack.