0pt/docs/07-dependencies.md

# 07 · Dependencies & repos to import

Curated. Each entry: what it is, where it slots in, how to install. **Do not import on reputation/star-count** — several items in earlier brainstorming were unverified or hallucinated (see Exclusions). Verify each repo exists and is maintained before adopting.

> **This is a plan, not a status report.** Every table below lists what *should* be imported for the target
> architecture in `docs/06-data-engine.md`. A **Status** column has been added to each table (verified this
> pass against `services/api/requirements.txt` and root `package.json` — the actual, current dependency
> manifests, read in full). Read the status column before assuming a package is already wired in; most of the
> data-engine tier (OCR, NER, quality checks, orchestration) is still **not installed**.

## Frontend (Next.js on Vercel)
| Package | Purpose | Install | Status |
|---|---|---|---|
| next, react, typescript | App Router base | `npx create-next-app@latest --ts` | **Installed** — `next@16.0.10`, `react@19.2.0`, `typescript@^5` in `package.json` |
| tailwindcss | styling (mono tokens) | per Next setup | **Installed** — `tailwindcss@^4.1.9` |
| shadcn/ui | base components | `npx shadcn@latest init` | **Installed via copy-in, not as a runtime dep** — no `shadcn` package in `package.json` (expected: the shadcn CLI copies component source into `components/ui/`); the underlying `@radix-ui/react-*` primitives it generates against are installed individually and present |
| @tanstack/react-table **or** ag-grid-react | data tables / catalog grid | `npm i @tanstack/react-table` | **Not installed** — neither package present |
| @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3 | file upload → R2 (presigned) | `npm i @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3` | **Not installed** |
| next-auth (Auth.js) + @auth/pg-adapter | single auth, Postgres-backed | `npm i next-auth @auth/pg-adapter` | **Not installed.** Matches a broader, confirmed gap: there is no auth/session system anywhere in this repo yet (`app/api/orders/route.ts`'s own `GET` handler has the literal comment "org scoping added with auth" — i.e. not yet). Several `app/api/0pt/*` routes currently read `tenant_id`/`project_id` straight from the request body as a result — a real, open security gap, not a deliberate design; see `docs/09-build-plan.md` Phase 3 for the planned fix (not built anywhere yet, Simon's call on design). |
| stripe / @stripe/stripe-js | checkout + portal | `npm i stripe @stripe/stripe-js` | **Partially installed** — `stripe@^22.2.3` (server SDK) is present; `@stripe/stripe-js` (client SDK) is not |
| resend / react-email | transactional email | `npm i resend @react-email/components` | **Not installed** |
| lenis *(optional)* | smooth scroll (keeps all effects) | `npm i lenis` | **Not installed** |
| recharts *(only if needed)* | charts inside DataArtifact/dashboards | `npm i recharts` | **Installed** — `recharts@2.15.4` |

> Catalog/search in v1 = Postgres full-text. Add Meilisearch later only past a few thousand rows.

## Backend / workers (Python, FastAPI — existing)

`services/api/requirements.txt` currently pins exactly 9 packages. Everything else in this table is still to
be added — this is the single biggest gap between this doc's plan and the running code (see
`docs/06-data-engine.md`'s "Pipeline stages" section for what that means stage-by-stage: only official-API
sourcing and rapidfuzz-based dedup are real today; extraction/OCR, NER, and quality-checking have no code at all).

| Package | Purpose | Install | Status |
|---|---|---|---|
| fastapi, uvicorn, pydantic | API (existing) | `pip install fastapi uvicorn pydantic` | **Installed** — `fastapi==0.115.6`, `uvicorn[standard]==0.34.0`, `pydantic==2.10.4` |
| httpx | polite HTTP fetch (identifies itself, obeys rate limits) | `pip install httpx` | **Installed and actively used** — `httpx==0.28.1`, every connector in `sources/connectors.py` and `sources/financials.py` goes through it |
| pyyaml | parse `sources.yaml` | `pip install pyyaml` | **Installed and used** — `pyyaml==6.0.2`, loaded once in `orchestrator/main.py` |
| rapidfuzz | dedup / fuzzy matching | `pip install rapidfuzz` | **Installed and used** — `rapidfuzz==3.11.0`, the only matching engine in `workers/resolve.py` |
| python-dotenv, tenacity | env loading, retry helper | — | **Installed**, `python-dotenv==1.0.1` / `tenacity==9.0.0` — present in `requirements.txt` but not yet visibly imported in the worker/orchestrator/sources files read this pass; not confirmed wired up |
| sqlalchemy / asyncpg | Postgres access (align `db.py`) | `pip install sqlalchemy asyncpg` | **`asyncpg==0.30.0` is installed but unused.** No file in `services/api` imports it or issues a query — confirmed by grep. `orchestrator/main.py` reads `DATABASE_URL` once, only to report a boolean in `/health`. Orders/jobs live in in-memory dicts that reset on restart. `sqlalchemy` is **not installed** at all. There is also no `db.py` anywhere in this repo — an older doc reference to `src/workers/db.py` is stale; that path does not exist. |
| alembic | migrations | `pip install alembic` | **Not installed.** Migrations currently ship as plain numbered SQL files in `sql/` (`001_init.sql`, `004_iceberg_write_boundary.sql`), applied ad hoc, not via Alembic. |
| celery + redis **or** existing adaptive_router/queue | job queue/concurrency | `pip install celery redis` | **Neither Celery nor a Python Redis client is installed.** `workers/queue.py` is a small in-process `asyncio.Queue` wrapper (single-worker dev only, by its own docstring). `infra/docker-compose.yml` does run a `redis` *container*, but nothing in `services/api` currently talks to it. |
| scrapy **or** crawlee | orchestrate **authorized** fetches | `pip install scrapy` | **Not installed.** No HTML-scraping library of any kind (also no `bs4`/`lxml`) exists in this codebase — confirmed by grep. Tier-2 "authorized public web" sourcing (docs/06) has no code path yet; only tier-1 official-API connectors exist. |
| playwright | public JS-rendered pages, politely | `pip install playwright && playwright install chromium` | **Not installed.** No `browser_worker.py` file exists anywhere in this repo either — an older doc reference to reconciling a FlareSolverr stage inside it is moot; there's nothing to reconcile in the current Python engine (see `docs/06-data-engine.md`'s Known Issue callout for where a bypass stack *did* briefly exist, on a different git remote). |
| unstructured | parse PDF/HTML/DOCX/images → elements | `pip install "unstructured[all-docs]"` | **Not installed.** |
| paddleocr / pytesseract | OCR cascade | `pip install paddleocr pytesseract` (+ tesseract bin) | **Not installed.** No OCR code or `ocr_worker` module exists anywhere in this repo. |
| spacy (+ models) / transformers | NER, entity resolution (multilingual) | `pip install spacy transformers` | **Not installed.** `workers/resolve.py`'s entity resolution is rapidfuzz string-similarity only — no NER, no multilingual normalization. |
| polars | fast tabular processing | `pip install polars` | **Not installed.** |
| great-expectations | data quality checks in pipeline | `pip install great-expectations` | **Not installed, and nothing equivalent exists either.** The entire QA stage in `workers/pipeline.py` is a single confidence-threshold filter (`r["confidence"] >= threshold`) — no validity/completeness/freshness/uniqueness checks, no quality report emitted or stored. This is a real, currently-open gap, not a doc typo: a broader, confirmed finding this session is that 0pt's data-quality/QA layer, once described elsewhere in this doc corpus as already built and tested (`src/services/quality-check-service.ts` etc.), does not exist in code at all — zero commits in this repo's history, any branch, for any of those files. Every doc that used to make that claim (`docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, `docs/P0_QUERY_SOURCE_FLOW.md`) has since been corrected. The Python engine independently confirms the same absence. |
| dbt-postgres *(optional)* | transform layer | `pip install dbt-postgres` | **Not installed.** |
| pgvector (Postgres ext) | embeddings/matching | `CREATE EXTENSION vector;` | The `pgvector/pgvector:pg16` image is used in `infra/docker-compose.yml`, so the extension is available in the container, but no code in `services/api` runs a vector query — not exercised by any of the files read this pass |
| boto3 *(or aioboto3)* | R2 (S3-compatible) | `pip install boto3` | **Not installed.** No R2/S3 upload code found in `services/api` this pass. |
| stripe (python) | webhook handling | `pip install stripe` | **Not installed** in `services/api/requirements.txt` (the JS `stripe` package in root `package.json` is separate and unrelated to this table) |

## Self-hosted services (docker-compose on the VPS)
| Service | Purpose | Image | Status |
|---|---|---|---|
| **NocoDB** | Airtable-style views over existing Postgres tables; embedded "Min data" | `nocodb/nocodb` | **Wired in `infra/docker-compose.yml`** (`nocodb/nocodb:latest`, points at the same `postgres` service) — not independently verified running/reachable this pass |
| **Metabase** | customer-facing dashboards; embedded | `metabase/metabase` | **Wired in `infra/docker-compose.yml`** (`metabase/metabase:latest`, same Postgres backend) — not independently verified running/reachable this pass |
| **Postgres** *(or Neon managed)* | backbone (+ pgvector) | `pgvector/pgvector:pg16` | **Wired in `infra/docker-compose.yml`** — but see the backend table above: the FastAPI service that should write to it currently does not (in-memory only) |
| **Redis** *(if Celery)* | queue | `redis` | **Container is wired in `infra/docker-compose.yml`**, but no Python code in `services/api` currently connects to it (no Celery, no `redis` client installed) |
| **Meilisearch** *(later)* | catalog search at scale | `getmeili/meilisearch` | Not in `infra/docker-compose.yml` yet — consistent with this doc's own "later" framing, not a gap |

## External managed (free/cheap tiers)
- **Vercel** — Next.js hosting.
- **Cloudflare R2** — object storage (presigned URLs). No R2 client code found in this pass (`boto3` not installed; no `@uppy/aws-s3` in `package.json`) — treat as not yet wired regardless of account status.
- **Stripe** — payments (test mode first). Server SDK (`stripe` npm package) is installed; whether a live/test key is actually configured was not verified this pass (no secret material read).
- **Resend** — email. Not installed (see frontend table).
- **Neon** *(optional)* — managed Postgres. `infra/docker-compose.yml` runs its own Postgres container instead; no evidence either way of a Neon instance actually in use.

## EXCLUSIONS — do not import

- **Bypass/stealth/evasion**: FlareSolverr, "CloakBrowser", undetected-chromedriver/undetected-playwright (as bypass), proxy/fingerprint-rotation engines, any "WAF bypass" repo. Prohibited by CLAUDE.md §1 — they destroy the honest-accuracy positioning and create legal exposure. **This is not hypothetical**: a real WAF-bypass/proxy-rotation/fake-identity scraper stack (`services/scrapers/*`) briefly existed on this repo's Gitea remote (`git.siax.io/sax3l/0pt`, three commits dated 2026-07-13) and has since been removed there via PR #1. It never touched `origin` (`github.com/sax3l/0pt`), which this doc corpus was verified against. See `docs/06-data-engine.md`'s Known Issue callout for the full, unresolved detail — which remote is canonical is Simon's call, not decided this pass.
- **Fake-account automation** on Apollo/Cognism/etc.
- **Unverified / likely-hallucinated** from earlier brainstorming: "CloakBrowser/CloakHQ", "Rota proxy engine", "OpenClaw". Do not adopt unless you confirm a real, maintained project — and even then, not if it falls in the bypass category.
- **Overlapping tools** already decided against (Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`. *Corrected 2026-09-06: Superset and OpenMetadata were previously listed here as rejected. They are canonical (BI engine and catalog respectively) — see the correction note under "Stack decisions — OUT" in `docs/02-architecture.md`.*

> Note on real-but-overhyped tools (Firecrawl, browser-use, etc.): they exist, but ignore inflated star-counts and only adopt if they fit a *compliant* role and don't duplicate the stack.