# 0pt.query_source — Bounded Source Query Flow > **Companion-doc note (corrected):** this doc originally called itself an > "additive companion to `docs/P0_DATASET_QUALITY_FLOW.md`." That file did > not exist anywhere in this repository at the time — `git log --all` for it > returns zero commits, on any branch, ever. This pass adds > `docs/P0_DATASET_QUALITY_FLOW.md` as a clearly-labeled **target spec, not > yet built** (see that file). Treat the two docs as siblings: this one > covers the real, working `0pt.query_source` read path; the other covers > the *intended, unbuilt* dataset-quality/QA path that several other docs in > this repo (wrongly) describe as already implemented. See "Known gaps" > below for the details of that wider mixup. Adds the capability to actually **read data** from a registered Source, not just register its metadata. ## What this is `0pt.query_source`: given a Source already registered via `0pt.create_source` (`source_type = "database"`, `adapter_id = "fabric_onelake_sql"`), run a caller-supplied SQL query against the customer's own Azure Fabric / OneLake SQL analytics endpoint and return rows — synchronously, bounded by a max-row and max-time ceiling. > Note: `0pt.create_source` (the capability id `source-service.ts` emits in > its AUD0 events) is **not** one of the 14 `0pt.*` capabilities currently > registered in `cl0ud`'s `packages/capability-registry/src/registry.ts` — > only `0pt.source_registry_list` (a read/list capability) is registered for > the Source domain. `0pt.query_source` itself *is* registered (see > "Capability registration" below). Whether `0pt.create_source` needs its own > registry entry, or was meant to be folded into an existing one, is unresolved > — flagging for Simon rather than guessing. **Tenant-agnostic by construction.** Any SIAX customer registers their own Fabric/OneLake warehouse (their own Entra tenant, their own service principal, their own SQL endpoint). Nothing in the adapter or service layer is hardcoded to one customer — every credential and endpoint is a parameter supplied per call, never an env var fallback. (This is orthogonal to the `tenant_id`/`project_id` handling gap described below — the *warehouse* credentials are correctly never defaulted from env; the *platform* tenant_id used for scoping/audit is a separate, unresolved problem.) ## Why bounded + synchronous, not just "run any query" This path is for small, interactive/preview-style reads (validate a Source is reachable, sample a table, preview a query before scheduling it). It is *not* a general query engine: - `max_rows`: default 1,000, hard ceiling 10,000 (`SOURCE_QUERY_DEFAULT_MAX_ROWS` / `SOURCE_QUERY_HARD_MAX_ROWS` in `source-query-service.ts`). Excess rows are truncated and the response is flagged `truncated: true`. - `timeout_ms`: default 10s, hard ceiling 30s (`SOURCE_QUERY_DEFAULT_TIMEOUT_MS` / `SOURCE_QUERY_HARD_MAX_TIMEOUT_MS`). A caller-requested value above the ceiling is clamped down, not rejected. - Anything heavier — large exports, long scans, scheduled/recurring pulls — is meant to go through **N0D as a job** instead. The real, working precedent for that "synchronous path vs. N0D job" split in this codebase is `src/services/iceberg-write-service.ts`: it takes an `input.heavy` flag and, when set, calls `this.n0d.createJob(...)` instead of doing the work inline. (An earlier version of this doc cited `quality-check-service.ts` for this same pattern — that file does not exist; see "Known gaps.") `SourceQueryService` does not itself request an N0D job for anything today — there is no "heavy" variant of `0pt.query_source` yet, only the bounded synchronous path described here. ## Architecture ``` Adapter: src/adapters/fabric-onelake-adapter.ts - acquireFabricAccessToken(): Entra service-principal OAuth2 client-credentials flow (POST login.microsoftonline.com/{tenantId} /oauth2/v2.0/token) — generalized from v0-datahub's scripts/ops/cloud-bootstrap.mjs `aadToken()`. - FabricOneLakeAdapter.query(): opens a TDS pool to the Fabric SQL analytics endpoint using that token (azure-active-directory-access-token auth), same shape as v0-datahub's lib/warehouse/fabric-sql.ts `buildConfig()`/`runFabricQuery()`, generalized to caller-supplied tenant/client/secret/endpoint/database instead of one customer's env vars. Service: src/services/source-query-service.ts - SourceQueryService.querySource(): looks up the Source via the existing SourceService.getSource() (read-only — does not modify source-service.ts), validates source_type/adapter_id, clamps max_rows/timeout_ms, races the adapter query against a timeout, truncates rows, emits an AUD0 event, always releases the adapter's connection. Route: app/api/0pt/sources/[sourceId]/query/route.ts - POST { query, params?, tenant_id, project_id, actor_id?, correlation_id?, max_rows?, timeout_ms?, connection_config } -> QuerySourceResult - 400 on missing query/tenant_id/project_id/connection_config - 404 when the source does not exist - 422 (SourceQueryUnsupportedError) when source_type != "database" or adapter_id != "fabric_onelake_sql" - 504 (SourceQueryTimeoutError) on timeout ``` ## Capability registration (CL0UD registry — verified this pass) `0pt.query_source` is registered in `cl0ud`'s `packages/capability-registry/src/registry.ts`: | Field | Value | |---|---| | `surfaces` | `['api']` only — no `cli`/`mcp`/`job` surface is enabled | | `sync` | `true` | | `requiresJob` | `false` | | `piiLevel` | `'high'` — "ad-hoc SQL against an arbitrary registered source can return any column, including PII" (registry's own comment) | | `riskLevel` | `'L2'` | | `mcpEnabled` / `cliEnabled` | both `false` | | `auditRequired` | `true` (comment: "emits `0pt.source_queried`") | | `approvalRequired` | `false` | This matches the platform-wide six-surface capability model: every `0pt.*` capability registered today exposes `api` only (plus `job` for a couple of the heaviest write paths) — none has a live CLI or MCP tool yet. Any claim of a working `siax 0pt query-source ...` CLI command or an MCP tool for this capability would be describing a target, not current reality. **Discrepancy found this pass:** the registry declares `riskLevel: 'L2'` for `0pt.query_source`, but `source-query-service.ts`'s own AUD0 emission (see AUD0 section below) uses `risk_level: "L4"` on every `0pt.source_queried` event — a two-level gap between what the capability is registered as and what it actually reports. Unresolved; needs a decision on which is correct (or whether the registry entry should simply be updated to `L4` to match the live code, given "external read with a tenant's own credentials" is arguably closer to `L4` than `L2`). ## AUD0 event `event_type: "0pt.source_queried"`, `capability_id: "0pt.query_source"`, `resource_type: "source"`, `risk_level: "L4"` in the code as written today (see the registry discrepancy noted above). For comparison, other real, verified `risk_level` usages elsewhere in this repo's services: `source-service.ts` and `dataset-service.ts` use `"L2"` for their own metadata-write events; `document-capture-service.ts` and `document-extraction-service.ts` use `"L3"`; `external-writeback-service.ts` (`0pt.write_external_record`, a real, tested service) uses `"L4"` for live writes to an external system with a tenant's credentials — that is the closest real precedent for this capability's choice of `"L4"`, not the `src/data-platform/services/catalog-service.ts` citation an earlier version of this doc made up (that path does not exist anywhere in this repo — confirmed via `find`). Emitted on both success (`ok:true, row_count, truncated, duration_ms`) and failure (`ok:false, error`) via the same `Aud0Client` pattern as `source-service.ts` / `external-writeback-service.ts`. Like every other `Aud0Client` use in this repo, this runs against the in-memory mock client — no live AUD0 ingestion endpoint is reachable from this codebase today (no `AUD0_URL` is read via `process.env` anywhere in the code). Treat this as "real code, mock-only, not live-verified," not as "integrated." ## Known gaps / deliberately left out - **`tenant_id`/`project_id` are read straight from the request body, not bound to an authenticated session.** `app/api/0pt/sources/[sourceId]/query/route.ts` requires `body.tenant_id` and `body.project_id` as plain JSON fields (see the route's own 400-check) and passes them straight through to `SourceQueryService.querySource()`, which uses them unchecked for AUD0 scoping. This is **not** a deliberate two-tier trust design — it is because **no auth/session system exists anywhere in this repo yet** (`app/api/orders/route.ts`'s own `GET` handler has the literal comment "org scoping added with auth" — i.e., not yet). Any caller can currently claim to be any tenant simply by putting a different `tenant_id` in the request body. This is one of several routes in this repo with the same gap: `lakehouse/mappings/route.ts` and `sources/rate-policies/route.ts` read `tenant_id`/`project_id` directly off the request body exactly like this route does (verified by reading both files); `document-capture-service.ts`'s own route (`app/api/0pt/documents/capture/route.ts`) does the same (`body.tenant_id`/`body.project_id`). `lakehouse/mappings/[mappingId]/writes/route.ts` is one hop removed — it takes no `tenant_id` field on its own POST body at all; `IcebergWriteService.requestIcebergWrite()` instead pulls `tenant_id`/ `project_id` off the previously-stored mapping record (looked up by `mappingId`), whose `tenant_id` was itself set, unauthenticated, via `lakehouse/mappings/route.ts`'s POST body. Same underlying trust-chain problem, one indirection further away — worth keeping the distinction precise rather than implying all five routes share identical code shape. The fix is an auth/session layer that server-binds `tenant_id` — that is Simon's call, tracked as Phase 3 of `docs/09-build-plan.md`, not built anywhere in this repo yet. Do not treat this route as tenant-isolated until that lands. - **`mssql` is not a declared dependency yet.** package.json and pnpm-lock.yaml already have unrelated uncommitted changes from a separate in-flight session at the time this capability was built, so adding a new dependency there was avoided to prevent a collision. The adapter loads `mssql` lazily via a non-literal dynamic import (works with any bundler/ typecheck without the package installed) and throws a clear, actionable error if it's missing. Once package.json is settled, `pnpm add mssql` is the only change needed to make `query()` hit a real Fabric endpoint — no code changes required. As of this pass, this means `0pt.query_source` cannot actually execute a live query against Fabric/OneLake yet — the whole path up to the driver load is real and typechecks, but `query()` will throw the "mssql package is not installed/resolvable" error on any real invocation until the dependency is added. - **In-memory `SourceService` is not shared across API route modules.** Same pre-existing limitation as `sources/route.ts` / `sources/[sourceId]/route.ts` (each Next.js route module instantiates its own `SourceService`, so a Source created via `POST /api/0pt/sources` is not yet visible to `POST /api/0pt/sources/[sourceId]/query` in the same running process). `document-capture-service.ts` hit the same issue and added a shared singleton for its own two routes; doing the same for Source would mean editing `source-service.ts` and/or the two existing `sources/route.ts` files, which was out of scope here to avoid touching that flagged area. **Correction:** an earlier version of this doc, and the route file's own header comment, said the real fix is "already pointed at by `sql/002_p0_dataset_quality.sql` (`opt_sources` table)." That file does not exist and never has — `git log --all` returns zero commits for it, on any branch. The `sql/` directory in this repo today contains only `001_init.sql` and `004_iceberg_write_boundary.sql`; no `002` or `003` migration has ever been created. The real fix is still "back `SourceService` with real persistence" — just not via a file that exists. (`app/api/0pt/sources/[sourceId]/query/route.ts`'s own header comment still makes this same broken reference; that is a code comment, not a doc, so it's out of scope for this pass to edit, but it should be fixed alongside whichever pass builds real persistence.) - **No CL0UD policy check before executing a query.** Unlike `external-writeback-service.ts` (`0pt.write_external_record`), which calls `Cl0udClient.evaluatePolicy()` before executing a write, `source-query-service.ts` has no `cl0ud` import and performs no policy check at all — confirmed by reading the file in full. This synchronous read path is scoped as a bounded preview read, not a governed job, but that's a design choice that has not been written down or reviewed, not an enforced policy. Worth revisiting once policy-gating on `0pt.query_source` is defined — and note the CL0UD registry already declares `piiLevel: 'high'` for this capability, which makes the absence of any policy check more notable, not less. - **No SQL rewriting/injection of `TOP`/`LIMIT`.** Row bounding is applied to whatever the adapter returns, not pushed into the query itself (arbitrary caller SQL can't be safely rewritten). A query that itself returns far more rows than `max_rows` still executes in full on the warehouse before being truncated client-side — only the *response* is bounded, not warehouse-side compute. Documented, not silently hidden. - **Cross-file fabrication this pass found and corrected here, but could not fix everywhere it appears.** Several files in this repo (some in this doc's own former text, some in code comments this pass does not own) reference `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`, `src/types/quality-check.ts`, `src/types/quality-report.ts`, `src/data-platform/services/catalog-service.ts`, `sql/002_p0_dataset_quality.sql`, and `sql/003_*.sql` as if they are real, already-built, already-tested files. `git log --all` for every one of those paths returns **zero commits, ever, on any branch** (`main`, `feature/0pt-env-example`, `ci/add-quality-gate` all checked). These files have never existed in this repository. This doc's own text used to make this same false claim in three places; all are corrected above. **Re-checked this pass:** `docs/11-lakehouse-write-plan.md` has since been corrected by a parallel pass — it now carries its own "Correction (documentation-accuracy pass)" callout and its item 4 explicitly reads "**NOT BUILT** ... there is no dedicated quality-check/quality-report service in this repo as of this writing," so it no longer needs the same fix. **`docs/ICEBERG_WRITE_BOUNDARY.md` has *also* since been fully corrected** (re-verified this pass, reading the current file end to end): its former internal contradiction — a top-of-file correction callout next to a body that still asserted the fabricated files as real two sections down — has been resolved. Its "Verification status" section now reads "this section previously pointed at `src/services/quality-check-service.ts` as a peer example ... that file does not exist," and its data-quality section (retitled "Data quality: not yet built (corrected)") now quotes its own old wording only inside a "this section previously read..." framing before stating plainly that those files don't exist and were never built. Neither section currently asserts the fabricated files as fact. Code comments in `app/api/0pt/sources/[sourceId]/query/route.ts` (line 14: "`sql/002_p0_dataset_quality.sql` already defines `opt_sources`") and `src/services/external-writeback-service.ts` (line 21: "... `quality-check-service.ts`/`document-extraction-service.ts` already follow") still carry the same false reference — confirmed still present as of this pass — and remain outside any of these docs' scope to fix (code, not docs; flagging for whoever next touches those two files). See `docs/P0_DATASET_QUALITY_FLOW.md` (added this pass) for the honest, target-spec-only replacement for what those files were supposed to do. ## Known issue: which 0pt repo is authoritative This repo has two remotes with diverged history — `github.com/sax3l/0pt` and `git.siax.io/sax3l/0pt` (Gitea) — that share origin lineage for at least part of the codebase but have each accumulated commits the other doesn't have. Which one is deployed to `0pt.app`, and whether they're meant to stay in sync, is unresolved and is Simon's call, not something this pass attempted to resolve. Noting it here since this doc describes code that lives in "the 0pt repo" as if there's exactly one.