CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
Implements Simon's specified pattern (dealer_stock_entry/deal -> watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10, CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP directly for analytics. See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale. Also corrects a real schema mismatch found while building this: production has no generic `updated_at` on any of the three target tables (verified live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/ created_at as insert-time watermarks instead (documented caveat: in-place updates are not caught without a schema change); app.listing has no timestamp column at all and is deliberately not covered yet. Vehicle dimension lookup is narrow by construction: keyed on exactly the vehicle_id set referenced by the batch just read, never a scan of the 98M-row vehiq.vehicle table (connector-level validate() refuses that table by name; runner-level tests assert the query only ever touches referenced IDs). Verified: fixture-backed connector/runner tests (watermark+limit, same-timestamp tie at a page boundary, crash/resume, empty-table case, narrow-lookup query-count assertions) + a real disposable Postgres container (caught two real Date-handling bugs fixtures could not) + a read-only proof against actual production (server10, vehiq-db): the extractor's exact SQL returns zero rows, no error, confirming it is safe to schedule with zero current effect. .gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured — safe to merge with zero production effect until explicitly activated per the doc's "what activating this requires". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
114 lines
14 KiB
Markdown
114 lines
14 KiB
Markdown
# P070 — VehIQ OLTP -> 0pt watermark extractor
|
|
|
|
Status: built, tested, verified read-only against production. **Not scheduled/activated in production.** Read "What activating this requires" before turning the workflow on.
|
|
|
|
## What this is
|
|
|
|
A controlled, incremental extractor implementing exactly the pattern Simon specified (`admin/vehiq` 2026-09-25 pass, "Locked architecture decisions from Simon" / `docs/adr/0035-bi-analytics-architecture-product-firewall.md`):
|
|
|
|
```
|
|
OLTP operational tables -> read-only analytical extraction -> watermark ->
|
|
0pt ingestion -> ClickHouse (opt_dataset_projection)
|
|
```
|
|
|
|
Source: `app.dealer_stock_entry` and `app.deal` on production Postgres (`vehiq-db`, server10) — read-only `SELECT`, never a schema/config change, never touching `wal_level` or replication, never a permanent replica. Sink: `SiaxAnalyticsEngine.ingestProjection` (BI-P009, `opt_dataset_projection` in ClickHouse). Vehicle dimension: a narrow, keyed lookup against `vehiq.vehicle` (98M rows / 85GB) scoped to exactly the `vehicle_id`s referenced by the `dealer_stock_entry` batch just read — never `SELECT * FROM vehicle`, never a full scan.
|
|
|
|
## Why this lives in `admin/0pt`, not `admin/vehiq`
|
|
|
|
The task that produced this work described building it as a new package inside `admin/vehiq` (e.g. `packages/opt-ingest`). Before writing any code, this repo's own `docs/adr/0035-bi-analytics-architecture-product-firewall.md` (decisions 1, 2, 10 — decided the same day, one-way-door, CI-enforced by `tools/validate_engine_firewall.py`) was read and is unambiguous:
|
|
|
|
> VehIQ's own codebase must **never** get its own ClickHouse or DuckDB credentials, and must never import a ClickHouse/DuckDB client directly. ... Product code must never hammer the live OLTP Postgres with analytical queries directly.
|
|
|
|
And `admin/vehiq`'s `docs/CURRENT_STATE.md` ("Connection details for a future API0 connector", same pass) says explicitly:
|
|
|
|
> VehIQ's own codebase must never see any of this. This is 0pt's data plane; only a future `admin/api0` connector should hold these details.
|
|
|
|
Building the extractor inside `admin/vehiq` would have failed `validate_engine_firewall.py` the moment it imported a ClickHouse client or held a `CLICKHOUSE_*` env var, and would have been the exact "temporary VehIQ-owned data-access surface" ADR-0035 names as the failure mode it exists to prevent. So this extractor was built here instead — `admin/0pt`, 0pt's own data plane, a sibling of `admin/api0` (the mechanics belong next to the P060-P068 ingestion plane and the BI-P009/P010 ClickHouse engine that already live in this repo). **`admin/vehiq` was not modified by this work.**
|
|
|
|
This is a deliberate deviation from the task's literal repo suggestion, made because the suggestion conflicted with Simon's own same-day, explicit, locked architecture decision discovered while reading the target repo — not a judgment call made against that decision.
|
|
|
|
## Real production schema does not match the task's assumed column names
|
|
|
|
The task (and `admin/vehiq`'s `docs/CURRENT_STATE.md`) assumed all three OLTP tables carry a generic `updated_at` ("the columns already exist"). Verified live against production (server10, `vehiq-db`, read-only `information_schema.columns`, 2026-09-25) — **this is not true**:
|
|
|
|
| table | live columns | usable watermark |
|
|
|---|---|---|
|
|
| `app.dealer_stock_entry` | `id, tenant_id, vehicle_id, lot_location, floorplan_status, listed_at, sold_at` | `listed_at` (NOT NULL, insert-time only) |
|
|
| `app.deal` | `id, tenant_id, party_id, vehicle_id, sale_price_minor, currency, status, created_at` | `created_at` (NOT NULL DEFAULT now(), insert-time only) |
|
|
| `app.listing` | `id, tenant_id, vehicle_id, asking_price_minor, currency, channels, status` | **none** |
|
|
|
|
**`app.listing` has no timestamp column of any kind today.** `admin/vehiq` migration `0084_listing_deal_lifecycle_timestamps.sql` (already written, adds `published_at`/`unpublished_at` to `listing` and `closed_at`/`listing_id` to `deal`) exists in the repo but is **not yet applied to production** (verified: those columns are absent live). Until it lands, there is no honest way to build a watermark connector for `app.listing` — one is not built here. **This extractor covers `dealer_stock_entry` and `deal` only, not `listing`.**
|
|
|
|
**Caveat on the two sources that ARE built:** `listed_at`/`created_at` are set once, at INSERT. Neither table has a column that advances when an *existing* row changes (`sold_at`/`floorplan_status` on `dealer_stock_entry`; `status` on `deal`). This extractor reliably sees every new row; it does **not** see an in-place UPDATE to a row it already read, because that row has already passed the watermark. Closing this gap needs either `listing_id`/`closed_at` (migration 0084, once applied) plus an equivalent for `dealer_stock_entry`, or a generic `updated_at` + trigger — a schema decision for Simon, not something this pass made unilaterally (per the hard "never a schema/config change" rule).
|
|
|
|
Also found and corrected while building: `vehiq.vehicle`'s primary key column is `vehicle_id`, not `id` (verified via schema introspection only — no vehicle data rows were read to establish this).
|
|
|
|
## Where the code lives
|
|
|
|
- `src/connectors/postgres-watermark-connector.ts` — the P061 `Connector` implementing `WHERE watermark_col > $1 OR (watermark_col = $1 AND key_col > $2) ORDER BY watermark_col, key_col LIMIT $3` (the `OR` tuple form, not a bare `>`, so a same-timestamp tie at a page boundary is not silently dropped — see the file's own header for the bug this avoids). Explicit column allow-list, never `SELECT *`. `validate()` structurally refuses any config pointed at `vehiq.vehicle`.
|
|
- `src/connectors/postgres-keyed-lookup-connector.ts` — the narrow vehicle-dimension connector. Config carries an already-resolved, finite `keys` array; `connect()` refuses an empty list rather than defaulting to a full read; every query is `WHERE vehicle_id = ANY($1)` chunked to `keys_per_query` (default 500).
|
|
- `src/connectors/postgres-query-impl.ts` — the real `postgres` (postgres.js) implementation of the connectors' injectable `QueryImpl`, used only by the orchestration script. Refuses to execute anything whose first keyword isn't `SELECT`, independent of the DB role's own privileges.
|
|
- `src/ingestion/projection-sink.ts` — adapts the P062 runner's `IngestionSink` onto `SiaxAnalyticsEngine.ingestProjection` (writes `opt_dataset_projection`). No ClickHouse import — only the `SiaxAnalyticsEngine` port type.
|
|
- `src/ingestion/vehiq-oltp-ingest-runner.ts` — the orchestration: `previewVehiqOltpSource` (one bounded read-only read, never commits, safe against production at any time) and `runVehiqOltpIngestion` (the real P062-runner path).
|
|
- `scripts/vehiq-oltp-ingest/run.ts` — thin CLI (`pnpm run vehiq:oltp-ingest`), env-driven, defaults to `preview` mode.
|
|
- Tests: `tests/connectors/postgres-watermark-connector.test.ts`, `tests/connectors/postgres-keyed-lookup-connector.test.ts`, `tests/ingestion/vehiq-oltp-ingest-runner.test.ts`.
|
|
|
|
## Checkpoint field mapping
|
|
|
|
Simon's own specified checkpoint columns — `source, table, last_updated_at, last_id, batch_count, checksum, ingested_at` — mapped onto the **existing** P062 `IngestionCheckpoint` (reused, not duplicated, per the "no double truths" rule):
|
|
|
|
| Simon's field | Canonical field |
|
|
|---|---|
|
|
| `source` | `IngestionCheckpoint.source_id` |
|
|
| `table` | `IngestionCheckpoint.partition_key` |
|
|
| `last_updated_at` | decoded from `IngestionCheckpoint.cursor` (JSON `WatermarkCursor`) |
|
|
| `last_id` | decoded from `IngestionCheckpoint.cursor` |
|
|
| `checksum` | decoded from `IngestionCheckpoint.cursor` — sha256 of the page just landed |
|
|
| `batch_count` | `IngestionCheckpoint.records_committed` (cumulative); a per-run page count is also in each run's `PartitionRunReport` |
|
|
| `ingested_at` | `IngestionCheckpoint.committed_at` |
|
|
|
|
Durable storage: `createIngestionRepository(DATABASE_URL)` — Postgres-backed when `DATABASE_URL` (0pt's own metadata store) is set, in-memory only in non-production/tests. `run` mode refuses to start without `DATABASE_URL` explicitly, so a scheduled run can never silently fall back to re-reading everything from the beginning.
|
|
|
|
## Testing
|
|
|
|
Fixture-backed (no real infra): watermark + limit correctness, a same-timestamp tie at a page boundary, resume-from-checkpoint after a simulated crash (via a sink that throws mid-run, mirroring `tests/p0/ingestion-runner.test.ts`'s own technique), the empty-table case (zero rows, zero envelopes, no throw), and the narrow vehicle-dimension lookup asserted at the *query* level (every id ever bound into a query is a subset of the referenced set; a vehicle never referenced by the batch is never queried).
|
|
|
|
Also verified against a **real, disposable Postgres container** (not just fixtures), both seeded and truncated-to-empty, before touching production — caught and fixed two real bugs the fixture-only tests could not have caught:
|
|
|
|
1. `timestamptz` columns come back from the real driver as JS `Date` objects, not strings — the initial cursor-encoding logic did `String(date)` (locale-dependent, useless as a stored watermark) instead of `.toISOString()`. Fixed; regression test added (`tests/connectors/postgres-watermark-connector.test.ts`, "real Postgres driver value shapes").
|
|
2. The same `Date`-vs-string mismatch tripped the P063 schema-conformance gate on every row (`field "listed_at" is object, expected string`) until the connector's `toRecord()` was made to normalize every `Date`-valued payload field to ISO before building the envelope.
|
|
|
|
**Verified read-only against actual production** (server10, `vehiq-db`, via `fleet`), 2026-09-25: the connector's exact SQL (`SELECT id, tenant_id, vehicle_id, lot_location, floorplan_status, listed_at, sold_at FROM app.dealer_stock_entry ORDER BY listed_at, id LIMIT 100` and the equivalent for `app.deal`) executed directly, read-only, and returned **zero rows, no error** — confirming both that the extractor's query shape is valid against the live schema and that there is genuinely nothing to extract yet. `vehiq.vehicle`'s `vehicle_id` primary-key column name was confirmed via `information_schema.columns` only — no vehicle data rows were read. **The compiled service itself was not run against production** — only its exact SQL, via read-only `psql`, matching this pass's "prove it does nothing, don't deploy a live service yet" scope.
|
|
|
|
## Deferred: full vehicle-dimension backfill
|
|
|
|
Not built or run in this pass, per the task's explicit instruction. If/when a full `dimension_vehicle` snapshot is ever wanted (as opposed to the permanent narrow per-batch lookup this extractor does forever), it needs its own deliberate decision:
|
|
|
|
- **Chunking**: page through `vehiq.vehicle` by primary key range (`vehicle_id > :last ORDER BY vehicle_id LIMIT :n`), never an unbounded scan — the same shape `PostgresWatermarkConnector` already implements, just pointed at a PK range instead of a timestamp (this connector's own `validate()` refuses `vehiq.vehicle` by name specifically to keep this a distinct, deliberate choice rather than an accidental default).
|
|
- **Throttle**: an explicit delay between chunks, watching server10's `iostat`/`vmstat`/CPU while it runs — 98M rows / 85GB is enough to matter even read-only.
|
|
- **Checksum + reconciliation**: a row-count and content checksum comparison between source and `opt_dataset_projection` after the backfill, recorded, before trusting it.
|
|
- **Watermark handoff**: the backfill's end position must hand off cleanly to the narrow per-batch lookup this extractor already does, so nothing is double-counted or gapped at the boundary.
|
|
|
|
This mirrors the numbered runbook `admin/vehiq`'s `docs/CURRENT_STATE.md` already wrote for the general CDC case (not executed there either) — the shape is consistent, deliberately, rather than inventing a second convention.
|
|
|
|
## Deferred: `app.listing` extraction
|
|
|
|
Blocked on a schema decision, not an engineering gap: `app.listing` has no timestamp column today. Unblocking it needs one of:
|
|
|
|
1. Apply `admin/vehiq` migration `0084_listing_deal_lifecycle_timestamps.sql` (adds `published_at`/`unpublished_at`) — a decision for whoever owns that repo's migration pipeline, not this pass.
|
|
2. Add a generic `updated_at` + trigger to `dealer_stock_entry`/`deal`/`listing` — closes both this gap and the "insert-only watermark misses in-place updates" caveat above in one move, but is a real schema change on production and needs Simon's explicit sign-off per the hard rule against touching production schema unsupervised.
|
|
|
|
Once either lands, adding a third `VehiqOltpSourceDef` entry to `VEHIQ_OLTP_SOURCES` (`src/ingestion/vehiq-oltp-ingest-runner.ts`) is the only code change needed — the connector, runner, sink and tests are all already generic across sources.
|
|
|
|
## What activating this in production actually requires
|
|
|
|
Nothing here is scheduled or running. To turn it on:
|
|
|
|
1. **A read-only Postgres role on `vehiq-db`** (server10) with `SELECT` on exactly `app.dealer_stock_entry` and `app.deal` (not the whole `app` schema, not `vehiq.vehicle` beyond what the narrow lookup needs) — creating this role is a production database change and should be coordinated explicitly, the same posture `admin/vehiq`'s own CDC runbook already takes for a bigger version of this same question. Its connection string becomes `VEHIQ_SOURCE_DATABASE_URL`.
|
|
2. **`DATABASE_URL`** — 0pt's own metadata Postgres, for durable checkpoints. Already exists for every other 0pt capability; reuse it, do not provision a new one for this.
|
|
3. **`OPT_ANALYTICS_CLICKHOUSE_BASE_URL` / `OPT_ANALYTICS_CLICKHOUSE_BASIC_AUTH`** — the already-deployed `opt-analytics-clickhouse` on server6 (internal-only, reachable from containers on the `coolify` network — see `admin/vehiq`'s `docs/CURRENT_STATE.md`, "0pt production ClickHouse 26.8 LTS"). A real least-privilege application user should be provisioned for this connector specifically, not the schema-admin `opt_admin` user mentioned there.
|
|
4. **`VEHIQ_INGEST_MODE=run`** on the scheduled workflow (default is `preview`, which never writes).
|
|
5. A decision on whether `.gitea/workflows/vehiq-oltp-ingest.yml`'s cron schedule should actually be enabled (it is present and correctly configured, but Gitea Actions secrets are not set on this repo by this pass — an unset `VEHIQ_SOURCE_DATABASE_URL` makes every scheduled run a clean no-op, so leaving the workflow file in place with no secrets configured is safe, but is not the same as "this is now extracting real data").
|
|
|
|
None of steps 1-3 were done in this pass — they require production credentials/role provisioning this pass does not have standing authority to create unsupervised.
|