feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070) #117

Merged

admin merged 3 commits from feat/vehiq-oltp-watermark-extractor into main 2026-09-26 21:58:36 +00:00

Conversation 1 Commits 3 Files Changed 13

+1995 -11

admin commented 2026-09-25 17:11:39 +00:00

Owner

Copy Link

Copy Source

What

Implements Simon's specified pattern — OLTP operational tables -> read-only analytical extraction -> watermark/updated_at -> 0pt ingestion -> ClickHouse — as a new P061 connector pair plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Full design/rationale in docs/P070_VEHIQ_OLTP_EXTRACTOR.md.

Why here, not admin/vehiq

admin/vehiq's own docs/adr/0035-bi-analytics-architecture-product-firewall.md (decisions 1/2/10, decided the same day this task was framed, CI-enforced by tools/validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP Postgres directly for analytics — and its own docs/CURRENT_STATE.md says explicitly this belongs in "a future admin/api0 connector" / 0pt's data plane. Built here instead, as a sibling of admin/api0, next to the existing P060-P068 ingestion plane and BI-P009/P010 ClickHouse engine. admin/vehiq was not touched.

Real-schema correction found while building

Verified live (read-only, information_schema.columns, server10 vehiq-db) that production has no generic updated_at on any of the three target tables, contrary to the task's assumption. dealer_stock_entry/deal use listed_at/created_at (insert-time only — documented caveat on in-place-update blindness). app.listing has no timestamp column at all and is not covered — blocked on a schema decision (migration 0084 or an updated_at+trigger), documented in P070 as a deferred, explicit next step, not silently worked around.

Testing

• Fixture-backed connector + runner tests: watermark/limit correctness, a same-timestamp tie at a page boundary, crash/resume from a durable checkpoint, the empty-table case, and the narrow vehicle-dimension lookup asserted at the query level (never touches an unreferenced vehicle row).
• Verified against a real disposable Postgres container (seeded, then truncated to empty) — caught and fixed two real bugs (Date object handling) the fixtures alone could not have caught.
• Verified read-only against actual production (server10, vehiq-db, via fleet): the extractor's exact SQL returns zero rows, no error — proving it's safe to leave scheduled with zero current effect.
• pnpm run typecheck / pnpm run test (1138 passed) / pnpm run deps:check all green, zero new findings from these files. pnpm run lint has 12 pre-existing errors in files this PR does not touch (advisory gate; not from this change).

Activation

.gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured on this repo — safe to merge with zero production effect. See P070's "What activating this in production actually requires" for the explicit remaining steps (a scoped read-only Postgres role on server10, DATABASE_URL, ClickHouse credentials, and flipping the mode) — none of which this PR does.

Opening for review rather than self-merging: this is new production-facing data-extraction logic touching a separate product's OLTP primary, even read-only — the kind of change that should get a second look per this task's own instruction, not the mechanical/self-mergeable kind.

🤖 Generated with Claude Code

## What Implements Simon's specified pattern — `OLTP operational tables -> read-only analytical extraction -> watermark/updated_at -> 0pt ingestion -> ClickHouse` — as a new P061 connector pair plugged into the existing P062 `IngestionRunner`, writing to `opt_dataset_projection` via `SiaxAnalyticsEngine.ingestProjection`. Full design/rationale in `docs/P070_VEHIQ_OLTP_EXTRACTOR.md`. ## Why here, not `admin/vehiq` `admin/vehiq`'s own `docs/adr/0035-bi-analytics-architecture-product-firewall.md` (decisions 1/2/10, decided the same day this task was framed, CI-enforced by `tools/validate_engine_firewall.py`) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP Postgres directly for analytics — and its own `docs/CURRENT_STATE.md` says explicitly this belongs in "a future `admin/api0` connector" / 0pt's data plane. Built here instead, as a sibling of `admin/api0`, next to the existing P060-P068 ingestion plane and BI-P009/P010 ClickHouse engine. `admin/vehiq` was not touched. ## Real-schema correction found while building Verified live (read-only, `information_schema.columns`, server10 `vehiq-db`) that production has **no generic `updated_at`** on any of the three target tables, contrary to the task's assumption. `dealer_stock_entry`/`deal` use `listed_at`/`created_at` (insert-time only — documented caveat on in-place-update blindness). `app.listing` has no timestamp column at all and is **not covered** — blocked on a schema decision (migration `0084` or an `updated_at`+trigger), documented in P070 as a deferred, explicit next step, not silently worked around. ## Testing - Fixture-backed connector + runner tests: watermark/limit correctness, a same-timestamp tie at a page boundary, crash/resume from a durable checkpoint, the empty-table case, and the narrow vehicle-dimension lookup asserted at the query level (never touches an unreferenced vehicle row). - Verified against a real disposable Postgres container (seeded, then truncated to empty) — caught and fixed two real bugs (`Date` object handling) the fixtures alone could not have caught. - Verified **read-only against actual production** (server10, `vehiq-db`, via `fleet`): the extractor's exact SQL returns zero rows, no error — proving it's safe to leave scheduled with zero current effect. - `pnpm run typecheck` / `pnpm run test` (1138 passed) / `pnpm run deps:check` all green, zero new findings from these files. `pnpm run lint` has 12 pre-existing errors in files this PR does not touch (advisory gate; not from this change). ## Activation `.gitea/workflows/vehiq-oltp-ingest.yml` is present, cron-scheduled, and defaults to `VEHIQ_INGEST_MODE=preview` (read-only, never writes) with no secrets configured on this repo — safe to merge with zero production effect. See P070's "What activating this in production actually requires" for the explicit remaining steps (a scoped read-only Postgres role on server10, `DATABASE_URL`, ClickHouse credentials, and flipping the mode) — none of which this PR does. Opening for review rather than self-merging: this is new production-facing data-extraction logic touching a separate product's OLTP primary, even read-only — the kind of change that should get a second look per this task's own instruction, not the mechanical/self-mergeable kind. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

admin added 1 commit 2026-09-25 17:11:40 +00:00

feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070) ... 07158099ee

Implements Simon's specified pattern (dealer_stock_entry/deal ->
watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector
pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged
into the existing P062 IngestionRunner, writing to opt_dataset_projection
via SiaxAnalyticsEngine.ingestProjection.

Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10,
CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever
holding a ClickHouse credential or hammering OLTP directly for analytics.
See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale.

Also corrects a real schema mismatch found while building this: production
has no generic `updated_at` on any of the three target tables (verified
live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/
created_at as insert-time watermarks instead (documented caveat: in-place
updates are not caught without a schema change); app.listing has no
timestamp column at all and is deliberately not covered yet.

Vehicle dimension lookup is narrow by construction: keyed on exactly the
vehicle_id set referenced by the batch just read, never a scan of the
98M-row vehiq.vehicle table (connector-level validate() refuses that table
by name; runner-level tests assert the query only ever touches referenced
IDs).

Verified: fixture-backed connector/runner tests (watermark+limit,
same-timestamp tie at a page boundary, crash/resume, empty-table case,
narrow-lookup query-count assertions) + a real disposable Postgres
container (caught two real Date-handling bugs fixtures could not) + a
read-only proof against actual production (server10, vehiq-db): the
extractor's exact SQL returns zero rows, no error, confirming it is safe
to schedule with zero current effect.

.gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and
defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no
secrets configured — safe to merge with zero production effect until
explicitly activated per the doc's "what activating this requires".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CODEOWNERS rules requested review from sax3l 2026-09-25 17:11:40 +00:00

admin added 1 commit 2026-09-26 14:18:48 +00:00

Merge branch 'main' into feat/vehiq-oltp-watermark-extractor ff7c5cdd66

siax-bot approved these changes 2026-09-26 21:55:12 +00:00

admin added 1 commit 2026-09-26 21:55:36 +00:00

Merge branch 'main' into feat/vehiq-oltp-watermark-extractor 6b733f7b9c

admin merged commit 9f52ceb1f7 into main 2026-09-26 21:58:36 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

sax3l

siax-bot

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

siax-bot siax-ci siax-healer (siax-healer (CI self-healing agent)) admin (Simon Axelsson)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: admin/0pt#117