adminandClaude Sonnet 5
35b6ec63bf
Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
51 lines
2.2 KiB
SQL
51 lines
2.2 KiB
SQL
-- 0pt Iceberg write-path ownership repair (docs/ICEBERG_WRITE_BOUNDARY.md)
|
|
-- Two tables: a REFERENCE from a 0pt dataset to its Iceberg table in ST0RE's
|
|
-- table registry, and 0pt's own source-rate/crawl-compliance policy (moved
|
|
-- out of N0D per ADR-0030's audit -- see docs/10-lakehouse-decision.md).
|
|
--
|
|
-- opt_dataset_iceberg_mappings.iceberg_table_id is intentionally NOT a
|
|
-- foreign key -- it references ST0RE's st0re_iceberg_tables.table_id, a
|
|
-- different service's database. 0pt holds only the reference, never a copy
|
|
-- of ST0RE's table/catalog/snapshot state (see the module doc comment on
|
|
-- src/clients/st0re-lakehouse-client.ts).
|
|
|
|
CREATE TABLE IF NOT EXISTS opt_dataset_iceberg_mappings (
|
|
dataset_iceberg_mapping_id TEXT PRIMARY KEY,
|
|
tenant_id TEXT NOT NULL,
|
|
project_id TEXT NOT NULL,
|
|
dataset_id TEXT NOT NULL REFERENCES opt_datasets (dataset_id),
|
|
iceberg_table_id TEXT NOT NULL,
|
|
namespace TEXT,
|
|
table_name TEXT,
|
|
mapping_status TEXT NOT NULL DEFAULT 'active',
|
|
mapped_by TEXT,
|
|
correlation_id TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
updated_at TIMESTAMPTZ
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS opt_dataset_iceberg_mappings_dataset_idx ON opt_dataset_iceberg_mappings (dataset_id);
|
|
CREATE INDEX IF NOT EXISTS opt_dataset_iceberg_mappings_tenant_project_idx ON opt_dataset_iceberg_mappings (tenant_id, project_id);
|
|
CREATE INDEX IF NOT EXISTS opt_dataset_iceberg_mappings_table_idx ON opt_dataset_iceberg_mappings (iceberg_table_id);
|
|
|
|
CREATE TABLE IF NOT EXISTS opt_source_rate_policies (
|
|
source_rate_policy_id TEXT PRIMARY KEY,
|
|
tenant_id TEXT NOT NULL,
|
|
project_id TEXT NOT NULL,
|
|
source_id TEXT NOT NULL REFERENCES opt_sources (source_id),
|
|
rate_limit_per_minute NUMERIC NOT NULL,
|
|
crawl_budget NUMERIC NOT NULL,
|
|
compliance_notes TEXT,
|
|
domain TEXT,
|
|
robots_ok BOOLEAN,
|
|
access TEXT,
|
|
status TEXT NOT NULL DEFAULT 'active',
|
|
correlation_id TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
updated_at TIMESTAMPTZ,
|
|
created_by TEXT
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS opt_source_rate_policies_source_idx ON opt_source_rate_policies (source_id);
|
|
CREATE INDEX IF NOT EXISTS opt_source_rate_policies_tenant_project_idx ON opt_source_rate_policies (tenant_id, project_id);
|