Simon AxelssonandClaude Sonnet 5
07b13bd833
Replaces the in-memory (per-instance, not even per-process-shared) Maps in SourceService/SourceRegistryService with real Postgres persistence, and adds the register/query/describe/health/rights/provenance/freshness HTTP routes that never existed (grep for SourceRegistryService in app/ previously returned nothing). - sql/002_source_registry.sql: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, opt_source_freshness. Numbered 002 (not 006) because 004's opt_source_rate_policies already forward-references opt_sources, which no sql/ file defined until now — fixes 004's pre-existing broken migration ordering (opt_datasets remains a separate, unrelated, still-open gap in 004, flagged not fixed). - src/repositories/*: SourceRepository / SourceRegistryRepository seams, each with an in-memory implementation (business logic extracted verbatim) and a Postgres implementation, chosen via resolveStoreMode — production still fails hard with no DATABASE_URL (Master-DoD §3), but a configured production now actually works instead of always being blocked. - SourceService / SourceRegistryService: storage moved behind the repository seam; getHealth/getFreshness are now async and tenant-scoped at the accessor itself (needed once they're reachable directly over HTTP, not only after an earlier caller's tenant check). - src/services/source-registry.ts: shared singleton instances for the sources/* route group, fixing the "each route module owned its own private registry" defect (source-rate-policies and .../query previously couldn't see a source registered via POST /api/0pt/sources). - New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id], GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health, GET .../freshness. - Business logic unchanged: fail-closed purpose gate, cross-tenant NOT_FOUND (no existence oracle, §50), bounded 100-row health history. Tested against a real Postgres container (not mocked) — tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249 pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the full HTTP route surface end-to-end against next dev + live Postgres. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
63 lines
2.9 KiB
SQL
63 lines
2.9 KiB
SQL
-- 0pt Iceberg write-path ownership repair (docs/ICEBERG_WRITE_BOUNDARY.md)
|
|
-- Two tables: a REFERENCE from a 0pt dataset to its Iceberg table in ST0RE's
|
|
-- table registry, and 0pt's own source-rate/crawl-compliance policy (moved
|
|
-- out of N0D per ADR-0030's audit -- see docs/10-lakehouse-decision.md).
|
|
--
|
|
-- opt_dataset_iceberg_mappings.iceberg_table_id is intentionally NOT a
|
|
-- foreign key -- it references ST0RE's st0re_iceberg_tables.table_id, a
|
|
-- different service's database. 0pt holds only the reference, never a copy
|
|
-- of ST0RE's table/catalog/snapshot state (see the module doc comment on
|
|
-- src/clients/st0re-lakehouse-client.ts).
|
|
--
|
|
-- Ordering note (added by the Source Registry Postgres-persistence pass):
|
|
-- opt_source_rate_policies below has always forward-referenced
|
|
-- `opt_sources (source_id)`, a table no sql/ file defined until
|
|
-- sql/002_source_registry.sql landed -- applying this file to a fresh
|
|
-- database before 002 would fail on that FK. 002 is now the one that must
|
|
-- run first (001 -> 002 -> 004 -> 005). Flagging, not fixing here: this
|
|
-- file's OTHER forward reference, `opt_dataset_iceberg_mappings.dataset_id
|
|
-- REFERENCES opt_datasets (dataset_id)` below, has the identical problem --
|
|
-- no sql/ file defines opt_datasets either -- and remains open; it is a
|
|
-- dataset-registry gap, unrelated to and out of scope for the Source
|
|
-- Registry work that added 002.
|
|
|
|
CREATE TABLE IF NOT EXISTS opt_dataset_iceberg_mappings (
|
|
dataset_iceberg_mapping_id TEXT PRIMARY KEY,
|
|
tenant_id TEXT NOT NULL,
|
|
project_id TEXT NOT NULL,
|
|
dataset_id TEXT NOT NULL REFERENCES opt_datasets (dataset_id),
|
|
iceberg_table_id TEXT NOT NULL,
|
|
namespace TEXT,
|
|
table_name TEXT,
|
|
mapping_status TEXT NOT NULL DEFAULT 'active',
|
|
mapped_by TEXT,
|
|
correlation_id TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
updated_at TIMESTAMPTZ
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS opt_dataset_iceberg_mappings_dataset_idx ON opt_dataset_iceberg_mappings (dataset_id);
|
|
CREATE INDEX IF NOT EXISTS opt_dataset_iceberg_mappings_tenant_project_idx ON opt_dataset_iceberg_mappings (tenant_id, project_id);
|
|
CREATE INDEX IF NOT EXISTS opt_dataset_iceberg_mappings_table_idx ON opt_dataset_iceberg_mappings (iceberg_table_id);
|
|
|
|
CREATE TABLE IF NOT EXISTS opt_source_rate_policies (
|
|
source_rate_policy_id TEXT PRIMARY KEY,
|
|
tenant_id TEXT NOT NULL,
|
|
project_id TEXT NOT NULL,
|
|
source_id TEXT NOT NULL REFERENCES opt_sources (source_id),
|
|
rate_limit_per_minute NUMERIC NOT NULL,
|
|
crawl_budget NUMERIC NOT NULL,
|
|
compliance_notes TEXT,
|
|
domain TEXT,
|
|
robots_ok BOOLEAN,
|
|
access TEXT,
|
|
status TEXT NOT NULL DEFAULT 'active',
|
|
correlation_id TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
updated_at TIMESTAMPTZ,
|
|
created_by TEXT
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS opt_source_rate_policies_source_idx ON opt_source_rate_policies (source_id);
|
|
CREATE INDEX IF NOT EXISTS opt_source_rate_policies_tenant_project_idx ON opt_source_rate_policies (tenant_id, project_id);
|