docs(er): reconcile ecosystem acquisition plan with canonical ER architecture

2026-09-07 18:22:26 +02:00
parent 93c01d2264
commit e2437ab09e
1 changed files with 197 additions and 0 deletions
@@ -0,0 +1,197 @@
+# 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan
+
+Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN  
+Date: 2026-09-07
+
+0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record.
+
+## 1. Ecosystem ownership
+
+### Work / j0b
+
+0pt may supply:
+- company/employer/end-client enrichment;
+- registry/company facts;
+- generic source acquisition;
+- buyer/supplier enrichment for procurement;
+- entity-resolution candidates/evidence;
+- public technology/company signals for account intelligence.
+
+j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions.
+
+### PRESS0
+
+0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions.
+
+### FUND0
+
+0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY.
+
+### FIT0 / CRM
+
+0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth.
+
+## 2. Provenance-first invariant
+
+Every externally derived fact or observation must preserve enough lineage to answer:
+- source/provider;
+- external identifier or URL where appropriate;
+- observed/fetched time;
+- parser/transform version;
+- freshness/confidence where meaningful;
+- rights/reuse/retention classification;
+- payload/artifact hash or ST0RE ref where raw retention is required.
+
+No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule.
+
+## 3. Lossless acquisition contract
+
+When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified.
+
+Target acquisition-receipt fields:
+- source/provider;
+- external ID/URL;
+- fetched/observed times;
+- payload/artifact hash/ref;
+- parser/version;
+- rights/reuse/retention classification;
+- data classification;
+- extracted observations/provenance;
+- correlation/tenant/project scope.
+
+## 4. Canonical entity-resolution architecture
+
+There must be one decision architecture, not competing production scorers.
+
+### 4.1 Synchronous candidate path
+
+The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides:
+- deterministic blocking;
+- explainable similarity evidence;
+- product hard-rule hooks;
+- review-band candidates;
+- stable-ID hard match/non-match rules at the product boundary.
+
+For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match.
+
+### 4.2 Python ingestion path
+
+The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering.
+
+### 4.3 Heavy/statistical ER
+
+Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract.
+
+## 5. Strong company identity keys
+
+Prefer strong keys where available:
+- LEI (global scope);
+- country + official registration number;
+- VAT/official registry identifiers with their correct jurisdictional scope;
+- other verified registry IDs.
+
+National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier.
+
+Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges.
+
+## 6. Human review boundary
+
+Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced.
+
+## 7. Public procurement enrichment
+
+Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence.
+
+## 8. Source-policy metadata
+
+Products may pass/receive policy metadata for:
+- access method;
+- reuse/redistribution;
+- retention;
+- polling/rate limits;
+- terms/license review date;
+- purpose/data classification.
+
+0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose.
+
+## 9. PRESS0 rights boundary
+
+Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision.
+
+## 10. Company verification boundary
+
+0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row.
+
+## 11. Market/account signals
+
+Useful dated observations include:
+- hiring activity;
+- public procurement notices/awards;
+- public technology/platform signals;
+- company growth/change events;
+- leadership/public contact changes where lawful;
+- relevant product/market announcements.
+
+Signals are provenance-bearing observations with observation time and freshness, not permanent assertions.
+
+## 12. API contract
+
+Generic typed capabilities should cover:
+- company/entity enrichment;
+- source acquisition receipt;
+- structured observations with provenance;
+- entity-resolution candidates;
+- batch ER job submission;
+- provider/source health/freshness;
+- durable dataset/version/search surfaces where implemented.
+
+Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling.
+
+## 13. Security/privacy
+
+Required controls include:
+- tenant/project/purpose scoping;
+- PII minimisation;
+- controlled egress/allowed-host policy;
+- SSRF protection for user-supplied URLs;
+- canonical secret storage;
+- no raw sensitive payloads in logs;
+- retention/deletion support;
+- provider/license/data-rights documentation;
+- fail-closed identity merge behavior;
+- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required.
+
+## 14. Production gate
+
+Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED.
+
+Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence.
+
+## 15. Implementation order
+
+1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners.
+2. Keep the synchronous TS ER engine as the canonical candidate/decision contract.
+3. Keep Python ingestion as a fail-closed adapter, never a second scorer.
+4. Complete durable acquisition/provenance/dataset-version/search contracts.
+5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review.
+6. Wire j0b company/end-client/buyer enrichment through typed APIs.
+7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth.
+8. Add PRESS0/FUND0 consumers only when their repos/contracts are real.
+9. Load/performance/backup/restore/live-verify every mission-critical path.
+
+## 16. Definition of Done per data capability
+
+- source/provider documented;
+- legal/license/data-use classification recorded;
+- provenance/freshness/confidence available where meaningful;
+- deterministic/idempotent normalization where expected;
+- entity resolution tested against labelled truth data;
+- fuzzy evidence cannot silently become identity truth;
+- tenant/purpose/security negative tests pass;
+- provider outage/schema drift handled fail closed or explicitly degraded;
+- raw artifact retention/deletion contract works where required;
+- exact production endpoint/dependency is live verified;
+- product consumes API/contract, not another product's DB;
+- status documentation reflects observed reality.
+
+Target ecosystem uses in this document do not claim they are already wired or live.