docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md

@@ -1,54 +1,56 @@
 # 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan
 
-Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN  
+Status: TARGET / IMPLEMENTATION CONTRACT  
 Date: 2026-09-07
 
-0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record.
+0pt remains the generic external-data acquisition/enrichment/provenance capability. Product domains own their business entities and decisions. 0pt supplies sourced observations/enrichment/entity-resolution inputs and must not become a second Work/Press/Fund/CRM SSOT.
 
-## 1. Ecosystem ownership
+## 1. Ecosystem uses
 
 ### Work / j0b
 
-0pt may supply:
 - company/employer/end-client enrichment;
 - registry/company facts;
-- generic source acquisition;
+- source acquisition where generic 0pt infrastructure is appropriate;
 - buyer/supplier enrichment for procurement;
-- entity-resolution candidates/evidence;
+- entity-resolution inputs;
 - public technology/company signals for account intelligence.
 
-j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions.
+j0b owns opportunities, roles, requirements, consultants, matches and submissions.
 
 ### PRESS0
 
-0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions.
+- public source monitoring/acquisition;
+- official releases/datasets;
+- source snapshots/enrichment;
+- factual dataset/source metadata.
+
+PRESS0 owns articles/citations/claims/editorial decisions.
 
 ### FUND0
 
-0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY.
+- issuer/company/public market facts used in project materials;
+- public registry/company enrichment.
 
-### FIT0 / CRM
+FUND0 owns project/disclosure/investor/data-room state.
 
-0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth.
+### FIT0/CRM
+
+- account enrichment and market/buying signals;
+- company identity resolution.
 
 ## 2. Provenance-first invariant
 
-Every externally derived fact or observation must preserve enough lineage to answer:
-- source/provider;
-- external identifier or URL where appropriate;
-- observed/fetched time;
-- parser/transform version;
-- freshness/confidence where meaningful;
-- rights/reuse/retention classification;
-- payload/artifact hash or ST0RE ref where raw retention is required.
+Retain the existing source hierarchy/principles and ensure each field/observation can carry source, observed/fetched time, freshness/confidence and transformation/version metadata.
 
-No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule.
+No enrichment should silently overwrite a higher-authority canonical fact without provenance/precedence rules.
 
 ## 3. Lossless acquisition contract
 
-When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified.
+Where a product needs raw upstream retention, 0pt should return/store an immutable acquisition/artifact reference plus metadata rather than only a lossy normalized subset. ST0RE custody is preferred for large raw payloads when integrated.
+
+Target acquisition receipt fields include:
 
-Target acquisition-receipt fields:
 - source/provider;
 - external ID/URL;
 - fetched/observed times;
@@ -56,55 +58,40 @@ Target acquisition-receipt fields:
 - parser/version;
 - rights/reuse/retention classification;
 - data classification;
-- extracted observations/provenance;
-- correlation/tenant/project scope.
+- extracted observations/provenance.
 
-## 4. Canonical entity-resolution architecture
+## 4. Entity resolution
 
-There must be one decision architecture, not competing production scorers.
+Add/use Splink or equivalent approved probabilistic entity-resolution capability where it materially improves cross-source linking.
 
-### 4.1 Synchronous candidate path
+Primary use cases:
 
-The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides:
-- deterministic blocking;
-- explainable similarity evidence;
-- product hard-rule hooks;
-- review-band candidates;
-- stable-ID hard match/non-match rules at the product boundary.
+- Brainville company = LinkedIn-export/company = JobTech employer = registry company = TED buyer = Twenty account;
+- duplicate/similar organizations across countries/registries;
+- person/consultant references when legally/purpose appropriate.
 
-For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match.
+Entity resolution returns candidates/scores/evidence. Product/canonical identity merge remains policy/domain controlled; do not auto-merge high-risk records solely on fuzzy score.
 
-### 4.2 Python ingestion path
-
-The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering.
-
-### 4.3 Heavy/statistical ER
-
-Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract.
-
-## 5. Strong company identity keys
+## 5. Company identity keys
 
 Prefer strong keys where available:
-- LEI (global scope);
+
 - country + official registration number;
-- VAT/official registry identifiers with their correct jurisdictional scope;
-- other verified registry IDs.
+- LEI;
+- VAT ID;
+- official registry IDs;
+- verified domains.
 
-National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier.
+Use probabilistic attributes only when strong IDs are missing/inconsistent.
 
-Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges.
+## 6. Public procurement enrichment
 
-## 6. Human review boundary
+Support acquisition/normalization references for buyer/supplier/award/contract datasets such as TED/OCDS/official sources. j0b procurement domain owns the commercial opportunity/access graph; 0pt supplies sourced external facts.
 
-Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced.
-
-## 7. Public procurement enrichment
-
-Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence.
-
-## 8. Source-policy metadata
+## 7. Source-policy metadata
 
 Products may pass/receive policy metadata for:
+
 - access method;
 - reuse/redistribution;
 - retention;
@@ -112,19 +99,20 @@ Products may pass/receive policy metadata for:
 - terms/license review date;
 - purpose/data classification.
 
-0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose.
+0pt must not globally assume all publicly reachable web content is reusable for every commercial/public purpose.
 
-## 9. PRESS0 rights boundary
+## 8. PRESS0 rights boundary
 
-Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision.
+Source acquisition/extraction is not publication permission. PRESS0 editorial/source-rights policy decides what may be quoted/reproduced/published. 0pt must preserve enough source metadata to support this decision.
 
-## 10. Company verification boundary
+## 9. Company verification boundary
 
-0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row.
+0pt may fetch/enrich official company facts; ID0 owns verification decisions/KYB status. j0b/CRM consume verification refs/status from ID0, not infer `verified` merely because 0pt found a matching registry row.
 
-## 11. Market/account signals
+## 10. Market/account signals
+
+Target observations useful to FIT0/CRM/j0b:
 
-Useful dated observations include:
 - hiring activity;
 - public procurement notices/awards;
 - public technology/platform signals;
@@ -132,66 +120,58 @@ Useful dated observations include:
 - leadership/public contact changes where lawful;
 - relevant product/market announcements.
 
-Signals are provenance-bearing observations with observation time and freshness, not permanent assertions.
+Signals are dated observations with provenance, not permanent assertions.
 
-## 12. API contract
+## 11. API contract
+
+Fit into actual 0pt architecture. Product consumers should be able to request/receive:
 
-Generic typed capabilities should cover:
 - company/entity enrichment;
 - source acquisition receipt;
 - structured observations with provenance;
 - entity-resolution candidates;
-- batch ER job submission;
-- provider/source health/freshness;
-- durable dataset/version/search surfaces where implemented.
+- provider/source health/freshness where applicable.
 
-Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling.
+Avoid bespoke endpoint copies for each product if generic typed capabilities suffice.
 
-## 13. Security/privacy
+## 12. Security/privacy
 
-Required controls include:
-- tenant/project/purpose scoping;
+- tenant/purpose scoping;
 - PII minimisation;
-- controlled egress/allowed-host policy;
-- SSRF protection for user-supplied URLs;
-- canonical secret storage;
+- egress/allowed-host controls;
+- SSRF prevention for user-supplied URLs;
+- secrets in canonical secret store;
 - no raw sensitive payloads in logs;
 - retention/deletion support;
-- provider/license/data-rights documentation;
-- fail-closed identity merge behavior;
-- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required.
+- provider/license/data-rights documentation.
 
-## 14. Production gate
+## 13. Current deployment gate
 
-Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED.
+Before j0b/PRESS0/FUND0 depend on 0pt as a mission-critical production service, reconcile the repo's current deployment/live endpoint status and close any documented routing/auth/hosting gap. Do not treat architecture/code as deployed merely because it exists.
 
-Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence.
+## 14. Implementation order
 
-## 15. Implementation order
+1. Reconcile current live deployment, auth and data/provenance APIs.
+2. Stabilise generic acquisition receipt/provenance contract.
+3. Add Splink/entity-resolution capability with strong-key-first strategy and reviewable candidates.
+4. Wire j0b company/end-client/buyer enrichment.
+5. Wire FIT0/CRM account signals.
+6. Add PRESS0 source/dataset acquisition when press0 exists.
+7. Add FUND0 public issuer facts when fund0 exists.
+8. Load/performance/backup/restore/live verify.
 
-1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners.
-2. Keep the synchronous TS ER engine as the canonical candidate/decision contract.
-3. Keep Python ingestion as a fail-closed adapter, never a second scorer.
-4. Complete durable acquisition/provenance/dataset-version/search contracts.
-5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review.
-6. Wire j0b company/end-client/buyer enrichment through typed APIs.
-7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth.
-8. Add PRESS0/FUND0 consumers only when their repos/contracts are real.
-9. Load/performance/backup/restore/live-verify every mission-critical path.
-
-## 16. Definition of Done per data capability
+## 15. Definition of Done per data capability
 
 - source/provider documented;
 - legal/license/data-use classification recorded;
-- provenance/freshness/confidence available where meaningful;
+- provenance/freshness/confidence available;
 - deterministic/idempotent normalization where expected;
-- entity resolution tested against labelled truth data;
-- fuzzy evidence cannot silently become identity truth;
-- tenant/purpose/security negative tests pass;
-- provider outage/schema drift handled fail closed or explicitly degraded;
+- entity resolution tested against labelled truth set;
+- tenant/purpose/security tests pass;
+- provider outage/schema drift handled;
 - raw artifact retention/deletion contract works where required;
-- exact production endpoint/dependency is live verified;
-- product consumes API/contract, not another product's DB;
-- status documentation reflects observed reality.
+- actual production endpoint/dependency is live verified;
+- product consumes API/contract, not DB;
+- status docs reflect reality.
 
-Target ecosystem uses in this document do not claim they are already wired or live.
+Target ecosystem uses in this document do not claim they are already wired.

services/api/tests/test_entity_resolution.py

@@ -0,0 +1,93 @@
+from workers.entity_resolution import (
+    EntityCandidate,
+    ResolutionDecision,
+    rank_resolution_candidates,
+    resolve_entity_pair,
+)
+
+
+def organization(entity_id: str, **overrides):
+    values = {
+        "entity_id": entity_id,
+        "entity_type": "organization",
+        "name": "Example Technology AB",
+        "source_ref": f"source:{entity_id}",
+        "registration_country": "SE",
+        "registration_number": None,
+        "website": None,
+        "city": "Stockholm",
+    }
+    values.update(overrides)
+    return EntityCandidate(**values)
+
+
+def person(entity_id: str, **overrides):
+    values = {
+        "entity_id": entity_id,
+        "entity_type": "person",
+        "name": "Anna Andersson",
+        "source_ref": f"source:{entity_id}",
+        "email": None,
+        "organization_id": None,
+    }
+    values.update(overrides)
+    return EntityCandidate(**values)
+
+
+def test_registration_number_is_a_strong_organization_identifier():
+    result = resolve_entity_pair(
+        organization("left", registration_number="556123-4567"),
+        organization("right", registration_number="5561234567"),
+    )
+    assert result.decision is ResolutionDecision.EXACT_LINK
+    assert result.score == 1.0
+    assert result.requires_human_review is False
+
+
+def test_conflicting_registration_numbers_fail_distinct_even_when_names_match():
+    result = resolve_entity_pair(
+        organization("left", registration_number="5561234567"),
+        organization("right", registration_number="5569999999"),
+    )
+    assert result.decision is ResolutionDecision.DISTINCT
+    assert result.score == 0.0
+
+
+def test_same_person_email_is_exact_link():
+    result = resolve_entity_pair(
+        person("left", email="Anna@example.com"),
+        person("right", email=" anna@EXAMPLE.com "),
+    )
+    assert result.decision is ResolutionDecision.EXACT_LINK
+    assert result.requires_human_review is False
+
+
+def test_fuzzy_organization_similarity_never_auto_merges():
+    result = resolve_entity_pair(
+        organization("left", name="Example Technology AB", website="example.se"),
+        organization("right", name="Example Technologies AB", website="https://www.example.se"),
+    )
+    assert result.score > 0.72
+    assert result.decision is ResolutionDecision.REVIEW
+    assert result.requires_human_review is True
+
+
+def test_different_entity_types_are_distinct():
+    result = resolve_entity_pair(
+        organization("org"),
+        person("person"),
+    )
+    assert result.decision is ResolutionDecision.DISTINCT
+
+
+def test_candidate_ranking_prefers_exact_then_review_and_is_stable():
+    query = organization("query", registration_number="5561234567", website="example.se")
+    candidates = [
+        organization("weak", name="Completely Different Company", registration_country="NO", city="Oslo"),
+        organization("review", name="Example Technologies AB", website="example.se"),
+        organization("exact", registration_number="556123-4567"),
+    ]
+    ranked = rank_resolution_candidates(query, candidates)
+    assert [result.right_id for result in ranked] == ["exact", "review", "weak"]
+    assert ranked[0].decision is ResolutionDecision.EXACT_LINK
+    assert ranked[1].decision is ResolutionDecision.REVIEW

services/api/workers/entity_resolution.py

@@ -0,0 +1,217 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Literal
+from urllib.parse import urlparse
+import re
+import unicodedata
+
+from rapidfuzz.fuzz import ratio
+
+EntityType = Literal["organization", "person"]
+
+
+class ResolutionDecision(str, Enum):
+    EXACT_LINK = "exact_link"
+    REVIEW = "review"
+    DISTINCT = "distinct"
+
+
+@dataclass(frozen=True)
+class EntityCandidate:
+    entity_id: str
+    entity_type: EntityType
+    name: str
+    source_ref: str
+    registration_country: str | None = None
+    registration_number: str | None = None
+    email: str | None = None
+    website: str | None = None
+    organization_id: str | None = None
+    city: str | None = None
+    metadata: dict[str, object] = field(default_factory=dict)
+
+
+@dataclass(frozen=True)
+class ResolutionResult:
+    left_id: str
+    right_id: str
+    decision: ResolutionDecision
+    score: float
+    reasons: tuple[str, ...]
+    requires_human_review: bool
+    algorithm_version: str = "entity-resolution-v1"
+
+
+def _norm(value: str | None) -> str:
+    if not value:
+        return ""
+    normalized = unicodedata.normalize("NFKD", value)
+    ascii_text = "".join(char for char in normalized if not unicodedata.combining(char))
+    return re.sub(r"[^a-z0-9]+", " ", ascii_text.lower()).strip()
+
+
+def _norm_registration_number(value: str | None) -> str:
+    return re.sub(r"[^0-9a-z]+", "", (value or "").lower())
+
+
+def _norm_email(value: str | None) -> str:
+    return (value or "").strip().lower()
+
+
+def _domain(value: str | None) -> str:
+    if not value:
+        return ""
+    raw = value.strip().lower()
+    if "@" in raw and "://" not in raw:
+        raw = raw.rsplit("@", 1)[-1]
+    if "://" not in raw:
+        raw = f"https://{raw}"
+    host = (urlparse(raw).hostname or "").lower()
+    return host.removeprefix("www.")
+
+
+def _exact_organization(left: EntityCandidate, right: EntityCandidate) -> ResolutionResult | None:
+    left_number = _norm_registration_number(left.registration_number)
+    right_number = _norm_registration_number(right.registration_number)
+    left_country = _norm(left.registration_country)
+    right_country = _norm(right.registration_country)
+    if left_number and right_number and left_country and right_country:
+        if left_country == right_country and left_number == right_number:
+            return ResolutionResult(
+                left.entity_id,
+                right.entity_id,
+                ResolutionDecision.EXACT_LINK,
+                1.0,
+                ("same registration country and registration number",),
+                False,
+            )
+        if left_country == right_country and left_number != right_number:
+            return ResolutionResult(
+                left.entity_id,
+                right.entity_id,
+                ResolutionDecision.DISTINCT,
+                0.0,
+                ("different registration numbers in the same country",),
+                False,
+            )
+    return None
+
+
+def _exact_person(left: EntityCandidate, right: EntityCandidate) -> ResolutionResult | None:
+    left_email = _norm_email(left.email)
+    right_email = _norm_email(right.email)
+    if left_email and right_email:
+        if left_email == right_email:
+            return ResolutionResult(
+                left.entity_id,
+                right.entity_id,
+                ResolutionDecision.EXACT_LINK,
+                1.0,
+                ("same normalized email address",),
+                False,
+            )
+    return None
+
+
+def resolve_entity_pair(left: EntityCandidate, right: EntityCandidate) -> ResolutionResult:
+    if left.entity_type != right.entity_type:
+        return ResolutionResult(
+            left.entity_id,
+            right.entity_id,
+            ResolutionDecision.DISTINCT,
+            0.0,
+            ("different entity types",),
+            False,
+        )
+
+    exact = _exact_organization(left, right) if left.entity_type == "organization" else _exact_person(left, right)
+    if exact is not None:
+        return exact
+
+    left_name = _norm(left.name)
+    right_name = _norm(right.name)
+    if not left_name or not right_name:
+        return ResolutionResult(
+            left.entity_id,
+            right.entity_id,
+            ResolutionDecision.REVIEW,
+            0.0,
+            ("missing normalized name prevents deterministic comparison",),
+            True,
+        )
+
+    name_similarity = ratio(left_name, right_name) / 100.0
+    reasons = [f"name similarity {name_similarity:.3f}"]
+    score = name_similarity * 0.65
+
+    if left.entity_type == "organization":
+        left_domain = _domain(left.website or left.email)
+        right_domain = _domain(right.website or right.email)
+        if left_domain and right_domain:
+            if left_domain == right_domain:
+                score += 0.25
+                reasons.append("same normalized domain")
+            else:
+                score -= 0.1
+                reasons.append("different normalized domains")
+        if _norm(left.registration_country) and _norm(left.registration_country) == _norm(right.registration_country):
+            score += 0.05
+            reasons.append("same registration country")
+        if _norm(left.city) and _norm(left.city) == _norm(right.city):
+            score += 0.05
+            reasons.append("same city")
+    else:
+        if left.organization_id and right.organization_id:
+            if left.organization_id == right.organization_id:
+                score += 0.25
+                reasons.append("same known organization")
+            else:
+                score -= 0.1
+                reasons.append("different known organizations")
+        left_domain = _domain(left.email)
+        right_domain = _domain(right.email)
+        if left_domain and right_domain and left_domain == right_domain:
+            score += 0.1
+            reasons.append("same email domain")
+
+    score = max(0.0, min(1.0, score))
+
+    # Only strong identifiers auto-link. Similarity-based matches remain operator-review candidates.
+    if score >= 0.72:
+        decision = ResolutionDecision.REVIEW
+        review = True
+    elif score <= 0.3:
+        decision = ResolutionDecision.DISTINCT
+        review = False
+    else:
+        decision = ResolutionDecision.REVIEW
+        review = True
+
+    return ResolutionResult(
+        left.entity_id,
+        right.entity_id,
+        decision,
+        round(score, 6),
+        tuple(reasons),
+        review,
+    )
+
+
+def rank_resolution_candidates(
+    query: EntityCandidate,
+    candidates: list[EntityCandidate],
+    *,
+    limit: int = 20,
+) -> list[ResolutionResult]:
+    results = [resolve_entity_pair(query, candidate) for candidate in candidates if candidate.entity_id != query.entity_id]
+    decision_order = {
+        ResolutionDecision.EXACT_LINK: 0,
+        ResolutionDecision.REVIEW: 1,
+        ResolutionDecision.DISTINCT: 2,
+    }
+    return sorted(
+        results,
+        key=lambda result: (decision_order[result.decision], -result.score, result.right_id),
+    )[: max(0, limit)]