diff --git a/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md index 842f9df..cfdcdc9 100644 --- a/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md +++ b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md @@ -1,54 +1,56 @@ # 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan -Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN +Status: TARGET / IMPLEMENTATION CONTRACT Date: 2026-09-07 -0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record. +0pt remains the generic external-data acquisition/enrichment/provenance capability. Product domains own their business entities and decisions. 0pt supplies sourced observations/enrichment/entity-resolution inputs and must not become a second Work/Press/Fund/CRM SSOT. -## 1. Ecosystem ownership +## 1. Ecosystem uses ### Work / j0b -0pt may supply: - company/employer/end-client enrichment; - registry/company facts; -- generic source acquisition; +- source acquisition where generic 0pt infrastructure is appropriate; - buyer/supplier enrichment for procurement; -- entity-resolution candidates/evidence; +- entity-resolution inputs; - public technology/company signals for account intelligence. -j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions. +j0b owns opportunities, roles, requirements, consultants, matches and submissions. ### PRESS0 -0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions. +- public source monitoring/acquisition; +- official releases/datasets; +- source snapshots/enrichment; +- factual dataset/source metadata. + +PRESS0 owns articles/citations/claims/editorial decisions. ### FUND0 -0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY. +- issuer/company/public market facts used in project materials; +- public registry/company enrichment. -### FIT0 / CRM +FUND0 owns project/disclosure/investor/data-room state. -0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth. +### FIT0/CRM + +- account enrichment and market/buying signals; +- company identity resolution. ## 2. Provenance-first invariant -Every externally derived fact or observation must preserve enough lineage to answer: -- source/provider; -- external identifier or URL where appropriate; -- observed/fetched time; -- parser/transform version; -- freshness/confidence where meaningful; -- rights/reuse/retention classification; -- payload/artifact hash or ST0RE ref where raw retention is required. +Retain the existing source hierarchy/principles and ensure each field/observation can carry source, observed/fetched time, freshness/confidence and transformation/version metadata. -No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule. +No enrichment should silently overwrite a higher-authority canonical fact without provenance/precedence rules. ## 3. Lossless acquisition contract -When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified. +Where a product needs raw upstream retention, 0pt should return/store an immutable acquisition/artifact reference plus metadata rather than only a lossy normalized subset. ST0RE custody is preferred for large raw payloads when integrated. + +Target acquisition receipt fields include: -Target acquisition-receipt fields: - source/provider; - external ID/URL; - fetched/observed times; @@ -56,55 +58,40 @@ Target acquisition-receipt fields: - parser/version; - rights/reuse/retention classification; - data classification; -- extracted observations/provenance; -- correlation/tenant/project scope. +- extracted observations/provenance. -## 4. Canonical entity-resolution architecture +## 4. Entity resolution -There must be one decision architecture, not competing production scorers. +Add/use Splink or equivalent approved probabilistic entity-resolution capability where it materially improves cross-source linking. -### 4.1 Synchronous candidate path +Primary use cases: -The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides: -- deterministic blocking; -- explainable similarity evidence; -- product hard-rule hooks; -- review-band candidates; -- stable-ID hard match/non-match rules at the product boundary. +- Brainville company = LinkedIn-export/company = JobTech employer = registry company = TED buyer = Twenty account; +- duplicate/similar organizations across countries/registries; +- person/consultant references when legally/purpose appropriate. -For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match. +Entity resolution returns candidates/scores/evidence. Product/canonical identity merge remains policy/domain controlled; do not auto-merge high-risk records solely on fuzzy score. -### 4.2 Python ingestion path - -The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering. - -### 4.3 Heavy/statistical ER - -Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract. - -## 5. Strong company identity keys +## 5. Company identity keys Prefer strong keys where available: -- LEI (global scope); + - country + official registration number; -- VAT/official registry identifiers with their correct jurisdictional scope; -- other verified registry IDs. +- LEI; +- VAT ID; +- official registry IDs; +- verified domains. -National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier. +Use probabilistic attributes only when strong IDs are missing/inconsistent. -Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges. +## 6. Public procurement enrichment -## 6. Human review boundary +Support acquisition/normalization references for buyer/supplier/award/contract datasets such as TED/OCDS/official sources. j0b procurement domain owns the commercial opportunity/access graph; 0pt supplies sourced external facts. -Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced. - -## 7. Public procurement enrichment - -Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence. - -## 8. Source-policy metadata +## 7. Source-policy metadata Products may pass/receive policy metadata for: + - access method; - reuse/redistribution; - retention; @@ -112,19 +99,20 @@ Products may pass/receive policy metadata for: - terms/license review date; - purpose/data classification. -0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose. +0pt must not globally assume all publicly reachable web content is reusable for every commercial/public purpose. -## 9. PRESS0 rights boundary +## 8. PRESS0 rights boundary -Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision. +Source acquisition/extraction is not publication permission. PRESS0 editorial/source-rights policy decides what may be quoted/reproduced/published. 0pt must preserve enough source metadata to support this decision. -## 10. Company verification boundary +## 9. Company verification boundary -0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row. +0pt may fetch/enrich official company facts; ID0 owns verification decisions/KYB status. j0b/CRM consume verification refs/status from ID0, not infer `verified` merely because 0pt found a matching registry row. -## 11. Market/account signals +## 10. Market/account signals + +Target observations useful to FIT0/CRM/j0b: -Useful dated observations include: - hiring activity; - public procurement notices/awards; - public technology/platform signals; @@ -132,66 +120,58 @@ Useful dated observations include: - leadership/public contact changes where lawful; - relevant product/market announcements. -Signals are provenance-bearing observations with observation time and freshness, not permanent assertions. +Signals are dated observations with provenance, not permanent assertions. -## 12. API contract +## 11. API contract + +Fit into actual 0pt architecture. Product consumers should be able to request/receive: -Generic typed capabilities should cover: - company/entity enrichment; - source acquisition receipt; - structured observations with provenance; - entity-resolution candidates; -- batch ER job submission; -- provider/source health/freshness; -- durable dataset/version/search surfaces where implemented. +- provider/source health/freshness where applicable. -Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling. +Avoid bespoke endpoint copies for each product if generic typed capabilities suffice. -## 13. Security/privacy +## 12. Security/privacy -Required controls include: -- tenant/project/purpose scoping; +- tenant/purpose scoping; - PII minimisation; -- controlled egress/allowed-host policy; -- SSRF protection for user-supplied URLs; -- canonical secret storage; +- egress/allowed-host controls; +- SSRF prevention for user-supplied URLs; +- secrets in canonical secret store; - no raw sensitive payloads in logs; - retention/deletion support; -- provider/license/data-rights documentation; -- fail-closed identity merge behavior; -- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required. +- provider/license/data-rights documentation. -## 14. Production gate +## 13. Current deployment gate -Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED. +Before j0b/PRESS0/FUND0 depend on 0pt as a mission-critical production service, reconcile the repo's current deployment/live endpoint status and close any documented routing/auth/hosting gap. Do not treat architecture/code as deployed merely because it exists. -Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence. +## 14. Implementation order -## 15. Implementation order +1. Reconcile current live deployment, auth and data/provenance APIs. +2. Stabilise generic acquisition receipt/provenance contract. +3. Add Splink/entity-resolution capability with strong-key-first strategy and reviewable candidates. +4. Wire j0b company/end-client/buyer enrichment. +5. Wire FIT0/CRM account signals. +6. Add PRESS0 source/dataset acquisition when press0 exists. +7. Add FUND0 public issuer facts when fund0 exists. +8. Load/performance/backup/restore/live verify. -1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners. -2. Keep the synchronous TS ER engine as the canonical candidate/decision contract. -3. Keep Python ingestion as a fail-closed adapter, never a second scorer. -4. Complete durable acquisition/provenance/dataset-version/search contracts. -5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review. -6. Wire j0b company/end-client/buyer enrichment through typed APIs. -7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth. -8. Add PRESS0/FUND0 consumers only when their repos/contracts are real. -9. Load/performance/backup/restore/live-verify every mission-critical path. - -## 16. Definition of Done per data capability +## 15. Definition of Done per data capability - source/provider documented; - legal/license/data-use classification recorded; -- provenance/freshness/confidence available where meaningful; +- provenance/freshness/confidence available; - deterministic/idempotent normalization where expected; -- entity resolution tested against labelled truth data; -- fuzzy evidence cannot silently become identity truth; -- tenant/purpose/security negative tests pass; -- provider outage/schema drift handled fail closed or explicitly degraded; +- entity resolution tested against labelled truth set; +- tenant/purpose/security tests pass; +- provider outage/schema drift handled; - raw artifact retention/deletion contract works where required; -- exact production endpoint/dependency is live verified; -- product consumes API/contract, not another product's DB; -- status documentation reflects observed reality. +- actual production endpoint/dependency is live verified; +- product consumes API/contract, not DB; +- status docs reflect reality. -Target ecosystem uses in this document do not claim they are already wired or live. \ No newline at end of file +Target ecosystem uses in this document do not claim they are already wired. \ No newline at end of file diff --git a/services/api/tests/test_entity_resolution.py b/services/api/tests/test_entity_resolution.py new file mode 100644 index 0000000..520bc4a --- /dev/null +++ b/services/api/tests/test_entity_resolution.py @@ -0,0 +1,93 @@ +from workers.entity_resolution import ( + EntityCandidate, + ResolutionDecision, + rank_resolution_candidates, + resolve_entity_pair, +) + + +def organization(entity_id: str, **overrides): + values = { + "entity_id": entity_id, + "entity_type": "organization", + "name": "Example Technology AB", + "source_ref": f"source:{entity_id}", + "registration_country": "SE", + "registration_number": None, + "website": None, + "city": "Stockholm", + } + values.update(overrides) + return EntityCandidate(**values) + + +def person(entity_id: str, **overrides): + values = { + "entity_id": entity_id, + "entity_type": "person", + "name": "Anna Andersson", + "source_ref": f"source:{entity_id}", + "email": None, + "organization_id": None, + } + values.update(overrides) + return EntityCandidate(**values) + + +def test_registration_number_is_a_strong_organization_identifier(): + result = resolve_entity_pair( + organization("left", registration_number="556123-4567"), + organization("right", registration_number="5561234567"), + ) + assert result.decision is ResolutionDecision.EXACT_LINK + assert result.score == 1.0 + assert result.requires_human_review is False + + +def test_conflicting_registration_numbers_fail_distinct_even_when_names_match(): + result = resolve_entity_pair( + organization("left", registration_number="5561234567"), + organization("right", registration_number="5569999999"), + ) + assert result.decision is ResolutionDecision.DISTINCT + assert result.score == 0.0 + + +def test_same_person_email_is_exact_link(): + result = resolve_entity_pair( + person("left", email="Anna@example.com"), + person("right", email=" anna@EXAMPLE.com "), + ) + assert result.decision is ResolutionDecision.EXACT_LINK + assert result.requires_human_review is False + + +def test_fuzzy_organization_similarity_never_auto_merges(): + result = resolve_entity_pair( + organization("left", name="Example Technology AB", website="example.se"), + organization("right", name="Example Technologies AB", website="https://www.example.se"), + ) + assert result.score > 0.72 + assert result.decision is ResolutionDecision.REVIEW + assert result.requires_human_review is True + + +def test_different_entity_types_are_distinct(): + result = resolve_entity_pair( + organization("org"), + person("person"), + ) + assert result.decision is ResolutionDecision.DISTINCT + + +def test_candidate_ranking_prefers_exact_then_review_and_is_stable(): + query = organization("query", registration_number="5561234567", website="example.se") + candidates = [ + organization("weak", name="Completely Different Company", registration_country="NO", city="Oslo"), + organization("review", name="Example Technologies AB", website="example.se"), + organization("exact", registration_number="556123-4567"), + ] + ranked = rank_resolution_candidates(query, candidates) + assert [result.right_id for result in ranked] == ["exact", "review", "weak"] + assert ranked[0].decision is ResolutionDecision.EXACT_LINK + assert ranked[1].decision is ResolutionDecision.REVIEW diff --git a/services/api/workers/entity_resolution.py b/services/api/workers/entity_resolution.py new file mode 100644 index 0000000..4e66d3d --- /dev/null +++ b/services/api/workers/entity_resolution.py @@ -0,0 +1,217 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from typing import Literal +from urllib.parse import urlparse +import re +import unicodedata + +from rapidfuzz.fuzz import ratio + +EntityType = Literal["organization", "person"] + + +class ResolutionDecision(str, Enum): + EXACT_LINK = "exact_link" + REVIEW = "review" + DISTINCT = "distinct" + + +@dataclass(frozen=True) +class EntityCandidate: + entity_id: str + entity_type: EntityType + name: str + source_ref: str + registration_country: str | None = None + registration_number: str | None = None + email: str | None = None + website: str | None = None + organization_id: str | None = None + city: str | None = None + metadata: dict[str, object] = field(default_factory=dict) + + +@dataclass(frozen=True) +class ResolutionResult: + left_id: str + right_id: str + decision: ResolutionDecision + score: float + reasons: tuple[str, ...] + requires_human_review: bool + algorithm_version: str = "entity-resolution-v1" + + +def _norm(value: str | None) -> str: + if not value: + return "" + normalized = unicodedata.normalize("NFKD", value) + ascii_text = "".join(char for char in normalized if not unicodedata.combining(char)) + return re.sub(r"[^a-z0-9]+", " ", ascii_text.lower()).strip() + + +def _norm_registration_number(value: str | None) -> str: + return re.sub(r"[^0-9a-z]+", "", (value or "").lower()) + + +def _norm_email(value: str | None) -> str: + return (value or "").strip().lower() + + +def _domain(value: str | None) -> str: + if not value: + return "" + raw = value.strip().lower() + if "@" in raw and "://" not in raw: + raw = raw.rsplit("@", 1)[-1] + if "://" not in raw: + raw = f"https://{raw}" + host = (urlparse(raw).hostname or "").lower() + return host.removeprefix("www.") + + +def _exact_organization(left: EntityCandidate, right: EntityCandidate) -> ResolutionResult | None: + left_number = _norm_registration_number(left.registration_number) + right_number = _norm_registration_number(right.registration_number) + left_country = _norm(left.registration_country) + right_country = _norm(right.registration_country) + if left_number and right_number and left_country and right_country: + if left_country == right_country and left_number == right_number: + return ResolutionResult( + left.entity_id, + right.entity_id, + ResolutionDecision.EXACT_LINK, + 1.0, + ("same registration country and registration number",), + False, + ) + if left_country == right_country and left_number != right_number: + return ResolutionResult( + left.entity_id, + right.entity_id, + ResolutionDecision.DISTINCT, + 0.0, + ("different registration numbers in the same country",), + False, + ) + return None + + +def _exact_person(left: EntityCandidate, right: EntityCandidate) -> ResolutionResult | None: + left_email = _norm_email(left.email) + right_email = _norm_email(right.email) + if left_email and right_email: + if left_email == right_email: + return ResolutionResult( + left.entity_id, + right.entity_id, + ResolutionDecision.EXACT_LINK, + 1.0, + ("same normalized email address",), + False, + ) + return None + + +def resolve_entity_pair(left: EntityCandidate, right: EntityCandidate) -> ResolutionResult: + if left.entity_type != right.entity_type: + return ResolutionResult( + left.entity_id, + right.entity_id, + ResolutionDecision.DISTINCT, + 0.0, + ("different entity types",), + False, + ) + + exact = _exact_organization(left, right) if left.entity_type == "organization" else _exact_person(left, right) + if exact is not None: + return exact + + left_name = _norm(left.name) + right_name = _norm(right.name) + if not left_name or not right_name: + return ResolutionResult( + left.entity_id, + right.entity_id, + ResolutionDecision.REVIEW, + 0.0, + ("missing normalized name prevents deterministic comparison",), + True, + ) + + name_similarity = ratio(left_name, right_name) / 100.0 + reasons = [f"name similarity {name_similarity:.3f}"] + score = name_similarity * 0.65 + + if left.entity_type == "organization": + left_domain = _domain(left.website or left.email) + right_domain = _domain(right.website or right.email) + if left_domain and right_domain: + if left_domain == right_domain: + score += 0.25 + reasons.append("same normalized domain") + else: + score -= 0.1 + reasons.append("different normalized domains") + if _norm(left.registration_country) and _norm(left.registration_country) == _norm(right.registration_country): + score += 0.05 + reasons.append("same registration country") + if _norm(left.city) and _norm(left.city) == _norm(right.city): + score += 0.05 + reasons.append("same city") + else: + if left.organization_id and right.organization_id: + if left.organization_id == right.organization_id: + score += 0.25 + reasons.append("same known organization") + else: + score -= 0.1 + reasons.append("different known organizations") + left_domain = _domain(left.email) + right_domain = _domain(right.email) + if left_domain and right_domain and left_domain == right_domain: + score += 0.1 + reasons.append("same email domain") + + score = max(0.0, min(1.0, score)) + + # Only strong identifiers auto-link. Similarity-based matches remain operator-review candidates. + if score >= 0.72: + decision = ResolutionDecision.REVIEW + review = True + elif score <= 0.3: + decision = ResolutionDecision.DISTINCT + review = False + else: + decision = ResolutionDecision.REVIEW + review = True + + return ResolutionResult( + left.entity_id, + right.entity_id, + decision, + round(score, 6), + tuple(reasons), + review, + ) + + +def rank_resolution_candidates( + query: EntityCandidate, + candidates: list[EntityCandidate], + *, + limit: int = 20, +) -> list[ResolutionResult]: + results = [resolve_entity_pair(query, candidate) for candidate in candidates if candidate.entity_id != query.entity_id] + decision_order = { + ResolutionDecision.EXACT_LINK: 0, + ResolutionDecision.REVIEW: 1, + ResolutionDecision.DISTINCT: 2, + } + return sorted( + results, + key=lambda result: (decision_order[result.decision], -result.score, result.right_id), + )[: max(0, limit)]