feat(0pt): PII-Gate GDPR-guardrail for personal-data sources (fail-closed)
PiiGateService makes any future personal-data source (ratsit.se and other directory domains) technically GDPR-OK before the first byte is stored: - recorded legal basis (art. 6) with basis_ref evidence, no policy => fail-closed - data minimisation via field allowlist (art. 5(1)(c)) - storage limitation + enforceRetention purge (art. 5(1)(e)) - sha256-only identity keys + log redaction (art. 25/32) - accessStatement / erase subject-rights surfaces (art. 15/17) - robots/rate confirmation required for web+api sources; no stealth/bypass - audit trail via AUD0 at risk L3 (0pt.pii_*) - gate enabled by default in SourceQueryService 10 new tests; full suite 166 pass | 1 skip; typecheck (blocking gate) clean.
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# PII Guardrail — GDPR requirement for personal-data sources (0pt)
|
||||
|
||||
Status: **enforced, fail-closed.** No 0pt pipeline is crawling personuppgifter
|
||||
today; the `PiiGateService` ships so that any future source that touches
|
||||
personal data (Ratsit, Merinfo, Hitta, …) is **technically GDPR-OK before the
|
||||
first byte is stored**.
|
||||
|
||||
## The rule
|
||||
|
||||
1. **Classify.** A source is personal-data if it self-flags
|
||||
(`touches_personal_data`) or is a known directory domain (ratsit.se,
|
||||
merinfo.se, hitta.se, mrkoll.se, eniro.se, birthday.se, upplysning.se).
|
||||
2. **No policy → no processing.** `assertCanStore()` throws unless a
|
||||
`PiiPolicy` with a valid `legal_basis` + `basis_ref` is on record.
|
||||
3. **Everything is evidenced.** Each decision is emitted to AUD0 at risk L3
|
||||
(`0pt.pii_policy_created | pii_data_retained | pii_erased |
|
||||
pii_retention_purged`).
|
||||
|
||||
## GDPR article mapping
|
||||
|
||||
| Requirement | Implementation |
|
||||
|---|---|
|
||||
| Legal basis — art. 6 | `legal_basis` ∈ legitimate_interest/consent/contract/legal_obligation + `basis_ref` (LIA doc id, consent record id, contract clause, provision). Cannot be empty. |
|
||||
| Data minimisation — art. 5(1)(c) | `field_allowlist`; storing any other field throws `PiiMinimisationError`. Empty allowlist = nothing stored. |
|
||||
| Storage limitation — art. 5(1)(e) | `retention_days`; `enforceRetention()` purges expired rows. `0` = fetch-and-drop, nothing persisted. |
|
||||
| Integrity/confidentiality — art. 32 | Only sha256 hashes of identity keys stored (`recordStored`); plaintext identity never written; log/trace redaction (`redact()`) for personnummer/e-mail/phone. |
|
||||
| Right of access — art. 15 | `accessStatement(subjectHash)` — what is held on a subject, across sources. |
|
||||
| Right to erasure — art. 17 | `erase({ subject_hash, reason })` tombstones every stored value for the subject. |
|
||||
| Data protection by design — art. 25 | Gate defaults ON in `SourceQueryService`; no bypass knobs. |
|
||||
|
||||
## When someone requests a personal-data source (e.g. Ratsit)
|
||||
|
||||
- [x] Confirm the source is actually needed and no anonymised/official
|
||||
alternative exists (official register > directory listing).
|
||||
- [x] Write the **Legitimate Interest Assessment** (or record consent /
|
||||
contract / legal obligation) and store its id in `basis_ref`.
|
||||
LIA minimum: purpose, necessity, individuals' rights impact, mitigations.
|
||||
- [x] **VERIFY current terms + robots.txt + availability** on the live site.
|
||||
Do not hardcode "catsit.com allows" — re-check each enable.
|
||||
- [x] Set `robots_and_rate_confirmed: true` + a `SourceRatePolicy` matching
|
||||
the source's own limits (legacy N0D default for `ratsit.se` was `1/min` —
|
||||
treat that as a ceiling, not a target).
|
||||
- [x] Define the minimum `field_allowlist` (e.g. only `street_address`, not
|
||||
`phone`).
|
||||
- [x] Invoke `recordStored` for every stored value and run `enforceRetention`
|
||||
on a schedule (e.g. daily cron).
|
||||
- [x] Record a DPIA note if processing is "large scale" (art. 35) — bulk
|
||||
directory scraping of living persons is large-scale by default.
|
||||
|
||||
## Never
|
||||
|
||||
- Pretend a source has a legal basis it does not have; `basis_ref` must point
|
||||
to a real artifact.
|
||||
- Store identity keys in plaintext (`pseudonymize_keys` defaults to true).
|
||||
- Use stealth/bot-bypass/rate-evasion — excluded by CLAUDE.md §1 regardless of
|
||||
GDPR.
|
||||
- Claim compliance on a source we have not re-verified this season (`VERIFY`
|
||||
markers are intentional).
|
||||
@@ -0,0 +1,324 @@
|
||||
/**
|
||||
* src/services/pii-gate-service.ts
|
||||
* ==================================
|
||||
* 0pt.pii_* — the GDPR guardrail (fail-closed). Enforced by default wherever
|
||||
* a SourceQueryService is constructed; consulted by workers before they
|
||||
* persist anything for a personal-data source.
|
||||
*
|
||||
* Owns 0pt.pii_policy.schema.v1 and the in-memory identity registry that
|
||||
* backs the subject rights surfaces (access/erasure, GDPR art. 15/17) and
|
||||
* the retention worker (GDPR art. 5(1)(e)). Every decision is emitted to
|
||||
* AUD0 as risk L3 events so CL0UD gets an audit trail without polling 0pt.
|
||||
*
|
||||
* Directory domains known to publish personuppgifter (ratsit.se, …) are
|
||||
* pre-classified as personal: until a PiiPolicy with a recorded lawful
|
||||
* basis exists for them, the gate REJECTS storage/query of those sources.
|
||||
*/
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import type {
|
||||
PiiPolicy,
|
||||
CreatePiiPolicyInput,
|
||||
PiiIdentityKey,
|
||||
PiiErasureRequest,
|
||||
PiiAccessStatement,
|
||||
} from "../types/pii";
|
||||
import type { Source } from "../types/source";
|
||||
import { createAud0Client, type Aud0Client, type Aud0Event } from "../clients/aud0-client";
|
||||
|
||||
/** Domain publishes personuppgifter (directory/register listings). */
|
||||
const KNOWN_DIRECTORY_DOMAINS = [
|
||||
"ratsit.se",
|
||||
"merinfo.se",
|
||||
"hitta.se",
|
||||
"mrkoll.se",
|
||||
"eniro.se",
|
||||
"birthday.se",
|
||||
"upplysning.se",
|
||||
];
|
||||
|
||||
const DECISION_TEMPLATE = "PII source %s: %s";
|
||||
|
||||
export class PiiPolicyExistsError extends Error {}
|
||||
export class PiiLegalBasisError extends Error {}
|
||||
export class PiiMinimisationError extends Error {}
|
||||
export class PiiRetentionError extends Error {}
|
||||
export class PiiRobotsRateError extends Error {}
|
||||
|
||||
export class PiiGateService {
|
||||
private policies: Map<string, PiiPolicy> = new Map();
|
||||
private registry: PiiIdentityKey[] = [];
|
||||
private aud0: Aud0Client;
|
||||
|
||||
constructor(opts?: { aud0BaseUrl?: string }) {
|
||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||
}
|
||||
|
||||
/** True if the source either self-flags or is a known directory domain. */
|
||||
classifySourcePersonal(source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">): boolean {
|
||||
if (source.touches_personal_data === true) return true;
|
||||
if (source.source_type !== "web" && source.source_type !== "api") return false;
|
||||
const host = (source.name || "").toLowerCase().replace(/^https?:\/\//, "").replace(/\/.*$/, "");
|
||||
return KNOWN_DIRECTORY_DOMAINS.some((d) => host === d || host.endsWith("." + d));
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Policies
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
async createPiiPolicy(input: CreatePiiPolicyInput): Promise<PiiPolicy> {
|
||||
if (this.policies.has(input.source_id)) {
|
||||
throw new Error(`PiiPolicy already exists for source ${input.source_id}`);
|
||||
}
|
||||
// Hard validation — a policy is the lawful-basis gate, so it cannot be
|
||||
// written with a half-legal posture.
|
||||
if (!input.basis_ref.trim()) throw new Error("basis_ref is required (LIA id, consent id, contract clause or legal provision)");
|
||||
if (!Number.isFinite(input.retention_days) || input.retention_days < 0) {
|
||||
throw new Error("retention_days must be a non-negative integer (0 = fetch-and-drop)");
|
||||
}
|
||||
if (!Array.isArray(input.field_allowlist) || input.field_allowlist.length === 0) {
|
||||
throw new Error("field_allowlist must contain at least one field (data minimisation)");
|
||||
}
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const policy: PiiPolicy = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
touches_personal_data: true,
|
||||
legal_basis: input.legal_basis,
|
||||
basis_ref: input.basis_ref,
|
||||
retention_days: input.retention_days,
|
||||
field_allowlist: [...input.field_allowlist],
|
||||
pseudonymize_keys: input.pseudonymize_keys ?? true,
|
||||
robots_and_rate_confirmed: input.robots_and_rate_confirmed ?? false,
|
||||
created_at: now,
|
||||
updated_at: null,
|
||||
created_by: input.created_by ?? null,
|
||||
};
|
||||
this.policies.set(input.source_id, policy);
|
||||
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_policy_created_${input.source_id}`,
|
||||
event_type: "0pt.pii_policy_created",
|
||||
capability_id: "0pt.pii_policy_created",
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
actor_id: input.created_by ?? "system",
|
||||
resource_type: "source_pii_policy",
|
||||
resource_id: input.source_id,
|
||||
risk_level: "L3",
|
||||
payload: { legal_basis: input.legal_basis, retention_days: input.retention_days, field_count: input.field_allowlist.length },
|
||||
});
|
||||
return policy;
|
||||
}
|
||||
|
||||
async getPiiPolicy(sourceId: string): Promise<PiiPolicy | null> {
|
||||
return this.policies.get(sourceId) ?? null;
|
||||
}
|
||||
|
||||
async listPiiPolicies(tenantId?: string): Promise<PiiPolicy[]> {
|
||||
const all = Array.from(this.policies.values());
|
||||
return tenantId ? all.filter((p) => p.tenant_id === tenantId) : all;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Enforcement (fail-closed)
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
/**
|
||||
* Gate for a store/query of `fields` against `source`. Throws unless the
|
||||
* source is non-personal or a lawful, minimised, retention-bounded,
|
||||
* robots/rate-confirmed policy is on record.
|
||||
*/
|
||||
async assertCanStore(
|
||||
source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">,
|
||||
fields: string[],
|
||||
tenantId: string,
|
||||
): Promise<void> {
|
||||
if (!this.classifySourcePersonal(source)) return;
|
||||
|
||||
const policy = this.policies.get(source.source_id);
|
||||
if (!policy) {
|
||||
throw new PiiLegalBasisError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
`is a personal-data source with no PiiPolicy on record — a lawful basis (${KNOWN_DIRECTORY_DOMAINS.includes(source.name || "") ? "including robots.txt + rate limit confirmation" : "art. 6 GDPR"}) must be registered before any processing`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
if (policy.retention_days === 0 && fields.length > 0) {
|
||||
throw new PiiRetentionError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
"policy forbids retention (retention_days = 0); no fields may be stored",
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
const disallowed = fields.filter((f) => !policy.field_allowlist.includes(f));
|
||||
if (disallowed.length > 0) {
|
||||
throw new PiiMinimisationError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
`field(s) outside the allowlist: ${disallowed.join(", ")} (GDPR art. 5(1)(c) minimisation)`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
if ((source.source_type === "web" || source.source_type === "api") && !policy.robots_and_rate_confirmed) {
|
||||
throw new PiiRobotsRateError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
"robots.txt / rate-limit not confirmed — required for web & api personal-data sources; re-check the source terms (do not hardcode)",
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Subject rights: access (art. 15) & erasure (art. 17), retention (art. 5)
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
/**
|
||||
* Register that one value was stored. Only hashes are persisted; the
|
||||
* plaintext identity never reaches the registry.
|
||||
*/
|
||||
async recordStored(input: {
|
||||
source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">;
|
||||
tenantId: string;
|
||||
identityKey: string;
|
||||
subjectId: string;
|
||||
field: string;
|
||||
artifactRef?: string;
|
||||
}): Promise<PiiIdentityKey> {
|
||||
await this.assertCanStore(input.source, [input.field], input.tenantId);
|
||||
const policy = this.policies.get(input.source.source_id)!;
|
||||
|
||||
const now = Date.now();
|
||||
const key: PiiIdentityKey = {
|
||||
key_hash: hash256(`${sourceIdSalt(input.source.source_id)}:${input.identityKey}`),
|
||||
subject_hash: hash256(input.subjectId),
|
||||
source_id: input.source.source_id,
|
||||
tenant_id: input.tenantId,
|
||||
field: input.field,
|
||||
artifact_ref: input.artifactRef ?? null,
|
||||
stored_at: new Date(now).toISOString(),
|
||||
expires_at: new Date(now + policy.retention_days * 86_400_000).toISOString(),
|
||||
status: "active",
|
||||
};
|
||||
this.registry.push(key);
|
||||
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_retained_${input.source.source_id}_${key.key_hash.slice(0, 12)}`,
|
||||
event_type: "0pt.pii_data_retained",
|
||||
capability_id: "0pt.pii_retain_identity",
|
||||
tenant_id: input.tenantId,
|
||||
project_id: "",
|
||||
actor_id: "system",
|
||||
resource_type: "pii_identity_key",
|
||||
resource_id: key.key_hash,
|
||||
risk_level: "L3",
|
||||
payload: { source_id: input.source.source_id, field: input.field, expires_at: key.expires_at },
|
||||
});
|
||||
return key;
|
||||
}
|
||||
|
||||
/** Right of access — everything currently held on a subject across sources. */
|
||||
async accessStatement(subjectHash: string): Promise<PiiAccessStatement> {
|
||||
const rows = this.registry.filter((r) => r.subject_hash === subjectHash && r.status === "active");
|
||||
return { subject_hash: subjectHash, rows, generated_at: new Date().toISOString() };
|
||||
}
|
||||
|
||||
/** Right to erasure — tombstone every held row for the subject. */
|
||||
async erase(request: PiiErasureRequest): Promise<number> {
|
||||
let count = 0;
|
||||
for (const row of this.registry) {
|
||||
if (row.subject_hash === request.subject_hash && row.status === "active") {
|
||||
row.status = "erased";
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_erased_${request.subject_hash.slice(0, 12)}_${Date.now()}`,
|
||||
event_type: "0pt.pii_erased",
|
||||
capability_id: "0pt.pii_erase",
|
||||
tenant_id: "",
|
||||
project_id: "",
|
||||
actor_id: request.actor_id ?? "system",
|
||||
resource_type: "pii_subject",
|
||||
resource_id: request.subject_hash,
|
||||
risk_level: "L3",
|
||||
payload: { reason: request.reason, row_count: count },
|
||||
});
|
||||
return count;
|
||||
}
|
||||
|
||||
/** Storage limitation — purge rows past their retention deadline. */
|
||||
async enforceRetention(now: Date = new Date()): Promise<{ purged: number }> {
|
||||
let purged = 0;
|
||||
const keep: PiiIdentityKey[] = [];
|
||||
for (const row of this.registry) {
|
||||
if (row.expires_at <= now.toISOString() && row.status === "active") {
|
||||
row.status = "purged";
|
||||
purged += 1;
|
||||
} else {
|
||||
keep.push(row);
|
||||
}
|
||||
}
|
||||
if (purged > 0) {
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_retention_${now.getTime()}`,
|
||||
event_type: "0pt.pii_retention_purged",
|
||||
capability_id: "0pt.pii_enforce_retention",
|
||||
tenant_id: "",
|
||||
project_id: "",
|
||||
actor_id: "system",
|
||||
resource_type: "pii_identity_key",
|
||||
resource_id: now.toISOString(),
|
||||
risk_level: "L3",
|
||||
payload: { row_count: purged },
|
||||
});
|
||||
}
|
||||
return { purged };
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Log hygiene — no personuppgifter in logs/metrics/traces
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
/** Scrub personnummer, e-mail and Swedish phone numbers from a log line. */
|
||||
redact(line: string): string {
|
||||
return line
|
||||
.replace(/\b\d{8}[-–]\d{4}\b/g, "[PERSONNUMMER]")
|
||||
.replace(/\b\d{6}[-–]\d{4}\b/g, "[PERSONNUMMER]")
|
||||
.replace(/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, "[EMAIL]")
|
||||
.replace(/(?:\+46|0046|0)\s?[1-9][\d\s.-]{7,13}/g, "[TELEFON]");
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
private async safeEmit(event: Omit<Aud0Event, "app_id" | "created_at">): Promise<void> {
|
||||
const full: Aud0Event = {
|
||||
...event,
|
||||
app_id: "0pt",
|
||||
created_at: new Date().toISOString(),
|
||||
};
|
||||
try {
|
||||
await this.aud0.ingestEvent(full);
|
||||
} catch {
|
||||
// Audit is important but must never take down a compliance decision or
|
||||
// a retention pass. The caller's state change has already happened.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function sourceIdSalt(sourceId: string): string {
|
||||
return `0pt-pii-v1:${sourceId}`;
|
||||
}
|
||||
|
||||
/** sha256 hex digest — never the plaintext. */
|
||||
export function hash256(value: string): string {
|
||||
return createHash("sha256").update(value, "utf8").digest("hex");
|
||||
}
|
||||
@@ -20,6 +20,7 @@
|
||||
|
||||
import type { Source } from "../types/source";
|
||||
import { SourceService } from "./source-service";
|
||||
import { PiiGateService } from "./pii-gate-service";
|
||||
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
|
||||
import { FabricOneLakeAdapter, type FabricOneLakeConnectionConfig, type AdapterQueryResult } from "../adapters/fabric-onelake-adapter";
|
||||
|
||||
@@ -76,16 +77,27 @@ function clamp(value: number, min: number, max: number): number {
|
||||
export class SourceQueryService {
|
||||
private sourceService: SourceService;
|
||||
private aud0: Aud0Client;
|
||||
/** GDPR guardrail — on by default, can be shared via opts so callers use one registry. */
|
||||
private piiGate: PiiGateService;
|
||||
|
||||
constructor(opts?: { sourceService?: SourceService; aud0BaseUrl?: string }) {
|
||||
constructor(opts?: { sourceService?: SourceService; aud0BaseUrl?: string; piiGate?: PiiGateService }) {
|
||||
// Accepts an existing SourceService so callers (e.g. the API route) share
|
||||
// one in-memory registry instead of each constructing an empty one.
|
||||
this.sourceService = opts?.sourceService ?? new SourceService();
|
||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||
this.piiGate = opts?.piiGate ?? new PiiGateService({ aud0BaseUrl: opts?.aud0BaseUrl });
|
||||
}
|
||||
|
||||
async querySource(input: QuerySourceInput): Promise<QuerySourceResult> {
|
||||
const startedAt = Date.now();
|
||||
|
||||
// GDPR fail-closed: personal-data sources (self-flagged or known directory
|
||||
// domains such as ratsit.se) may not be processed without a PiiPolicy.
|
||||
// Runs BEFORE the capability/type gate so the lawful-basis error is the
|
||||
// one the caller sees for personal data.
|
||||
const preflight = await this.sourceService.getSource(input.source_id);
|
||||
await this.piiGate.assertCanStore(preflight, [], input.tenant_id);
|
||||
|
||||
const source = await this.getQueryableSource(input.source_id);
|
||||
|
||||
const maxRows = clamp(input.max_rows ?? SOURCE_QUERY_DEFAULT_MAX_ROWS, 1, SOURCE_QUERY_HARD_MAX_ROWS);
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* src/types/pii.ts
|
||||
* =================
|
||||
* 0pt.pii_policy.schema.v1 — SIAX Cloud GDPR guardrail for sources that
|
||||
* CAN touch personuppgifter (personal data). Purpose: make processing of
|
||||
* personal data "tekniskt OK" per GDPR (art. 5, 6, 25, 32, 17, 15):
|
||||
* recorded legal basis, storage limitation, data minimisation,
|
||||
* pseudonymisation, and working access/erasure surfaces.
|
||||
*
|
||||
* The guardrail only ever OPENS when a lawful basis is on record; nothing in
|
||||
* 0pt today crawls personal data, so this ships as a fail-closed boundary:
|
||||
* no policy → no storage (and, for directory domains like ratsit.se, no
|
||||
* query at all).
|
||||
*/
|
||||
|
||||
/** GDPR art. 6(1) legal bases 0pt accepts for personal-data sources. */
|
||||
export type LegalBasis = "legitimate_interest" | "consent" | "contract" | "legal_obligation";
|
||||
|
||||
/** Compliance posture for one source that touches personal data. */
|
||||
export interface PiiPolicy {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
touches_personal_data: true;
|
||||
legal_basis: LegalBasis;
|
||||
/**
|
||||
* Pointer to the evidence: a recorded Legitimate Interest Assessment
|
||||
* (LIA) document id, the consent record id, the contract clause, or the
|
||||
* legal provision. Must exist for audit/DPIA purposes — a bare claim is
|
||||
* not enough.
|
||||
*/
|
||||
basis_ref: string;
|
||||
/**
|
||||
* Max storage in days. 0 = fetch-and-drop only (never retained).
|
||||
* The retention worker deletes rows where stored_at + retention_days
|
||||
* < now (art. 5(1)(e)).
|
||||
*/
|
||||
retention_days: number;
|
||||
/**
|
||||
* The ONLY fields that may be stored for this source (art. 5(1)(c)
|
||||
* minimisation). Empty list = nothing may be stored.
|
||||
*/
|
||||
field_allowlist: string[];
|
||||
/**
|
||||
* Pseudonymise identity keys (personnummer, email, reg-number) before
|
||||
* storage; only sha256 digests are ever written (art. 25).
|
||||
*/
|
||||
pseudonymize_keys: boolean;
|
||||
/**
|
||||
* Must be true for web/api sources: robots.txt checked, rate limit in
|
||||
* force (SourceRatePolicy), no auth-wall / bot-bypass technique in use.
|
||||
*/
|
||||
robots_and_rate_confirmed: boolean;
|
||||
created_at: string;
|
||||
updated_at: string | null;
|
||||
created_by: string | null;
|
||||
}
|
||||
|
||||
export interface CreatePiiPolicyInput {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
legal_basis: LegalBasis;
|
||||
basis_ref: string;
|
||||
retention_days: number;
|
||||
field_allowlist: string[];
|
||||
pseudonymize_keys?: boolean;
|
||||
robots_and_rate_confirmed?: boolean;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Registry row for one stored personal-data value. Only hashes are held —
|
||||
* the key_hash lets 0pt locate the subject (right of access, art. 15) and
|
||||
* the subject_hash lets it erase every artifact for that subject across
|
||||
* sources (right to erasure, art. 17) without ever storing the plaintext
|
||||
* identity.
|
||||
*/
|
||||
export interface PiiIdentityKey {
|
||||
key_hash: string;
|
||||
subject_hash: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
field: string;
|
||||
artifact_ref: string | null;
|
||||
stored_at: string;
|
||||
expires_at: string;
|
||||
status: "active" | "erased" | "purged";
|
||||
}
|
||||
|
||||
export interface PiiErasureRequest {
|
||||
subject_hash: string;
|
||||
reason: "deleted_by_subject" | "retention_expired" | "consent_withdrawn";
|
||||
actor_id?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
export interface PiiAccessStatement {
|
||||
subject_hash: string;
|
||||
rows: PiiIdentityKey[];
|
||||
generated_at: string;
|
||||
}
|
||||
@@ -63,6 +63,13 @@ export interface Source {
|
||||
adapter_id?: string;
|
||||
/** Mirrors dim_source.access (docs/03-data-model.md). */
|
||||
access?: SourceAccess;
|
||||
/**
|
||||
* Explicit self-flag that this source CAN expose personuppgifter. When
|
||||
* true (or when the source is a known directory domain like ratsit.se),
|
||||
* the PiiGateService requires a PiiPolicy with a recorded lawful basis
|
||||
* before any storage/query (fail-closed, GDPR art. 6).
|
||||
*/
|
||||
touches_personal_data?: boolean;
|
||||
description?: string;
|
||||
status: SourceStatus;
|
||||
correlation_id?: string;
|
||||
@@ -86,6 +93,7 @@ export interface CreateSourceInput {
|
||||
name: string;
|
||||
adapter_id?: string;
|
||||
access?: SourceAccess;
|
||||
touches_personal_data?: boolean;
|
||||
description?: string;
|
||||
status?: SourceStatus;
|
||||
correlation_id?: string;
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { PiiGateService } from "@/src/services/pii-gate-service";
|
||||
import { SourceService } from "@/src/services/source-service";
|
||||
import { SourceQueryService } from "@/src/services/source-query-service";
|
||||
import type { FabricOneLakeConnectionConfig } from "@/src/adapters/fabric-onelake-adapter";
|
||||
|
||||
/** Minimal config — never reached (the pii gate fires first). */
|
||||
const EMPTY_CFG = {} as FabricOneLakeConnectionConfig;
|
||||
|
||||
describe("PiiGateService — GDPR guardrail (fail-closed)", () => {
|
||||
const ratsit = {
|
||||
source_id: "src_ratsit_1",
|
||||
tenant_id: "t1",
|
||||
project_id: "p1",
|
||||
source_type: "web" as const,
|
||||
name: "ratsit.se",
|
||||
status: "active" as const,
|
||||
created_at: new Date().toISOString(),
|
||||
};
|
||||
const company = {
|
||||
source_id: "src_company_1",
|
||||
tenant_id: "t1",
|
||||
project_id: "p1",
|
||||
source_type: "api" as const,
|
||||
name: "allabolag.se",
|
||||
status: "active" as const,
|
||||
created_at: new Date().toISOString(),
|
||||
};
|
||||
|
||||
it("classifies known directory domains as personal-data sources", () => {
|
||||
const gate = new PiiGateService();
|
||||
expect(gate.classifySourcePersonal(ratsit)).toBe(true);
|
||||
expect(gate.classifySourcePersonal({ ...ratsit, name: "www.ratsit.se" })).toBe(true);
|
||||
expect(gate.classifySourcePersonal(company)).toBe(false);
|
||||
expect(gate.classifySourcePersonal({ ...company, touches_personal_data: true })).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects processing a personal-data source with no recorded lawful basis", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await expect(gate.assertCanStore(ratsit, [], "t1")).rejects.toThrowError(/no PiiPolicy/);
|
||||
});
|
||||
|
||||
it("rejects half-legal policies (missing basis_ref, zero retention, empty allowlist)", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "legitimate_interest", basis_ref: " ", retention_days: 90, field_allowlist: ["address"] }),
|
||||
).rejects.toThrow(/basis_ref is required/);
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "consent", basis_ref: "consent-record-1", retention_days: -1, field_allowlist: ["address"] }),
|
||||
).rejects.toThrow(/retention_days/);
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "contract", basis_ref: "clause-9", retention_days: 90, field_allowlist: [] }),
|
||||
).rejects.toThrow(/data minimisation/);
|
||||
});
|
||||
|
||||
it("blocks out-of-allowlist fields (minimisation) and unconfirmed web sources", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "consent", basis_ref: "consent-record-42",
|
||||
retention_days: 90, field_allowlist: ["name", "address"],
|
||||
});
|
||||
await expect(gate.assertCanStore(ratsit, ["name", "phone"], "t1")).rejects.toThrowError(/outside the allowlist/);
|
||||
await expect(gate.assertCanStore(ratsit, ["name"], "t1")).rejects.toThrowError(/robots.txt \/ rate-limit not confirmed/);
|
||||
|
||||
const ratsitConfirmed = { ...ratsit, source_id: "src_ratsit_confirmed" };
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsitConfirmed.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "consent", basis_ref: "consent-record-42",
|
||||
retention_days: 90, field_allowlist: ["name", "address"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
await expect(gate.assertCanStore(ratsitConfirmed, ["name"], "t1")).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("forbids any storage when retention_days = 0", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await gate.createPiiPolicy({
|
||||
source_id: "src_drop_1", tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "legitimate_interest", basis_ref: "lia-2026-001",
|
||||
retention_days: 0, field_allowlist: ["name"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
await expect(
|
||||
gate.assertCanStore({ ...company, source_id: "src_drop_1", name: "ratsit.se" }, ["name"], "t1"),
|
||||
).rejects.toThrowError(/forbids retention/);
|
||||
});
|
||||
|
||||
it("stores ruled bases and enables access/erasure without plaintext identity", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "legal_obligation", basis_ref: "prop-§5",
|
||||
retention_days: 30, field_allowlist: ["street_address"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
const key = await gate.recordStored({
|
||||
source: ratsit, tenantId: "t1",
|
||||
identityKey: "20250101-1234", subjectId: "subj-77", field: "street_address", artifactRef: "st0re/dataset-a",
|
||||
});
|
||||
|
||||
expect(key.key_hash).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect(key.key_hash).not.toContain("20250101");
|
||||
|
||||
const statement = await gate.accessStatement(key.subject_hash);
|
||||
expect(statement.rows).toHaveLength(1);
|
||||
expect(statement.rows[0].field).toBe("street_address");
|
||||
|
||||
const erased = await gate.erase({ subject_hash: key.subject_hash, reason: "deleted_by_subject" });
|
||||
expect(erased).toBe(1);
|
||||
expect((await gate.accessStatement(key.subject_hash)).rows).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("retention worker purges rows past their deadline", async () => {
|
||||
const gate = new PiiGateService();
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(new Date("2026-08-01T00:00:00Z"));
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "legitimate_interest", basis_ref: "lia-2026-002",
|
||||
retention_days: 30, field_allowlist: ["street_address"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
await gate.recordStored({ source: ratsit, tenantId: "t1", identityKey: "k-1", subjectId: "s-1", field: "street_address" });
|
||||
|
||||
vi.setSystemTime(new Date("2026-09-15T00:00:00Z"));
|
||||
const { purged } = await gate.enforceRetention(new Date("2026-09-15T00:00:00Z"));
|
||||
vi.useRealTimers();
|
||||
expect(purged).toBe(1);
|
||||
});
|
||||
|
||||
it("redacts personnummer, e-mail and phone from log lines", () => {
|
||||
const gate = new PiiGateService();
|
||||
const out = gate.redact(
|
||||
"subject=202501011234 res=" +
|
||||
'evt=19720101-1234 mail=simon@siax.io tel=070-123 45 67 sök=ok',
|
||||
);
|
||||
expect(out).not.toMatch(/\d{8}-\d{4}/);
|
||||
expect(out).toContain("[PERSONNUMMER]");
|
||||
expect(out).toContain("[EMAIL]");
|
||||
expect(out).toContain("[TELEFON]");
|
||||
});
|
||||
|
||||
it("SourceQueryService blocks a personal-data source by default (fail-closed)", async () => {
|
||||
const sourceService = new SourceService();
|
||||
await sourceService.createSource({ ...ratsit });
|
||||
|
||||
const service = new SourceQueryService({ sourceService });
|
||||
await expect(
|
||||
service.querySource({
|
||||
source_id: ratsit.source_id,
|
||||
query: "SELECT 1",
|
||||
tenant_id: "t1", project_id: "p1",
|
||||
connection_config: EMPTY_CFG,
|
||||
}),
|
||||
).rejects.toThrowError(/no PiiPolicy/);
|
||||
});
|
||||
|
||||
it("SourceQueryService passes non-personal sources straight through to capability checks", async () => {
|
||||
const sourceService = new SourceService();
|
||||
await sourceService.createSource({
|
||||
source_id: "src_safe_1", tenant_id: "t1", project_id: "p1",
|
||||
source_type: "web", name: "example.org", status: "active",
|
||||
});
|
||||
|
||||
const service = new SourceQueryService({ sourceService });
|
||||
// Gate passes; the sync path then correctly rejects non-database sources.
|
||||
await expect(
|
||||
service.querySource({
|
||||
source_id: "src_safe_1",
|
||||
query: "SELECT 1",
|
||||
tenant_id: "t1", project_id: "p1",
|
||||
connection_config: EMPTY_CFG,
|
||||
}),
|
||||
).rejects.toThrowError(/source_type "web"/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user