Simon AxelssonandClaude Sonnet 5
dc6aaa0e10
CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
Tenant isolation (security-critical): every app/api/0pt/** route reading
tenant_id from client-controlled query/body never cross-checked it against
the token middleware.ts already verified (x-0pt-verified-tenant). Adds a
shared assertTenantMatchesVerified guard and wires it into all 20+ affected
routes. Also closes two full-leak-by-omission bugs found in this pass
(documents/capture and lakehouse/mappings GETs returned every tenant's rows
when tenant_id was simply omitted) and two "no tenant check at all" reads
(rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone.
Duplicate-delivery idempotency (correctness-critical): POST /api/orders now
honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres +
in-memory); the Python orchestrator's POST /orders/{id}/run — the actual
delivery trigger — is now idempotent under an asyncio lock, closing a real
concurrent-request race, not just the sequential-retry case.
Engine-duplication resolution: adaptive_router.py's docstring wrongly
invited a bypass ("see b0ts for scraping") contradicting the product's own
compliance stance; corrected, and given real (additive, not escalating)
block-signal tracking so a rate-limited/blocked source is excluded from the
plan instead of hammered. Also fixes a previously-undiscovered production
bug: the Python engine's only call into the TS quality bridge
(X-Internal-Token) was unconditionally 401'd by middleware.ts, which never
checked that credential — the advisory quality bridge was unreachable in any
environment enforcing auth.
Restore drill: executed for real against throwaway Docker Postgres
containers (not simulated). Found and fixed two real blockers along the way:
sql/004 referenced opt_datasets, which no migration ever created (chain was
never actually runnable end to end); lib/db.ts hardcoded ssl:"require",
making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev
container) unreachable. After both fixes: full migration chain applies
cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts +
content checksums), and real application code (lib/orders-store.ts,
SourceRegistryService's Postgres repository) reads the restored data back
correctly, including cross-tenant isolation still holding.
82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS +
55 Python) still green; typecheck clean on every changed/added file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
78 lines
3.7 KiB
Python
78 lines
3.7 KiB
Python
"""
|
|
P014: adaptive_router.py previously had ZERO test coverage and a docstring
|
|
("all methods, no compliance restrictions. See b0ts for scraping") that
|
|
directly contradicted this product's own compliance stance and pointed at a
|
|
forbidden, separate, non-compliant repo. This file tests the corrected
|
|
version: plan_for_market's static per-market plan, and the new block-
|
|
tracking (record_blocked/is_blocked/cooldown) that makes the engine actually
|
|
back off a source that just told it to stop, instead of retrying it with no
|
|
memory at all.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from workers.adaptive_router import AdaptiveRouter, MARKET_SOURCES, plan_for_market
|
|
|
|
|
|
class TestPlanForMarket:
|
|
def test_returns_the_full_static_plan_when_nothing_is_blocked(self):
|
|
router = AdaptiveRouter()
|
|
assert plan_for_market("GB", router=router) == MARKET_SOURCES["GB"]
|
|
|
|
def test_unknown_market_falls_back_to_gleif_only(self):
|
|
router = AdaptiveRouter()
|
|
assert plan_for_market("ZZ", router=router) == ["gleif"]
|
|
|
|
def test_defaults_to_the_process_wide_router_when_none_is_passed(self):
|
|
# Just proves the call doesn't blow up and returns a real plan --
|
|
# not asserting anything about global state here (other tests use
|
|
# their own injected router precisely to avoid cross-test pollution
|
|
# of the shared default_router).
|
|
assert plan_for_market("FI") == MARKET_SOURCES["FI"]
|
|
|
|
|
|
class TestBlockTracking:
|
|
def test_a_freshly_blocked_source_is_excluded_from_the_plan(self):
|
|
router = AdaptiveRouter()
|
|
router.record_blocked("GB", "companies_house")
|
|
plan = plan_for_market("GB", router=router)
|
|
assert "companies_house" not in plan
|
|
assert "gleif" in plan # only the blocked source is removed, not the whole market
|
|
|
|
def test_is_blocked_is_false_before_any_block_is_recorded(self):
|
|
router = AdaptiveRouter()
|
|
assert router.is_blocked("GB", "companies_house") is False
|
|
|
|
def test_blocking_is_scoped_per_market_not_global_to_the_source_id(self):
|
|
# gleif appears in every market's plan -- blocking it for GB must not
|
|
# silently remove it from FI/SE/etc too.
|
|
router = AdaptiveRouter()
|
|
router.record_blocked("GB", "gleif")
|
|
assert "gleif" not in plan_for_market("GB", router=router)
|
|
assert "gleif" in plan_for_market("FI", router=router)
|
|
assert "gleif" in plan_for_market("SE", router=router)
|
|
|
|
def test_cooldown_expires_and_the_source_becomes_eligible_again(self):
|
|
router = AdaptiveRouter(cooldown_seconds=60.0)
|
|
router.record_blocked("GB", "companies_house", now=1000.0)
|
|
assert router.is_blocked("GB", "companies_house", now=1030.0) is True # still within cooldown
|
|
assert router.is_blocked("GB", "companies_house", now=1061.0) is False # cooldown elapsed
|
|
# And the plan reflects that:
|
|
assert "companies_house" in plan_for_market("GB", router=router)
|
|
|
|
def test_clear_forgets_every_recorded_block(self):
|
|
router = AdaptiveRouter()
|
|
router.record_blocked("GB", "companies_house")
|
|
router.record_blocked("NO", "brreg")
|
|
router.clear()
|
|
assert plan_for_market("GB", router=router) == MARKET_SOURCES["GB"]
|
|
assert plan_for_market("NO", router=router) == MARKET_SOURCES["NO"]
|
|
|
|
def test_two_router_instances_never_share_state(self):
|
|
# Regression guard for a shared-mutable-default bug class: each
|
|
# AdaptiveRouter() must own its own dict, not a class-level one.
|
|
a = AdaptiveRouter()
|
|
b = AdaptiveRouter()
|
|
a.record_blocked("GB", "companies_house")
|
|
assert a.is_blocked("GB", "companies_house") is True
|
|
assert b.is_blocked("GB", "companies_house") is False
|