admin/0pt
5
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

chore(compliance): ta bort den forbjudna WAF-bypass-/evasionstacken (CLAUDE.md §1) #7

Closed
admin wants to merge 1 commits from compliance/remove-evasion-stack into main
pull from: compliance/remove-evasion-stack
merge into: :main
Conversation 0 Commits 1 Files Changed 17
+85 -1647
1 Commits
Author SHA1 Message Date
Simon AxelssonandClaude Opus 5 e848992020 chore(compliance): remove the banned WAF-bypass/evasion stack (CLAUDE.md §1)
CI (SIAX Cloud) / security (pull_request) Successful in 19s
Details
CI (SIAX Cloud) / quality (pull_request) Successful in 41s
Details
CI (SIAX Cloud) / security (push) Successful in 15s
Details
CI (SIAX Cloud) / quality (push) Successful in 40s
Details 
CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to
evade bans, stealth/undetected browsers, scraping behind login walls and
automated fake-account creation. The tree had drifted from that policy since
commit 4a164c5 ("refactor: remove compliance restrictions"). This restores the
tree to the repo's own stated policy.

Removed (services/scrapers/):
- waf_bypass.py    ProxyRotator + free proxy pool, Tor SOCKS5 rotation,
                   FlareSolverr client, cloudscraper, curl_cffi impersonation
                   tiers, Cloudflare-challenge detection, Playwright fallback
- engine.py        multi-method bypass chain + CF-block detection
- engine_v2.py     METHOD_RANKING per-source bypass table + orchestration
- run_pipeline.py  batch orchestrator driving ScraperEngine
- run_benchmark.py benchmarked bypass methods against ~32 sites
- clickhouse_pipeline.py  sink for the pipeline (scrape_log.cf_blocked)
- s3_sync.py       shipped pipeline output to MinIO
- Dockerfile, docker-compose.yml  ran `python -m scrapers.run_pipeline`
- requirements.txt pinned curl_cffi + cloudscraper
- __pycache__/engine*.pyc  compiled bytecode of both engines (importable)

Removed (services/infra/docker-compose.data-stack.yml):
- flaresolverr (ghcr.io/flaresolverr/flaresolverr)
- tor-proxy    (dperson/torproxy)
clickhouse/trino/superset/postgres are untouched.

Fixed:
- adaptive_router.py: restored the compliant docstring (recovered verbatim from
  the tracked .pyc, which predates 4a164c5). The code had no bypass tier.
- ARCHITECTURE.md: S9 and §1 described the removed code as present; updated.

Preserved deliberately: services/scrapers/output/ (brreg data, not capability)
and the compliant connector layer services/api/sources/connectors.py.

Verification: vitest 128 passed / 1 skipped across 17 files (unchanged),
tsc -p tsconfig.check.json 0 errors, next build exit 0.
src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-07 23:06:53 +02:00