Ta bort otillåten bypass-/stealth-skrapkod (CLAUDE.md §1) #1

2026-07-19T17:04:25Z

admin commented

2026-07-19 17:04:25 +00:00

Tar bort services/scrapers/ (FlareSolverr, cloudscraper, Tor SOCKS5-rotation, fake-identity-generering, curl_cffi TLS-fingerprint-impersonation) och services/infra/ (ClickHouse-pipeline byggd enbart för denna data), som planterades i tre commits 2026-07-13 (afd301f, bb6be0f, 4a164c5) utan att någon dokumentation (README/CLAUDE.md/docs) refererade dit — direkt brott mot CLAUDE.md paragraf 1 (No bot-bypass/stealth, excluded by design). Återställer adaptive_router.py byte-för-byte till versionen före 4a164c5. services/scrapers/README.md innehöll dessutom en klartext Hetzner Server2-IP + SSH-nyckelsökväg, ett separat konfidentialitetsbrott, borttaget med resten.

Verifierat: repo-brett grep efter bypass-mönster ger nu noll träffar i körbar kod, python3 -m py_compile på den återställda filen ger exit 0, diff mot pre-4a164c5-versionen är identisk.

Mergar inte själv — för din granskning.

Tar bort services/scrapers/ (FlareSolverr, cloudscraper, Tor SOCKS5-rotation, fake-identity-generering, curl_cffi TLS-fingerprint-impersonation) och services/infra/ (ClickHouse-pipeline byggd enbart för denna data), som planterades i tre commits 2026-07-13 (afd301f, bb6be0f, 4a164c5) utan att någon dokumentation (README/CLAUDE.md/docs) refererade dit — direkt brott mot CLAUDE.md paragraf 1 (No bot-bypass/stealth, excluded by design). Återställer adaptive_router.py byte-för-byte till versionen före 4a164c5. services/scrapers/README.md innehöll dessutom en klartext Hetzner Server2-IP + SSH-nyckelsökväg, ett separat konfidentialitetsbrott, borttaget med resten. Verifierat: repo-brett grep efter bypass-mönster ger nu noll träffar i körbar kod, python3 -m py_compile på den återställda filen ger exit 0, diff mot pre-4a164c5-versionen är identisk. Mergar inte själv — för din granskning.

admin added 1 commit 2026-07-19 17:04:25 +00:00

fix: remove non-compliant scraper stack, restore compliant-only sourcing a63530bbe1

Three commits landed directly on main (afd301f, bb6be0f, 4a164c5, all
2026-07-13) that violate CLAUDE.md §1 ("Non-negotiable constraints") and
this repo's own services/api/README.md ("Deliberately excluded — never
add here"):

- services/scrapers/waf_bypass.py: FlareSolverr, cloudscraper, Tor SOCKS5
  rotation, free-proxy-pool harvesting, fake identity generation, TLS
  fingerprint impersonation (curl_cffi) — the exact bypass/stealth taxonomy
  the constraint enumerates.
- services/infra/clickhouse/* + docker-compose.data-stack.yml: pipeline
  built solely to ingest that scraper's output.
- services/scrapers/engine.py, engine_v2.py, run_pipeline.py, s3_sync.py,
  run_benchmark.py, Dockerfile, docker-compose.yml, requirements.txt,
  __pycache__/*.pyc, output/*.json fixtures: the rest of the standalone
  scraper stack.
- services/scrapers/README.md: pointed at an external "b0ts" repo for
  "deep scraping" and committed a live Hetzner Server2 IP + SSH key path
  in plaintext — a confidentiality-rule violation independent of the
  compliance one.
- .gitea/workflows/ci.yml: a no-op "Scraper CI" stub added solely for
  this code (echoes a string, checks nothing).
- services/api/workers/adaptive_router.py: reverted to the pre-4a164c5
  compliant version. The removed version dropped the ACCESS_RANK /
  compliant-only docstring and replaced MARKET_SOURCES with ~25 source
  ids (proff, gulesider, personensuche, whitepages, northdata, ...) none
  of which exist in sources.yaml or connectors.py — dangling references
  to sources with no registered licence/robots/access record.

None of the removed source ids were ever registered as compliant
connectors, and no doc (README.md, CLAUDE.md, docs/**) references
services/scrapers, services/infra, clickhouse, or b0ts — this stack was
wired in from outside the documented architecture, not part of any
planned build-plan phase.

Verified:
- Repo-wide grep for flaresolverr|stealth|undetected|proxy.rotat|
  waf_bypass|cloudscraper|fake.identity|b0ts now only matches the
  compliant docs/code that document the exclusion (CLAUDE.md,
  services/api/README.md, services/api/sources/connectors.py,
  adaptive_router.py's own docstring).
- python3 -m py_compile services/api/workers/adaptive_router.py: OK.
- adaptive_router.py diffed byte-for-byte against git show 4a164c5^ —
  identical restore.

NOT verified (environment limitation, not introduced by this change):
- `pnpm install` cannot complete in this sandbox — no outbound network
  access to registry.npmjs.org (all package fetches ETIMEDOUT). No
  tsc/next build/vitest could therefore be run. This is a pre-existing
  environment constraint: this change touches zero .ts/.tsx files, and
  tests/p0/*.test.ts already import `vitest`, which is not present in
  package.json or pnpm-lock.yaml (0 occurrences) — that test suite was
  not runnable before this change either.

Not in scope of this change: services/api's own __pycache__/*.pyc files
predate this commit and are a separate, pre-existing hygiene issue.

admin referenced this issue from a commit

2026-07-28 10:56:15 +00:00

docs: stop the README reading as if the banned scraper stack were already gone

admin referenced this pull request

2026-07-28 10:56:56 +00:00

unify: merge the GitHub line into Gitea, settled against real service contracts #6

admin referenced this pull request

2026-08-07 21:08:12 +00:00

chore(compliance): ta bort den forbjudna WAF-bypass-/evasionstacken (CLAUDE.md §1) #7

admin referenced this issue from a commit

2026-08-09 09:31:06 +00:00

ci: add typecheck/lint/test quality gate (#1)