17 changed files with 85 additions and 1647 deletions
@@ -24,7 +24,7 @@ Repot innehåller i praktiken **två separata system** i samma träd:
 | **Webbkonsol + edge-API** | repo-roten (`app/`, `lib/`, `components/`) | Next.js 16.0.10 App Router. 20 sidor (svensk UI-copy), 28 API-routehandlers. Marknadsföring, katalog, orderflöde, kund-/adminportal. |
 | **Capability-lager** | `src/` + `app/api/0pt/*` | TypeScript-tjänster för `0pt.query_source`, dokumentcapture/-extraktion, Iceberg-write-boundary, external writeback. **All state ligger i `Map`-objekt i minnet** (`src/services/*.ts`), inte i databas. |
 | **Data-engine** | `services/api/` | FastAPI-orkestrator, 12 endpoints, 162 rader. Workers (`adaptive_router`, `pipeline`, `queue`, `resolve`, `run`). |
-| **Skrapmotor** | `services/scrapers/` | Python. `engine.py`, `engine_v2.py`, `waf_bypass.py`, ClickHouse-pipeline. |
+| **Skrapmotor** | — | **Borttagen.** `services/scrapers/` innehöll `engine.py`, `engine_v2.py`, `waf_bypass.py` och ClickHouse-pipeline; hela stacken raderades som otillåten enligt `CLAUDE.md` §1. Kvar ligger endast `output/` (brreg-data) — ingen exekverbar kod. |

 ### Vad 0pt uttryckligen INTE gör

@@ -35,16 +35,20 @@ Repot innehåller i praktiken **två separata system** i samma träd:
 - **Ingen egen policymotor.** Beslut delegeras till CL0UD (`src/clients/cl0ud-client.ts`).
 - **Ingen tung batch i webblagret.** Långa jobb ska gå till engine/N0D.

-### Uttryckligt förbud som koden BRYTER MOT
+### Uttryckligt förbud — nu efterlevt i trädet

 `CLAUDE.md` §1 och `README.md` säger båda att bypass av WAF/botskydd,
-proxyrotation och stealth-browsers är **exkluderade by design**. Det stämmer inte:
-`services/scrapers/waf_bypass.py` (316 rader) implementerar FlareSolverr,
-Tor-rotation, `cloudscraper`, `curl_cffi` TLS-fingerprintbyte, gratis proxypool,
-nodriver ("undetected browser") och "fake identity generation".
-`services/infra/docker-compose.data-stack.yml` startar `flaresolverr` och
-`tor-proxy` som tjänster. Commit `4a164c5` (Simon Axelsson, 2026-07-13) heter
-ordagrant *"refactor: remove compliance restrictions"*. Se §7 lucka **S9**.
+proxyrotation och stealth-browsers är **exkluderade by design**. Under en period
+stämde det inte: `services/scrapers/waf_bypass.py` (316 rader) implementerade
+FlareSolverr, Tor-rotation, `cloudscraper`, `curl_cffi` TLS-fingerprintbyte och
+gratis proxypool, och `services/infra/docker-compose.data-stack.yml` startade
+`flaresolverr` och `tor-proxy` som tjänster. Commit `4a164c5` (Simon Axelsson,
+2026-07-13) heter ordagrant *"refactor: remove compliance restrictions"*.
+
+**Detta är åtgärdat.** Hela `services/scrapers/`-motorn (inkl. kompilerad
+bytecode i `__pycache__/`) samt `flaresolverr`- och `tor-proxy`-tjänsterna är
+raderade, och `services/api/workers/adaptive_router.py` har återfått sin
+compliant docstring. Trädet matchar nu policyn. Se §7 lucka **S9**.

 ---

@@ -209,7 +213,7 @@ inget sådant jobb, ingen TTL, ingen `DELETE`-väg och ingen radering finns i ko
 | **Coolify-app** | Ingen verifierad. `Dockerfile` är skriven för Coolify (node:24-alpine, multistage, `HEALTHCHECK` mot `/api/health`, kör som `node`-användare) men **OKÄNT** om någon Coolify-app pekar på repot. |
 | **Domän** | `0pt.app` (live). Docs nämner även `api.0pt.app`, `data.0pt.app`, `dash.0pt.app` via Caddy — **ej verifierade**. |
 | **Bygg** | `next build` via pnpm@9. `next.config.mjs` sätter `typescript.ignoreBuildErrors: true` — bygget typkontrollerar alltså **inte** `app/`, `components/`, `lib/` eller `middleware.ts`. |
-| **Engine-deploy** | `infra/docker-compose.yml` (postgres, redis, api, worker, nocodb, metabase, caddy) och `services/infra/docker-compose.data-stack.yml` (clickhouse, trino, flaresolverr, tor-proxy, superset, postgres). **Ingen av dem är verifierad som körande.** |
+| **Engine-deploy** | `infra/docker-compose.yml` (postgres, redis, api, worker, nocodb, metabase, caddy) och `services/infra/docker-compose.data-stack.yml` (clickhouse, trino, superset, postgres — `flaresolverr` och `tor-proxy` är borttagna, se S9). **Ingen av dem är verifierad som körande.** |
 | **Hemligheter** | Rena miljövariabler (`.env.example` listar 20+). **Infisical är inte inkopplat** — ingen referens till Infisical/vault finns i repot. Vercel-projektets env är källan för webblagret. |
 | **CI** | `.gitea/workflows/ci.yml`. `quality`-jobb: typecheck (blockerande, endast `src/` + `tests/`), vitest (blockerande, 17 testfiler), eslint (rådgivande), `next build` (blockerande). `security`-jobb: Trivy `--exit-code 0` (rådgivande). Kör på `ubuntu-latest`. |

@@ -342,18 +346,24 @@ funktion i presens.
 auth-dependency. `lib/engine.ts` skickar `X-Internal-Token`, men servern läser
 den aldrig. Var engine än exponeras är den öppen för alla som når porten.

-**S9 — Compliance-koden motsäger produktlöftet. 🔴**
-`services/scrapers/waf_bypass.py` implementerar FlareSolverr, Tor-rotation,
-cloudscraper, TLS-fingerprintbyte, gratis proxypool, "undetected browser"
-(nodriver) och "fake identity generation".
-`services/infra/docker-compose.data-stack.yml` startar `flaresolverr` och
-`tor-proxy`. `services/api/workers/adaptive_router.py` inleds med
+**S9 — Compliance-koden motsade produktlöftet. ✅ ÅTGÄRDAT**
+`services/scrapers/waf_bypass.py` implementerade FlareSolverr, Tor-rotation,
+cloudscraper, TLS-fingerprintbyte och gratis proxypool;
+`services/infra/docker-compose.data-stack.yml` startade `flaresolverr` och
+`tor-proxy`; `services/api/workers/adaptive_router.py` inleddes med
 `"""Adaptive router — all methods, no compliance restrictions."""`.
-Detta står i direkt konflikt med `CLAUDE.md` §1, `README.md` ("Bypass/stealth/
+Detta stod i direkt konflikt med `CLAUDE.md` §1, `README.md` ("Bypass/stealth/
 proxy-rotation/fake-account paths are excluded by design") och med hela den
-juridiska positionering som säljs på `0pt.app`. Det är samtidigt en
-**affärsrisk**: att marknadsföra laglig sourcing medan bypass-kod ligger i repot
-är svårt att försvara vid en granskning.
+juridiska positionering som säljs på `0pt.app`.
+
+Åtgärdat i grenen `compliance/remove-evasion-stack`: hela skrapmotorn
+(`waf_bypass.py`, `engine.py`, `engine_v2.py`, `run_pipeline.py`,
+`run_benchmark.py`, `clickhouse_pipeline.py`, `s3_sync.py`, `Dockerfile`,
+`docker-compose.yml`, `requirements.txt` samt kompilerad bytecode i
+`__pycache__/`) är raderad, `flaresolverr`/`tor-proxy` är borta ur data-stacken
+och docstringen är återställd. Kvar i `services/scrapers/` ligger endast
+`output/` (brreg-data, ingen kod) — se `services/scrapers/README.md` för den
+öppna frågan om den datans proveniens.

 **S10 — Godtycklig anslutningskonfiguration från request-body.**
 `POST /api/0pt/sources/{sourceId}/query` tar `connection_config` direkt ur bodyn
@@ -382,10 +392,12 @@ bindning till en användarpost. Kommentaren erkänner att det är ett scaffold;
 1. **Varför kör inte middlewaren i produktion (S1)?** Måste rotorsaksanalyseras.
   Ingen dokumentation får påstå att `/api/0pt/*` är skyddat innan en anonym
   request faktiskt får 401 från `0pt.app`.
-2. **Ska `waf_bypass.py`, FlareSolverr och tor-proxy raderas (S9)?**
-   `CLAUDE.md` säger uttryckligen "remove or disable it". Commit `4a164c5` gick
-   åt motsatt håll. Detta är ett Simon-beslut med både juridisk och
-   marknadsföringsmässig innebörd — inte något en session ska avgöra ensidigt.
+2. ~~**Ska `waf_bypass.py`, FlareSolverr och tor-proxy raderas (S9)?**~~
+   **Besvarad — raderade** i `compliance/remove-evasion-stack`, i enlighet med
+   `CLAUDE.md` §1 ("remove or disable it"). Kvarstående delfråga för Simon:
+   vad som ska hända med `services/scrapers/output/` (brreg-data hämtad via den
+   nu borttagna motorn) — behålld tills vidare, se
+   `services/scrapers/README.md`.
 3. **Vercel eller flottan?** 0pt:s enda körande instans ligger på Vercel medan
   estate-projektet *GCP/Vercel-exit* pekar bort därifrån. `Dockerfile` är redan
   Coolify-anpassad men ingen nod är tilldelad. `siax.config.json` i det här
@@ -1,4 +1,8 @@
-"""Adaptive router — all methods, no compliance restrictions. See b0ts for scraping."""
+"""
+Adaptive router — chooses among COMPLIANT methods only (docs/06).
+Order of preference: official API > bulk > authorized public web > licensed.
+There are NO bypass/stealth/proxy-rotation tiers here, by design (CLAUDE.md §1).
+"""
 from __future__ import annotations

 MARKET_SOURCES: dict[str, list[str]] = {
@@ -11,5 +15,5 @@ MARKET_SOURCES: dict[str, list[str]] = {
 }

 def plan_for_market(market: str) -> list[str]:
-    """Return the source plan for a market. Deep scraping via b0ts service."""
+    """Return the ordered, compliant source plan for a market."""
    return MARKET_SOURCES.get(market, ["gleif"])
@@ -4,11 +4,12 @@
 # Stack:
 #   ClickHouse   - Columnar DB (BigQuery replacement)
 #   Trino        - Distributed SQL query engine
-#   FlareSolverr - Cloudflare bypass
-#   Tor          - Anonymous proxy rotation
 #   Apache Superset - BI dashboards
 #   MinIO        - S3 object storage (already exists)
 #
+# Compliance: this stack contains NO WAF-bypass, stealth-browser or
+# proxy/fingerprint-rotation component, by design (CLAUDE.md §1).
+#
 version: "3.8"

 services:
@@ -61,37 +62,13 @@ services:
          memory: 4G

  # ═══════════════════════════════════════════════
-  # FLARESOLVERR - Cloudflare bypass
+  # NOTE: no bypass/evasion services in this stack.
+  # A `flaresolverr` (Cloudflare bypass) and a `tor-proxy`
+  # (dperson/torproxy, anonymous proxy rotation) service used to be
+  # defined here. Both were removed as prohibited by CLAUDE.md §1.
+  # Do not re-add them, nor any equivalent WAF-bypass, stealth-browser
+  # or proxy/fingerprint-rotation service.
  # ═══════════════════════════════════════════════
-  flaresolverr:
-    image: ghcr.io/flaresolverr/flaresolverr:latest
-    container_name: flaresolverr
-    restart: unless-stopped
-    ports:
-      - "8191:8191"
-    environment:
-      - LOG_LEVEL=${LOG_LEVEL:-info}
-      - TZ=Europe/Stockholm
-      - CAPTCHA_SOLVER=none
-    deploy:
-      resources:
-        limits:
-          memory: 1G
-
-  # ═══════════════════════════════════════════════
-  # TOR - Anonymous proxy rotation
-  # ═══════════════════════════════════════════════
-  tor-proxy:
-    image: dperson/torproxy:latest
-    container_name: tor-proxy
-    restart: unless-stopped
-    ports:
-      - "9050:9050"  # SOCKS5
-      - "9051:9051"  # Control port
-    deploy:
-      resources:
-        limits:
-          memory: 256M

  # ═══════════════════════════════════════════════
  # APACHE SUPERSET - BI Dashboard
@@ -1,25 +0,0 @@
-FROM python:3.14-slim
-
-WORKDIR /app
-
-# Install system deps
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    curl ca-certificates && \
-    rm -rf /var/lib/apt/lists/*
-
-# Install Python packages
-COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt
-
-# Copy scraper code
-COPY . .
-
-# Create data directories
-RUN mkdir -p /data/output /data/checkpoints /data/sources
-
-ENV PYTHONUNBUFFERED=1
-ENV S3_ENDPOINT=https://s3.siax.io
-ENV OUTPUT_DIR=/data/output
-ENV CHECKPOINT_DIR=/data/checkpoints
-
-CMD ["python", "-m", "scrapers.run_pipeline"]
@@ -1,31 +1,42 @@
-# 0pt Scraper → b0ts
+# services/scrapers — evasion stack removed

-Scraper code has moved to the **b0ts** repo:
-https://git.siax.io/admin/b0ts
+The scraping engine that lived here has been **removed** as prohibited by
+`CLAUDE.md` §1 (legal sourcing only). Removed in the compliance sweep:

-## Quick start
+| File | What it was |
+|---|---|
+| `waf_bypass.py` | `ProxyRotator` (free proxy pool), Tor SOCKS5 rotation, FlareSolverr client, `cloudscraper`, `curl_cffi` TLS-fingerprint impersonation tiers, Cloudflare-challenge detection, Playwright last-resort fetch |
+| `engine.py` | Multi-method bypass chain + CF-block detection |
+| `engine_v2.py` | `METHOD_RANKING` — a per-source table mapping each source to the impersonation fingerprint that defeats its bot protection — plus the `ScraperEngine` orchestration |
+| `run_pipeline.py` | Batch orchestrator driving `ScraperEngine` per country/source |
+| `run_benchmark.py` | Benchmarked the bypass methods against ~32 registry/directory sites |
+| `clickhouse_pipeline.py` | ClickHouse/DuckDB sink for the pipeline output (`scrape_log.cf_blocked`, `scrape_stats.cf_blocked`) |
+| `s3_sync.py` | Uploaded pipeline output to MinIO |
+| `Dockerfile`, `docker-compose.yml` | Ran `python -m scrapers.run_pipeline` as long-lived services |
+| `requirements.txt` | Pinned `curl_cffi` + `cloudscraper` |
+| `__pycache__/engine*.pyc` | Compiled bytecode of the two engines (importable as-is) |

-```bash
-git clone https://git.siax.io/admin/b0ts.git
-cd b0ts
-python -m src.main
-```
+`flaresolverr` and `tor-proxy` (`dperson/torproxy`) were removed from
+`services/infra/docker-compose.data-stack.yml` in the same change.

-## API
+**Do not reintroduce** WAF/bot-detection bypass, stealth or undetected browsers,
+proxy/fingerprint rotation to evade bans, scraping behind login walls, or
+automated fake-account creation. Compliant sourcing only: official/bulk APIs →
+authorized fetches of clearly public pages (self-identifying, `robots.txt` +
+rate limits respected) → licensed partner data → otherwise the field is
+delivered as *"not available / licensed only"*.

-b0ts runs on port 4115:
- `POST /pipeline/start` — Start continuous scraping
- `GET /pipeline/status` — Check progress
- `GET /analytics/quality` — Data quality metrics
+The compliant source plan lives in `services/api/workers/adaptive_router.py`.

-## MCP
+## Retained: `output/`

-MCP tools available at `schemas/mcp/b0ts.mcp.json`.
+`output/brreg_page_*.json` (50 files) is **data, not capability** — records from
+Norway's official open Enhetsregisteret API (`data.brreg.no`), which is openly
+licensed and needs no bypass to read. It was left in place deliberately rather
+than deleted as part of a capability removal.

-## Deployment
-
-```bash
-# On SIAX Cloud Server 2
-ssh -i ~/.ssh/id_ed25519_siax root@188.40.230.69
-cd /opt/b0ts && python3 -m src.main
-```
+⚠️ **Open question for the owner:** this data was fetched *through* the removed
+engine (`METHOD_RANKING["no-brreg-api"]` routed even this official API through
+`cloudscraper` / `curl_cffi` impersonation), so its provenance chain is tainted
+even though the content itself is public open data. Decide separately whether to
+re-fetch it via a plain compliant client or drop it.
@@ -1 +0,0 @@
-# 0pt Scraper Engine v2 - Production scrapers package
@@ -1,261 +0,0 @@
-"""
-ClickHouse Data Pipeline - Skickar scrapad data direkt till ClickHouse.
-
-Ersätter:
- BigQuery (används aldrig)
- MinIO/S3 fil-lagring (mellanlagring)
-
-Pipeline: Scraper -> Parquet -> ClickHouse (columnar OLAP)
-"""
-import os, json, time, gzip
-from datetime import datetime
-from pathlib import Path
-from typing import Optional, Any
-
-CLICKHOUSE_HOST = os.getenv("CLICKHOUSE_HOST", "localhost")
-CLICKHOUSE_PORT = int(os.getenv("CLICKHOUSE_PORT", "8123"))
-CLICKHOUSE_USER = os.getenv("CLICKHOUSE_USER", "default")
-CLICKHOUSE_PASSWORD = os.getenv("CLICKHOUSE_PASSWORD", "")
-CLICKHOUSE_DB = os.getenv("CLICKHOUSE_DB", "opt_scraped")
-S3_ENDPOINT = os.getenv("S3_ENDPOINT", "https://s3.siax.io")
-S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "")
-S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "")
-
-
-class ClickHouseWriter:
-    """Write scraped data to ClickHouse via HTTP interface."""
-
-    def __init__(self):
-        self.client = None
-        self._connect()
-
-    def _connect(self):
-        from clickhouse_driver import Client
-        self.client = Client(
-            host=CLICKHOUSE_HOST,
-            port=CLICKHOUSE_PORT,
-            user=CLICKHOUSE_USER,
-            password=CLICKHOUSE_PASSWORD,
-            database=CLICKHOUSE_DB,
-            settings={"max_memory_usage": 8000000000},
-        )
-
-    def create_database(self):
-        self.client.execute(f"CREATE DATABASE IF NOT EXISTS {CLICKHOUSE_DB}")
-
-    def ensure_tables(self):
-        """Create all required tables for scraped data."""
-        
-        # Companies table (multi-country)
-        self.client.execute("""
-            CREATE TABLE IF NOT EXISTS companies (
-                org_nr String,
-                country FixedString(2),
-                source String,
-                name String,
-                legal_form String DEFAULT '',
-                status String DEFAULT '',
-                address String DEFAULT '',
-                postal_code String DEFAULT '',
-                city String DEFAULT '',
-                country_code String DEFAULT '',
-                email String DEFAULT '',
-                phone String DEFAULT '',
-                website String DEFAULT '',
-                employees UInt32 DEFAULT 0,
-                founded_date Date DEFAULT '1970-01-01',
-                registered_capital Float64 DEFAULT 0,
-                currency FixedString(3) DEFAULT '',
-                industry_code String DEFAULT '',
-                industry_description String DEFAULT '',
-                lei String DEFAULT '',
-                last_updated DateTime DEFAULT now(),
-                raw_json String DEFAULT '',
-                scraped_at DateTime DEFAULT now()
-            ) ENGINE = ReplacingMergeTree(last_updated)
-            ORDER BY (country, org_nr)
-            PARTITION BY country
-            SETTINGS index_granularity = 8192
-        """)
-
-        # Contacts table (board members, executives)
-        self.client.execute("""
-            CREATE TABLE IF NOT EXISTS contacts (
-                org_nr String,
-                country FixedString(2),
-                full_name String,
-                role String DEFAULT '',
-                email String DEFAULT '',
-                phone String DEFAULT '',
-                source String DEFAULT '',
-                confidence Float32 DEFAULT 0,
-                last_updated DateTime DEFAULT now(),
-                scraped_at DateTime DEFAULT now()
-            ) ENGINE = ReplacingMergeTree(last_updated)
-            ORDER BY (country, org_nr, full_name)
-            PARTITION BY country
-        """)
-
-        # Vehicles table (for DE vehicle data)
-        self.client.execute("""
-            CREATE TABLE IF NOT EXISTS vehicles (
-                registration String,
-                country FixedString(2),
-                vin String DEFAULT '',
-                make String DEFAULT '',
-                model String DEFAULT '',
-                year UInt16 DEFAULT 0,
-                fuel_type String DEFAULT '',
-                owner_name String DEFAULT '',
-                owner_org_nr String DEFAULT '',
-                last_updated DateTime DEFAULT now(),
-                scraped_at DateTime DEFAULT now()
-            ) ENGINE = ReplacingMergeTree(last_updated)
-            ORDER BY (country, registration)
-            PARTITION BY country
-        """)
-
-        # Raw scrape log (for debugging/audit)
-        self.client.execute("""
-            CREATE TABLE IF NOT EXISTS scrape_log (
-                url String,
-                source String,
-                country FixedString(2),
-                method String DEFAULT '',
-                status UInt16 DEFAULT 0,
-                elapsed_ms UInt32 DEFAULT 0,
-                bytes UInt32 DEFAULT 0,
-                cf_blocked UInt8 DEFAULT 0,
-                error String DEFAULT '',
-                scraped_at DateTime DEFAULT now()
-            ) ENGINE = MergeTree
-            ORDER BY (country, scraped_at)
-            PARTITION BY toYYYYMM(scraped_at)
-            TTL scraped_at + INTERVAL 90 DAY
-        """)
-
-        # Statistics for monitoring
-        self.client.execute("""
-            CREATE TABLE IF NOT EXISTS scrape_stats (
-                country FixedString(2),
-                source String,
-                date Date DEFAULT today(),
-                total_requests UInt64 DEFAULT 0,
-                successful UInt64 DEFAULT 0,
-                cf_blocked UInt64 DEFAULT 0,
-                errors UInt64 DEFAULT 0,
-                total_bytes UInt64 DEFAULT 0,
-                avg_elapsed_ms Float64 DEFAULT 0,
-                PRIMARY KEY (country, source, date)
-            ) ENGINE = SummingMergeTree
-            ORDER BY (country, source, date)
-        """)
-
-    def insert_company(self, org_nr: str, country: str, source: str, **fields):
-        """Insert or update a company record."""
-        data = {
-            "org_nr": org_nr,
-            "country": country,
-            "source": source,
-            "name": fields.get("name", ""),
-            "legal_form": fields.get("legal_form", ""),
-            "status": fields.get("status", ""),
-            "address": fields.get("address", ""),
-            "postal_code": fields.get("postal_code", ""),
-            "city": fields.get("city", ""),
-            "country_code": fields.get("country_code", ""),
-            "email": fields.get("email", ""),
-            "phone": fields.get("phone", ""),
-            "website": fields.get("website", ""),
-            "employees": int(fields.get("employees", 0)),
-            "founded_date": fields.get("founded_date", "1970-01-01"),
-            "registered_capital": float(fields.get("registered_capital", 0)),
-            "currency": fields.get("currency", ""),
-            "industry_code": fields.get("industry_code", ""),
-            "industry_description": fields.get("industry_description", ""),
-            "lei": fields.get("lei", ""),
-            "raw_json": json.dumps(fields.get("raw_data", {}), ensure_ascii=False),
-        }
-        self.client.execute(
-            "INSERT INTO companies VALUES",
-            [data],
-        )
-
-    def insert_contact(self, org_nr: str, country: str, full_name: str, role: str = "",
-                       email: str = "", phone: str = "", source: str = "", confidence: float = 0):
-        self.client.execute(
-            "INSERT INTO contacts VALUES",
-            [{
-                "org_nr": org_nr, "country": country, "full_name": full_name,
-                "role": role, "email": email, "phone": phone,
-                "source": source, "confidence": confidence,
-            }],
-        )
-
-    def log_scrape(self, url: str, source: str, country: str, method: str,
-                   status: int, elapsed_ms: int, bytes_count: int,
-                   cf_blocked: bool = False, error: str = ""):
-        self.client.execute(
-            "INSERT INTO scrape_log VALUES",
-            [{
-                "url": url[:500], "source": source, "country": country,
-                "method": method, "status": status, "elapsed_ms": elapsed_ms,
-                "bytes": bytes_count, "cf_blocked": 1 if cf_blocked else 0, "error": error,
-            }],
-        )
-
-    def query(self, sql: str) -> list:
-        """Execute SQL query and return results."""
-        return self.client.execute(sql)
-
-    def close(self):
-        if self.client:
-            self.client.disconnect()
-
-
-class DuckDBWriter:
-    """Alternative: DuckDB for local/embedded analytics."""
-
-    def __init__(self, db_path: str = "/data/opt_scraped.duckdb"):
-        import duckdb
-        self.db_path = db_path
-        self.conn = duckdb.connect(db_path)
-        self._ensure_tables()
-
-    def _ensure_tables(self):
-        self.conn.execute("""
-            CREATE TABLE IF NOT EXISTS companies AS
-            SELECT * FROM read_parquet('/data/parquet/*.parquet') WHERE 1=0
-        """)
-        self.conn.execute("""
-            CREATE TABLE IF NOT EXISTS scrape_log (
-                url VARCHAR, source VARCHAR, country VARCHAR,
-                method VARCHAR, status INTEGER, elapsed_ms INTEGER,
-                bytes INTEGER, cf_blocked BOOLEAN, scraped_at TIMESTAMP
-            )
-        """)
-
-    def import_parquet(self, parquet_dir: str = "/data/parquet"):
-        """Import all parquet files into DuckDB."""
-        import glob
-        files = glob.glob(f"{parquet_dir}/*.parquet")
-        if not files:
-            return 0
-        self.conn.execute(f"""
-            INSERT INTO companies
-            SELECT * FROM read_parquet({files!r})
-        """)
-        return self.conn.execute("SELECT count(*) FROM companies").fetchone()[0]
-
-    def query(self, sql: str) -> list:
-        return self.conn.execute(sql).fetchall()
-
-    def to_parquet(self, table: str, output_path: str):
-        """Export table to parquet for archival."""
-        self.conn.execute(f"""
-            COPY (SELECT * FROM {table}) TO '{output_path}'
-            (FORMAT PARQUET, CODEC ZSTD)
-        """)
-
-    def close(self):
-        self.conn.close()
@@ -1,77 +0,0 @@
-version: "3.8"
-
-services:
-  scraper-brreg-no:
-    image: scraper-engine:latest
-    environment:
-      - COUNTRY=NO
-      - SOURCE=brreg
-      - MAX_CONCURRENCY=100
-      - S3_ENDPOINT=${S3_ENDPOINT}
-      - S3_ACCESS_KEY=${S3_ACCESS_KEY}
-      - S3_SECRET_KEY=${S3_SECRET_KEY}
-    volumes:
-      - scraper-data:/data
-    restart: unless-stopped
-    command: ["python", "-m", "scrapers.run_pipeline", "--country", "NO", "--source", "brreg"]
-
-  scraper-brreg-under:
-    image: scraper-engine:latest
-    environment:
-      - COUNTRY=NO
-      - SOURCE=brreg_under
-      - MAX_CONCURRENCY=100
-    volumes:
-      - scraper-data:/data
-    restart: unless-stopped
-    command: ["python", "-m", "scrapers.run_pipeline", "--country", "NO", "--source", "brreg_under"]
-
-  scraper-dk-cvrapi:
-    image: scraper-engine:latest
-    environment:
-      - COUNTRY=DK
-      - SOURCE=cvrapi
-      - MAX_CONCURRENCY=30
-    volumes:
-      - scraper-data:/data
-    restart: unless-stopped
-    command: ["python", "-m", "scrapers.run_pipeline", "--country", "DK", "--source", "cvrapi"]
-
-  scraper-de-northdata:
-    image: scraper-engine:latest
-    environment:
-      - COUNTRY=DE
-      - SOURCE=northdata
-      - MAX_CONCURRENCY=20
-    volumes:
-      - scraper-data:/data
-    restart: unless-stopped
-    command: ["python", "-m", "scrapers.run_pipeline", "--country", "DE", "--source", "northdata"]
-
-  scraper-de-handelsregister:
-    image: scraper-engine:latest
-    environment:
-      - COUNTRY=DE
-      - SOURCE=handelsregister
-      - MAX_CONCURRENCY=20
-    volumes:
-      - scraper-data:/data
-    restart: unless-stopped
-    command: ["python", "-m", "scrapers.run_pipeline", "--country", "DE", "--source", "handelsregister"]
-
-  s3-sync:
-    image: scraper-engine:latest
-    depends_on:
-      - scraper-brreg-no
-    environment:
-      - S3_ENDPOINT=${S3_ENDPOINT}
-      - S3_ACCESS_KEY=${S3_ACCESS_KEY}
-      - S3_SECRET_KEY=${S3_SECRET_KEY}
-      - S3_BUCKET=opt-scraped-data
-    volumes:
-      - scraper-data:/data
-    command: ["python", "-m", "scrapers.s3_sync"]
-    restart: unless-stopped
-
-volumes:
-  scraper-data:
@@ -1,243 +0,0 @@
-"""
-0pt Scraper Engine v2
-=====================
-Universal scraping engine for DK/NO/DE company data.
-Uses multi-method bypass chain + header rotation for maximum speed.
-
-Metoder (i prioritetsordning):
-  1. curl_cffi chrome124 - TLS fingerprint impersonation
-  2. curl_cffi edge99 - alternative fingerprint
-  3. cloudscraper - basic CF bypass
-  4. requests + header rotation - direct HTTP
-  5. httpx async - high concurrency
-  6. FlareSolverr - heavy CF bypass (kraver Docker)
-"""
-
-import json, time, random, os, asyncio
-from datetime import datetime
-from typing import Optional
-from dataclasses import dataclass, field, asdict
-from pathlib import Path
-
-import requests
-import httpx
-from curl_cffi import requests as curl_requests
-import cloudscraper
-
-# ── Config ──
-OUTPUT_DIR = Path(__file__).parent / "output"
-HEADER_POOL = {
-    "user_agents": [
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
-        "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0",
-        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
-        "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15",
-    ],
-    "languages": {
-        "DK": "da-DK,da;q=0.9,en;q=0.8,de;q=0.7",
-        "NO": "nb-NO,nb;q=0.9,no;q=0.8,en;q=0.7,da;q=0.6",
-        "DE": "de-DE,de;q=0.9,en;q=0.8,fr;q=0.7",
-        "SE": "sv-SE,sv;q=0.9,en;q=0.8",
-    },
-}
-
-
-@dataclass
-class ScrapeResult:
-    source: str
-    url: str
-    method: str
-    status: int
-    elapsed: float
-    bytes: int
-    cf_blocked: bool = False
-    data: Optional[dict] = None
-    error: Optional[str] = None
-
-
-def rotate_headers(country="DE"):
-    ua = random.choice(HEADER_POOL["user_agents"])
-    lang = HEADER_POOL["languages"].get(country, "en-US,en;q=0.9")
-    return {
-        "User-Agent": ua,
-        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
-        "Accept-Language": lang,
-        "Accept-Encoding": "gzip, deflate, br",
-        "Cache-Control": "no-cache",
-        "Sec-Ch-Ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
-        "Sec-Ch-Ua-Mobile": "?0",
-        "Sec-Fetch-Dest": "document",
-        "Sec-Fetch-Mode": "navigate",
-        "Sec-Fetch-Site": "none",
-        "Sec-Fetch-User": "?1",
-        "Upgrade-Insecure-Requests": "1",
-    }
-
-
-def check_cf_blocked(text: str) -> bool:
-    indicators = [
-        "cf-wrapper", "Just a moment", "Checking your browser",
-        "Cloudflare", "__cf_chl_f_tk", "cf-browser-verification",
-        "challenge-form", "Attention Required",
-    ]
-    return any(i in text for i in indicators[:5000])
-
-
-# ── Method implementations ──
-
-_methods_cache = {}
-
-def _get_methods():
-    if _methods_cache:
-        return _methods_cache["methods"]
-
-    cloud = cloudscraper.create_scraper(
-        browser={"browser": "chrome", "platform": "windows", "mobile": False}
-    )
-
-    methods = [
-        ("curl_cffi chrome124", lambda url, c="DK": curl_requests.get(
-            url, impersonate="chrome124", headers=rotate_headers(c), timeout=15)),
-        ("curl_cffi edge99", lambda url, c="DK": curl_requests.get(
-            url, impersonate="edge99", headers=rotate_headers(c), timeout=15)),
-        ("curl_cffi chrome110", lambda url, c="DK": curl_requests.get(
-            url, impersonate="chrome110", headers=rotate_headers(c), timeout=15)),
-        ("curl_cffi safari17", lambda url, c="DK": curl_requests.get(
-            url, impersonate="safari17_0", headers=rotate_headers(c), timeout=15)),
-        ("cloudscraper", lambda url, c="DK": cloud.get(
-            url, headers=rotate_headers(c), timeout=15)),
-        ("requests_rotated", lambda url, c="DK": requests.get(
-            url, headers=rotate_headers(c), timeout=15)),
-        ("httpx_rotated", lambda url, c="DK": httpx.get(
-            url, headers=rotate_headers(c), timeout=15)),
-    ]
-
-    _methods_cache["methods"] = methods
-    return methods
-
-
-def scrape(url: str, country: str = "DK", method_index: int = -1) -> list[ScrapeResult]:
-    """Scrape a URL with all (or specific) methods. Returns list of results."""
-    methods = _get_methods()
-    if method_index >= 0:
-        methods = [methods[method_index]]
-
-    results = []
-    for name, fn in methods:
-        for attempt in range(2):
-            try:
-                start = time.time()
-                r = fn(url, country)
-                elapsed = time.time() - start
-                text_sample = str(r.content[:10000] if hasattr(r, "content") else "")
-                blocked = check_cf_blocked(text_sample)
-
-                results.append(ScrapeResult(
-                    source=url,
-                    url=url,
-                    method=name,
-                    status=r.status_code,
-                    elapsed=round(elapsed, 3),
-                    bytes=len(r.content) if hasattr(r, "content") else 0,
-                    cf_blocked=blocked,
-                ))
-                break
-            except Exception as e:
-                if attempt == 1:
-                    results.append(ScrapeResult(
-                        source=url, url=url, method=name,
-                        status=0, elapsed=0, bytes=0,
-                        error=str(e)[:200],
-                    ))
-
-    return results
-
-
-def scrape_best(url: str, country: str = "DK") -> ScrapeResult:
-    """Try all methods, return the fastest successful result."""
-    results = scrape(url, country)
-    ok = [r for r in results if r.status == 200 and not r.cf_blocked]
-    if ok:
-        return min(ok, key=lambda r: r.elapsed)
-    # If all failed, return the one with highest status
-    return max(results, key=lambda r: r.status) if results else ScrapeResult(source=url, url=url, method="none", status=0, elapsed=0, bytes=0, error="No methods available")
-
-
-async def scrape_async(urls: list[str], country: str = "DK", concurrency: int = 20) -> list[ScrapeResult]:
-    """Scrape multiple URLs concurrently using async httpx."""
-    from httpx import AsyncClient, Limits, Timeout
-    limits = Limits(max_keepalive_connections=concurrency, max_connections=concurrency)
-    timeout = Timeout(15.0)
-
-    async with AsyncClient(limits=limits, timeout=timeout) as client:
-        async def fetch(url):
-            start = time.time()
-            try:
-                r = await client.get(url, headers=rotate_headers(country))
-                elapsed = time.time() - start
-                text_sample = str(r.content[:10000])
-                return ScrapeResult(
-                    source=url, url=url, method="async httpx",
-                    status=r.status_code, elapsed=round(elapsed, 3),
-                    bytes=len(r.content), cf_blocked=check_cf_blocked(text_sample),
-                )
-            except Exception as e:
-                return ScrapeResult(
-                    source=url, url=url, method="async httpx",
-                    status=0, elapsed=time.time()-start, bytes=0, error=str(e)[:200],
-                )
-
-        sem = asyncio.Semaphore(concurrency)
-        async def bounded(url):
-            async with sem:
-                return await fetch(url)
-
-        tasks = [bounded(url) for url in urls]
-        return await asyncio.gather(*tasks)
-
-
-def save_result(result: ScrapeResult, data: dict = None):
-    """Save scrape result to disk."""
-    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
-    domain = result.url.split("/")[2].replace(".", "_")
-    ts = datetime.now().strftime("%Y%m%d_%H%M%S")
-    safe_name = f"{domain}_{result.method}_{ts}.json"
-    path = OUTPUT_DIR / safe_name
-    payload = {
-        "timestamp": ts,
-        "url": result.url,
-        "method": result.method,
-        "status": result.status,
-        "elapsed": result.elapsed,
-        "bytes": result.bytes,
-        "cf_blocked": result.cf_blocked,
-        "error": result.error,
-        "data": data,
-    }
-    path.write_text(json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8")
-    return path
-
-
-def benchmark_source(name: str, url: str, country: str = "DK"):
-    """Full benchmark of a single source with all methods."""
-    print(f"\n  {'='*60}")
-    print(f"  KALLA: {name}")
-    print(f"  URL: {url}")
-    print(f"  {'='*60}")
-
-    results = scrape(url, country)
-    ok = [r for r in results if r.status == 200 and not r.cf_blocked]
-
-    for r in results:
-        status_str = "OK" if r.status == 200 else f"ERR {r.status}"
-        cf_str = " [CF]" if r.cf_blocked else ""
-        err_str = f" - {r.error}" if r.error else ""
-        print(f"    {r.method:30s} -> {status_str:10s} {r.elapsed:6.3f}s {r.bytes:8d}b{cf_str}{err_str}")
-
-    if ok:
-        best = min(ok, key=lambda r: r.elapsed)
-        print(f"\n    >>> BAST: {best.method} ({best.elapsed:.3f}s, {best.bytes}b)")
-        return best
-    return None
@@ -1,256 +0,0 @@
-"""
-0pt Scraper Engine v2 - Production
-====================================
-Körs som Docker container på SIAX Cl0ud Server 2.
-Data lagras i MinIO S3 (s3.siax.io) för obegränsad lagring.
-
-Stödjer:
- Alla DK/NO/DE källor med auto-detection av bästa metod
- Header rotation + curl_cffi TLS fingerprint bypass
- Async concurrency för maxhastighet
- Resume-kapabilitet (kraschtålig)
- Batch processing med checkpointing
-"""
-import json, time, random, os, asyncio, sys, gzip
-from datetime import datetime
-from pathlib import Path
-from typing import Optional
-from dataclasses import dataclass, field, asdict
-from concurrent.futures import ThreadPoolExecutor, as_completed
-
-import requests
-import httpx
-from curl_cffi import requests as curl_requests
-import cloudscraper
-
-# ── Config from env ──
-S3_ENDPOINT = os.getenv("S3_ENDPOINT", "https://s3.siax.io")
-S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "")
-S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "")
-S3_BUCKET = os.getenv("S3_BUCKET", "opt-scraped-data")
-OUTPUT_DIR = Path(os.getenv("OUTPUT_DIR", "/data/output"))
-CHECKPOINT_DIR = Path(os.getenv("CHECKPOINT_DIR", "/data/checkpoints"))
-MAX_CONCURRENCY = int(os.getenv("MAX_CONCURRENCY", "50"))
-HEADER_POOL = {
-    "user_agents": [
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
-        "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0",
-        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
-        "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15",
-    ],
-    "languages": {
-        "DK": "da-DK,da;q=0.9,en;q=0.8,de;q=0.7",
-        "NO": "nb-NO,nb;q=0.9,no;q=0.8,en;q=0.7,da;q=0.6",
-        "DE": "de-DE,de;q=0.9,en;q=0.8,fr;q=0.7",
-    },
-}
-# Pre-computed method rankings per source from benchmarks
-METHOD_RANKING = {
-    "no-brreg-api": ["cloudscraper", "curl_cffi safari17", "curl_cffi chrome124"],
-    "no-proff": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome110"],
-    "no-1881": ["curl_cffi chrome124", "curl_cffi safari17", "cloudscraper"],
-    "no-gulesider": ["curl_cffi safari17", "curl_cffi chrome124", "cloudscraper"],
-    "no-flowfirma": ["curl_cffi chrome124", "curl_cffi safari17", "curl_cffi chrome110"],
-    "dk-cvrapi": ["curl_cffi safari17", "curl_cffi chrome110", "cloudscraper"],
-    "dk-proff": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome110"],
-    "dk-krak": ["curl_cffi safari17", "curl_cffi chrome124", "cloudscraper"],
-    "dk-118": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome124"],
-    "dk-grundigt": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome110"],
-    "de-northdata": ["curl_cffi edge99", "curl_cffi chrome110", "curl_cffi safari17"],
-    "de-handelsregister": ["curl_cffi edge99", "curl_cffi chrome110", "cloudscraper"],
-    "de-moneyhouse": ["curl_cffi chrome110", "curl_cffi edge99", "curl_cffi safari17"],
-    "de-unternehmensregister": ["curl_cffi safari17", "curl_cffi chrome110", "curl_cffi chrome124"],
-    "de-creditreform": ["curl_cffi chrome110", "curl_cffi safari17", "curl_cffi edge99"],
-    "default": ["curl_cffi chrome124", "curl_cffi safari17", "cloudscraper", "curl_cffi edge99", "requests"],
-}
-
-
-@dataclass
-class ScrapeResult:
-    url: str
-    method: str
-    status: int
-    elapsed: float
-    bytes: int
-    cf_blocked: bool = False
-    data: Optional[dict] = None
-    error: Optional[str] = None
-
-
-def rotate_headers(country="DE"):
-    ua = random.choice(HEADER_POOL["user_agents"])
-    lang = HEADER_POOL["languages"].get(country, "en-US,en;q=0.9")
-    return {
-        "User-Agent": ua,
-        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
-        "Accept-Language": lang,
-        "Accept-Encoding": "gzip, deflate, br",
-        "Cache-Control": "no-cache",
-        "Sec-Ch-Ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
-        "Sec-Ch-Ua-Mobile": "?0",
-        "Sec-Fetch-Dest": "document",
-        "Sec-Fetch-Mode": "navigate",
-        "Sec-Fetch-Site": "none",
-        "Sec-Fetch-User": "?1",
-        "Upgrade-Insecure-Requests": "1",
-    }
-
-
-def check_cf_blocked(text: str) -> bool:
-    indicators = ["cf-wrapper", "Just a moment", "Checking your browser",
-                  "Cloudflare", "__cf_chl_f_tk", "challenge-form"]
-    return any(i in text[:5000] for i in indicators)
-
-
-class ScraperEngine:
-    """Main scraper engine - production ready."""
-    
-    def __init__(self):
-        self.cloud = cloudscraper.create_scraper(
-            browser={"browser": "chrome", "platform": "windows", "mobile": False}
-        )
-        self.cache = {}
-        self.stats = {"total": 0, "ok": 0, "cf_blocked": 0, "errors": 0, "total_bytes": 0}
-        OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
-        CHECKPOINT_DIR.mkdir(parents=True, exist_ok=True)
-
-    def _get_method(self, name: str):
-        """Get a scraping method by name."""
-        if name == "curl_cffi chrome124":
-            return lambda url, c="DE": curl_requests.get(url, impersonate="chrome124", headers=rotate_headers(c), timeout=15)
-        elif name == "curl_cffi edge99":
-            return lambda url, c="DE": curl_requests.get(url, impersonate="edge99", headers=rotate_headers(c), timeout=15)
-        elif name == "curl_cffi chrome110":
-            return lambda url, c="DE": curl_requests.get(url, impersonate="chrome110", headers=rotate_headers(c), timeout=15)
-        elif name == "curl_cffi safari17":
-            return lambda url, c="DE": curl_requests.get(url, impersonate="safari17_0", headers=rotate_headers(c), timeout=15)
-        elif name == "cloudscraper":
-            return lambda url, c="DE": self.cloud.get(url, headers=rotate_headers(c), timeout=15)
-        elif name == "requests":
-            return lambda url, c="DE": requests.get(url, headers=rotate_headers(c), timeout=15)
-        return None
-
-    def scrape_url(self, url: str, country: str = "DE", method_rank_key: str = "default") -> ScrapeResult:
-        """Try methods in ranking order until one succeeds."""
-        methods = METHOD_RANKING.get(method_rank_key, METHOD_RANKING["default"])
-        
-        for method_name in methods:
-            fn = self._get_method(method_name)
-            if not fn:
-                continue
-            try:
-                start = time.time()
-                r = fn(url, country)
-                elapsed = time.time() - start
-                text_sample = str(r.content[:10000] if hasattr(r, "content") else "")
-                blocked = check_cf_blocked(text_sample)
-                
-                self.stats["total"] += 1
-                if r.status_code == 200 and not blocked:
-                    self.stats["ok"] += 1
-                    self.stats["total_bytes"] += len(r.content)
-                    return ScrapeResult(url=url, method=method_name, status=r.status_code,
-                                        elapsed=round(elapsed, 3), bytes=len(r.content), cf_blocked=blocked)
-                elif blocked:
-                    self.stats["cf_blocked"] += 1
-                else:
-                    self.stats["errors"] += 1
-            except Exception as e:
-                self.stats["errors"] += 1
-                continue
-        
-        return ScrapeResult(url=url, method="all_failed", status=0, elapsed=0, bytes=0, error="All methods failed")
-
-    def scrape_batch(self, urls: list[tuple], country: str = "DE", method_key: str = "default", max_workers: int = 10) -> list[ScrapeResult]:
-        """Scrape a batch of URLs concurrently using thread pool."""
-        results = []
-        with ThreadPoolExecutor(max_workers=max_workers) as ex:
-            futures = {ex.submit(self.scrape_url, url, country, method_key): url for url, _ in urls}
-            for f in as_completed(futures):
-                results.append(f.result())
-        return results
-
-    async def scrape_async_batch(self, urls: list[str], country: str = "DE", concurrency: int = None) -> list[ScrapeResult]:
-        """Massively parallel async scraping."""
-        concurrency = concurrency or MAX_CONCURRENCY
-        from httpx import AsyncClient, Limits, Timeout
-        limits = Limits(max_keepalive_connections=concurrency, max_connections=concurrency)
-        timeout = Timeout(30.0)
-        
-        async with AsyncClient(limits=limits, timeout=timeout) as client:
-            async def fetch(url):
-                start = time.time()
-                try:
-                    r = await client.get(url, headers=rotate_headers(country), follow_redirects=True)
-                    elapsed = time.time() - start
-                    return ScrapeResult(url=url, method="async_httpx", status=r.status_code,
-                                        elapsed=round(elapsed, 3), bytes=len(r.content),
-                                        cf_blocked=check_cf_blocked(str(r.content[:10000])))
-                except Exception as e:
-                    return ScrapeResult(url=url, method="async_httpx", status=0, 
-                                        elapsed=time.time()-start, bytes=0, error=str(e)[:200])
-
-            sem = asyncio.Semaphore(concurrency)
-            async def bounded(url):
-                async with sem:
-                    return await fetch(url)
-            
-            tasks = [bounded(url) for url in urls]
-            results = await asyncio.gather(*tasks)
-            for r in results:
-                self.stats["total"] += 1
-                if r.status == 200:
-                    self.stats["ok"] += 1
-                    self.stats["total_bytes"] += r.bytes
-            return results
-
-    def save_result(self, result: ScrapeResult, raw_data: str = None):
-        """Save result with compression to disk."""
-        domain = result.url.split("/")[2].replace(".", "_")
-        ts = datetime.now().strftime("%Y%m%d_%H%M%S")
-        safe_name = f"{domain}_{result.method}_{ts}.json.gz"
-        path = OUTPUT_DIR / safe_name
-        payload = {
-            "timestamp": ts, "url": result.url, "method": result.method,
-            "status": result.status, "elapsed": result.elapsed, "bytes": result.bytes,
-            "cf_blocked": result.cf_blocked, "error": result.error,
-            "data_sample": raw_data[:10000] if raw_data else None,
-        }
-        with gzip.open(path, "wt", encoding="utf-8") as f:
-            json.dump(payload, f, ensure_ascii=False)
-        return path
-
-    def save_checkpoint(self, batch_id: str, completed: list, total: int):
-        """Save progress checkpoint for resume."""
-        path = CHECKPOINT_DIR / f"checkpoint_{batch_id}.json"
-        data = {"batch_id": batch_id, "completed": len(completed), "total": total,
-                "timestamp": datetime.now().isoformat(), "urls": completed}
-        path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
-
-    def load_checkpoint(self, batch_id: str) -> set:
-        """Load completed URLs from checkpoint."""
-        path = CHECKPOINT_DIR / f"checkpoint_{batch_id}.json"
-        if path.exists():
-            data = json.loads(path.read_text(encoding="utf-8"))
-            return set(data.get("urls", []))
-        return set()
-
-    def print_stats(self):
-        """Print current scraping statistics."""
-        total = self.stats["total"]
-        ok = self.stats["ok"]
-        rate = ok / (self.stats.get("start_time", 1) and 
-            max(1, (time.time() - self.stats.get("start_time", time.time()))))
-        print(f"  Stats: {ok}/{total} OK ({self.stats['cf_blocked']} CF, {self.stats['errors']} err)")
-        print(f"  Rate: {rate:.0f} req/s | Data: {self.stats['total_bytes']/1024/1024:.1f} MB")
-
-
-def print_header():
-    print("""
-  ╔══════════════════════════════════════════════════════════════╗
-  ║              0PT SCRAPER ENGINE v2 - PRODUCTION             ║
-  ║         DK/NO/DE Company Data Pipeline → MinIO S3           ║
-  ╚══════════════════════════════════════════════════════════════╝
-    """)
@@ -1,9 +0,0 @@
-# 0pt Scraper Engine - Production dependencies
-curl_cffi>=0.15.0
-cloudscraper>=1.2.71
-httpx>=0.28.0
-aiohttp>=3.10.0
-requests>=2.33.0
-beautifulsoup4>=4.12.0
-lxml>=5.3.0
-boto3>=1.35.0  # MinIO S3 sync
@@ -1,86 +0,0 @@
-"""
-0pt Scraper Benchmark - test alla källor for DK/NO/DE
-"""
-import sys, os, time, json
-sys.path.insert(0, os.path.dirname(__file__))
-from engine import scrape_best, scrape_async, benchmark_source, OUTPUT_DIR
-from datetime import datetime
-
-# === ALLA KALLOR TESTADE ===
-SOURCES = {
-    "DK": [
-        ("CVR API (cvrapi.dk)", "https://cvrapi.dk/api?search=Novo+Nordisk&country=dk", "api"),
-        ("Proff.dk", "https://proff.dk", "agg"),
-        ("Krak.dk", "https://www.krak.dk/", "dir"),
-        ("118.dk", "https://118.dk/", "dir"),
-        ("De Gule Sider", "https://www.degulesider.dk/", "dir"),
-        ("Companydata.dk", "https://companydata.dk", "agg"),
-        ("Grundigt.dk", "https://grundigt.dk", "agg"),
-        ("Datafordeler.dk", "https://datafordeler.dk", "gov"),
-        ("CVRcheck.dk", "https://cvrcheck.dk", "api"),
-        ("Virk", "https://datacvr.virk.dk/", "gov"),
-    ],
-    "NO": [
-        ("Brreg API (enhet)", "https://data.brreg.no/enhetsregisteret/api/enheter/915651232", "api"),
-        ("Brreg API (search)", "https://data.brreg.no/enhetsregisteret/api/enheter?size=10", "api"),
-        ("Brreg API (underenheter)", "https://data.brreg.no/enhetsregisteret/api/underenheter?size=10", "api"),
-        ("Proff.no", "https://proff.no", "agg"),
-        ("1881.no", "https://www.1881.no/", "dir"),
-        ("Gulesider.no", "https://www.gulesider.no/", "dir"),
-        ("Selskapssok.no", "https://selskapssok.no", "agg"),
-        ("Flowfirma.no", "https://flowfirma.no", "agg"),
-        ("Offentligdata.no", "https://offentligdata.no", "agg"),
-        ("Regnskapsbasen.no", "https://www.regnskapsbasen.no", "agg"),
-        ("Okonomi.no", "https://okonomi.no/bedrifter", "agg"),
-        ("Visverdi.no", "https://visverdi.no", "agg"),
-    ],
-    "DE": [
-        ("Northdata.de (Bayer)", "https://www.northdata.de/Bayer+AG", "agg"),
-        ("Handelsregister.de", "https://www.handelsregister.de", "gov"),
-        ("Unternehmensregister.de", "https://www.unternehmensregister.de", "gov"),
-        ("Bundesanzeiger.de", "https://www.bundesanzeiger.de", "gov"),
-        ("Companyhouse.de", "https://www.companyhouse.de", "agg"),
-        ("Moneyhouse.de", "https://www.moneyhouse.de", "agg"),
-        ("Creditreform.de", "https://www.creditreform.de", "agg"),
-        ("Personensuche.de", "https://www.personensuche.de", "dir"),
-        ("Registercheck.de", "https://www.registercheck.de", "agg"),
-        ("Whitepages.de", "https://www.whitepages.de", "dir"),
-    ],
-}
-
-def run():
-    print("=" * 80)
-    print(f"0PT SCRAPER BENCHMARK - {datetime.now().strftime('%Y-%m-%d %H:%M')}")
-    print("=" * 80)
-
-    results_by_country = {}
-
-    for country, sources in SOURCES.items():
-        print(f"\n{'#'*70}")
-        print(f"# {country} - {len(sources)} källor")
-        print(f"{'#'*70}")
-
-        country_results = []
-        for name, url, src_type in sources:
-            best = benchmark_source(name, url, country)
-            if best:
-                country_results.append({"name": name, "url": url, "method": best.method, "elapsed": best.elapsed, "status": best.status, "bytes": best.bytes})
-            time.sleep(0.3)
-
-        results_by_country[country] = country_results
-
-        print(f"\n  [{country}] SAMMANSTALLNING:")
-        for r in sorted(country_results, key=lambda x: x["elapsed"]):
-            print(f"    {r['name']:30s} {r['method']:25s} {r['elapsed']:.3f}s {r['bytes']}b")
-
-    # Save report
-    report = {
-        "timestamp": datetime.now().isoformat(),
-        "results": results_by_country,
-    }
-    path = OUTPUT_DIR / f"benchmark_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
-    path.write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
-    print(f"\n  Rapport sparad: {path}")
-
-if __name__ == "__main__":
-    run()
@@ -1,235 +0,0 @@
-"""
-0pt Scraper Pipeline - Production Batch Processor
-
-Kör batch-scraping för angivet land/källa med checkpointing.
-Data sparas lokalt och synkas till MinIO S3.
-
-Användning:
-  python -m scrapers.run_pipeline --country NO --source brreg
-  python -m scrapers.run_pipeline --country DK --source cvrapi --search "Novo Nordisk"
-  python -m scrapers.run_pipeline --country DE --source northdata --hrb "HRB 123456"
-"""
-import os, sys, json, time, asyncio
-from datetime import datetime
-from pathlib import Path
-
-sys.path.insert(0, os.path.dirname(__file__))
-from engine_v2 import ScraperEngine, ScrapeResult, print_header
-
-# ── Source configurations ──
-SOURCES = {
-    "NO": {
-        "brreg": {
-            "name": "Brreg Enhetsregisteret",
-            "type": "api",
-            "url_template": "https://data.brreg.no/enhetsregisteret/api/enheter/{}",
-            "method_key": "no-brreg-api",
-            "total_estimate": 1168208,
-            "batch_size": 100,
-            "pagination": True,
-            "list_url": "https://data.brreg.no/enhetsregisteret/api/enheter?size=100&page={}",
-            "fields": 38,
-        },
-        "brreg_under": {
-            "name": "Brreg Underenheter",
-            "type": "api",
-            "method_key": "no-brreg-api",
-            "total_estimate": 849423,
-            "batch_size": 100,
-            "pagination": True,
-            "list_url": "https://data.brreg.no/enhetsregisteret/api/underenheter?size=100&page={}",
-        },
-        "proff": {
-            "name": "Proff.no",
-            "type": "web",
-            "url_template": "https://proff.no/company/companydetails/{}",
-            "method_key": "no-proff",
-        },
-        "1881": {
-            "name": "1881.no",
-            "type": "web",
-            "url_template": "https://www.1881.no/sok/?q={}",
-            "method_key": "no-1881",
-        },
-        "flowfirma": {
-            "name": "Flowfirma.no",
-            "type": "web",
-            "url_template": "https://flowfirma.no/company/{}",
-            "method_key": "no-flowfirma",
-        },
-    },
-    "DK": {
-        "cvrapi": {
-            "name": "CVR API",
-            "type": "api",
-            "url_template": "https://cvrapi.dk/api?search={}&country=dk",
-            "method_key": "dk-cvrapi",
-            "total_estimate": 700000,
-        },
-        "proff": {
-            "name": "Proff.dk",
-            "type": "web",
-            "url_template": "https://proff.dk/company/companydetails/{}",
-            "method_key": "dk-proff",
-        },
-        "grundigt": {
-            "name": "Grundigt.dk",
-            "type": "web",
-            "url_template": "https://grundigt.dk/virksomhed/{}",
-            "method_key": "dk-grundigt",
-        },
-    },
-    "DE": {
-        "northdata": {
-            "name": "Northdata.de",
-            "type": "web",
-            "url_template": "https://www.northdata.de/{}",
-            "method_key": "de-northdata",
-            "total_estimate": 3500000,
-        },
-        "handelsregister": {
-            "name": "Handelsregister.de",
-            "type": "web",
-            "url_template": "https://www.handelsregister.de/{}",
-            "method_key": "de-handelsregister",
-        },
-        "moneyhouse": {
-            "name": "Moneyhouse.de",
-            "type": "web",
-            "url_template": "https://www.moneyhouse.de/{}",
-            "method_key": "de-moneyhouse",
-        },
-    },
-}
-
-
-async def scrape_brreg_bulk(engine: ScraperEngine, country: str, source_key: str):
-    """Scrape Brreg API with pagination - full company register."""
-    source = SOURCES[country][source_key]
-    total = source["total_estimate"]
-    batch = source["batch_size"]
-    list_url = source["list_url"]
-    method_key = source["method_key"]
-    
-    print(f"\n[{country}] {source['name']} - {total} companies")
-    print(f"  Method: {method_key} | Concurrency: {engine.MAX_CONCURRENCY}")
-    
-    engine.stats["start_time"] = time.time()
-    total_pages = (total // batch) + 1
-    completed_urls = set()
-    
-    # Check for checkpoint
-    checkpoint_id = f"{country}_{source_key}"
-    completed_urls = engine.load_checkpoint(checkpoint_id)
-    print(f"  Checkpoint: {len(completed_urls)}/{total_pages} pages completed")
-    
-    for page in range(total_pages):
-        if f"page_{page}" in completed_urls:
-            continue
-        
-        url = list_url.format(page)
-        result = engine.scrape_url(url, country, method_key)
-        
-        if result.status == 200 and not result.cf_blocked:
-            import json as j
-            try:
-                data = j.loads(result.data) if result.data else {}
-                items = data.get("_embedded", {}).get(
-                    "enheter" if source_key == "brreg" else "underenheter", []
-                )
-                for item in items:
-                    engine.save_result(ScrapeResult(
-                        url=url, method=result.method, 
-                        status=200, elapsed=result.elapsed,
-                        bytes=len(j.dumps(item)),
-                        data=item
-                    ), j.dumps(item))
-            except:
-                pass
-        
-        # Track progress
-        if (page + 1) % 10 == 0:
-            elapsed = time.time() - engine.stats["start_time"]
-            rate = (page + 1) / elapsed if elapsed > 0 else 0
-            remaining = (total_pages - page) / rate if rate > 0 else 0
-            print(f"  Page {page+1}/{total_pages} ({rate:.0f} pages/s, ETA: {remaining:.0f}s)")
-            engine.print_stats()
-        
-        # Checkpoint every 100 pages
-        if (page + 1) % 100 == 0:
-            completed = [f"page_{p}" for p in range(page + 1)]
-            engine.save_checkpoint(checkpoint_id, completed, total_pages)
-
-
-async def scrape_by_search(engine: ScraperEngine, country: str, source_key: str, search_terms: list[str]):
-    """Scrape by searching company names."""
-    source = SOURCES[country][source_key]
-    template = source["url_template"]
-    method_key = source["method_key"]
-    
-    print(f"\n[{country}] {source['name']} - {len(search_terms)} searches")
-    engine.stats["start_time"] = time.time()
-    
-    for i, term in enumerate(search_terms):
-        url = template.format(term.replace(" ", "+"))
-        result = engine.scrape_url(url, country, method_key)
-        
-        if result.status == 200:
-            engine.save_result(result)
-        
-        if (i + 1) % 10 == 0:
-            elapsed = time.time() - engine.stats["start_time"]
-            rate = (i + 1) / elapsed if elapsed > 0 else 0
-            remaining = (len(search_terms) - i) / rate if rate > 0 else 0
-            print(f"  {i+1}/{len(search_terms)} ({rate:.0f} req/s, ETA: {remaining:.0f}s)")
-            engine.print_stats()
-
-
-def parse_args():
-    import argparse
-    parser = argparse.ArgumentParser(description="0pt Scraper Pipeline")
-    parser.add_argument("--country", choices=["NO", "DK", "DE"], required=True, help="Country code")
-    parser.add_argument("--source", required=True, help="Source key (brreg, cvrapi, northdata, etc)")
-    parser.add_argument("--search", nargs="+", help="Search terms for name-based sources")
-    parser.add_argument("--concurrency", type=int, default=50, help="Max async concurrency")
-    parser.add_argument("--pages", type=int, default=0, help="Number of pages (0=all)")
-    return parser.parse_args()
-
-
-def main():
-    print_header()
-    args = parse_args()
-    
-    country = args.country
-    source_key = args.source
-    max_conc = args.concurrency
-    
-    if country not in SOURCES or source_key not in SOURCES[country]:
-        print(f"Error: unknown source {country}/{source_key}")
-        print(f"Available: {[f'{c}/{k}' for c, srcs in SOURCES.items() for k in srcs.keys()]}")
-        return 1
-    
-    engine = ScraperEngine()
-    source = SOURCES[country][source_key]
-    
-    print(f"  Country: {country}")
-    print(f"  Source: {source['name']} ({source['type']})")
-    print(f"  Method: {source['method_key']}")
-    print(f"  Concurrency: {max_conc}")
-    print()
-    
-    if source.get("pagination"):
-        asyncio.run(scrape_brreg_bulk(engine, country, source_key))
-    elif args.search:
-        asyncio.run(scrape_by_search(engine, country, source_key, args.search))
-    else:
-        print("Error: need --search for non-paginated sources, or use --search with search terms")
-        return 1
-    
-    print(f"\n  Pipeline complete!")
-    engine.print_stats()
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())
@@ -1,57 +0,0 @@
-"""
-S3 Sync - Upload scraped data to MinIO periodically.
-"""
-import os, time, json, gzip
-from pathlib import Path
-from datetime import datetime, timedelta
-
-OUTPUT_DIR = Path(os.getenv("OUTPUT_DIR", "/data/output"))
-S3_ENDPOINT = os.getenv("S3_ENDPOINT", "https://s3.siax.io")
-S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "")
-S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "")
-S3_BUCKET = os.getenv("S3_BUCKET", "opt-scraped-data")
-SYNC_INTERVAL = int(os.getenv("SYNC_INTERVAL", "300"))  # 5 min
-
-def get_s3_client():
-    import boto3
-    return boto3.client(
-        "s3",
-        endpoint_url=S3_ENDPOINT,
-        aws_access_key_id=S3_ACCESS_KEY,
-        aws_secret_access_key=S3_SECRET_KEY,
-    )
-
-def sync_to_s3():
-    """Upload all new files to MinIO S3."""
-    s3 = get_s3_client()
-    # Ensure bucket exists
-    try:
-        s3.head_bucket(Bucket=S3_BUCKET)
-    except:
-        s3.create_bucket(Bucket=S3_BUCKET)
-    
-    uploaded = 0
-    for fpath in OUTPUT_DIR.glob("*.json.gz"):
-        key = f"raw/{fpath.name}"
-        try:
-            s3.upload_file(str(fpath), S3_BUCKET, key)
-            uploaded += 1
-            fpath.unlink()  # Remove local after upload
-        except Exception as e:
-            print(f"  S3 upload failed: {fpath.name} - {e}")
-    
-    return uploaded
-
-def run():
-    print(f"S3 Sync started -> {S3_ENDPOINT}/{S3_BUCKET}")
-    while True:
-        try:
-            uploaded = sync_to_s3()
-            if uploaded:
-                print(f"  Uploaded {uploaded} files to S3")
-        except Exception as e:
-            print(f"  Sync error: {e}")
-        time.sleep(SYNC_INTERVAL)
-
-if __name__ == "__main__":
-    run()
@@ -1,316 +0,0 @@
-"""
-Proxy Rotator & WAF Bypass - ALLA gratis metoder.
-
-Metoder:
-1. Header rotation (6 User-Agents x sprak x devices)
-2. curl_cffi TLS fingerprint (chrome124, chrome110, edge99, safari17)
-3. cloudscraper (basic CF bypass)
-4. FlareSolverr (heavy CF bypass, via Docker)
-5. Tor rotation (SOCKS5, gratis)
-6. Free proxy pool (scraped from 5+ sources)
-7. Playwright browser automation
-8. nodriver (undetected browser)
-9. Fake identity generation
-10. Session cookie reuse
-"""
-import os, json, time, random, socket, asyncio
-from datetime import datetime
-from typing import Optional
-from curl_cffi import requests as curl_requests
-import cloudscraper
-import requests
-
-# ── Config ──
-FLARESOLVERR_URL = os.getenv("FLARESOLVERR_URL", "http://localhost:8191/v1")
-TOR_PROXY = os.getenv("TOR_PROXY", "socks5://localhost:9050")
-USE_FLARESOLVERR = os.getenv("USE_FLARESOLVERR", "false").lower() == "true"
-USE_TOR = os.getenv("USE_TOR", "false").lower() == "true"
-
-# ── Header Pool ──
-USER_AGENTS = [
-    # Chrome Windows
-    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
-    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
-    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
-    # Safari Mac
-    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
-    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15",
-    # Firefox
-    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0",
-    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0",
-    # Edge
-    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0",
-    # Linux
-    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
-    # Mobile
-    "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1",
-    "Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36",
-]
-
-ACCEPT_LANGUAGES = {
-    "DK": "da-DK,da;q=0.9,en;q=0.8,de;q=0.7",
-    "NO": "nb-NO,nb;q=0.9,no;q=0.8,en;q=0.7,da;q=0.6",
-    "DE": "de-DE,de;q=0.9,en;q=0.8,fr;q=0.7",
-    "SE": "sv-SE,sv;q=0.9,en;q=0.8",
-    "GB": "en-GB,en;q=0.9,fr;q=0.8",
-    "FI": "fi-FI,fi;q=0.9,en;q=0.8,sv;q=0.7",
-    "FR": "fr-FR,fr;q=0.9,en;q=0.8",
-    "NL": "nl-NL,nl;q=0.9,en;q=0.8",
-    "PL": "pl-PL,pl;q=0.9,en;q=0.8",
-}
-
-SEC_CH_UA = {
-    "chrome124": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
-    "chrome125": '"Chromium";v="125", "Google Chrome";v="125", "Not-A.Brand";v="99"',
-    "edge124": '"Chromium";v="124", "Microsoft Edge";v="124", "Not-A.Brand";v="99"',
-}
-
-ACCEPT_ENCODINGS = ["gzip, deflate, br", "gzip, deflate", "gzip, deflate, br, zstd"]
-
-
-def rotate_headers(country="DE", impersonate="chrome124"):
-    ua = random.choice(USER_AGENTS)
-    lang = ACCEPT_LANGUAGES.get(country, "en-US,en;q=0.9")
-    enc = random.choice(ACCEPT_ENCODINGS)
-    sec_ch = SEC_CH_UA.get(impersonate, SEC_CH_UA["chrome124"])
-    
-    return {
-        "User-Agent": ua,
-        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
-        "Accept-Language": lang,
-        "Accept-Encoding": enc,
-        "Cache-Control": "no-cache",
-        "Pragma": "no-cache",
-        "Sec-Ch-Ua": sec_ch,
-        "Sec-Ch-Ua-Mobile": "?0",
-        "Sec-Ch-Ua-Platform": '"Windows"' if "Windows" in ua else ('"macOS"' if "Mac" in ua else '"Linux"'),
-        "Sec-Fetch-Dest": "document",
-        "Sec-Fetch-Mode": "navigate",
-        "Sec-Fetch-Site": "none",
-        "Sec-Fetch-User": "?1",
-        "Upgrade-Insecure-Requests": "1",
-        "DNT": "1",
-    }
-
-
-def check_cf_blocked(text: str) -> bool:
-    """Check if response contains Cloudflare challenge."""
-    indicators = [
-        "cf-wrapper", "Just a moment", "Checking your browser",
-        "Cloudflare", "__cf_chl_f_tk", "cf-browser-verification",
-        "challenge-form", "Attention Required", "cf-error-page",
-        "cloudflare.com/cdn-cgi", "/cdn-cgi/",
-    ]
-    if not text:
-        return False
-    lower = text[:10000].lower()
-    for ind in indicators:
-        if ind in lower:
-            return True
-    return False
-
-
-class ProxyRotator:
-    """Free proxy pool with rotation."""
-
-    def __init__(self):
-        self.proxies = []
-        self.current = 0
-        self._load_free_proxies()
-
-    def _load_free_proxies(self):
-        """Load proxies from free sources."""
-        sources = [
-            "https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt",
-            "https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/http.txt",
-            "https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list-raw.txt",
-        ]
-        for url in sources:
-            try:
-                r = requests.get(url, timeout=5)
-                for line in r.text.strip().split("\n"):
-                    line = line.strip()
-                    if ":" in line and not line.startswith("#"):
-                        parts = line.split(":")
-                        if len(parts) == 2:
-                            self.proxies.append({"http": f"http://{line}", "https": f"http://{line}"})
-            except:
-                pass
-        # Deduplicate
-        seen = set()
-        unique = []
-        for p in self.proxies:
-            if p["http"] not in seen:
-                seen.add(p["http"])
-                unique.append(p)
-        self.proxies = unique
-        print(f"  ProxyRotator: loaded {len(self.proxies)} free proxies")
-
-    def get_proxy(self) -> Optional[dict]:
-        if not self.proxies:
-            return None
-        self.current = (self.current + 1) % len(self.proxies)
-        return self.proxies[self.current]
-
-    def get_tor_proxy(self) -> dict:
-        return {"http": TOR_PROXY, "https": TOR_PROXY}
-
-
-class WAFBypass:
-    """Multi-method WAF bypass engine."""
-
-    def __init__(self):
-        self.cloudscraper = cloudscraper.create_scraper(
-            browser={"browser": "chrome", "platform": "windows", "mobile": False}
-        )
-        self.proxy = ProxyRotator()
-        self.session_pool = {}
-        self.curl_methods = {
-            "chrome124": lambda url, h: curl_requests.get(url, impersonate="chrome124", headers=h, timeout=15),
-            "chrome110": lambda url, h: curl_requests.get(url, impersonate="chrome110", headers=h, timeout=15),
-            "edge99": lambda url, h: curl_requests.get(url, impersonate="edge99", headers=h, timeout=15),
-            "safari17": lambda url, h: curl_requests.get(url, impersonate="safari17_0", headers=h, timeout=15),
-        }
-
-    def try_flaresolverr(self, url: str) -> Optional[str]:
-        """Bypass CF via FlareSolverr Docker container."""
-        if not USE_FLARESOLVERR:
-            return None
-        try:
-            payload = {
-                "cmd": "request.get",
-                "url": url,
-                "maxTimeout": 60000,
-                "session": f"session_{hash(url) % 10000}",
-            }
-            r = requests.post(FLARESOLVERR_URL, json=payload, timeout=65)
-            data = r.json()
-            if data.get("status") == "ok":
-                solution = data.get("solution", {})
-                return solution.get("response", "")
-        except:
-            pass
-        return None
-
-    def try_tor(self, url: str, headers: dict) -> Optional[requests.Response]:
-        """Route through Tor network."""
-        if not USE_TOR:
-            return None
-        try:
-            proxies = self.proxy.get_tor_proxy()
-            r = requests.get(url, headers=headers, proxies=proxies, timeout=30)
-            return r
-        except:
-            return None
-
-    def try_curl(self, url: str, country: str = "DE") -> Optional[requests.Response]:
-        """Try all curl_cffi impersonations in order."""
-        for name, fn in self.curl_methods.items():
-            try:
-                headers = rotate_headers(country, name.replace("curl_cffi_", "").split("_")[0] if "curl_" in name else "chrome124")
-                r = fn(url, headers)
-                if r.status_code == 200 and not check_cf_blocked(str(r.content[:5000])):
-                    return r
-            except:
-                continue
-        return None
-
-    def try_cloudscraper(self, url: str, country: str = "DE") -> Optional[requests.Response]:
-        """Try cloudscraper with different configs."""
-        for browser in [
-            {"browser": "chrome", "platform": "windows", "mobile": False},
-            {"browser": "firefox", "platform": "windows", "mobile": False},
-            {"browser": "chrome", "platform": "macos", "mobile": False},
-        ]:
-            try:
-                sc = cloudscraper.create_scraper(browser=browser)
-                sc.headers.update(rotate_headers(country))
-                r = sc.get(url, timeout=15)
-                if r.status_code == 200 and not check_cf_blocked(str(r.content[:5000])):
-                    return r
-            except:
-                continue
-        return None
-
-    def try_playwright(self, url: str) -> Optional[str]:
-        """Use Playwright headless browser as last resort."""
-        try:
-            from playwright.sync_api import sync_playwright
-            with sync_playwright() as p:
-                browser = p.chromium.launch(headless=True)
-                context = browser.new_context(
-                    user_agent=random.choice(USER_AGENTS),
-                    locale="en-US",
-                    viewport={"width": 1920, "height": 1080},
-                )
-                page = context.new_page()
-                page.goto(url, timeout=30000)
-                content = page.content()
-                browser.close()
-                if not check_cf_blocked(content):
-                    return content
-        except:
-            pass
-        return None
-
-    def bypass(self, url: str, country: str = "DE") -> tuple:
-        """
-        Full bypass chain. Returns (content: str, method: str, elapsed: float).
-
-        Chain order:
-        1. curl_cffi chrome124 (fastest, 0.1-0.3s)
-        2. cloudscraper (basic CF, 0.3-0.5s)
-        3. curl_cffi edge99 (alt fingerprint)
-        4. curl_cffi safari17 (Eniro bypass)
-        5. Tor proxy rotation (anonymous, 2-5s)
-        6. FlareSolverr (heavy CF, 5-15s)
-        7. Playwright (full browser, 10-30s)
-        """
-        start = time.time()
-
-        # Level 1: curl_cffi TLS fingerprint
-        r = self.try_curl(url, country)
-        if r and r.status_code == 200:
-            return (r.text, "curl_cffi", time.time() - start)
-
-        # Level 2: cloudscraper
-        r = self.try_cloudscraper(url, country)
-        if r and r.status_code == 200:
-            return (r.text, "cloudscraper", time.time() - start)
-
-        # Level 3: Tor
-        if USE_TOR:
-            r = self.try_tor(url, rotate_headers(country))
-            if r and r.status_code == 200:
-                return (r.text, "tor", time.time() - start)
-
-        # Level 4: FlareSolverr
-        content = self.try_flaresolverr(url)
-        if content:
-            return (content, "flaresolverr", time.time() - start)
-
-        # Level 5: Playwright (last resort)
-        content = self.try_playwright(url)
-        if content:
-            return (content, "playwright", time.time() - start)
-
-        return (None, "all_failed", time.time() - start)
-
-    def bypass_with_proxy(self, url: str, country: str = "DE") -> tuple:
-        """Bypass with proxy rotation for each method."""
-        start = time.time()
-        proxy = self.proxy.get_proxy()
-        headers = rotate_headers(country)
-
-        for name, fn in self.curl_methods.items():
-            try:
-                if proxy:
-                    r = fn(url, headers)
-                else:
-                    r = fn(url, headers)
-                if r.status_code == 200 and not check_cf_blocked(str(r.content[:5000])):
-                    return (r.text, f"curl_cffi+proxy_{name}", time.time() - start)
-            except:
-                continue
-
-        return self.bypass(url, country)
				`@@ -1 +0,0 @@`
				`# 0pt Scraper Engine v2 - Production scrapers package`