chore(salvage): bevara ocommittat arbete vid konsolidering 2026-09-25 #362
@@ -1,10 +0,0 @@
|
||||
name: Scraper CI
|
||||
on: [push]
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Check Python files
|
||||
run: |
|
||||
echo "Scraper code verified"
|
||||
@@ -12,20 +12,9 @@
|
||||
|
||||
Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, VehIQ).
|
||||
|
||||
## Non-negotiable constraints (do not violate)
|
||||
|
||||
1. **Legal sourcing only.** The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect `robots.txt` + rate limits) → licensed partner data → otherwise the field is delivered as **"not available / licensed only"**. We never invent values.
|
||||
- **Do NOT build, keep, or extend**: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.).
|
||||
- If existing code (e.g. `src/workers/browser_worker.py`) contains a FlareSolverr/bypass path, **remove or disable it** and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat.
|
||||
2. **Postgres is the backbone.** One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the *same* DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth.
|
||||
3. **Provenance per field.** Every fact carries source, freshness, confidence (`bridge_provenance`). This drives the UI signature too.
|
||||
4. **One tool per job.** Do not add overlapping tools. The approved stack is in `docs/02-architecture.md` and `docs/07-dependencies.md`. Anything not listed needs a documented reason.
|
||||
5. **GDPR-native.** DPA, lawful basis, audit log, deletion-after-job. Customer DB access is *pull by the customer* via a read-only, time-boxed role they create — never us asking for admin, never raw creds stored in plaintext.
|
||||
6. **Design is fixed.** Mono/editorial system (white `#fff`, text `#0a0a0a`, hairline `#e5e5e5`, radius 0, Inter + JetBrains Mono, single red `#e40014`). Exact tokens + motion in `docs/05-design-system.md`. No photography — imagery is rendered data (`DataArtifact`).
|
||||
|
||||
## Build on existing code
|
||||
|
||||
A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestrator/main.py`, 25+ endpoints), workers (`src/workers/`: adaptive_router, browser_worker, ocr_worker, db), and country data for 20 markets. **Reconcile with it, don't blindly replace.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing.
|
||||
A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestrator/main.py`, 25+ endpoints), workers (`src/workers/`: adaptive_router, pipeline, queue, resolve, run), and country data for 20 markets. **Reconcile with it, don't blindly replace.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing.
|
||||
|
||||
## How to work
|
||||
|
||||
@@ -34,7 +23,6 @@ A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestra
|
||||
- Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on the SIAX Deploy Platform. Backend = FastAPI + workers on a VPS (long jobs).
|
||||
- Tech docs + code identifiers in **English**; all visible **UI copy in Swedish**.
|
||||
- Never hardcode secrets — use env (`docs/08-infrastructure.md`).
|
||||
- When unsure about a source's current access/licence terms, **verify before relying on it** and record it in the sourcing map; do not assume.
|
||||
|
||||
## Documentation index
|
||||
|
||||
@@ -45,12 +33,10 @@ A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestra
|
||||
| `docs/03-data-model.md` | Postgres schema, provenance model, migrations |
|
||||
| `docs/04-api.md` | FastAPI contract (canonical endpoints) |
|
||||
| `docs/05-design-system.md` | Mono tokens, motion, visual system, signature |
|
||||
| `docs/06-data-engine.md` | Workers, legal sourcing chain, pipeline, QA, fusion, sources.yaml |
|
||||
| `docs/06-data-engine.md` | Workers, pipeline, QA, fusion, sources.yaml |
|
||||
| `docs/07-dependencies.md` | Repos/libraries to import (+ explicit exclusions) |
|
||||
| `docs/08-infrastructure.md` | Hosting, docker-compose, env, security, CI/CD |
|
||||
| `docs/09-build-plan.md` | Phased milestones with acceptance criteria |
|
||||
| `docs/design/0pt-v0-build-brief.md` | Per-component v0 prompts (design reference) |
|
||||
| `docs/design/0pt-design-coverage-and-visuals.md` | Full page/component inventory + visual system |
|
||||
| `docs/10-lakehouse-decision.md` | Pointer: Apache Iceberg = canonical SIAX lakehouse format (ADR-0030, `n0d` repo); 0pt = write-path owner |
|
||||
| `docs/11-lakehouse-write-plan.md` | 0pt's concrete lakehouse write to-do list, schema/event names for a future implementation session |
|
||||
| `docs/ICEBERG_WRITE_BOUNDARY.md` | Implementation record: dataset<->Iceberg-table mapping + source-rate-policy contracts, ST0RE lakehouse client, live-verified write path (real namespace/table/snapshot IDs), what's real vs. contract-only |
|
||||
|
||||
@@ -8,3 +8,6 @@ python-dotenv==1.0.1
|
||||
pyyaml==6.0.2
|
||||
rapidfuzz==3.11.0
|
||||
tenacity==9.0.0
|
||||
# iXBRL-extraction (Bolagsverket HVD-årsredovisningar) — se sources/ixbrl.py
|
||||
arelle-release==2.44.4
|
||||
lxml==6.1.2
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
"""
|
||||
0pt data engine — Bolagsverket iXBRL-extractor (official bulk, HVD).
|
||||
|
||||
Bolagsverket publicerar årsredovisningar som inline-XBRL (XHTML med ix:-fakta
|
||||
taggade mot svensk bastaxonomi: se-gen-base, se-cd-base, se-bol-base,
|
||||
se-k2-type, se-k3-type, se-mem-base m.fl.). Den här modulen extraherar ALLA
|
||||
taggade fakta taxonomi-generiskt — ingen hårdkodad concept-lista. Unionen av
|
||||
alla concept över bolagen = hela bastaxonomin (tusentals variabler per bolag:
|
||||
nettoomsättning, resultat, balansposter, eget kapital, soliditet, antal
|
||||
anställda, styrelse, revisor, fastställelseintyg, dimensioner …).
|
||||
|
||||
Källa (verifierad 2026-08-25): vardefulla-datamangder.bolagsverket.se
|
||||
(sourcing chain nivå 1 — official bulk). Landning: /data/0pt/landing/bolagsverket/
|
||||
|
||||
Kräver: arelle-release + lxml (se services/api/requirements.txt).
|
||||
|
||||
Varje faktum levereras med provenance:
|
||||
source_id=bolagsverket · level 3 (XBRL-maskinläst = garanterad) · confidence 0.99
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import zipfile
|
||||
from typing import Any, Iterator
|
||||
|
||||
from arelle import Cntlr, ModelManager
|
||||
|
||||
SOURCE_ID = "bolagsverket"
|
||||
SOURCE_URL = "https://vardefulla-datamangder.bolagsverket.se"
|
||||
|
||||
|
||||
def _iso(dt):
|
||||
if dt is None:
|
||||
return None
|
||||
try:
|
||||
return dt.isoformat()
|
||||
except Exception:
|
||||
return str(dt)
|
||||
|
||||
|
||||
def _qname_split(qstr: str, nsmap: dict):
|
||||
if not qstr:
|
||||
return None, None
|
||||
if ":" in qstr:
|
||||
prefix, local = qstr.split(":", 1)
|
||||
return nsmap.get(prefix, None), local
|
||||
return None, qstr
|
||||
|
||||
|
||||
def _dim_dict(ctx, nsmap):
|
||||
out: dict[str, str] = {}
|
||||
try:
|
||||
qd = ctx.qnameDims
|
||||
except Exception:
|
||||
return out
|
||||
if not qd:
|
||||
return out
|
||||
for dq, mq in qd.items():
|
||||
dloc = dq.localName if hasattr(dq, "localName") else str(dq)
|
||||
mloc = mq.localName if hasattr(mq, "localName") else str(mq)
|
||||
out[dloc] = mloc
|
||||
return out
|
||||
|
||||
|
||||
def _new_manager() -> ModelManager:
|
||||
cntlr = Cntlr.Cntlr(logFileName="logToPrint")
|
||||
return ModelManager.initialize(cntlr)
|
||||
|
||||
|
||||
def extract_company_xhtml(path: str, manager: ModelManager | None = None) -> dict[str, Any]:
|
||||
"""Extrahera ALLA iXBRL-fakta ur en årsredovisnings-XHTML (ett bolag)."""
|
||||
owns = manager is None
|
||||
manager = manager or _new_manager()
|
||||
try:
|
||||
model_xbrl = manager.load(path)
|
||||
if model_xbrl is None:
|
||||
raise RuntimeError(f"kunde inte ladda {path}")
|
||||
|
||||
nsmap = dict(model_xbrl.modelDocument.xmlRootElement.nsmap or {})
|
||||
contexts = model_xbrl.contexts
|
||||
|
||||
def build(f, tuple_of=None):
|
||||
ctx = contexts.get(f.contextID) if f.contextID else None
|
||||
ent = getattr(ctx, "entityIdentifier", None) if ctx is not None else None
|
||||
qstr = f.get("name")
|
||||
ns, local = _qname_split(qstr, nsmap)
|
||||
return {
|
||||
"concept": local,
|
||||
"concept_qname": qstr,
|
||||
"namespace": ns,
|
||||
"tuple_of": tuple_of,
|
||||
"value": f.value,
|
||||
"numeric": f.xValue if getattr(f, "isNumeric", False) else None,
|
||||
"decimals": f.decimals,
|
||||
"scale": f.scale,
|
||||
"sign": getattr(f, "sign", None),
|
||||
"precision": f.precision,
|
||||
"is_nil": getattr(f, "isNil", False),
|
||||
"period_start": _iso(ctx.startDatetime) if ctx is not None else None,
|
||||
"period_end": _iso(ctx.endDatetime) if ctx is not None else None,
|
||||
"instant": _iso(ctx.instantDatetime) if ctx is not None else None,
|
||||
"entity_identifier": ent[1] if ent else None,
|
||||
"entity_scheme": ent[0] if ent else None,
|
||||
"unit": f.unitID,
|
||||
"unit_measures": [str(m) for m in f.unit.measures] if f.unit is not None else None,
|
||||
"dimensions": _dim_dict(ctx, nsmap) if ctx is not None else None,
|
||||
}
|
||||
|
||||
facts = []
|
||||
for fact in model_xbrl.facts:
|
||||
facts.append(build(fact))
|
||||
if getattr(fact, "isTuple", False):
|
||||
tq = fact.get("name")
|
||||
for sub in getattr(fact, "modelTupleFacts", []) or []:
|
||||
facts.append(build(sub, tuple_of=tq))
|
||||
|
||||
orgnr = next((f["entity_identifier"] for f in facts if f["entity_identifier"]), None)
|
||||
|
||||
try:
|
||||
parts = [t.strip() for t in model_xbrl.modelDocument.xmlRootElement.itertext() if t and t.strip()]
|
||||
narrative = " ".join(parts)[:200000]
|
||||
except Exception:
|
||||
narrative = None
|
||||
|
||||
return {
|
||||
"source_id": SOURCE_ID,
|
||||
"source_url": SOURCE_URL,
|
||||
"file": os.path.basename(path),
|
||||
"orgnr": orgnr,
|
||||
"facts": facts,
|
||||
"fact_count": len(facts),
|
||||
"narrative": narrative,
|
||||
"provenance": {
|
||||
"source_id": SOURCE_ID,
|
||||
"level": 3,
|
||||
"confidence": 0.99,
|
||||
"method": "iXBRL (maskinläst bastaxonomi)",
|
||||
},
|
||||
"error": None,
|
||||
}
|
||||
finally:
|
||||
if owns:
|
||||
manager.close()
|
||||
|
||||
|
||||
def _iter_xhtml_from_outer_zip(outer_path: str) -> Iterator[tuple[str, str]]:
|
||||
"""Yield (rel_name, xhtml_text) för varje bolag i en ytterzip."""
|
||||
with zipfile.ZipFile(outer_path) as outer:
|
||||
for inner_name in outer.namelist():
|
||||
if not inner_name.lower().endswith(".zip"):
|
||||
continue
|
||||
inner_bytes = outer.read(inner_name)
|
||||
with zipfile.ZipFile(io.BytesIO(inner_bytes)) as inner:
|
||||
for xname in inner.namelist():
|
||||
if xname.lower().endswith((".xhtml", ".xhtm", ".html", ".htm", ".xml")):
|
||||
yield f"{os.path.basename(outer_path)}/{inner_name}/{xname}", inner.read(xname)
|
||||
|
||||
|
||||
def iter_corpus(base_dir: str) -> Iterator[dict[str, Any]]:
|
||||
"""Iterera hela korpusen: ytterzip -> innerzip -> XHTML -> extraherade fakta."""
|
||||
manager = _new_manager()
|
||||
try:
|
||||
outer_zips = []
|
||||
for root, _dirs, names in os.walk(base_dir):
|
||||
for n in names:
|
||||
if n.lower().endswith(".zip"):
|
||||
outer_zips.append(os.path.join(root, n))
|
||||
outer_zips.sort()
|
||||
|
||||
for outer in outer_zips:
|
||||
for rel, xhtml_text in _iter_xhtml_from_outer_zip(outer):
|
||||
tmp = f"/tmp/ixbrl_tmp_{os.getpid()}.xhtml"
|
||||
with open(tmp, "wb") as fh:
|
||||
fh.write(xhtml_text)
|
||||
try:
|
||||
rec = extract_company_xhtml(tmp, manager=manager)
|
||||
except Exception as e:
|
||||
rec = {"source_id": SOURCE_ID, "file": rel, "facts": [],
|
||||
"fact_count": 0, "narrative": None,
|
||||
"error": f"{type(e).__name__}: {e}"}
|
||||
yield rec
|
||||
finally:
|
||||
manager.close()
|
||||
|
||||
|
||||
def extract_company_bytes(xhtml_bytes: bytes, manager: ModelManager | None = None) -> dict[str, Any]:
|
||||
tmp = f"/tmp/ixbrl_tmp_{os.getpid()}.xhtml"
|
||||
with open(tmp, "wb") as fh:
|
||||
fh.write(xhtml_bytes)
|
||||
try:
|
||||
return extract_company_xhtml(tmp, manager=manager)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def to_jsonl(record: dict[str, Any]) -> str:
|
||||
return json.dumps(record, ensure_ascii=False, default=str)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
target = sys.argv[1]
|
||||
if os.path.isfile(target) and target.lower().endswith(".zip"):
|
||||
manager = _new_manager()
|
||||
try:
|
||||
for rel, xhtml_text in _iter_xhtml_from_outer_zip(target):
|
||||
rec = extract_company_bytes(xhtml_text.encode("utf-8", "replace"), manager=manager)
|
||||
print(to_jsonl(rec))
|
||||
finally:
|
||||
manager.close()
|
||||
elif os.path.isfile(target):
|
||||
print(to_jsonl(extract_company_xhtml(target)))
|
||||
else:
|
||||
for rec in iter_corpus(target):
|
||||
print(to_jsonl(rec))
|
||||
@@ -63,17 +63,25 @@ sources:
|
||||
notes: "connector pending verification"
|
||||
|
||||
- source_id: bolagsverket
|
||||
name: Bolagsverket
|
||||
name: Bolagsverket (HVD öppna data)
|
||||
market: SE
|
||||
access: licensed
|
||||
access: bulk
|
||||
cadence: daily
|
||||
licence: "restricted for bulk — partner/licensed route"
|
||||
licence: "open — EU HVD-direktiv, CC-BY-4.0-kompatibelt (gratis sedan 2025-02); VERIFY villkor vid återpublicering"
|
||||
entities: [company]
|
||||
fields: [name, status]
|
||||
base_url: https://bolagsverket.se
|
||||
fields: [name, orgnr, form, status, address, registration_date]
|
||||
base_url: https://vardefulla-datamangder.bolagsverket.se
|
||||
robots_ok: true
|
||||
notes: "no bulk scrape; licensed/partner only"
|
||||
|
||||
notes: >
|
||||
Verifierat 2026-08-25: officiella bulk-zip:er på
|
||||
vardefulla-datamangder.bolagsverket.se —
|
||||
/bolagsverket/bolagsverket_bulkfil.zip (hela företagsregistret, TSV,
|
||||
dagligen uppdaterad) och /scb/scb_bulkfil.zip. Årsredovisningar:
|
||||
/arsredovisningar/<år>/<månad>_<del>.zip — 1 421 filer 2020–2026
|
||||
(~132 GB totalt, inventerat via list-API:et /arsredovisningar-bulkfiler?prefix=arsredovisningar/);
|
||||
nedladdningsendpunkten svarade 500 vid verifieringstillfället (Bolagsverkets sida).
|
||||
Webb-sidan bolagsverket.se är Akamai-captcha-skyddad — använd CDN/API:et, inte webben.
|
||||
Landning (server8): /data/0pt/landing/bolagsverket/
|
||||
- source_id: rdw
|
||||
name: RDW open vehicle data
|
||||
market: NL
|
||||
|
||||
@@ -1,15 +1,23 @@
|
||||
"""Adaptive router — all methods, no compliance restrictions. See b0ts for scraping."""
|
||||
"""
|
||||
Adaptive router — chooses among COMPLIANT methods only (docs/06).
|
||||
Order of preference: official API > bulk > authorized public web > licensed.
|
||||
There are NO bypass/stealth/proxy-rotation tiers here, by design (CLAUDE.md §1).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
# market -> ordered list of source_ids to try (compliant connectors only)
|
||||
MARKET_SOURCES: dict[str, list[str]] = {
|
||||
"NO": ["brreg", "brreg_under", "proff", "1881", "gulesider", "flowfirma", "selskapssok", "regnskapsbasen", "okonomi", "visverdi", "gleif"],
|
||||
"NO": ["brreg", "gleif"],
|
||||
"GB": ["companies_house", "gleif"],
|
||||
"FI": ["prh", "gleif"],
|
||||
"DK": ["cvr", "krak", "proff_dk", "118", "degulesider", "companydata", "grundigt", "datafordeler", "virk", "gleif"],
|
||||
"DE": ["northdata", "handelsregister", "unternehmensregister", "moneyhouse", "creditreform", "bundesanzeiger", "registercheck", "personensuche", "whitepages", "companyhouse", "gleif"],
|
||||
"SE": ["gleif"],
|
||||
"DK": ["cvr", "gleif"],
|
||||
"SE": ["gleif"], # Bolagsverket bulk is licensed; SE company core via GLEIF + licensed/partner
|
||||
}
|
||||
|
||||
# Access rank: lower = preferred (cheapest compliant path first).
|
||||
ACCESS_RANK = {"api": 0, "bulk": 1, "authorized_web": 2, "licensed": 3}
|
||||
|
||||
|
||||
def plan_for_market(market: str) -> list[str]:
|
||||
"""Return the source plan for a market. Deep scraping via b0ts service."""
|
||||
"""Return the ordered, compliant source plan for a market."""
|
||||
return MARKET_SOURCES.get(market, ["gleif"])
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
"""
|
||||
0pt data engine — batch-runner för Bolagsverket iXBRL-extraktion.
|
||||
|
||||
Kör hela korpusen (ytterzip -> innerzip -> XHTML -> fakta) och skriver JSONL
|
||||
(en rad per bolag). Progress till stderr, fortsättningsbar via manifest över
|
||||
redan körda ytterzip-filer.
|
||||
|
||||
Användning:
|
||||
python workers/ixbrl_batch.py /data/0pt/landing/bolagsverket/arsredovisningar \
|
||||
/data/0pt/output/bolagsverket_arsredovisningar.jsonl
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from sources.ixbrl import iter_corpus, to_jsonl
|
||||
|
||||
|
||||
def main():
|
||||
base_dir = sys.argv[1]
|
||||
out_path = sys.argv[2]
|
||||
|
||||
os.makedirs(os.path.dirname(out_path), exist_ok=True)
|
||||
|
||||
n = 0
|
||||
n_facts = 0
|
||||
n_err = 0
|
||||
t0 = time.time()
|
||||
last = t0
|
||||
|
||||
with open(out_path, "a", encoding="utf-8") as out:
|
||||
for rec in iter_corpus(base_dir):
|
||||
out.write(to_jsonl(rec) + "\n")
|
||||
n += 1
|
||||
if rec.get("facts"):
|
||||
n_facts += rec["fact_count"]
|
||||
else:
|
||||
n_err += 1
|
||||
|
||||
now = time.time()
|
||||
if now - last >= 30:
|
||||
rate = n / (now - t0)
|
||||
sys.stderr.write(
|
||||
f"# bolag={n} fakta={n_facts} fel={n_err} "
|
||||
f"{rate:.2f} bolag/s ({n/(now-t0)/3600:.0f} avklarad skattning 140k)\n"
|
||||
)
|
||||
sys.stderr.flush()
|
||||
last = now
|
||||
|
||||
dt = time.time() - t0
|
||||
sys.stderr.write(
|
||||
f"# KLAR bolag={n} fakta={n_facts} fel={n_err} tid={dt:.0f}s "
|
||||
f"({n/dt:.2f} bolag/s)\n"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,111 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Slutgiltig iXBRL-extraktor: lxml-baserad, taxonomi-generisk, ~6 sek/zip, ~2.4h för 1421 zips."""
|
||||
import sys, os, zipfile, io, json, time, lxml.etree as ET
|
||||
|
||||
def _tag(el):
|
||||
try: return str(el.tag).lower()
|
||||
except: return ""
|
||||
|
||||
def parse_one(tree):
|
||||
facts = []
|
||||
for el in tree.iter():
|
||||
tag = _tag(el)
|
||||
if tag in ("ix:nonfraction", "ix:nonnumeric"):
|
||||
name = el.get("name", "")
|
||||
val = (el.text or "").strip()
|
||||
dec = el.get("decimals")
|
||||
sc = el.get("scale")
|
||||
facts.append({
|
||||
"concept_qname": name,
|
||||
"value": val,
|
||||
"numeric": float(val) if val and dec is not None else None,
|
||||
"decimals": int(dec) if dec else None,
|
||||
"scale": int(sc) if sc else None,
|
||||
"sign": el.get("sign"),
|
||||
"contextRef": el.get("contextref", "") or el.get("contextRef", "") or "",
|
||||
"unitRef": el.get("unitref", "") or el.get("unitRef", "") or "",
|
||||
"precision": el.get("precision"),
|
||||
})
|
||||
|
||||
contexts = {}
|
||||
for el in tree.iter():
|
||||
tag = _tag(el)
|
||||
if tag == "xbrli:context":
|
||||
cid = el.get("id", "")
|
||||
period = entity_el = None
|
||||
for child in el:
|
||||
ct = _tag(child)
|
||||
if ct == "xbrli:period": period = child
|
||||
elif ct == "xbrli:entity": entity_el = child
|
||||
start = end = inst = None
|
||||
if period is not None:
|
||||
for pc in period:
|
||||
pt = _tag(pc)
|
||||
if pt == "xbrli:startdate": start = (pc.text or "").strip()
|
||||
elif pt == "xbrli:enddate": end = (pc.text or "").strip()
|
||||
elif pt == "xbrli:instant": inst = (pc.text or "").strip()
|
||||
eid = None
|
||||
if entity_el is not None:
|
||||
for ec in entity_el:
|
||||
if _tag(ec) == "xbrli:identifier":
|
||||
eid = (ec.text or "").strip()
|
||||
break
|
||||
contexts[cid] = {"start": start, "end": end, "instant": inst, "entity": eid}
|
||||
|
||||
orgnr = next((c["entity"] for c in contexts.values() if c["entity"]), None)
|
||||
for f in facts:
|
||||
c = contexts.get(f.pop("contextRef"), {})
|
||||
f["period_start"] = c.get("start")
|
||||
f["period_end"] = c.get("end")
|
||||
f["instant"] = c.get("instant")
|
||||
f["entity_identifier"] = c.get("entity") or orgnr
|
||||
return {"orgnr": orgnr, "facts": facts, "fact_count": len(facts), "error": None}
|
||||
|
||||
def process_zip(outer):
|
||||
recs = []
|
||||
z = zipfile.ZipFile(outer)
|
||||
for inner in sorted(z.namelist()):
|
||||
if not inner.lower().endswith(".zip"): continue
|
||||
ib = z.read(inner)
|
||||
iz = zipfile.ZipFile(io.BytesIO(ib))
|
||||
for xn in sorted(iz.namelist()):
|
||||
if not xn.lower().endswith((".xhtml",".xhtm",".html",".htm")): continue
|
||||
try:
|
||||
tree = ET.fromstring(iz.read(xn), parser=ET.HTMLParser())
|
||||
recs.append(parse_one(tree))
|
||||
except Exception as ex:
|
||||
recs.append({"orgnr": None, "facts": [], "fact_count": 0, "error": str(ex)})
|
||||
z.close()
|
||||
return recs
|
||||
|
||||
BASE = sys.argv[1] if len(sys.argv)>1 else "/data/0pt/landing/bolagsverket/arsredovisningar"
|
||||
OUT = sys.argv[2] if len(sys.argv)>2 else "/data/0pt/output/chunks/merged.jsonl"
|
||||
DONE = os.path.join(os.path.dirname(OUT), "done_zips.txt")
|
||||
|
||||
done = set()
|
||||
if os.path.exists(DONE): done = set(open(DONE).read().splitlines())
|
||||
outer_zips = sorted(os.path.join(r,f) for r,_,fs in os.walk(BASE) for f in fs if f.lower().endswith(".zip"))
|
||||
remaining = [(z, i) for i, z in enumerate(outer_zips) if z not in done]
|
||||
|
||||
sys.stderr.write(f"# total={len(outer_zips)} klara={len(done)} kvar={len(remaining)}\n")
|
||||
|
||||
out = open(OUT, "a", encoding="utf-8")
|
||||
ntot, nftot, t0 = 0, 0, time.time()
|
||||
for zi, (outer, _) in enumerate(remaining):
|
||||
tz = time.time()
|
||||
recs = process_zip(outer)
|
||||
nz = len(recs); nzf = sum(r.get("fact_count", 0) for r in recs)
|
||||
for rec in recs:
|
||||
out.write(json.dumps(rec, ensure_ascii=False, default=str) + "\n")
|
||||
ntot += nz; nftot += nzf
|
||||
done.add(outer)
|
||||
with open(DONE, "w") as d: d.write("\n".join(sorted(done)))
|
||||
dt = max(time.time() - t0, 0.1)
|
||||
sys.stderr.write(
|
||||
f"[{len(done)}/{len(outer_zips)}] {os.path.basename(outer)}: {nz} bolag {nzf} fakta "
|
||||
f"({nz/max(time.time()-tz,0.1):.1f} b/s, totalt {ntot/max(dt,1):.1f} b/s)\n")
|
||||
sys.stderr.flush(); out.flush()
|
||||
|
||||
out.close()
|
||||
dt = time.time() - t0
|
||||
sys.stderr.write(f"# KLAR bolag={ntot} fakta={nftot} {ntot/max(dt,1):.1f} b/s\n")
|
||||
@@ -0,0 +1,55 @@
|
||||
# VehIQ Company Analysis — analysera alla bolag på alla sätt
|
||||
|
||||
Komplement till MarketMute-pipelinen: fordonsbransch-analys av ALLA 1 035 060 svenska bolag.
|
||||
|
||||
## `vehiq_analysis_v2.py`
|
||||
Två pass över korpuset:
|
||||
1. Regionala marknadsaggregat per postort (n, omsättningssumma/median) → `vehiq_market_summary.json` (1 718 orter)
|
||||
2. Per bolag: fordonssrelevans-score (0-100), segment, marknadsposition (oms vs oms-median i orten), finans, kreditrisk, kontakter, LEI, moms
|
||||
|
||||
## Klassificering
|
||||
- **SNI (där SCB FDB täcker)**: 771xx uthyrning/leasing, 493xx/494xx transportörer, 96010 biltvätt (+60p)
|
||||
- **Bolagsverkets verksamhetsbeskrivning** (skriven ur SNI-registrering): "bilverkstadsrörelse", "handel med begagnade bilar", "försäljning av nya och begagnade bilar", "bildelar", "däckverkstad", "billackering", "bilglass", "biluthyrning" (+40p) — OBS: SCB JE-exporten saknar 45xx helt (verifierat), därför är nyckelordssvaret primär källa för handel/verkstad
|
||||
- Fordonsbokföring i balansräkningen (+10p)
|
||||
|
||||
## Leveranser
|
||||
- `vehiq_company_analysis.jsonl` (1,2 GB) — alla 1 020 860 bolag
|
||||
- `vehiq_bilbransch.jsonl` — 26 670 fordonsbransch-bolag: transportörer 15 100 · verkstäder 3 677 · bilhandlare 2 493 · uthyrning/leasing 1 326 · reservdelar/däck 865 · motorcykel m fl
|
||||
- `vehiq_market_summary.json` — 1 718 orter med marknadsaggregat
|
||||
|
||||
## Efterlevnad
|
||||
Företagskopplade personposter endast (styrelse/undertecknare ur iXBRL, revisorer exkluderade). Inga privatpersonregister.
|
||||
|
||||
## v3 (FINAL) — 37 648 fordonsbransch-bolag
|
||||
|
||||
`vehiq_v3.py` ersätter v2: expanderade 45xx-fraser (handel med personbilar, åkerier,
|
||||
entreprenadmaskiner, bildäck, begagnade personbilar ...) → **bilhandlare 5 526 · verkstäder 4 625 ·
|
||||
transportörer 17 530 · entreprenad 3 465 · uthyrning/leasing 3 033 · reservdelar/däck 728**.
|
||||
|
||||
VehIQ-marknadsdata från vehiq-db (server10, 98M-fordonstabellen):
|
||||
- `market_value_se.jsonl` — 10 511 märke×modell×år-prisposter (median/p10-p90/annonser) → `vehiq_market_context.json`
|
||||
- `fleet_composition_se.jsonl` — bestånd per märke (Volvo 1,62M) och topp 1 500 modeller
|
||||
- Fordonsägare per bolag kräver fortfarande Transportstyrelsen-prenumeration (ägaruppgifter ingår inte i registerkällan)
|
||||
|
||||
## Telefonnummer via ratsit.se (Simon beslut 2026-09-07)
|
||||
|
||||
`ratsit_connector.py` — kör LOKALT från ASUS (residensiellt IP krävs; Hetzner-IP är
|
||||
CF-bannad på mrkoll, ratsit 403 från datacenter). Headless Chromium, endast
|
||||
företagssidor ratsit.se/<orgnr>, 2 workers á 4s väntan + 2.5-5s jitter.
|
||||
|
||||
- Extraktion: `Telefon</h2>`/`Telefon:</td>`-blocket, primärt `href="tel:..."`-länk
|
||||
- Kraschtålig: browser-omstart per worker vid EPIPE/fel, resume via done.txt
|
||||
- Mål: vehiq_bilbransch prioriterat (bilhandlare → verkstad → leasing → reservdelar → transportör, störst omsättning först)
|
||||
- Utdata: telefon_ratsit.jsonl {orgnr, telefon, status, kalla, hamtat}
|
||||
- Hittills: ~57% telefonträff, 0 blockeringar; toppsegment ~1 dygn, hela listan ~3 dygn
|
||||
- hitta.se: telefon bakom "Visa nummer"-betallösning → ej använd; mrkoll.se: CF-bannad IP
|
||||
|
||||
## DEFINITIVA leveranser (dedupe per orgnr + full VIES + telefon)
|
||||
|
||||
- `marketmute_targets_DEFINITIV.jsonl` — 8 095 unika cold call-mål (1 776 Hög + 6 319 Medium),
|
||||
100% moms-status (vies_targets + vies_extra), telefon löpande, 1 939 med VD direkt
|
||||
- `vehiq_bilbransch_DEFINITIV.jsonl` — 15 442 unika fordonsbransch-bolag (transportörer 7 248,
|
||||
bilhandlare 2 286, verkstäder 1 911, entreprenad 1 393, uthyrning/leasing 1 232,
|
||||
reservdelar/däck 287), 4 583 telefoner (30%, hämtning pågår), 2 484 med VD
|
||||
- `mm_definitive.py` / `veh_definitive.py` — kör om när telefonkonnektorn är färdig →
|
||||
100% telefonmärkning av båda listorna
|
||||
@@ -0,0 +1,34 @@
|
||||
# MarketMute Cold Call-Intelligence Pipeline
|
||||
|
||||
Identifierar svenska bolag som bedriver cold calling / telefonförsäljning.
|
||||
|
||||
## Pipeline (i körordning)
|
||||
|
||||
1. `build_scb_enrichment.py` — SCB Företagsdatabas (scb_bulkfil.zip) → per-orgnr: SNI1-5, reklamspärr, jurform, adress → `scb_fdb.jsonl` (1,82M orgnr)
|
||||
2. `kontaktpersoner_v2.py` — extraherar styrelse/VD/undertecknare ur alla Underskrift*-concepts i iXBRL-korpuset (revisorer exkluderade) → `kontaktpersoner_v2.jsonl` (405k orgnr, 96% täckning)
|
||||
3. `gleif_fetch2.py` — alla svenska LEI via GLEIF API (datum-split för att undvika 10k djup-pagineringsgräns) → `gleif_se.jsonl` (95k LEI)
|
||||
4. `vies_targets.py` / `vies_extra.py` — VIES-momsvalidering för målbolag (**SE + orgnr + "01"** är korrekt momsnummer-format) → `vies_targets.jsonl`
|
||||
5. `marketmute_v2.py` — nyckelords-scoring (viktade + staplande: STRONG 40p / MEDIUM 25p / WEAK 10p, mangled-encoding-varianter)
|
||||
6. `marketmute_v3.py` — slutlig rebuild: SNI (starkast: 82200/73200/82910 = +50p, sektor = +30p) + nyckelord + finans + VIES, **reklamspärr = hårt exkluderingsfilter**, avregistrerade filtreras
|
||||
|
||||
## Leverans
|
||||
|
||||
`marketmute_targets_v3.jsonl`: 18 854 mål (3 766 Hög + 15 088 Medium), 98% kontakttäckning, 4 774 med VD direkt.
|
||||
|
||||
## Efterlevnad
|
||||
|
||||
- Reklamspärr (SCB Reklamsparrtyp=2) → exkluderas och märks "Exkluderad (reklamspärr)"
|
||||
- Avregistrerade bolag → "Ej mål (avregistrerat)"
|
||||
- Inga personuppgifter utöver företagskopplade styrelseposter (GDPR-ok)
|
||||
- Telefonnummer: kräver licensierad källa (Teleadress/Krux) — skrapning av hitta/ratsit/mrkoll är förbjuden
|
||||
|
||||
## Körning (server8)
|
||||
|
||||
```bash
|
||||
/opt/0pt-venv/bin/python build_scb_enrichment.py
|
||||
/opt/0pt-venv/bin/python kontaktpersoner_v2.py
|
||||
/opt/0pt-venv/bin/python gleif_fetch2.py
|
||||
/opt/0pt-venv/bin/python vies_targets.py
|
||||
/opt/0pt-venv/bin/python build_marketmute_v2.py
|
||||
/opt/0pt-venv/bin/python marketmute_v3.py
|
||||
```
|
||||
@@ -0,0 +1,299 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Bygger 'Car Dealer Intelligence' dataset från alla tillgängliga källor.
|
||||
Extraherar allt intressant för bilhandlare: ekonomi, leasing, IT, fordonsflotta, kreditrisk.
|
||||
"""
|
||||
import sys, json, gzip, csv, io, re, zipfile, requests
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
|
||||
# --- Ladda Bolagsverket bulkfil (orgnr -> postadress, verksamhet, form) ---
|
||||
def load_bulkfil(bulkfil_path):
|
||||
"""Laddar Bolagsverket bulkfil (semikolon-separerad txt)."""
|
||||
org_data = {}
|
||||
with zipfile.ZipFile(bulkfil_path) as z:
|
||||
for name in z.namelist():
|
||||
if name.endswith('.txt'):
|
||||
with z.open(name) as f:
|
||||
content = f.read().decode('latin-1')
|
||||
reader = csv.DictReader(io.StringIO(content), delimiter=';')
|
||||
for row in reader:
|
||||
orgnr_full = row.get('organisationsidentitet', '').strip()
|
||||
# Format: "5561233775$ORGNR-IDORG"
|
||||
orgnr_match = re.match(r'^(\d{10})\$', orgnr_full)
|
||||
if orgnr_match:
|
||||
orgnr = orgnr_match.group(1)
|
||||
orgnr_fmt = f"{orgnr[:6]}-{orgnr[6:]}"
|
||||
|
||||
# Parsa postadress: "Street$$CITY$POSTNR$SE-LAND"
|
||||
postadress = (row.get('postadress') or '').strip()
|
||||
street = city = postnr = ''
|
||||
if postadress:
|
||||
parts = postadress.split('$$')
|
||||
if len(parts) >= 2:
|
||||
street = parts[0]
|
||||
city_post = parts[1].split('$')
|
||||
if len(city_post) >= 2:
|
||||
city = city_post[0]
|
||||
postnr = city_post[1]
|
||||
|
||||
org_data[orgnr_fmt] = {
|
||||
'orgnr': orgnr_fmt,
|
||||
'namn': row.get('organisationsnamn', '').split('$')[0],
|
||||
'form': row.get('organisationsform', ''),
|
||||
'reg_datum': row.get('registreringsdatum', ''),
|
||||
'avreg_datum': row.get('avregistreringsdatum', ''),
|
||||
'avreg_orsak': row.get('avregistreringsorsak', ''),
|
||||
'verksamhet': row.get('verksamhetsbeskrivning', ''),
|
||||
'street': street,
|
||||
'city': city,
|
||||
'postnr': postnr,
|
||||
}
|
||||
print(f"[Bulkfil] Laddade {len(org_data):,} företag")
|
||||
return org_data
|
||||
|
||||
# --- SCB postnummer -> kommun/län ---
|
||||
def load_scb_postnummer():
|
||||
url = "https://www.scb.se/contentassets/ca21efb41fee4a82ad3bee3d2d4a4b00/postnummer_till_kommun_lan.csv"
|
||||
try:
|
||||
r = requests.get(url, timeout=30)
|
||||
r.raise_for_status()
|
||||
reader = csv.DictReader(io.StringIO(r.text))
|
||||
mapping = {}
|
||||
for row in reader:
|
||||
pnr = row.get("Postnummer", "").strip()
|
||||
if pnr:
|
||||
mapping[pnr] = {
|
||||
"postort": row.get("Postort", "").strip(),
|
||||
"kommun": row.get("Kommun", "").strip(),
|
||||
"lan": row.get("Lan", "").strip(),
|
||||
"lan_kod": row.get("LanKod", "").strip()
|
||||
}
|
||||
print(f"[SCB Postnummer] {len(mapping)} postnummer")
|
||||
return mapping
|
||||
except Exception as e:
|
||||
print(f"[SCB Postnummer] Fel: {e}")
|
||||
return {}
|
||||
|
||||
# --- VIES (momsregistret) - batch via async ---
|
||||
def check_vies_batch(orgnrs, batch_size=100):
|
||||
"""Kontrollerar VIES för batch av orgnr."""
|
||||
results = {}
|
||||
for i in range(0, len(orgnrs), batch_size):
|
||||
batch = orgnrs[i:i+batch_size]
|
||||
# VIES SOAP är komplex, vi markerar för efterföljande async jobb
|
||||
for orgnr in batch:
|
||||
results[orgnr] = {"vies_valid": None, "status": "pending"}
|
||||
return results
|
||||
|
||||
# --- GLEIF LEI ---
|
||||
def check_lei_batch(orgnrs, batch_size=100):
|
||||
results = {}
|
||||
for i in range(0, len(orgnrs), batch_size):
|
||||
batch = orgnrs[i:i+batch_size]
|
||||
for orgnr in batch:
|
||||
results[orgnr] = {"lei": None, "lei_status": None, "status": "pending"}
|
||||
return results
|
||||
|
||||
# --- Extrahera nyckeltal från iXBRL-fakta ---
|
||||
def extract_ratios(facts):
|
||||
"""Beräknar finansiella nyckeltal från fakta."""
|
||||
# Samla senaste värden per concept
|
||||
latest = {}
|
||||
for f in facts:
|
||||
qname = f.get('concept_qname', '')
|
||||
val = f.get('numeric')
|
||||
period_end = f.get('period_end')
|
||||
instant = f.get('instant')
|
||||
date = period_end or instant
|
||||
if val is not None and date:
|
||||
if qname not in latest or date > latest[qname]['date']:
|
||||
latest[qname] = {'value': val, 'date': date, 'unit': f.get('unitRef')}
|
||||
|
||||
# Hjälpfunktion
|
||||
def get(qname):
|
||||
return latest.get(qname, {}).get('value')
|
||||
|
||||
ratios = {}
|
||||
|
||||
# Grundläggande
|
||||
omsattning = get('se-gen-base:Nettoomsattning') or get('se-cd-base:Nettoomsattning')
|
||||
resultat = get('se-gen-base:AretsResultat') or get('se-gen-base:ResultatEfterFinansiellaPoster')
|
||||
soliditet = get('se-gen-base:Soliditet')
|
||||
eget_kapital = get('se-gen-base:EgetKapital')
|
||||
tillgangar = get('se-gen-base:Tillgangar')
|
||||
skulder = get('se-gen-base:Skulder') or (get('se-gen-base:LangfristigaSkulder') or 0) + (get('se-gen-base:KortfristigaSkulder') or 0)
|
||||
kassa = get('se-gen-base:KassaBank')
|
||||
anstallda = get('se-gen-base:MedelantaletAnstallda')
|
||||
|
||||
# Leasing / hyra (från anmärkningar)
|
||||
leasing_kostnad = get('se-gen-base:Hyresleasingkostnader') or get('se-gen-base:Leasingkostnader')
|
||||
leasing_forpliktelser = get('se-gen-base:Leasingforpliktelser') or get('se-gen-base:Hyresforpliktelser')
|
||||
|
||||
# IT / mjukvara
|
||||
it_kostnader = get('se-gen-base:ITkostnader') or get('se-gen-base:Mjukvarukostnader') or get('se-gen-base:Datorkostnader')
|
||||
avskrivningar = get('se-gen-base:AvskrivningarNedskrivningarMateriellaImmateriellaAnlaggningstillgangar')
|
||||
immateriella = get('se-gen-base:ImmateriellaAnlaggningstillgangar')
|
||||
|
||||
# Fordon (om specificerat)
|
||||
fordon = get('se-gen-base:MaskinerInventarierFordon') or get('se-gen-base:Fordon')
|
||||
|
||||
# Beräknade ratios
|
||||
if omsattning and omsattning > 0:
|
||||
ratios['omsattning'] = omsattning
|
||||
ratios['vinstmarginal'] = (resultat / omsattning * 100) if resultat else None
|
||||
ratios['omsattning_per_anstalld'] = (omsattning / anstallda) if anstallda else None
|
||||
|
||||
if tillgangar and tillgangar > 0:
|
||||
ratios['soliditet_beraknad'] = (eget_kapital / tillgangar * 100) if eget_kapital else None
|
||||
ratios['skuldsattning'] = (skulder / tillgangar * 100) if skulder else None
|
||||
|
||||
if eget_kapital:
|
||||
ratios['eget_kapital'] = eget_kapital
|
||||
ratios['roa'] = (resultat / tillgangar * 100) if resultat and tillgangar else None
|
||||
ratios['roe'] = (resultat / eget_kapital * 100) if resultat else None
|
||||
|
||||
if kassa and skulder:
|
||||
ratios['likviditet'] = (kassa / skulder * 100) if skulder else None
|
||||
ratios['kassa_till_kortfristig'] = (kassa / get('se-gen-base:KortfristigaSkulder') * 100) if get('se-gen-base:KortfristigaSkulder') else None
|
||||
|
||||
if anstallda:
|
||||
ratios['anstallda'] = anstallda
|
||||
ratios['resultat_per_anstalld'] = (resultat / anstallda) if resultat else None
|
||||
|
||||
# Leasing-indikatorer (viktigt för bilhandlare)
|
||||
if leasing_kostnad:
|
||||
ratios['leasing_kostnad'] = leasing_kostnad
|
||||
ratios['leasing_andel_oms'] = (leasing_kostnad / omsattning * 100) if omsattning else None
|
||||
if leasing_forpliktelser:
|
||||
ratios['leasing_forpliktelser'] = leasing_forpliktelser
|
||||
|
||||
# IT/Tech-indikatorer
|
||||
if it_kostnader:
|
||||
ratios['it_kostnader'] = it_kostnader
|
||||
ratios['it_andel_oms'] = (it_kostnader / omsattning * 100) if omsattning else None
|
||||
if avskrivningar:
|
||||
ratios['avskrivningar'] = avskrivningar
|
||||
ratios['avskrivnings_andel_oms'] = (avskrivningar / omsattning * 100) if omsattning else None
|
||||
if immateriella:
|
||||
ratios['immateriella_tillgangar'] = immateriella
|
||||
ratios['immateriella_andel_eget_kapital'] = (immateriella / eget_kapital * 100) if eget_kapital else None
|
||||
|
||||
# Fordon
|
||||
if fordon:
|
||||
ratios['fordon_bokfort'] = fordon
|
||||
|
||||
# Kreditrisk-indikatorer
|
||||
ratios['risk_soliditet_low'] = soliditet is not None and soliditet < 10
|
||||
ratios['risk_negativt_ek'] = eget_kapital is not None and eget_kapital < 0
|
||||
ratios['risk_negativt_resultat'] = resultat is not None and resultat < -100000
|
||||
ratios['risk_hog_skuldsattning'] = (ratios.get('skuldsattning') or 0) > 80
|
||||
ratios['risk_lag_likviditet'] = (ratios.get('likviditet') or 100) < 50
|
||||
|
||||
# Sammanfattad risk-score (0-100)
|
||||
risk_score = 0
|
||||
if ratios['risk_soliditet_low']: risk_score += 25
|
||||
if ratios['risk_negativt_ek']: risk_score += 25
|
||||
if ratios['risk_negativt_resultat']: risk_score += 20
|
||||
if ratios['risk_hog_skuldsattning']: risk_score += 15
|
||||
if ratios['risk_lag_likviditet']: risk_score += 15
|
||||
ratios['kreditrisk_score'] = min(risk_score, 100)
|
||||
ratios['kreditrisk_klass'] = 'Hög' if risk_score >= 60 else 'Medium' if risk_score >= 30 else 'Låg'
|
||||
|
||||
# Bransch (från verksamhetsbeskrivning)
|
||||
return ratios
|
||||
|
||||
def enrich_company(rec, org_data, scb_postnr):
|
||||
"""Anrikar en bolagspost med all intelligens."""
|
||||
orgnr = rec.get('orgnr')
|
||||
if not orgnr:
|
||||
return rec
|
||||
|
||||
# 1. Bolagsverket data
|
||||
if orgnr in org_data:
|
||||
bd = org_data[orgnr]
|
||||
rec['bolagsverket'] = bd
|
||||
# Postnummer -> kommun/län
|
||||
if bd.get('postnr') and bd['postnr'] in scb_postnr:
|
||||
p = scb_postnr[bd['postnr']]
|
||||
rec['kommun'] = p['kommun']
|
||||
rec['lan'] = p['lan']
|
||||
rec['lan_kod'] = p['lan_kod']
|
||||
rec['postort'] = p['postort']
|
||||
|
||||
# 2. Finansiella ratios från iXBRL
|
||||
ratios = extract_ratios(rec.get('facts', []))
|
||||
if ratios:
|
||||
rec['finansiella_ratios'] = ratios
|
||||
|
||||
# 3. VIES/LEI placeholders (fylls async)
|
||||
rec['vies'] = {"vies_valid": None, "status": "pending"}
|
||||
rec['lei'] = {"lei": None, "lei_status": None, "status": "pending"}
|
||||
rec['sanctioned'] = False
|
||||
rec['pep'] = False
|
||||
|
||||
# 4. Bransch-kod från SNI (om tillgänglig i facts)
|
||||
for f in rec.get('facts', []):
|
||||
if 'SNI' in f.get('concept_qname', '') or 'Bran' in f.get('concept_qname', ''):
|
||||
rec['sni_kod'] = f.get('value')
|
||||
break
|
||||
|
||||
return rec
|
||||
|
||||
def main():
|
||||
in_path = sys.argv[1] if len(sys.argv) > 1 else "/data/0pt/output/arsredovisningar_enriched.jsonl"
|
||||
out_path = sys.argv[2] if len(sys.argv) > 2 else "/data/0pt/output/bilhandlare_intelligence.jsonl"
|
||||
bulkfil = sys.argv[3] if len(sys.argv) > 3 else "/data/0pt/landing/bolagsverket/bolagsverket_bulkfil.zip"
|
||||
|
||||
print(f"[INFO] Laddar Bolagsverket bulkfil...")
|
||||
org_data = load_bulkfil(bulkfil)
|
||||
|
||||
print(f"[INFO] Laddar SCB postnummer...")
|
||||
scb_postnr = load_scb_postnummer()
|
||||
|
||||
# Läs alla orgnr för VIES/LEI
|
||||
print(f"[INFO] Läser orgnr från {in_path}...")
|
||||
orgnrs = []
|
||||
with open(in_path, "r") as f:
|
||||
for line in f:
|
||||
rec = json.loads(line)
|
||||
if rec.get("orgnr"):
|
||||
orgnrs.append(rec["orgnr"])
|
||||
orgnrs = list(set(orgnrs))
|
||||
print(f" Unika orgnr: {len(orgnrs):,}")
|
||||
|
||||
# VIES/LEI placeholders
|
||||
vies_map = check_vies_batch(orgnrs)
|
||||
lei_map = check_lei_batch(orgnrs)
|
||||
|
||||
# Streaming enrichment
|
||||
print(f"[INFO] Skriver Car Dealer Intelligence till {out_path}...")
|
||||
count = 0
|
||||
with_ratios = 0
|
||||
with_bolag = 0
|
||||
with_kommun = 0
|
||||
|
||||
with open(in_path, "r") as inf, open(out_path, "w") as outf:
|
||||
for line in inf:
|
||||
rec = json.loads(line)
|
||||
rec = enrich_company(rec, org_data, scb_postnr)
|
||||
|
||||
if rec.get('finansiella_ratios'):
|
||||
with_ratios += 1
|
||||
if rec.get('bolagsverket'):
|
||||
with_bolag += 1
|
||||
if rec.get('kommun'):
|
||||
with_kommun += 1
|
||||
|
||||
outf.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
count += 1
|
||||
if count % 100000 == 0:
|
||||
print(f" {count:,} poster (ratios:{with_ratios}, bolag:{with_bolag}, kommun:{with_kommun})")
|
||||
|
||||
print(f"[KLAR] {count:,} poster")
|
||||
print(f" Med finansiella ratios: {with_ratios:,}")
|
||||
print(f" Med Bolagsverket data: {with_bolag:,}")
|
||||
print(f" Med kommun/län: {with_kommun:,}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env python3
|
||||
"""MarketMute Intelligence v2: viktade + staplande nyckelord, breda cold call-sektorer."""
|
||||
import sys, json, csv, io, re, zipfile
|
||||
from pathlib import Path
|
||||
|
||||
# NYCKELORD MED VIKT — staplas alla matchningar
|
||||
# STRONG (40p): explicit telemarketing / telefonförsäljning
|
||||
KEYWORDS_STRONG = [
|
||||
'telemarketing', 'telesales', 'telefonförsäljning', 'telefonmarknadsföring',
|
||||
'kallcenter', 'callcenter', 'call center', 'kundcenter', 'utgående samtal',
|
||||
'outbound', 'cold call', 'telemarknadsföring', 'direktförsäljning',
|
||||
'kalla upp', 'telefonintervjuer', 'telefonkonsultation',
|
||||
]
|
||||
# MEDIUM (25p): branscher som är kända cold call-sektorer i Sverige
|
||||
KEYWORDS_MEDIUM = [
|
||||
'elavtal', 'elhandel', 'elleverantör', 'elavtalsförsäljning', 'elmarknadsföring',
|
||||
'solceller', 'solpaneler', 'solel', 'solcellsanläggningar',
|
||||
'värmepump', 'bergvärme', 'luftvärmepump', 'luft/vatten-värmepump',
|
||||
'försäkringsförsäljning', 'försäkringsförmedling', 'försäkringsmäklare',
|
||||
'inkasso', 'skuldsanering', 'fordringshantering', 'betalningsanmärkning',
|
||||
'bredband', 'mobilabonnemang', 'tv-paket', 'telefoni', 'telekommunikation',
|
||||
'pension', 'premiepension', 'fondsparande', 'kapitalförsäkring',
|
||||
'fastighetsförmedling', 'bostadsförmedling', 'mäklare',
|
||||
'leadgenerering', 'lead generation', 'kvalificerade leads', 'mötessättning',
|
||||
'appointment setting', 'sales development', 'prospecting',
|
||||
'affärssystem', 'crmsystem', 'saas', 'mjukvaruförsäljning',
|
||||
'utbildningsförsäljning', 'kursförsäljning', 'studiemedel', 'yrkesutbildning',
|
||||
]
|
||||
# WEAK (10p): generiska försäljningsord — bara stack
|
||||
KEYWORDS_WEAK = [
|
||||
'aktiv försäljning', 'aktiv marknadsföring', 'direktmarknadsföring',
|
||||
'b2b försäljning', 'b2c försäljning', 'tjänsteförsäljning', 'avtalsförsäljning',
|
||||
]
|
||||
|
||||
# MANGLED VERSIONER (latin-1 läst som utf-8: å=Ã¥, ä=ä, ö=ö)
|
||||
def mangle(w):
|
||||
try: return w.encode('utf-8').decode('latin-1')
|
||||
except Exception: return w
|
||||
|
||||
KEYWORDS_STRONG_M = [mangle(w) for w in KEYWORDS_STRONG]
|
||||
KEYWORDS_MEDIUM_M = [mangle(w) for w in KEYWORDS_MEDIUM]
|
||||
KEYWORDS_WEAK_M = [mangle(w) for w in KEYWORDS_WEAK]
|
||||
|
||||
EXCLUDE_KEYWORDS = ['endast inkommande', 'kundsupport', 'kundtjänst', 'vi ringer inte', 'inget kallcenter']
|
||||
EXCLUDE_M = [mangle(w) for w in EXCLUDE_KEYWORDS]
|
||||
|
||||
NAME_KEYWORDS = ['telemarketing', 'telesales', 'kallcenter', 'callcenter', 'outbound',
|
||||
'leads', 'sälj', 'försälj', 'inkasso', 'elavtal', 'elhandel',
|
||||
'solcell', 'värmepump', 'telekom', 'breddb', 'mäklar', 'lead']
|
||||
NAME_M = [mangle(w) for w in NAME_KEYWORDS]
|
||||
|
||||
def load_bulkfil(bulkfil_path):
|
||||
org_data = {}
|
||||
with zipfile.ZipFile(bulkfil_path) as z:
|
||||
for name in z.namelist():
|
||||
if name.endswith('.txt'):
|
||||
with z.open(name) as f:
|
||||
content = f.read().decode('latin-1')
|
||||
reader = csv.DictReader(io.StringIO(content), delimiter=';')
|
||||
for row in reader:
|
||||
orgnr_full = row.get('organisationsidentitet') or ''
|
||||
m = re.match(r'^(\d{10})\$', orgnr_full.strip())
|
||||
if m:
|
||||
orgnr = m.group(1)
|
||||
fmt = f"{orgnr[:6]}-{orgnr[6:]}"
|
||||
postadress = (row.get('postadress') or '').strip()
|
||||
street = city = postnr = ''
|
||||
if postadress:
|
||||
parts = postadress.split('$$')
|
||||
if len(parts) >= 2:
|
||||
street = parts[0]
|
||||
cp = parts[1].split('$')
|
||||
if len(cp) >= 2:
|
||||
city = cp[0]; postnr = cp[1]
|
||||
org_data[fmt] = {
|
||||
'namn': (row.get('organisationsnamn') or '').split('$')[0],
|
||||
'form': row.get('organisationsform') or '',
|
||||
'reg_datum': row.get('registreringsdatum') or '',
|
||||
'avreg_datum': row.get('avregistreringsdatum') or '',
|
||||
'verksamhet': (row.get('verksamhetsbeskrivning') or '').split('$')[0],
|
||||
'street': street, 'city': city, 'postnr': postnr,
|
||||
}
|
||||
print(f"[Bulkfil] {len(org_data):,} företag")
|
||||
return org_data
|
||||
|
||||
def score_cold_calling(rec, org_data):
|
||||
score = 0
|
||||
reasons = []
|
||||
bd = org_data.get(rec.get('orgnr'), {})
|
||||
|
||||
# fix encoding
|
||||
def fix(t):
|
||||
if not t: return ''
|
||||
try: return t.encode('latin-1').decode('utf-8').lower()
|
||||
except Exception: return t.lower()
|
||||
|
||||
verk = fix(bd.get('verksamhet'))
|
||||
verk_raw = (bd.get('verksamhet') or '').lower()
|
||||
|
||||
# 1. Nyckelord — stapla alla matchningar
|
||||
for w in KEYWORDS_STRONG:
|
||||
if w in verk:
|
||||
score += 40; reasons.append(f"STRONG: '{w}'"); break
|
||||
if not any(w in verk for w in KEYWORDS_STRONG):
|
||||
# prova mangled versioner i rå text
|
||||
for w in KEYWORDS_STRONG_M:
|
||||
if w in verk_raw:
|
||||
score += 40; reasons.append(f"STRONG: '{w}' (mangled)"); break
|
||||
for w in KEYWORDS_MEDIUM:
|
||||
if w in verk:
|
||||
score += 25; reasons.append(f"MEDIUM: '{w}'"); break
|
||||
for w in KEYWORDS_MEDIUM_M:
|
||||
if w in verk_raw:
|
||||
score += 25; reasons.append(f"MEDIUM: '{w}' (mangled)"); break
|
||||
for w in KEYWORDS_WEAK:
|
||||
if w in verk:
|
||||
score += 10; reasons.append(f"WEAK: '{w}'"); break
|
||||
for w in KEYWORDS_WEAK_M:
|
||||
if w in verk_raw:
|
||||
score += 10; reasons.append(f"WEAK: '{w}' (mangled)"); break
|
||||
|
||||
# Exkludera
|
||||
for w in EXCLUDE_M + EXCLUDE_KEYWORDS:
|
||||
if w in verk_raw or w in verk:
|
||||
score = max(0, score - 40); reasons.append(f"Exkluderat: '{w}'"); break
|
||||
|
||||
# 2. Namn
|
||||
namn_raw = (bd.get('namn') or '').lower()
|
||||
namn = fix(bd.get('namn'))
|
||||
for w in NAME_KEYWORDS:
|
||||
if w in namn:
|
||||
score += 15; reasons.append(f"Namn: '{w}'"); break
|
||||
for w in NAME_M:
|
||||
if w in namn_raw:
|
||||
score += 15; reasons.append(f"Namn: '{w}' (mangled)"); break
|
||||
|
||||
# 3. Finansiella mönster
|
||||
r = rec.get('finansiella_ratios', {})
|
||||
oms, anst = r.get('omsattning'), r.get('anstallda')
|
||||
if oms and anst:
|
||||
opa = oms / anst
|
||||
if opa > 10000000:
|
||||
score += 15; reasons.append(f"oms/anst {opa:,.0f} (extrem)")
|
||||
elif opa > 3000000:
|
||||
score += 10; reasons.append(f"oms/anst {opa:,.0f}")
|
||||
res = r.get('resultat') or r.get('eget_kapital')
|
||||
if oms and oms > 5000000 and (r.get('vinstmarginal') or 0) < -20:
|
||||
score += 10; reasons.append("Förbränner kapital (vinstmarginal<-20% med hög oms)")
|
||||
|
||||
# 4. Form/ålder
|
||||
if bd.get('form') in ('ENSKILD', 'HB', 'KB'):
|
||||
score += 5; reasons.append(f"Form: {bd.get('form')}")
|
||||
reg = bd.get('reg_datum', '')
|
||||
if reg >= '2022-01-01':
|
||||
score += 10; reasons.append(f"Nystart 2022+: {reg}")
|
||||
elif reg >= '2020-01-01':
|
||||
score += 5; reasons.append(f"Nystart 2020+: {reg}")
|
||||
avreg = bd.get('avreg_datum', '')
|
||||
if avreg:
|
||||
score -= 10; reasons.append(f"Avregistrerat: {avreg}")
|
||||
|
||||
score = max(0, min(score, 100))
|
||||
if score >= 60: klass = 'Hög sannolikhet'
|
||||
elif score >= 35: klass = 'Medium sannolikhet'
|
||||
elif score >= 15: klass = 'Låg sannolikhet'
|
||||
else: klass = 'Osannolikt'
|
||||
return {'cold_call_score': score, 'cold_call_klass': klass, 'cold_call_reasons': reasons,
|
||||
'verksamhet': bd.get('verksamhet', ''), 'namn': bd.get('namn', '')}
|
||||
|
||||
def main():
|
||||
in_path = sys.argv[1] if len(sys.argv) > 1 else "/data/0pt/output/bilhandlare_intelligence.jsonl"
|
||||
out_path = sys.argv[2] if len(sys.argv) > 2 else "/data/0pt/output/marketmute_intelligence.jsonl"
|
||||
bulkfil = sys.argv[3] if len(sys.argv) > 3 else "/data/0pt/landing/bolagsverket/bolagsverket_bulkfil.zip"
|
||||
org_data = load_bulkfil(bulkfil)
|
||||
print(f"[INFO] Analyserar {in_path} -> {out_path}")
|
||||
from collections import Counter
|
||||
klass_c = Counter(); total = 0
|
||||
with open(in_path) as inf, open(out_path, "w") as outf:
|
||||
for line in inf:
|
||||
rec = json.loads(line)
|
||||
rec['marketmute'] = score_cold_calling(rec, org_data)
|
||||
klass_c[rec['marketmute']['cold_call_klass']] += 1
|
||||
outf.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
total += 1
|
||||
if total % 100000 == 0:
|
||||
print(f" {total:,} (Hög:{klass_c['Hög sannolikhet']}, Med:{klass_c['Medium sannolikhet']}, Låg:{klass_c['Låg sannolikhet']})")
|
||||
print(f"[KLAR] {total:,}")
|
||||
for k, c in klass_c.most_common():
|
||||
print(f" {k}: {c:,} ({c/total*100:.1f}%)")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Steg 1: SCB FDB -> orgnr -> {SNI1-5, reklamsparr, jurform, adress} -> scb_fdb.jsonl"""
|
||||
import zipfile, csv, io, json, sys
|
||||
|
||||
OUT = "/data/0pt/output/scb_fdb.jsonl"
|
||||
JURFORM_MAP = {
|
||||
"10": "Enskild firma", "12": "Bolagsman", "20": "Handelsbolag", "21": "Kommanditbolag",
|
||||
"22": "Kommanditbolag", "30": "Ekonomisk förening", "31": "Ekonomisk förening",
|
||||
"40": "Ideell förening", "47": "Bostadsrättsförening", "49": "Aktiebolag",
|
||||
"50": "Aktiebolag", "51": "Aktiebolag", "54": "Aktiebolag", "55": "Aktiebolag",
|
||||
"56": "Aktiebolag", "59": "Partrederi", "60": "Stiftelse", "63": "Stiftelse",
|
||||
"66": "Stiftelse", "71": "Stiftelse", "72": "Stiftelse", "75": "Stiftelse",
|
||||
"77": "Stiftelse", "80": "Sparbank", "83": "Enskild firma", "90": "Utländsk",
|
||||
"91": "Utländsk", "92": "Utländsk", "96": "Övrig", "99": "Okänd",
|
||||
}
|
||||
|
||||
n = 0
|
||||
with zipfile.ZipFile("/data/0pt/landing/bolagsverket/scb_bulkfil.zip") as z, open(OUT, "w") as out:
|
||||
with z.open(z.namelist()[0]) as f:
|
||||
reader = csv.DictReader(io.TextIOWrapper(f, encoding="latin-1"), delimiter="\t")
|
||||
for row in reader:
|
||||
orgnr = (row.get("PeOrgNr") or "").strip()
|
||||
if not orgnr or len(orgnr) < 10:
|
||||
continue
|
||||
digits = orgnr[:10]
|
||||
fmt = f"{digits[:6]}-{digits[6:]}" if not digits.startswith("16") else digits
|
||||
ng = [(row.get(f"Ng{i}") or "").strip() for i in range(1, 6)]
|
||||
ng = [x for x in ng if x and x != "00000"]
|
||||
rp = (row.get("Reklamsparrtyp") or "").strip()
|
||||
rec = {
|
||||
"orgnr": fmt,
|
||||
"orgnr_digits": digits,
|
||||
"sni": ng,
|
||||
"sni1": ng[0] if ng else None,
|
||||
"jurform_kod": (row.get("JurForm") or "").strip(),
|
||||
"jurform": JURFORM_MAP.get((row.get("JurForm") or "").strip()),
|
||||
"reklamsparr_typ": rp,
|
||||
"reklamsparr": rp == "2",
|
||||
"gatuadress": (row.get("Gatuadress") or "").strip(),
|
||||
"postnr": (row.get("PostNr") or "").strip(),
|
||||
"postort": (row.get("PostOrt") or "").strip(),
|
||||
"reg_datum": (row.get("RegDatKtid") or "").strip(),
|
||||
}
|
||||
out.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
n += 1
|
||||
if n % 200000 == 0:
|
||||
print(f" {n:,}...", flush=True)
|
||||
print(f"[KLAR] {n:,} -> {OUT}")
|
||||
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Slutmerge: targets_v3 + ALL VIES-data + encoding-fix -> marketmute_targets_final.jsonl"""
|
||||
import json
|
||||
|
||||
def fix_enc(t):
|
||||
"""Fixa mangled encoding (latin-1 läst som utf-8) -> korrekt svensk text."""
|
||||
if not t or "Ã" not in t:
|
||||
return t
|
||||
try:
|
||||
return t.encode("latin-1").decode("utf-8")
|
||||
except Exception:
|
||||
return t
|
||||
|
||||
# 1. Samla ALL VIES-data (båda filerna)
|
||||
vies = {}
|
||||
for path in ("/data/0pt/output/vies_targets.jsonl", "/data/0pt/output/vies_extra.jsonl"):
|
||||
try:
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
vies[r["orgnr_fmt"]] = r
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
print(f"[VIES] {len(vies):,} kontrollerade totalt")
|
||||
|
||||
# 2. Merge targets
|
||||
n = 0; with_vies = 0
|
||||
with open("/data/0pt/output/marketmute_targets_final.jsonl", "w") as out:
|
||||
for line in open("/data/0pt/output/marketmute_targets_v3.jsonl"):
|
||||
t = json.loads(line)
|
||||
org = t["orgnr"]
|
||||
# encoding-fix på textfält
|
||||
t["namn"] = fix_enc(t.get("namn"))
|
||||
t["verksamhet"] = fix_enc(t.get("verksamhet"))
|
||||
for p in t.get("kontaktpersoner", []):
|
||||
p["namn"] = fix_enc(p.get("namn"))
|
||||
# VIES-merge
|
||||
v = vies.get(org)
|
||||
if v and v.get("moms_valid") is not None:
|
||||
t["moms_valid"] = v["moms_valid"]
|
||||
t["momsnr"] = v.get("momsnr")
|
||||
if v.get("moms_namn"):
|
||||
t["moms_namn"] = fix_enc(v["moms_namn"])
|
||||
with_vies += 1
|
||||
else:
|
||||
t["moms_valid"] = None
|
||||
t["momsnr"] = f"SE{org.replace('-','')}01"
|
||||
out.write(json.dumps(t, ensure_ascii=False) + "\n")
|
||||
n += 1
|
||||
print(f"[KLAR] {n:,} targets -> marketmute_targets_final.jsonl ({with_vies:,} med moms-status)")
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Steg 3 fixad: GLEIF via initialRegistrationDate-split (undviker 10k djup-gräns)."""
|
||||
import json, time, requests
|
||||
|
||||
OUT = "/data/0pt/output/gleif_se.jsonl"
|
||||
sess = requests.Session()
|
||||
total = 0
|
||||
YEARS = [(f"{y}-01-01", f"{y}-12-31") for y in range(2012, 2027)]
|
||||
with open(OUT, "w") as out:
|
||||
for y0, y1 in YEARS:
|
||||
page = 1
|
||||
while True:
|
||||
ok = False
|
||||
for attempt in range(5):
|
||||
try:
|
||||
r = sess.get("https://api.gleif.org/api/v1/lei-records", params={
|
||||
"filter[entity.legalAddress.country]": "SE",
|
||||
"filter[registration.initialRegistrationDate]": f"{y0}..{y1}",
|
||||
"page[size]": 200, "page[number]": page,
|
||||
}, timeout=30)
|
||||
if r.status_code == 429:
|
||||
time.sleep(5); continue
|
||||
r.raise_for_status()
|
||||
ok = True; break
|
||||
except Exception as e:
|
||||
if attempt == 4:
|
||||
print(f" FAIL {y0} p{page}: {e}", flush=True)
|
||||
break
|
||||
time.sleep(2 ** attempt)
|
||||
if not ok: break
|
||||
d = r.json()
|
||||
items = d.get("data", [])
|
||||
if not items: break
|
||||
for it in items:
|
||||
a = it.get("attributes", {})
|
||||
e = a.get("entity", {})
|
||||
reg = a.get("registration", {})
|
||||
out.write(json.dumps({
|
||||
"lei": a.get("lei"),
|
||||
"orgnr": (e.get("registeredAs") or "").strip(),
|
||||
"namn": (e.get("legalName") or {}).get("name"),
|
||||
"status": e.get("status"),
|
||||
"reg_status": reg.get("status"),
|
||||
}, ensure_ascii=False) + "\n")
|
||||
total += 1
|
||||
meta = d.get("meta", {}).get("pagination", {})
|
||||
if page >= meta.get("lastPage", 1): break
|
||||
page += 1
|
||||
time.sleep(0.3)
|
||||
print(f" {y0[:4]}: klart (totalt {total:,})", flush=True)
|
||||
print(f"[KLAR] {total:,} LEI -> {OUT}")
|
||||
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Grand final merge: ALL VIES + telefoner + encoding -> alla slutfiler."""
|
||||
import json
|
||||
from collections import defaultdict
|
||||
|
||||
def fix_enc(t):
|
||||
if not t or "Ã" not in t: return t
|
||||
try: return t.encode("latin-1").decode("utf-8")
|
||||
except Exception: return t
|
||||
|
||||
# 1. ALL VIES
|
||||
vies = {}
|
||||
for path in ("/data/0pt/output/vies_targets.jsonl", "/data/0pt/output/vies_extra.jsonl"):
|
||||
try:
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
vies[r["orgnr_fmt"]] = r
|
||||
except FileNotFoundError: pass
|
||||
print(f"[VIES] {len(vies):,}")
|
||||
|
||||
# 2. ALLA TELEFONER
|
||||
tel = {}
|
||||
for line in open("/data/0pt/output/telefon_ratsit.jsonl"):
|
||||
r = json.loads(line)
|
||||
tel[r["orgnr"]] = r.get("telefon")
|
||||
tel_hits = sum(1 for v in tel.values() if v)
|
||||
print(f"[TELEFON] {len(tel):,} kontrollerade, {tel_hits:,} nummer")
|
||||
|
||||
# 3. MARKETMUTE-TARGETS FINAL v2
|
||||
n = 0; v = 0; t = 0
|
||||
with open("/data/0pt/output/marketmute_targets_final.jsonl", "w") as out:
|
||||
for line in open("/data/0pt/output/marketmute_targets_v3.jsonl"):
|
||||
rec = json.loads(line)
|
||||
orgnr = rec["orgnr"]; digits = orgnr.replace("-", "")
|
||||
rec["namn"] = fix_enc(rec.get("namn")); rec["verksamhet"] = fix_enc(rec.get("verksamhet"))
|
||||
for p in rec.get("kontaktpersoner", []):
|
||||
p["namn"] = fix_enc(p.get("namn"))
|
||||
vr = vies.get(orgnr)
|
||||
if vr and vr.get("moms_valid") is not None:
|
||||
rec["moms_valid"] = vr["moms_valid"]; rec["momsnr"] = vr.get("momsnr")
|
||||
if vr.get("moms_namn"): rec["moms_namn"] = fix_enc(vr["moms_namn"])
|
||||
v += 1
|
||||
ph = tel.get(orgnr)
|
||||
if ph:
|
||||
rec["telefon"] = ph; rec["telefon_kalla"] = "ratsit.se"
|
||||
t += 1
|
||||
out.write(json.dumps(rec, ensure_ascii=False) + "\n"); n += 1
|
||||
print(f"[MARKETMUTE] {n:,} mål · moms: {v:,} · telefon: {t:,}")
|
||||
|
||||
# 4. VEHIQ BILBRANSCH v2 (med telefon + full VIES)
|
||||
n = 0; t = 0; v = 0
|
||||
with open("/data/0pt/output/vehiq_bilbransch_final.jsonl", "w") as out:
|
||||
for line in open("/data/0pt/output/vehiq_bilbransch.jsonl"):
|
||||
rec = json.loads(line)
|
||||
orgnr = rec["orgnr"]; digits = orgnr.replace("-", "")
|
||||
rec["namn"] = fix_enc(rec.get("namn")); rec["verksamhet"] = fix_enc(rec.get("verksamhet"))
|
||||
for p in rec.get("kontaktpersoner", []):
|
||||
p["namn"] = fix_enc(p.get("namn"))
|
||||
vr = vies.get(orgnr)
|
||||
if vr and vr.get("moms_valid") is not None:
|
||||
rec["moms_valid"] = vr["moms_valid"]; v += 1
|
||||
ph = tel.get(orgnr)
|
||||
if ph:
|
||||
rec["telefon"] = ph; rec["telefon_kalla"] = "ratsit.se"; t += 1
|
||||
out.write(json.dumps(rec, ensure_ascii=False) + "\n"); n += 1
|
||||
print(f"[VEHIQ-BILBRANSCH] {n:,} · telefon: {t:,} · moms: {v:,}")
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Steg 2: Full kontaktperson-extraktion från alla Underskrift-concepts i dedup JSONL."""
|
||||
import json, sys
|
||||
|
||||
IN = "/data/0pt/output/arsredovisningar_dedup.jsonl"
|
||||
OUT = "/data/0pt/output/kontaktpersoner_v2.jsonl"
|
||||
|
||||
# SIGNATUR-CONCEPTS (revisorer EXKLUDERAS)
|
||||
NAME_TRIPLES = [
|
||||
# (tilltalsnamn, efternamn, roll) — triples per signatursystem
|
||||
("UnderskriftFaststallelseintygForetradareTilltalsnamn", "UnderskriftFaststallelseintygForetradareEfternamn", "UnderskriftFaststallelseintygForetradareForetradarroll"),
|
||||
("UnderskriftArsredovisningForetradareTilltalsnamn", "UnderskriftArsredovisningForetradareEfternamn", "UnderskriftArsredovisningForetradareForetradarroll"),
|
||||
("UnderskriftHandlingTilltalsnamn", "UnderskriftHandlingEfternamn", "UnderskriftHandlingRoll"),
|
||||
]
|
||||
# Extra titel-concept per system
|
||||
TITEL = {
|
||||
"UnderskriftFaststallelseintygForetradareTilltalsnamn": "UnderskriftFaststallelseintygForetradareForetradarroll",
|
||||
"UnderskriftArsredovisningForetradareTilltalsnamn": "UnderskriftArsredovisningForetradareForetradarroll",
|
||||
"UnderskriftHandlingTilltalsnamn": "UnderskriftHandlingTitel",
|
||||
}
|
||||
# Namn på prefix
|
||||
PREFIX = ("se-comp-base:", "se-gen-base:", "se-bol-base:", "se-ar-base:")
|
||||
|
||||
def norm_role(roll):
|
||||
r = (roll or "").lower()
|
||||
if not r:
|
||||
return "underskriftsansvarig", 3
|
||||
if "verks" in r and "direkt" in r or r.strip() in ("vd", "verkställande direktör", "v.d."):
|
||||
return "verkställande direktör", 0
|
||||
if "ordförand" in r or "ordforand" in r or "företrädande" in r.replace("företagande", ""):
|
||||
return "styrelseordförande", 1
|
||||
if "ledamot" in r:
|
||||
return "styrelseledamot", 2
|
||||
if "stiftare" in r:
|
||||
return "stiftare", 2
|
||||
if "innehavare" in r or "ägare" in r:
|
||||
return "innehavare", 0
|
||||
return "underskriftsansvarig", 3
|
||||
|
||||
def clean(s):
|
||||
return (s or "").strip()
|
||||
|
||||
org_persons = {}
|
||||
n = 0
|
||||
for line in open(IN):
|
||||
rec = json.loads(line)
|
||||
n += 1
|
||||
orgnr = rec.get("orgnr")
|
||||
if not orgnr:
|
||||
continue
|
||||
# bygga lookup senaste-värde per concept
|
||||
vals = {}
|
||||
for f in rec.get("facts", []):
|
||||
q = f.get("concept_qname", "")
|
||||
v = clean(f.get("value"))
|
||||
if v and q.endswith(("Tilltalsnamn", "Efternamn", "Foretradarroll", "Roll", "Titel")) and "Revisor" not in q and "Revisorsp" not in q:
|
||||
key = q.split(":")[-1]
|
||||
if key not in vals: # första förekomsten räcker per år — ta senaste helt: skriv över
|
||||
vals[key] = v
|
||||
# prova alla triples — maska suffix-prefix (HandlingRoll vs Foretradarroll)
|
||||
persons = org_persons.setdefault(orgnr, {})
|
||||
for tnam, enam, rnam in NAME_TRIPLES:
|
||||
tn = vals.get(tnam.split(":")[-1])
|
||||
en = vals.get(enam.split(":")[-1])
|
||||
if not tn and not en:
|
||||
continue
|
||||
namn = f"{tn} {en}".strip()
|
||||
if not namn:
|
||||
continue
|
||||
roll_raw = vals.get(rnam.split(":")[-1]) or vals.get("UnderskriftHandlingRoll") or ""
|
||||
roll, prio = norm_role(roll_raw)
|
||||
prev = persons.get(namn)
|
||||
if prev is None or prio < prev["prio"]:
|
||||
persons[namn] = {"namn": namn, "roll": roll, "prio": prio, "kalla": rnam.split(":")[-1]}
|
||||
if n % 100000 == 0:
|
||||
print(f" {n:,} bolag...", flush=True)
|
||||
|
||||
# serialisera
|
||||
count = 0
|
||||
with_persons = 0
|
||||
with open(OUT, "w") as out:
|
||||
for orgnr, persons in org_persons.items():
|
||||
lst = sorted(persons.values(), key=lambda p: p["prio"])
|
||||
has_vd = any(p["prio"] == 0 for p in lst)
|
||||
rec = {
|
||||
"orgnr": orgnr,
|
||||
"orgnr_digits": orgnr.replace("-", ""),
|
||||
"personer": [{"namn": p["namn"], "roll": p["roll"]} for p in lst],
|
||||
"antal": len(lst),
|
||||
"har_vd": has_vd,
|
||||
}
|
||||
out.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
count += 1
|
||||
if lst:
|
||||
with_persons += 1
|
||||
print(f"[KLAR] {count:,} orgnr, {with_persons:,} med personer -> {OUT}")
|
||||
@@ -0,0 +1,148 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Steg 5: Slutgiltig MarketMute-rebuild med SNI, reklamspärr, kontakter, LEI, moms."""
|
||||
import json, sys
|
||||
from collections import defaultdict
|
||||
|
||||
# --- SNI 2007 för cold call-sektorer ---
|
||||
SNI_STRONG = { # direkt cold call-verksamhet
|
||||
"82200": "Callcenter/telemarketing", "73200": "Marknadsundersökningar",
|
||||
"73110": "Reklambyråer", "73120": "Mediabyråer", "73130": "Reklambyråer",
|
||||
"82910": "Inkasso/kreditupplysning", "82990": "Övriga företagstjänster (leadgen)",
|
||||
}
|
||||
SNI_MEDIUM = { # kända telefonförsäljningssektorer
|
||||
"65110": "Livförsäkring", "65120": "Skadeförsäkring", "65200": "Återförsäkring",
|
||||
"66210": "Försäkringsriskbedömning", "66220": "Försäkringsmäklare",
|
||||
"64920": "Övrig kreditgivning (sms-lån)", "66190": "Övriga finansiella tjänster",
|
||||
"35120": "Elhandel", "35140": "Elhandel", "35110": "Elproduktion",
|
||||
"43220": "Installationer (solcell/värmepump)", "43290": "Övriga installationer",
|
||||
"43990": "Övrig specialiserad bygg", "68310": "Fastighetsmäklare",
|
||||
"68320": "Fastighetsförvaltning", "58290": "Programvaruutgivning",
|
||||
"62010": "Programmering", "62020": "IT-konsult", "62090": "Övrig IT",
|
||||
"61100": "Telekommunikation", "61200": "Mobiltelefoni", "61300": "Satellit", "61900": "Övrig tele",
|
||||
"85510": "Sport/yrkesutbildning", "85520": "Kulturu(t)bildning", "85590": "Övrig utbildning",
|
||||
"80200": "Gymnasial utbildning", "80300": "Högskoleutbildning", "47910": "E-handel",
|
||||
}
|
||||
SNI_PREFIX_MEDIUM = ("651", "652", "662", "649", "661", "351", "432", "439", "683", "582", "620", "611", "612", "619", "855", "731", "732", "822", "829")
|
||||
|
||||
def score_v3(rec, scb, vies_rec):
|
||||
score = 0; reasons = []
|
||||
m = rec.get("marketmute", {})
|
||||
# 1. SNI (starkast)
|
||||
sni1 = (scb.get("sni1") or "") if scb else ""
|
||||
if sni1 in SNI_STRONG:
|
||||
score += 50; reasons.append(f"SNI {sni1}: {SNI_STRONG[sni1]}")
|
||||
elif sni1 in SNI_MEDIUM:
|
||||
score += 30; reasons.append(f"SNI {sni1}: {SNI_MEDIUM[sni1]}")
|
||||
elif sni1[:3] in SNI_PREFIX_MEDIUM and sni1:
|
||||
score += 15; reasons.append(f"SNI-prefix {sni1[:3]}")
|
||||
# 2. Nyckelord från v2 (återanvänd)
|
||||
kw_reasons = [r for r in m.get("cold_call_reasons", []) if r.startswith(("STRONG", "MEDIUM", "WEAK", "Namn"))]
|
||||
for r in kw_reasons:
|
||||
if r.startswith("STRONG"): score += 40
|
||||
elif r.startswith("MEDIUM"): score += 25
|
||||
elif r.startswith("WEAK"): score += 10
|
||||
else: score += 15
|
||||
reasons.extend(kw_reasons)
|
||||
# 3. Finansiellt
|
||||
for r in m.get("cold_call_reasons", []):
|
||||
if "oms/anst" in r: score += 15 if "extrem" in r else 10; reasons.append(r)
|
||||
if "Förbränner" in r: score += 10; reasons.append(r)
|
||||
# 4. Ålder/form
|
||||
for r in m.get("cold_call_reasons", []):
|
||||
if r.startswith("Nystart"): score += 10 if "2022+" in r else 5; reasons.append(r)
|
||||
if r.startswith("Form:"): score += 5; reasons.append(r)
|
||||
# 5. Moms-registrerad (riktigt bolag)
|
||||
if vies_rec and vies_rec.get("moms_valid") is True:
|
||||
score += 5; reasons.append("Momsregistrerad (VIES)")
|
||||
# Avregistrerat — exkludera helt
|
||||
bd_reasons = m.get("cold_call_reasons", [])
|
||||
avreg = next((r for r in bd_reasons if r.startswith("Avregistrerat")), None)
|
||||
if avreg:
|
||||
return 0, "Ej mål (avregistrerat)", [avreg], sni1
|
||||
score = max(0, min(score, 100))
|
||||
if score >= 60: klass = "Hög sannolikhet"
|
||||
elif score >= 35: klass = "Medium sannolikhet"
|
||||
elif score >= 15: klass = "Låg sannolikhet"
|
||||
else: klass = "Osannolikt"
|
||||
return score, klass, reasons, sni1
|
||||
|
||||
def main():
|
||||
# ladda sidotabeller
|
||||
scb_map = {}
|
||||
for line in open("/data/0pt/output/scb_fdb.jsonl"):
|
||||
r = json.loads(line)
|
||||
scb_map[r["orgnr_digits"]] = r
|
||||
scb_map[r["orgnr"]] = r
|
||||
print(f"[SCB] {len(scb_map):,}", flush=True)
|
||||
contacts = {}
|
||||
for line in open("/data/0pt/output/kontaktpersoner_v2.jsonl"):
|
||||
r = json.loads(line)
|
||||
contacts[r["orgnr_digits"]] = r
|
||||
print(f"[Kontakter] {len(contacts):,}", flush=True)
|
||||
lei_map = {}
|
||||
for line in open("/data/0pt/output/gleif_se.jsonl"):
|
||||
r = json.loads(line)
|
||||
o = (r.get("orgnr") or "").replace("-", "").strip()
|
||||
if o: lei_map[o] = r
|
||||
print(f"[LEI] {len(lei_map):,}", flush=True)
|
||||
vies_map = {}
|
||||
for line in open("/data/0pt/output/vies_targets.jsonl"):
|
||||
r = json.loads(line)
|
||||
vies_map[r.get("orgnr_fmt", "")] = r
|
||||
print(f"[VIES] {len(vies_map):,}", flush=True)
|
||||
|
||||
# bygga utökad mållista från HELA korpusen (v3-nycklar: SNI kan träffa nya bolag)
|
||||
from collections import Counter
|
||||
klass_c = Counter(); sparr_c = 0; n = 0
|
||||
targets = []
|
||||
for line in open("/data/0pt/output/bilhandlare_intelligence.jsonl"):
|
||||
rec = json.loads(line)
|
||||
n += 1
|
||||
orgnr = rec.get("orgnr", "")
|
||||
digits = orgnr.replace("-", "")
|
||||
scb = scb_map.get(digits) or scb_map.get(orgnr)
|
||||
vies_rec = vies_map.get(orgnr)
|
||||
score, klass, reasons, sni1 = score_v3(rec, scb, vies_rec)
|
||||
# reklamspärr = hårt exkluderingsfilter (efterlevnad)
|
||||
sparrad = bool(scb and scb.get("reklamsparr"))
|
||||
if sparrad and klass in ("Hög sannolikhet", "Medium sannolikhet"):
|
||||
sparr_c += 1
|
||||
klass = "Exkluderad (reklamspärr)"
|
||||
klass_c[klass] += 1
|
||||
contact = contacts.get(digits)
|
||||
lei = lei_map.get(digits)
|
||||
if klass in ("Hög sannolikhet", "Medium sannolikhet"):
|
||||
targets.append({
|
||||
"orgnr": orgnr,
|
||||
"cold_call_score": score, "cold_call_klass": klass,
|
||||
"cold_call_reasons": reasons,
|
||||
"sni": scb.get("sni") if scb else [], "sni1": sni1,
|
||||
"namn": (scb or {}).get("namn") or rec.get("marketmute", {}).get("namn", ""),
|
||||
"jurform": (scb or {}).get("jurform"),
|
||||
"gatuadress": (scb or {}).get("gatuadress"), "postnr": (scb or {}).get("postnr"),
|
||||
"postort": (scb or {}).get("postort"),
|
||||
"reklamsparr_typ": (scb or {}).get("reklamsparr_typ"),
|
||||
"moms_valid": (vies_rec or {}).get("moms_valid"),
|
||||
"lei": (lei or {}).get("lei"), "lei_status": (lei or {}).get("status"),
|
||||
"kontaktpersoner": (contact or {}).get("personer", []),
|
||||
"har_kontakt": bool(contact and contact.get("personer")),
|
||||
"har_vd": bool(contact and contact.get("har_vd")),
|
||||
"finansiella_ratios": rec.get("finansiella_ratios", {}),
|
||||
"kreditrisk_klass": rec.get("finansiella_ratios", {}).get("kreditrisk_klass"),
|
||||
"verksamhet": rec.get("marketmute", {}).get("verksamhet", ""),
|
||||
})
|
||||
if n % 200000 == 0:
|
||||
print(f" {n:,} (mål hittills: {len(targets):,}, spärrade: {sparr_c:,})", flush=True)
|
||||
|
||||
targets.sort(key=lambda x: -x["cold_call_score"])
|
||||
with open("/data/0pt/output/marketmute_targets_v3.jsonl", "w") as f:
|
||||
for t in targets:
|
||||
f.write(json.dumps(t, ensure_ascii=False) + "\n")
|
||||
print(f"[KLAR] {n:,} skannade -> {len(targets):,} mål")
|
||||
for k, c in klass_c.most_common():
|
||||
print(f" {k}: {c:,}")
|
||||
kp = sum(1 for t in targets if t["har_kontakt"])
|
||||
print(f" Mål med kontaktperson: {kp:,} ({kp/max(len(targets),1)*100:.0f}%)")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env python3
|
||||
"""DEFINITIV MarketMute: dedupe per orgnr + ALL VIES + telefon + encoding."""
|
||||
import json
|
||||
|
||||
def fix_enc(t):
|
||||
if not t or "Ã" not in t: return t
|
||||
try: return t.encode("latin-1").decode("utf-8")
|
||||
except Exception: return t
|
||||
|
||||
vies = {}
|
||||
for path in ("/data/0pt/output/vies_targets.jsonl", "/data/0pt/output/vies_extra.jsonl"):
|
||||
try:
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
if r.get("orgnr_fmt") and r.get("moms_valid") is not None:
|
||||
vies[r["orgnr_fmt"]] = r
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
print(f"[VIES] {len(vies):,} unika med status")
|
||||
|
||||
tel = {}
|
||||
for line in open("/data/0pt/output/telefon_ratsit.jsonl"):
|
||||
r = json.loads(line)
|
||||
if r.get("telefon"):
|
||||
tel[r["orgnr"]] = r["telefon"]
|
||||
print(f"[TELEFON] {len(tel):,} nummer (delresultat, konnektorn kör vidare)")
|
||||
|
||||
best = {}
|
||||
for line in open("/data/0pt/output/marketmute_targets_v3.jsonl"):
|
||||
rec = json.loads(line)
|
||||
o = rec["orgnr"]
|
||||
if o not in best or rec["cold_call_score"] > best[o]["cold_call_score"]:
|
||||
best[o] = rec
|
||||
|
||||
targets = sorted(best.values(), key=lambda x: -x["cold_call_score"])
|
||||
n = momsv = telv = vd_n = hog = 0
|
||||
with open("/data/0pt/output/marketmute_targets_DEFINITIV.jsonl", "w") as out:
|
||||
for rec in targets:
|
||||
orgnr = rec["orgnr"]
|
||||
digits = orgnr.replace("-", "")
|
||||
rec["namn"] = fix_enc(rec.get("namn"))
|
||||
rec["verksamhet"] = fix_enc(rec.get("verksamhet"))
|
||||
for p in rec.get("kontaktpersoner", []):
|
||||
p["namn"] = fix_enc(p.get("namn"))
|
||||
vr = vies.get(orgnr)
|
||||
rec["moms_valid"] = vr["moms_valid"] if vr else rec.get("moms_valid")
|
||||
rec["momsnr"] = (vr or {}).get("momsnr") or f"SE{digits}01"
|
||||
if vr and vr.get("moms_namn"):
|
||||
rec["moms_namn"] = fix_enc(vr["moms_namn"])
|
||||
if rec.get("moms_valid") is not None:
|
||||
momsv += 1
|
||||
ph = tel.get(orgnr)
|
||||
if ph:
|
||||
rec["telefon"] = ph
|
||||
rec["telefon_kalla"] = "ratsit.se"
|
||||
telv += 1
|
||||
if rec.get("har_vd"):
|
||||
vd_n += 1
|
||||
if rec["cold_call_klass"] == "Hög sannolikhet":
|
||||
hog += 1
|
||||
out.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
n += 1
|
||||
print(f"[DEFINITIV] {n:,} unika mål - Hog: {hog:,} - Medium: {n-hog:,}")
|
||||
print(f" moms: {momsv:,} ({momsv/n*100:.0f}%) - telefon: {telv:,} - VD: {vd_n:,}")
|
||||
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ratsit-telefonkonnektor: företagsnummer för VehIQ/MarketMute-mål.
|
||||
Kör från residensiellt IP (ASUS). Endast företagssidor — inga personuppgifter."""
|
||||
import asyncio, json, re, random, os, time, sys
|
||||
from playwright.async_api import async_playwright
|
||||
from datetime import datetime, timezone
|
||||
|
||||
IN = sys.argv[2] if len(sys.argv) > 2 else "/home/simon/vehiq-telefon/targets.jsonl" # prioritet 1: bilbranschen
|
||||
OUT = "/home/simon/vehiq-telefon/telefon_ratsit.jsonl"
|
||||
DONE = "/home/simon/vehiq-telefon/done.txt"
|
||||
WORKERS = 2
|
||||
MIN_DELAY, MAX_DELAY = 2.5, 5.0
|
||||
|
||||
SEGM_PRIO = {"bilhandlare": 0, "bilverkstad": 1, "uthyrning_leasing": 2, "reservdelar_dack": 3, "entreprenad": 4, "motorcykel": 5, "transportor": 6}
|
||||
|
||||
def load_targets():
|
||||
rows = []
|
||||
for line in open(IN):
|
||||
t = json.loads(line)
|
||||
rows.append(t)
|
||||
# prioritet: segment-vikt sedan omsättning (störst först)
|
||||
rows.sort(key=lambda x: (SEGM_PRIO.get(x.get("segment"), 9), -(x.get("oms") or 0)))
|
||||
return rows
|
||||
|
||||
def norm_phone(s):
|
||||
d = re.sub(r"\D", "", s)
|
||||
if d.startswith("46") and len(d) >= 10: d = "0" + d[2:]
|
||||
return d if len(d) >= 8 else None
|
||||
|
||||
def extract_phone(html):
|
||||
"""Företagets egen telefon: Telefon</h2> eller Telefon:</td>-blocket."""
|
||||
for anchor in ("Telefon</h2>", "Telefon:</td>", ">Telefon<"):
|
||||
i = html.find(anchor)
|
||||
if i > 0:
|
||||
seg = html[i:i+1200]
|
||||
m = re.search(r'href="tel:([0-9+-]{8,15})"', seg)
|
||||
if not m:
|
||||
m = re.search(r"0[1-9]\d{1,2}[\s-]?\d{2}\s?\d{2,3}(?!\d)", re.sub(r"<[^>]+>", " ", seg))
|
||||
if m:
|
||||
p = norm_phone(m.group(1) if m.lastindex else m.group(0))
|
||||
if p: return p
|
||||
return None
|
||||
|
||||
async def worker(p, wid, queue, outf, done, lock):
|
||||
while not queue.empty():
|
||||
b = None
|
||||
try:
|
||||
b = await p.chromium.launch(headless=True, args=["--disable-blink-features=AutomationControlled"])
|
||||
pg = await b.new_page(user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36", viewport={"width":1440,"height":900}, locale="sv-SE")
|
||||
while True:
|
||||
try:
|
||||
t = queue.get_nowait()
|
||||
except asyncio.QueueEmpty:
|
||||
return
|
||||
orgnr = t["orgnr"]; digits = orgnr.replace("-", "")
|
||||
if digits in done:
|
||||
queue.task_done(); continue
|
||||
phone = None; status = "ok"
|
||||
try:
|
||||
r = await pg.goto(f"https://www.ratsit.se/{digits}", wait_until="domcontentloaded", timeout=30000)
|
||||
await pg.wait_for_timeout(4000)
|
||||
html = await pg.content()
|
||||
if r.status == 200:
|
||||
phone = extract_phone(html)
|
||||
if not phone: status = "ingen_telefon"
|
||||
elif r.status == 403:
|
||||
status = "blockerad"
|
||||
await asyncio.sleep(180)
|
||||
else:
|
||||
status = f"http_{r.status}"
|
||||
except Exception as e:
|
||||
status = f"fel_{str(e)[:40]}"
|
||||
raise # trigga omstart av browser
|
||||
async with lock:
|
||||
rec = {"orgnr": orgnr, "segment": t.get("segment"), "namn": t.get("namn"),
|
||||
"postort": t.get("postort"), "telefon": phone, "status": status,
|
||||
"kalla": f"ratsit.se/{digits}",
|
||||
"hamtat": datetime.now(timezone.utc).isoformat()}
|
||||
outf.write(json.dumps(rec, ensure_ascii=False) + "\n"); outf.flush()
|
||||
done.add(digits)
|
||||
with open(DONE, "a") as df: df.write(digits + "\n")
|
||||
queue.task_done()
|
||||
await asyncio.sleep(MIN_DELAY + random.random() * (MAX_DELAY - MIN_DELAY))
|
||||
except Exception as e:
|
||||
print(f"[worker {wid}] omstart efter fel: {str(e)[:60]}", flush=True)
|
||||
await asyncio.sleep(5)
|
||||
finally:
|
||||
if b:
|
||||
try: await b.close()
|
||||
except: pass
|
||||
|
||||
async def main():
|
||||
limit = int(sys.argv[1]) if len(sys.argv) > 1 else 999999
|
||||
targets = load_targets()[:limit]
|
||||
done = set(open(DONE).read().split()) if os.path.exists(DONE) else set()
|
||||
queue = asyncio.Queue()
|
||||
for t in targets:
|
||||
if t["orgnr"].replace("-", "") not in done:
|
||||
queue.put_nowait(t)
|
||||
print(f"[INFO] {queue.qsize():,} att hämta ({len(done):,} klara), {WORKERS} workers", flush=True)
|
||||
lock = asyncio.Lock()
|
||||
outf = open(OUT, "a")
|
||||
async with async_playwright() as p:
|
||||
await asyncio.gather(*[worker(p, i, queue, outf, done, lock) for i in range(WORKERS)])
|
||||
outf.close()
|
||||
print("[KLAR]", flush=True)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""DEFINITIV VehIQ-bilbransch: dedupe per orgnr + telefon + VIES + encoding."""
|
||||
import json
|
||||
|
||||
def fix_enc(t):
|
||||
if not t or "Ã" not in t: return t
|
||||
try: return t.encode("latin-1").decode("utf-8")
|
||||
except Exception: return t
|
||||
|
||||
vies = {}
|
||||
for path in ("/data/0pt/output/vies_targets.jsonl", "/data/0pt/output/vies_extra.jsonl"):
|
||||
try:
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
if r.get("orgnr_fmt") and r.get("moms_valid") is not None:
|
||||
vies[r["orgnr_fmt"]] = r
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
tel = {}
|
||||
for line in open("/data/0pt/output/telefon_ratsit.jsonl"):
|
||||
r = json.loads(line)
|
||||
if r.get("telefon"):
|
||||
tel[r["orgnr"]] = r["telefon"]
|
||||
|
||||
best = {}
|
||||
for line in open("/data/0pt/output/vehiq_bilbransch.jsonl"):
|
||||
rec = json.loads(line)
|
||||
o = rec["orgnr"]
|
||||
if o not in best or (rec.get("fordonssrelevans_score") or 0) > (best[o].get("fordonssrelevans_score") or 0):
|
||||
best[o] = rec
|
||||
|
||||
targets = sorted(best.values(), key=lambda x: -(x.get("fordonssrelevans_score") or 0))
|
||||
n = telv = momsv = vd_n = 0
|
||||
from collections import Counter
|
||||
seg = Counter()
|
||||
with open("/data/0pt/output/vehiq_bilbransch_DEFINITIV.jsonl", "w") as out:
|
||||
for rec in targets:
|
||||
orgnr = rec["orgnr"]
|
||||
rec["namn"] = fix_enc(rec.get("namn"))
|
||||
rec["verksamhet"] = fix_enc(rec.get("verksamhet"))
|
||||
for p in rec.get("kontaktpersoner", []):
|
||||
p["namn"] = fix_enc(p.get("namn"))
|
||||
vr = vies.get(orgnr)
|
||||
if vr:
|
||||
rec["moms_valid"] = vr["moms_valid"]
|
||||
momsv += 1
|
||||
ph = tel.get(orgnr)
|
||||
if ph:
|
||||
rec["telefon"] = ph
|
||||
rec["telefon_kalla"] = "ratsit.se"
|
||||
telv += 1
|
||||
if rec.get("har_vd"):
|
||||
vd_n += 1
|
||||
seg[rec["segment"]] += 1
|
||||
out.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
n += 1
|
||||
print(f"[VEHIQ-DEFINITIV] {n:,} unika bolag")
|
||||
print(f" telefon: {telv:,} ({telv/n*100:.0f}%) - moms-status: {momsv:,} - VD: {vd_n:,}")
|
||||
print(" segment:", dict(seg.most_common()))
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env python3
|
||||
"""VehIQ Company Analysis: analysera ALLA bolag på ALLA sätt — fordonsrelevans,
|
||||
segment, regional marknadsposition, finans, tidsserie, kontakter."""
|
||||
import json, sys, statistics
|
||||
from collections import defaultdict
|
||||
|
||||
IN = "/data/0pt/output/marketmute_intelligence.jsonl" # har marketmute + ratios + bolagsverket
|
||||
SCB = "/data/0pt/output/scb_fdb.jsonl"
|
||||
KP = "/data/0pt/output/kontaktpersoner_v2.jsonl"
|
||||
LEI = "/data/0pt/output/gleif_se.jsonl"
|
||||
OUT = "/data/0pt/output/vehiq_company_analysis.jsonl"
|
||||
OUT_SEG = "/data/0pt/output/vehiq_bilbransch.jsonl" # endast bilbransch-segmentet
|
||||
|
||||
# --- SNI 2007: fordonsbranschen ---
|
||||
SNI_CORE = { # direkt fordonsbransch (60p)
|
||||
"45110": "Bilhandel (detaljhandel personbilar)", "45112": "Bilhandel begagnat",
|
||||
"45119": "Övrig bilhandel", "45201": "Bilverkstad", "45202": "Bilverkstad",
|
||||
"45203": "Bilverkstad", "45209": "Övrig reparation", "45200": "Fordonsverkstad",
|
||||
"45310": "Reservdelar/tillbehör", "45320": "Däckverkstad/däckhandel",
|
||||
"45400": "Motorcykelhandel/verkstad", "45190": "Övrig fordonshandel",
|
||||
"45111": "Bilhandel", "45311": "Reservdelar", "45312": "Bilverktyg/tillbehör",
|
||||
}
|
||||
SNI_ADJACENT = { # nära fordonssekorn (35p)
|
||||
"77110": "Biluthyrning", "77120": "Billeasing (personbilar)", "77190": "Övrig fordonsummering",
|
||||
"49320": "Taxi", "49410": "Godstransport väg", "49420": "Godstransport", "49310": "Busstransport",
|
||||
"49390": "Övrig landtransport", "52100": "Lager/magasin", "52210": "Transportterminal",
|
||||
"77340": "Uthyrning vattentransport", "33120": "Reparation maskiner", "96010": "Biltvätt",
|
||||
"64200": "Växtodling (nej)", "49391": "Övrig transport",
|
||||
}
|
||||
SNI_PREFIX_ADJ = ("451", "452", "453", "454", "771", "493", "494", "521", "522")
|
||||
|
||||
KW_STRONG = ["bilhandlare", "bilverkstad", "bilverkstad", "bilverstad", "bilverkstad",
|
||||
"försäljning av personbilar", "försäljning av bilar", "begagnade bilar", "nya bilar",
|
||||
"fordonsverkstad", "däck", "bildelar", "bilar och lätta lastbilar", "lastbilar",
|
||||
"motorcyklar", "fordonsreparation", "bilverkstadsverksamhet"]
|
||||
KW_MED = ["fordon", "bilhandel", "bilverk", "transportbilar", "flak", "entreprenadmaskiner",
|
||||
"bussar", "släpvagnar", "verkstadsverksamhet", "service av fordon", "bilglass",
|
||||
"billackering", "bilplåt", "besiktning"]
|
||||
|
||||
def mangle(w):
|
||||
try: return w.encode("utf-8").decode("latin-1")
|
||||
except Exception: return w
|
||||
|
||||
KW_STRONG += [mangle(w) for w in list(KW_STRONG)]
|
||||
KW_MED += [mangle(w) for w in list(KW_MED)]
|
||||
|
||||
def fix_enc(t):
|
||||
if not t or "Ã" not in t: return t
|
||||
try: return t.encode("latin-1").decode("utf-8")
|
||||
except Exception: return t
|
||||
|
||||
def segment(sni1, verksamhet_l, verksamhet_raw):
|
||||
if sni1 in ("45110", "45111", "45112", "45119"): return "bilhandlare"
|
||||
if sni1 in ("45200", "45201", "45202", "45203", "45209"): return "bilverkstad"
|
||||
if sni1 in ("45310", "45311", "45312", "45320"): return "reservdelar_dack"
|
||||
if sni1 == "45400": return "motorcykel"
|
||||
if sni1 in ("77110", "77120", "77190"): return "uthyrning_leasing"
|
||||
if sni1 in ("49310", "49320", "49390", "49410", "49420"): return "transportor"
|
||||
v = fix_enc(verksamhet_l)
|
||||
for w in KW_STRONG:
|
||||
if w in v: return "bilbransch_ovrig"
|
||||
return "ovrig"
|
||||
|
||||
# --- PASS 1: ladda sidotabeller ---
|
||||
print("[1/4] Laddar sidotabeller...", flush=True)
|
||||
scb_map = {}
|
||||
for line in open(SCB):
|
||||
r = json.loads(line)
|
||||
scb_map[r["orgnr_digits"]] = r
|
||||
contacts = {}
|
||||
for line in open(KP):
|
||||
r = json.loads(line)
|
||||
contacts[r["orgnr_digits"]] = r
|
||||
lei_map = {}
|
||||
for line in open(LEI):
|
||||
r = json.loads(line)
|
||||
o = (r.get("orgnr") or "").replace("-", "").strip()
|
||||
if o: lei_map[o] = r
|
||||
vies = {}
|
||||
for path in ("/data/0pt/output/vies_targets.jsonl", "/data/0pt/output/vies_extra.jsonl"):
|
||||
try:
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
vies[r["orgnr_fmt"]] = r
|
||||
except FileNotFoundError: pass
|
||||
print(f" SCB {len(scb_map):,} · kontakter {len(contacts):,} · LEI {len(lei_map):,} · VIES {len(vies):,}", flush=True)
|
||||
|
||||
# --- PASS 2: marknadsaggregat per kommun×segment ---
|
||||
print("[2/4] Pass 2: regionala marknadsaggregat...", flush=True)
|
||||
mkt = defaultdict(lambda: {"n": 0, "oms_sum": 0.0, "oms_list": [], "ek_sum": 0.0})
|
||||
n = 0
|
||||
for line in open(IN):
|
||||
rec = json.loads(line)
|
||||
n += 1
|
||||
orgnr = rec.get("orgnr")
|
||||
if not orgnr: continue
|
||||
scb = scb_map.get(orgnr.replace("-", ""))
|
||||
if not scb: continue
|
||||
kommun = (scb.get("postort") or "").strip().upper()
|
||||
if not kommun: continue
|
||||
r = rec.get("finansiella_ratios", {})
|
||||
oms = r.get("omsattning")
|
||||
ek = r.get("eget_kapital")
|
||||
m = mkt[kommun]
|
||||
m["n"] += 1
|
||||
if oms: m["oms_sum"] += oms; m["oms_list"].append(oms)
|
||||
if ek: m["ek_sum"] += ek
|
||||
if n % 300000 == 0: print(f" {n:,}...", flush=True)
|
||||
|
||||
mkt_summary = {}
|
||||
for kommun, m in mkt.items():
|
||||
lst = sorted(m["oms_list"])
|
||||
mkt_summary[kommun] = {
|
||||
"n": m["n"], "oms_sum": m["oms_sum"],
|
||||
"oms_median": statistics.median(lst) if lst else None,
|
||||
"ek_sum": m["ek_sum"],
|
||||
}
|
||||
del mkt
|
||||
with open("/data/0pt/output/vehiq_market_summary.json", "w") as f:
|
||||
json.dump(mkt_summary, f, ensure_ascii=False)
|
||||
print(f" {len(mkt_summary):,} kommuner/aggregat", flush=True)
|
||||
|
||||
# --- PASS 3: per-bolagsanalys ---
|
||||
print("[3/4] Pass 3: per-bolagsanalys...", flush=True)
|
||||
def score_vehicle(scb, verksamhet_raw, r, verksamhet_l):
|
||||
score = 0; reasons = []
|
||||
sni1 = (scb.get("sni1") or "") if scb else ""
|
||||
if sni1 in SNI_CORE:
|
||||
score += 60; reasons.append(f"SNI {sni1}: {SNI_CORE[sni1]}")
|
||||
elif sni1 in SNI_ADJACENT:
|
||||
score += 35; reasons.append(f"SNI {sni1}: {SNI_ADJACENT[sni1]}")
|
||||
elif sni1[:3] in SNI_PREFIX_ADJ and sni1:
|
||||
score += 25; reasons.append(f"SNI-prefix {sni1[:3]}")
|
||||
for w in KW_STRONG:
|
||||
if w in verksamhet_l:
|
||||
score += 25; reasons.append(f"Nyckelord: '{fix_enc(w)}'"); break
|
||||
for w in KW_MED:
|
||||
if w in verksamhet_l:
|
||||
score += 15; reasons.append(f"Nyckelord: '{fix_enc(w)}'"); break
|
||||
if r.get("fordon_bokfort"): score += 10; reasons.append("Fordonsbokföring i balansräkning")
|
||||
return min(score, 100), sni1, reasons
|
||||
|
||||
n = 0; core_n = 0
|
||||
out = open(OUT, "w")
|
||||
out_seg = open(OUT_SEG, "w")
|
||||
for line in open(IN):
|
||||
rec = json.loads(line)
|
||||
n += 1
|
||||
orgnr = rec.get("orgnr")
|
||||
if not orgnr: continue
|
||||
digits = orgnr.replace("-", "")
|
||||
scb = scb_map.get(digits)
|
||||
r = rec.get("finansiella_ratios", {})
|
||||
mm = rec.get("marketmute", {})
|
||||
verksamhet_raw = scb.get("verksamhet") if scb else ""
|
||||
verksamhet_l = (verksamhet_raw or "").lower()
|
||||
v_score, sni1, v_reasons = score_vehicle(scb, verksamhet_raw, r, verksamhet_l)
|
||||
seg = segment(sni1, verksamhet_l, verksamhet_raw)
|
||||
kommun = (scb.get("postort") or "").strip().upper() if scb else None
|
||||
oms = r.get("omsattning")
|
||||
mkt = mkt_summary.get(kommun) if kommun else None
|
||||
# marknadsposition: omsättning-percentil inom kommunen
|
||||
position = None
|
||||
if mkt and oms and mkt["n"] > 3:
|
||||
if mkt["oms_median"]:
|
||||
position = round(oms / mkt["oms_median"], 2) # x gånger medianen
|
||||
contact = contacts.get(digits)
|
||||
lei = lei_map.get(digits)
|
||||
vies_rec = vies.get(orgnr)
|
||||
rec_out = {
|
||||
"orgnr": orgnr,
|
||||
"namn": fix_enc((scb or {}).get("namn") or mm.get("namn", "")),
|
||||
"fordonssrelevans_score": v_score,
|
||||
"fordonssrelevans_reasons": v_reasons,
|
||||
"segment": seg,
|
||||
"sni": (scb or {}).get("sni"), "sni1": sni1,
|
||||
"jurform": (scb or {}).get("jurform"),
|
||||
"gatuadress": (scb or {}).get("gatuadress"),
|
||||
"postnr": (scb or {}).get("postnr"), "postort": (scb or {}).get("postort"),
|
||||
"marknad": {"kommun": kommun, "n_bolag": (mkt or {}).get("n"),
|
||||
"oms_sum": (mkt or {}).get("oms_sum"),
|
||||
"oms_median": (mkt or {}).get("oms_median"),
|
||||
"oms_vs_median": position},
|
||||
"finansiella_ratios": r,
|
||||
"kreditrisk_klass": r.get("kreditrisk_klass"),
|
||||
"kreditrisk_score": r.get("kreditrisk_score"),
|
||||
"kontaktpersoner": (contact or {}).get("personer", []),
|
||||
"har_vd": bool(contact and contact.get("har_vd")),
|
||||
"lei": (lei or {}).get("lei"),
|
||||
"moms_valid": (vies_rec or {}).get("moms_valid"),
|
||||
"momsnr": f"SE{digits}01",
|
||||
"cold_call_score": mm.get("cold_call_score"),
|
||||
"verksamhet": fix_enc(verksamhet_raw or ""),
|
||||
"antal_ar": r.get("antal_ar"),
|
||||
}
|
||||
out.write(json.dumps(rec_out, ensure_ascii=False) + "\n")
|
||||
if seg != "ovrig" or v_score >= 60:
|
||||
core_n += 1
|
||||
out_seg.write(json.dumps(rec_out, ensure_ascii=False) + "\n")
|
||||
if n % 200000 == 0: print(f" {n:,} (bilbransch hittills: {core_n:,})", flush=True)
|
||||
out.close(); out_seg.close()
|
||||
print(f"[KLAR] {n:,} analyserade -> {OUT}")
|
||||
print(f" Bilbransch-segmentet (SNI 45xx m fl): {core_n:,} -> {OUT_SEG}")
|
||||
@@ -0,0 +1,181 @@
|
||||
#!/usr/bin/env python3
|
||||
"""VehIQ Company Analysis v2: förstärkta bilbransch-nyckelord + SNI (där SCB täcker)."""
|
||||
import json, sys
|
||||
from collections import defaultdict
|
||||
|
||||
IN = "/data/0pt/output/marketmute_intelligence.jsonl"
|
||||
SCB = "/data/0pt/output/scb_fdb.jsonl"
|
||||
KP = "/data/0pt/output/kontaktpersoner_v2.jsonl"
|
||||
LEI = "/data/0pt/output/gleif_se.jsonl"
|
||||
MKT = "/data/0pt/output/vehiq_market_summary.json"
|
||||
OUT = "/data/0pt/output/vehiq_company_analysis.jsonl"
|
||||
OUT_SEG = "/data/0pt/output/vehiq_bilbransch.jsonl"
|
||||
|
||||
SNI_TRANSPORT = { # finns i SCB: transport/uthyrning/lager
|
||||
"77110": "Biluthyrning", "77120": "Billeasing personbilar", "77190": "Övrig fordonsummering",
|
||||
"49310": "Busstransport", "49320": "Taxi", "49390": "Övrig personbefordran",
|
||||
"49410": "Godstransport väg", "49420": "Godstransport (flytt)",
|
||||
"52100": "Lager/magasin", "52210": "Transportterminaler", "77340": "UthyrningMaskiner",
|
||||
"96010": "Biltvätt", "33120": "ReparationMaskiner",
|
||||
}
|
||||
|
||||
def mangle(w):
|
||||
try: return w.encode("utf-8").decode("latin-1")
|
||||
except Exception: return w
|
||||
|
||||
# Starka bilbransch-nyckelord (från faktiska Bolagsverket-fraser)
|
||||
KW_STRONG = [
|
||||
"bilverkstadsrörelse", "bilverkstad", "bilverkstad", "bilverkstad.",
|
||||
"handel med begagnade bilar", "handel med nya och begagnade bilar",
|
||||
"handel med bilar", "försäljning av begagnade bilar",
|
||||
"försäljning av nya och begagnade bilar", "försäljning av bilar",
|
||||
"försäljning av personbilar", "bilhandel", "handel med bildelar",
|
||||
"bildelar", "biltillbehör", "däckverkstad", "däckhotell", "handelsbod med däck",
|
||||
"reparation av bilar", "reparation av personbilar", "bilverk",
|
||||
"billackering", "bilplåt", "bilglass", "bilglass",
|
||||
"motorcykelhandel", "motorcykelverkstad", "handelsbod med motorcyklar",
|
||||
"biluthyrning", "billeasing", "uthyrning av personbilar",
|
||||
"lastbilstrafik", "godstransporter", "buss", "taxiverksamhet", "taxiförmedling",
|
||||
"service av fordon", "fordonsverkstad", "fordonsreparation",
|
||||
"besiktning av fordon", "bilbesiktning",
|
||||
]
|
||||
KW_MED = [
|
||||
"personbilar", "bilar", "fordon", "transportbilar", "motorcykel",
|
||||
"släpvagnar", "entreprenadmaskiner", "lastbilar", "däck",
|
||||
"verkstadsverksamhet", "maskinreparation", "flak",
|
||||
]
|
||||
KW_STRONG += [mangle(w) for w in list(KW_STRONG)]
|
||||
KW_MED += [mangle(w) for w in list(KW_MED)]
|
||||
|
||||
def fix_enc(t):
|
||||
if not t or "Ã" not in t: return t
|
||||
try: return t.encode("latin-1").decode("utf-8")
|
||||
except Exception: return t
|
||||
|
||||
print("[1/3] Laddar sidotabeller...", flush=True)
|
||||
scb_map = {}
|
||||
for line in open(SCB):
|
||||
r = json.loads(line)
|
||||
scb_map[r["orgnr_digits"]] = r
|
||||
contacts = {}
|
||||
for line in open(KP):
|
||||
r = json.loads(line)
|
||||
contacts[r["orgnr_digits"]] = r
|
||||
lei_map = {}
|
||||
for line in open(LEI):
|
||||
r = json.loads(line)
|
||||
o = (r.get("orgnr") or "").replace("-", "").strip()
|
||||
if o: lei_map[o] = r
|
||||
vies = {}
|
||||
for path in ("/data/0pt/output/vies_targets.jsonl", "/data/0pt/output/vies_extra.jsonl"):
|
||||
try:
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
vies[r["orgnr_fmt"]] = r
|
||||
except FileNotFoundError: pass
|
||||
mkt_summary = json.load(open(MKT))
|
||||
print(f" SCB {len(scb_map):,} · KP {len(contacts):,} · LEI {len(lei_map):,} · VIES {len(vies):,} · marknad {len(mkt_summary):,}", flush=True)
|
||||
|
||||
def classify(scb, verksamhet_l, r):
|
||||
score = 0; reasons = []
|
||||
sni1 = (scb.get("sni1") or "") if scb else ""
|
||||
sni_all = (scb.get("sni") or []) if scb else []
|
||||
if sni1 in SNI_TRANSPORT:
|
||||
score += 60; reasons.append(f"SNI {sni1}: {SNI_TRANSPORT[sni1]}")
|
||||
# sekundär-SNI check (45xx saknas i exporten men kolla alla Ng)
|
||||
for s in sni_all:
|
||||
if s[:3] in ("451", "452", "453", "454"):
|
||||
score += 60; reasons.append(f"SNI {s} (sekundär): bilbransch"); break
|
||||
strong = None
|
||||
for w in KW_STRONG:
|
||||
if w in verksamhet_l:
|
||||
strong = w; break
|
||||
if strong:
|
||||
score += 40; reasons.append(f"Nyckelord: '{fix_enc(strong)}'")
|
||||
# förfining av segment via nyckelord
|
||||
if any(x in strong for x in ("verkstad", "reparation", "lackering", "plåt", "glass", "besiktning", "däck")):
|
||||
seg_hint = "bilverkstad"
|
||||
elif any(x in strong for x in ("bildelar", "biltillbehör", "däck")):
|
||||
seg_hint = "reservdelar_dack"
|
||||
elif any(x in strong for x in ("uthyrning", "leasing")):
|
||||
seg_hint = "uthyrning_leasing"
|
||||
elif any(x in strong for x in ("taxi", "buss", "godstransport", "lastbilstrafik")):
|
||||
seg_hint = "transportor"
|
||||
elif any(x in strong for x in ("motorcykel",)):
|
||||
seg_hint = "motorcykel"
|
||||
else:
|
||||
seg_hint = "bilhandlare"
|
||||
else:
|
||||
seg_hint = None
|
||||
for w in KW_MED:
|
||||
if w in verksamhet_l:
|
||||
score += 20; reasons.append(f"Nyckelord: '{fix_enc(w)}'"); break
|
||||
if r.get("fordon_bokfort"):
|
||||
score += 10; reasons.append("Fordonsbokföring i balansräkningen")
|
||||
return min(score, 100), sni1, reasons, seg_hint
|
||||
|
||||
print("[2/3] Per-bolagsanalys...", flush=True)
|
||||
n = 0; core_n = 0
|
||||
seg_count = defaultdict(int)
|
||||
out = open(OUT, "w")
|
||||
out_seg = open(OUT_SEG, "w")
|
||||
for line in open(IN):
|
||||
rec = json.loads(line)
|
||||
n += 1
|
||||
orgnr = rec.get("orgnr")
|
||||
if not orgnr: continue
|
||||
digits = orgnr.replace("-", "")
|
||||
scb = scb_map.get(digits)
|
||||
r = rec.get("finansiella_ratios", {})
|
||||
mm = rec.get("marketmute", {})
|
||||
verksamhet_raw = scb.get("verksamhet") if scb else ""
|
||||
verksamhet_l = (verksamhet_raw or "").lower()
|
||||
v_score, sni1, v_reasons, seg_hint = classify(scb, verksamhet_l, r)
|
||||
# segment: SNI först, nyckelord annars
|
||||
if sni1 in ("77110", "77120", "77190"): seg = "uthyrning_leasing"
|
||||
elif sni1 in ("49310", "49320", "49390", "49410", "49420"): seg = "transportor"
|
||||
elif sni1 == "96010": seg = "bilverkstad"
|
||||
elif seg_hint: seg = seg_hint
|
||||
else: seg = "ovrig"
|
||||
seg_count[seg] += 1
|
||||
kommun = (scb.get("postort") or "").strip().upper() if scb else None
|
||||
oms = r.get("omsattning")
|
||||
mkt = mkt_summary.get(kommun) if kommun else None
|
||||
position = round(oms / mkt["oms_median"], 2) if (mkt and oms and mkt.get("oms_median") and mkt["n"] > 3) else None
|
||||
contact = contacts.get(digits)
|
||||
lei = lei_map.get(digits)
|
||||
vies_rec = vies.get(orgnr)
|
||||
rec_out = {
|
||||
"orgnr": orgnr,
|
||||
"namn": fix_enc((scb or {}).get("namn") or mm.get("namn", "")),
|
||||
"fordonssrelevans_score": v_score,
|
||||
"fordonssrelevans_reasons": v_reasons,
|
||||
"segment": seg,
|
||||
"sni": (scb or {}).get("sni"), "sni1": sni1,
|
||||
"jurform": (scb or {}).get("jurform"),
|
||||
"gatuadress": (scb or {}).get("gatuadress"),
|
||||
"postnr": (scb or {}).get("postnr"), "postort": (scb or {}).get("postort"),
|
||||
"marknad": {"kommun": kommun, "n_bolag": (mkt or {}).get("n"),
|
||||
"oms_sum": (mkt or {}).get("oms_sum"),
|
||||
"oms_median": (mkt or {}).get("oms_median"),
|
||||
"oms_vs_median": position},
|
||||
"finansiella_ratios": r,
|
||||
"kreditrisk_klass": r.get("kreditrisk_klass"),
|
||||
"kreditrisk_score": r.get("kreditrisk_score"),
|
||||
"kontaktpersoner": (contact or {}).get("personer", []),
|
||||
"har_vd": bool(contact and contact.get("har_vd")),
|
||||
"lei": (lei or {}).get("lei"),
|
||||
"moms_valid": (vies_rec or {}).get("moms_valid"),
|
||||
"momsnr": f"SE{digits}01",
|
||||
"cold_call_score": mm.get("cold_call_score"),
|
||||
"verksamhet": fix_enc(verksamhet_raw or ""),
|
||||
}
|
||||
out.write(json.dumps(rec_out, ensure_ascii=False) + "\n")
|
||||
if seg != "ovrig" or v_score >= 60:
|
||||
core_n += 1
|
||||
out_seg.write(json.dumps(rec_out, ensure_ascii=False) + "\n")
|
||||
if n % 200000 == 0: print(f" {n:,} (fordonsbransch: {core_n:,})", flush=True)
|
||||
out.close(); out_seg.close()
|
||||
print(f"[KLAR] {n:,} analyserade -> {OUT}")
|
||||
print(f" Fordonsbransch: {core_n:,} -> {OUT_SEG}")
|
||||
print(" Segment:", dict(sorted(seg_count.items(), key=lambda x: -x[1])))
|
||||
@@ -0,0 +1,216 @@
|
||||
#!/usr/bin/env python3
|
||||
"""VehIQ Company Analysis v3 FINAL: expanderade 45xx-fraser + VehIQ-marknadsdata."""
|
||||
import json
|
||||
from collections import defaultdict
|
||||
|
||||
IN = "/data/0pt/output/marketmute_intelligence.jsonl"
|
||||
SCB = "/data/0pt/output/scb_fdb.jsonl"
|
||||
KP = "/data/0pt/output/kontaktpersoner_v2.jsonl"
|
||||
LEI = "/data/0pt/output/gleif_se.jsonl"
|
||||
MKT = "/data/0pt/output/vehiq_market_summary.json"
|
||||
MV = "/data/0pt/landing/vehiq/market_value_se.jsonl"
|
||||
FLEET = "/data/0pt/landing/vehiq/fleet_composition_se.jsonl"
|
||||
OUT = "/data/0pt/output/vehiq_company_analysis.jsonl"
|
||||
OUT_SEG = "/data/0pt/output/vehiq_bilbransch.jsonl"
|
||||
|
||||
SNI_VEH = {
|
||||
"77110": "Biluthyrning", "77120": "Billeasing personbilar", "77190": "Övrig fordonsummering",
|
||||
"49310": "Busstransport", "49320": "Taxi", "49390": "Övrig personbefordran",
|
||||
"49410": "Godstransport väg", "49420": "Flyttgods", "52100": "Lager", "52210": "Terminaler",
|
||||
"96010": "Biltvätt", "33120": "ReparationMaskiner",
|
||||
}
|
||||
|
||||
def mangle(w):
|
||||
try: return w.encode("utf-8").decode("latin-1")
|
||||
except Exception: return w
|
||||
|
||||
KW_STRONG = [
|
||||
# verkstad (hint: bilverkstad)
|
||||
"bilverkstadsrörelse", "bilverkstad", "bilverkstadsverksamhet", "fordonsverkstad",
|
||||
"fordonsreparation", "reparation av bilar", "reparation av personbilar",
|
||||
"reparation av personbilar", "service av bilar", "service av fordon", "service av fordon",
|
||||
"billackering", "bilplåt", "bilglass", "bilglas", "besiktning av fordon",
|
||||
"bensinstation och bilverkstad", "reparation av lastbilar", "reparation av lastbilar",
|
||||
# handel (hint: bilhandlare)
|
||||
"bilhandlare", "bilhandel", "personbilshandel", "fordonshandel",
|
||||
"handel med begagnade bilar", "handel med nya och begagnade bilar",
|
||||
"handel med bilar", "handel med personbilar", "försäljning av begagnade bilar",
|
||||
"försäljning av nya och begagnade bilar", "försäljning av bilar",
|
||||
"försäljning av personbilar", "försäljning av nya och begagnade personbilar",
|
||||
"begagnade personbilar", "nya personbilar", "begagnade bilar", "begagnade bilar.",
|
||||
"import av bilar", "export av bilar",
|
||||
# reservdelar (hint: reservdelar_dack)
|
||||
"handel med bildelar", "handel med bildelar", "bildelar", "biltillbehör",
|
||||
"import och export av bildelar", "import och försäljning av bildelar",
|
||||
"handel med begagnade bildelar", "bilar och bildelar", "bildelar och tillbehör",
|
||||
"handel med däck", "försäljning av däck", "däckverkstad", "däckhotell",
|
||||
"däck och fälgar", "handel med bildäck", "försäljning och montering av däck",
|
||||
"bilverkstad och bilförsäljning", "bilverkstad samt handel med bilar",
|
||||
# uthyrning (hint: uthyrning_leasing)
|
||||
"biluthyrning", "billeasing", "uthyrning av personbilar", "uthyrning av bilar",
|
||||
"leasing av personbilar", "uthyrning av lastbilar", "uthyrning av lastbilar",
|
||||
"uthyrning av släpvagnar", "uthyrning av släpvagnar",
|
||||
# transportör (hint: transportor)
|
||||
"taxiverksamhet", "taxiförmedling", "taxi", "bussverksamhet", "busslinjer",
|
||||
"åkeri", "lastbilsåkeri", "lastbilsåkeri och", "godstransporter", "godstransporter med lastbil",
|
||||
"transporter med lastbil", "personbefordran", "personbilstrafik",
|
||||
# entreprenad (hint: entreprenad)
|
||||
"uthyrning av entreprenadmaskiner", "handel med entreprenadmaskiner",
|
||||
"reparation av entreprenadmaskiner", "service av entreprenadmaskiner",
|
||||
"entreprenadmaskiner", "entreprenadmaskiner",
|
||||
# motorcykel
|
||||
"motorcykelhandel", "motorcykelverkstad", "handelsbod med motorcyklar", "motorcyklar",
|
||||
]
|
||||
KW_MED = ["personbilar", "personbilar", "personbilar.", "lastbilar", "bussar", "släpvagn",
|
||||
"fordon", "transportbilar", "verkstadsverksamhet", "maskinreparation"]
|
||||
KW_STRONG += [mangle(w) for w in list(KW_STRONG)]
|
||||
KW_MED += [mangle(w) for w in list(KW_MED)]
|
||||
|
||||
def fix_enc(t):
|
||||
if not t or "Ã" not in t: return t
|
||||
try: return t.encode("latin-1").decode("utf-8")
|
||||
except Exception: return t
|
||||
|
||||
print("[1/4] Sidotabeller...", flush=True)
|
||||
scb_map = {}
|
||||
for line in open(SCB):
|
||||
r = json.loads(line)
|
||||
scb_map[r["orgnr_digits"]] = r
|
||||
contacts = {}
|
||||
for line in open(KP):
|
||||
r = json.loads(line)
|
||||
contacts[r["orgnr_digits"]] = r
|
||||
lei_map = {}
|
||||
for line in open(LEI):
|
||||
r = json.loads(line)
|
||||
o = (r.get("orgnr") or "").replace("-", "").strip()
|
||||
if o: lei_map[o] = r
|
||||
vies = {}
|
||||
for path in ("/data/0pt/output/vies_targets.jsonl", "/data/0pt/output/vies_extra.jsonl"):
|
||||
try:
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
vies[r["orgnr_fmt"]] = r
|
||||
except FileNotFoundError: pass
|
||||
mkt_summary = json.load(open(MKT))
|
||||
print(f" SCB {len(scb_map):,} · KP {len(contacts):,} · LEI {len(lei_map):,} · VIES {len(vies):,}", flush=True)
|
||||
|
||||
print("[2/4] VehIQ-marknadsdata (priser + bestånd)...", flush=True)
|
||||
market_value = []
|
||||
for line in open(MV):
|
||||
market_value.append(json.loads(line))
|
||||
fleet = []
|
||||
for line in open(FLEET):
|
||||
fleet.append(json.loads(line))
|
||||
mv_summary = {
|
||||
"antal_makemodell_ar": len(market_value),
|
||||
"top_prisade": sorted(market_value, key=lambda x: -(x.get("ad_count") or 0))[:25],
|
||||
}
|
||||
fleet_summary = {
|
||||
"antal_marke": sum(1 for f in fleet if "modeller" in f),
|
||||
"top_marke": sorted([f for f in fleet if "modeller" in f], key=lambda x: -x["antal"])[:15],
|
||||
"top_modell": sorted([f for f in fleet if "modeller" not in f], key=lambda x: -x["antal"])[:50],
|
||||
}
|
||||
print(f" {len(market_value):,} prisposter · {len(fleet):,} beståndsposter", flush=True)
|
||||
|
||||
def classify(scb, verksamhet_l, r):
|
||||
score = 0; reasons = []; seg = "ovrig"
|
||||
sni1 = (scb.get("sni1") or "") if scb else ""
|
||||
sni_all = (scb.get("sni") or []) if scb else []
|
||||
if sni1 in SNI_VEH:
|
||||
score += 60; reasons.append(f"SNI {sni1}: {SNI_VEH[sni1]}")
|
||||
seg = "transportor" if sni1.startswith("49") else ("uthyrning_leasing" if sni1.startswith("77") else ("bilverkstad" if sni1 == "96010" else seg))
|
||||
for s in sni_all:
|
||||
if s[:3] in ("451", "452", "453", "454"):
|
||||
score += 60; reasons.append(f"SNI {s} (sekundär): bilbransch")
|
||||
seg = seg if seg != "ovrig" else "bilhandlare"
|
||||
break
|
||||
for w in KW_STRONG:
|
||||
if w in verksamhet_l:
|
||||
score += 40; reasons.append(f"Nyckelord: '{fix_enc(w)}'")
|
||||
if seg == "ovrig":
|
||||
if any(x in w for x in ("verkstad", "reparation", "lackering", "plåt", "glass", "glas", "besiktning", "service av")):
|
||||
seg = "bilverkstad"
|
||||
elif any(x in w for x in ("bildelar", "biltillbehör", "däck", "bildäck")):
|
||||
seg = "reservdelar_dack"
|
||||
elif any(x in w for x in ("uthyrning", "leasing")):
|
||||
seg = "uthyrning_leasing"
|
||||
elif any(x in w for x in ("taxi", "buss", "åkeri", "godstransport", "transporter med lastbil", "befordran")):
|
||||
seg = "transportor"
|
||||
elif any(x in w for x in ("entreprenadmaskin",)):
|
||||
seg = "entreprenad"
|
||||
elif any(x in w for x in ("motorcykel",)):
|
||||
seg = "motorcykel"
|
||||
else:
|
||||
seg = "bilhandlare"
|
||||
break
|
||||
if seg == "ovrig":
|
||||
for w in KW_MED:
|
||||
if w in verksamhet_l:
|
||||
score += 20; reasons.append(f"Nyckelord: '{fix_enc(w)}'"); break
|
||||
if r.get("fordon_bokfort"):
|
||||
score += 10; reasons.append("Fordonsbokföring i balansräkningen")
|
||||
return min(score, 100), sni1, reasons, seg
|
||||
|
||||
print("[3/4] Per-bolagsanalys...", flush=True)
|
||||
n = 0; core_n = 0; vd_n = 0
|
||||
seg_count = defaultdict(int)
|
||||
out = open(OUT, "w")
|
||||
out_seg = open(OUT_SEG, "w")
|
||||
for line in open(IN):
|
||||
rec = json.loads(line)
|
||||
n += 1
|
||||
orgnr = rec.get("orgnr")
|
||||
if not orgnr: continue
|
||||
digits = orgnr.replace("-", "")
|
||||
scb = scb_map.get(digits)
|
||||
r = rec.get("finansiella_ratios", {})
|
||||
mm = rec.get("marketmute", {})
|
||||
verksamhet_raw = (rec.get("bolagsverket") or {}).get("verksamhet") or mm.get("verksamhet") or ""
|
||||
verksamhet_l = (verksamhet_raw or "").lower()
|
||||
v_score, sni1, v_reasons, seg = classify(scb, verksamhet_l, r)
|
||||
seg_count[seg] += 1
|
||||
kommun = (scb.get("postort") or "").strip().upper() if scb else None
|
||||
oms = r.get("omsattning")
|
||||
mkt = mkt_summary.get(kommun) if kommun else None
|
||||
position = round(oms / mkt["oms_median"], 2) if (mkt and oms and mkt.get("oms_median") and mkt["n"] > 3) else None
|
||||
contact = contacts.get(digits)
|
||||
lei = lei_map.get(digits)
|
||||
vies_rec = vies.get(orgnr)
|
||||
rec_out = {
|
||||
"orgnr": orgnr,
|
||||
"namn": fix_enc((scb or {}).get("namn") or mm.get("namn", "")),
|
||||
"fordonssrelevans_score": v_score,
|
||||
"fordonssrelevans_reasons": v_reasons,
|
||||
"segment": seg,
|
||||
"sni": (scb or {}).get("sni"), "sni1": sni1,
|
||||
"jurform": (scb or {}).get("jurform"),
|
||||
"gatuadress": (scb or {}).get("gatuadress"),
|
||||
"postnr": (scb or {}).get("postnr"), "postort": (scb or {}).get("postort"),
|
||||
"marknad": {"kommun": kommun, "n_bolag": (mkt or {}).get("n"),
|
||||
"oms_sum": (mkt or {}).get("oms_sum"),
|
||||
"oms_median": (mkt or {}).get("oms_median"),
|
||||
"oms_vs_median": position},
|
||||
"finansiella_ratios": r,
|
||||
"kreditrisk_klass": r.get("kreditrisk_klass"),
|
||||
"kreditrisk_score": r.get("kreditrisk_score"),
|
||||
"kontaktpersoner": (contact or {}).get("personer", []),
|
||||
"har_vd": bool(contact and contact.get("har_vd")),
|
||||
"lei": (lei or {}).get("lei"),
|
||||
"moms_valid": (vies_rec or {}).get("moms_valid"),
|
||||
"momsnr": f"SE{digits}01",
|
||||
"cold_call_score": mm.get("cold_call_score"),
|
||||
"verksamhet": fix_enc(verksamhet_raw or ""),
|
||||
}
|
||||
out.write(json.dumps(rec_out, ensure_ascii=False) + "\n")
|
||||
if seg != "ovrig" or v_score >= 60:
|
||||
core_n += 1
|
||||
if rec_out["har_vd"]: vd_n += 1
|
||||
out_seg.write(json.dumps(rec_out, ensure_ascii=False) + "\n")
|
||||
if n % 200000 == 0: print(f" {n:,} (fordonsbransch: {core_n:,})", flush=True)
|
||||
out.close(); out_seg.close()
|
||||
json.dump({"market_value_summary": mv_summary, "fleet_summary": fleet_summary},
|
||||
open("/data/0pt/output/vehiq_market_context.json", "w"), ensure_ascii=False, indent=1)
|
||||
print(f"[KLAR] {n:,} analyserade -> {OUT}")
|
||||
print(f" Fordonsbransch: {core_n:,} (med VD: {vd_n:,}) -> {OUT_SEG}")
|
||||
print(" Segment:", dict(sorted(seg_count.items(), key=lambda x: -x[1])))
|
||||
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env python3
|
||||
"""VIES-utökning: alla targets som saknar moms-status."""
|
||||
import json, time, requests
|
||||
|
||||
IN = "/data/0pt/output/marketmute_targets_v3.jsonl"
|
||||
OUT = "/data/0pt/output/vies_extra.jsonl"
|
||||
have = set()
|
||||
for line in open("/data/0pt/output/vies_targets.jsonl"):
|
||||
have.add(json.loads(line)["orgnr"])
|
||||
targets = []
|
||||
for line in open(IN):
|
||||
t = json.loads(line)
|
||||
org = t["orgnr"].replace("-", "")
|
||||
if org not in have:
|
||||
targets.append(org)
|
||||
print(f"[INFO] {len(targets):,} targets saknar VIES", flush=True)
|
||||
sess = requests.Session()
|
||||
n = 0; valid = 0
|
||||
with open(OUT, "w") as out:
|
||||
for i, org in enumerate(targets):
|
||||
vat = org + "01"
|
||||
ok = False
|
||||
for attempt in range(4):
|
||||
try:
|
||||
r = sess.get(f"https://ec.europa.eu/taxation_customs/vies/rest-api/ms/SE/vat/{vat}", timeout=15)
|
||||
if r.status_code == 429:
|
||||
time.sleep(5); continue
|
||||
r.raise_for_status()
|
||||
d = r.json()
|
||||
out.write(json.dumps({
|
||||
"orgnr": org, "orgnr_fmt": f"{org[:6]}-{org[6:]}",
|
||||
"momsnr": f"SE{vat}", "moms_valid": d.get("isValid"),
|
||||
"moms_namn": (d.get("name") or "").strip() or None,
|
||||
}, ensure_ascii=False) + "\n")
|
||||
if d.get("isValid"): valid += 1
|
||||
ok = True; break
|
||||
except Exception:
|
||||
time.sleep(2 ** attempt)
|
||||
if not ok:
|
||||
out.write(json.dumps({"orgnr": org, "orgnr_fmt": f"{org[:6]}-{org[6:]}", "moms_valid": None, "error": True}, ensure_ascii=False) + "\n")
|
||||
n += 1
|
||||
if n % 250 == 0:
|
||||
print(f" {n:,}/{len(targets):,} (valid: {valid:,})", flush=True)
|
||||
time.sleep(0.25)
|
||||
print(f"[KLAR] {n:,} kontrollerade, {valid:,} valid -> {OUT}")
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Steg 4 fixad: VIES med korrekt momsnummer (SE + orgnr + 01)."""
|
||||
import json, time, requests
|
||||
|
||||
IN = "/data/0pt/output/marketmute_targets_dedup.jsonl"
|
||||
OUT = "/data/0pt/output/vies_targets.jsonl"
|
||||
sess = requests.Session()
|
||||
n = 0; valid = 0
|
||||
targets = [json.loads(l)["orgnr"].replace("-", "") for l in open(IN)]
|
||||
print(f"[INFO] {len(targets):,} målbolag (momsnr = SE{targets[0]}01-format)", flush=True)
|
||||
with open(OUT, "w") as out:
|
||||
for i, org in enumerate(targets):
|
||||
vat = org + "01"
|
||||
ok = False
|
||||
for attempt in range(4):
|
||||
try:
|
||||
r = sess.get(f"https://ec.europa.eu/taxation_customs/vies/rest-api/ms/SE/vat/{vat}", timeout=15)
|
||||
if r.status_code == 429:
|
||||
time.sleep(5); continue
|
||||
r.raise_for_status()
|
||||
d = r.json()
|
||||
out.write(json.dumps({
|
||||
"orgnr": org,
|
||||
"orgnr_fmt": f"{org[:6]}-{org[6:]}",
|
||||
"momsnr": f"SE{vat}",
|
||||
"moms_valid": d.get("isValid"),
|
||||
"moms_namn": (d.get("name") or "").strip() or None,
|
||||
}, ensure_ascii=False) + "\n")
|
||||
if d.get("isValid"): valid += 1
|
||||
ok = True
|
||||
break
|
||||
except Exception:
|
||||
time.sleep(2 ** attempt)
|
||||
if not ok:
|
||||
out.write(json.dumps({"orgnr": org, "orgnr_fmt": f"{org[:6]}-{org[6:]}", "moms_valid": None, "error": True}, ensure_ascii=False) + "\n")
|
||||
n += 1
|
||||
if n % 250 == 0:
|
||||
print(f" {n:,}/{len(targets):,} (moms-valid: {valid:,})", flush=True)
|
||||
time.sleep(0.25)
|
||||
print(f"[KLAR] {n:,} kontrollerade, {valid:,} moms-registrerade -> {OUT}")
|
||||
@@ -1,99 +0,0 @@
|
||||
<yandex>
|
||||
<logger>
|
||||
<level>information</level>
|
||||
<console>true</console>
|
||||
</logger>
|
||||
|
||||
<http_port>8123</http_port>
|
||||
<tcp_port>9000</tcp_port>
|
||||
|
||||
<max_connections>4096</max_connections>
|
||||
<max_concurrent_queries>100</max_concurrent_queries>
|
||||
<max_server_memory_usage_to_ram_ratio>0.75</max_server_memory_usage_to_ram_ratio>
|
||||
<max_memory_usage_for_all_queries>0</max_memory_usage_for_all_queries>
|
||||
|
||||
<mark_cache_size>5368709120</mark_cache_size>
|
||||
<path>/var/lib/clickhouse/</path>
|
||||
<tmp_path>/var/lib/clickhouse/tmp/</tmp_path>
|
||||
<user_files_path>/var/lib/clickhouse/user_files/</user_files_path>
|
||||
|
||||
<users_config>users.xml</users_config>
|
||||
<default_profile>default</default_profile>
|
||||
<default_database>opt_scraped</default_database>
|
||||
|
||||
<timezone>Europe/Stockholm</timezone>
|
||||
|
||||
<query_log>
|
||||
<database>system</database>
|
||||
<table>query_log</table>
|
||||
<partition_by>toYYYYMM(event_date)</partition_by>
|
||||
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
|
||||
</query_log>
|
||||
|
||||
<distributed_ddl>
|
||||
<path>/clickhouse/task_queue/ddl</path>
|
||||
</distributed_ddl>
|
||||
|
||||
<merge_tree>
|
||||
<max_suspicious_broken_parts>5</max_suspicious_broken_parts>
|
||||
<min_bytes_for_wide_part>0</min_bytes_for_wide_part>
|
||||
<max_bytes_to_merge_at_max_space_in_pool>10737418240</max_bytes_to_merge_at_max_space_in_pool>
|
||||
</merge_tree>
|
||||
|
||||
<!-- S3 integration for MinIO -->
|
||||
<storage_configuration>
|
||||
<disks>
|
||||
<default>
|
||||
<path>/var/lib/clickhouse/</path>
|
||||
</default>
|
||||
<s3>
|
||||
<type>s3</type>
|
||||
<endpoint>https://s3.siax.io/clickhouse-data/</endpoint>
|
||||
<access_key_id>CHANGE_ME</access_key_id>
|
||||
<secret_access_key>CHANGE_ME</secret_access_key>
|
||||
<region>eu-central</region>
|
||||
<metadata_path>/var/lib/clickhouse/disks/s3/</metadata_path>
|
||||
<send_cache>true</send_cache>
|
||||
</s3>
|
||||
</disks>
|
||||
<policies>
|
||||
<s3_only>
|
||||
<volumes>
|
||||
<main>
|
||||
<disk>s3</disk>
|
||||
</main>
|
||||
</volumes>
|
||||
</s3_only>
|
||||
<tiered>
|
||||
<volumes>
|
||||
<hot>
|
||||
<disk>default</disk>
|
||||
<max_data_part_size>1073741824</max_data_part_size>
|
||||
</hot>
|
||||
<cold>
|
||||
<disk>s3</disk>
|
||||
</cold>
|
||||
</volumes>
|
||||
</tiered>
|
||||
</policies>
|
||||
</storage_configuration>
|
||||
|
||||
<part_log>
|
||||
<database>system</database>
|
||||
<table>part_log</table>
|
||||
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
|
||||
</part_log>
|
||||
|
||||
<query_thread_log>
|
||||
<database>system</database>
|
||||
<table>query_thread_log</table>
|
||||
<partition_by>toYYYYMM(event_date)</partition_by>
|
||||
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
|
||||
</query_thread_log>
|
||||
|
||||
<asynchronous_insert_log>
|
||||
<database>system</database>
|
||||
<table>asynchronous_insert_log</table>
|
||||
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
|
||||
</asynchronous_insert_log>
|
||||
</yandex>
|
||||
@@ -1,54 +0,0 @@
|
||||
<yandex>
|
||||
<profiles>
|
||||
<default>
|
||||
<max_memory_usage>8000000000</max_memory_usage>
|
||||
<use_uncompressed_cache>0</use_uncompressed_cache>
|
||||
<load_balancing>random</load_balancing>
|
||||
<max_query_size>10485760</max_query_size>
|
||||
<group_by_overflow_mode>any</group_by_overflow_mode>
|
||||
<timeout_before_checking_execution_speed>0</timeout_before_checking_execution_speed>
|
||||
</default>
|
||||
<readonly>
|
||||
<readonly>1</readonly>
|
||||
</readonly>
|
||||
</profiles>
|
||||
|
||||
<users>
|
||||
<default>
|
||||
<password>CHANGE_ME</password>
|
||||
<networks>
|
||||
<ip>::/0</ip>
|
||||
</networks>
|
||||
<profile>default</profile>
|
||||
<quota>default</quota>
|
||||
<grants>
|
||||
<select>1</select>
|
||||
<insert>1</insert>
|
||||
<alter>1</alter>
|
||||
<create>1</create>
|
||||
<drop>1</drop>
|
||||
</grants>
|
||||
</default>
|
||||
<readonly>
|
||||
<password></password>
|
||||
<networks>
|
||||
<ip>::/0</ip>
|
||||
</networks>
|
||||
<profile>readonly</profile>
|
||||
<quota>default</quota>
|
||||
</readonly>
|
||||
</users>
|
||||
|
||||
<quotas>
|
||||
<default>
|
||||
<interval>
|
||||
<duration>3600</duration>
|
||||
<queries>10000</queries>
|
||||
<errors>100</errors>
|
||||
<result_rows>10000000000</result_rows>
|
||||
<read_rows>100000000000</read_rows>
|
||||
<execution_time>3600</execution_time>
|
||||
</interval>
|
||||
</default>
|
||||
</quotas>
|
||||
</yandex>
|
||||
@@ -1,137 +0,0 @@
|
||||
# SIAX Cloud Data Stack - Complete BigQuery Replacement
|
||||
# All OPEN SOURCE, self-hosted on SIAX Cloud Server 2
|
||||
#
|
||||
# Stack:
|
||||
# ClickHouse - Columnar DB (BigQuery replacement)
|
||||
# Trino - Distributed SQL query engine
|
||||
# FlareSolverr - Cloudflare bypass
|
||||
# Tor - Anonymous proxy rotation
|
||||
# Apache Superset - BI dashboards
|
||||
# MinIO - S3 object storage (already exists)
|
||||
#
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
# ═══════════════════════════════════════════════
|
||||
# CLICKHOUSE - BigQuery ersättning
|
||||
# ═══════════════════════════════════════════════
|
||||
clickhouse:
|
||||
image: clickhouse/clickhouse-server:24.12-alpine
|
||||
container_name: clickhouse
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8123:8123" # HTTP API (REST)
|
||||
- "9000:9000" # Native TCP
|
||||
- "9090:9090" # Prometheus metrics
|
||||
volumes:
|
||||
- clickhouse-data:/var/lib/clickhouse
|
||||
- clickhouse-logs:/var/log/clickhouse-server
|
||||
- ./clickhouse/config.xml:/etc/clickhouse-server/config.xml:ro
|
||||
- ./clickhouse/users.xml:/etc/clickhouse-server/users.xml:ro
|
||||
environment:
|
||||
- CLICKHOUSE_DB=opt_scraped
|
||||
- CLICKHOUSE_USER=default
|
||||
- CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD:-changeme}
|
||||
ulimits:
|
||||
nofile:
|
||||
soft: 262144
|
||||
hard: 262144
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 8G
|
||||
reservations:
|
||||
memory: 4G
|
||||
|
||||
# ═══════════════════════════════════════════════
|
||||
# TRINO - Distributed SQL (optional, for complex queries)
|
||||
# ═══════════════════════════════════════════════
|
||||
trino:
|
||||
image: trinodb/trino:470
|
||||
container_name: trino
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./trino/etc:/etc/trino:ro
|
||||
- trino-data:/data/trino
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4G
|
||||
|
||||
# ═══════════════════════════════════════════════
|
||||
# FLARESOLVERR - Cloudflare bypass
|
||||
# ═══════════════════════════════════════════════
|
||||
flaresolverr:
|
||||
image: ghcr.io/flaresolverr/flaresolverr:latest
|
||||
container_name: flaresolverr
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8191:8191"
|
||||
environment:
|
||||
- LOG_LEVEL=${LOG_LEVEL:-info}
|
||||
- TZ=Europe/Stockholm
|
||||
- CAPTCHA_SOLVER=none
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 1G
|
||||
|
||||
# ═══════════════════════════════════════════════
|
||||
# TOR - Anonymous proxy rotation
|
||||
# ═══════════════════════════════════════════════
|
||||
tor-proxy:
|
||||
image: dperson/torproxy:latest
|
||||
container_name: tor-proxy
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9050:9050" # SOCKS5
|
||||
- "9051:9051" # Control port
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 256M
|
||||
|
||||
# ═══════════════════════════════════════════════
|
||||
# APACHE SUPERSET - BI Dashboard
|
||||
# ═══════════════════════════════════════════════
|
||||
superset:
|
||||
image: apache/superset:latest
|
||||
container_name: superset
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8088:8088"
|
||||
environment:
|
||||
- SUPERSET_SECRET_KEY=${SUPERSET_SECRET_KEY:-changeme}
|
||||
- SUPERSET_DATABASE_URI=postgresql://superset:${POSTGRES_PASSWORD:-changeme}@postgres:5432/superset
|
||||
volumes:
|
||||
- superset-data:/app/superset_home
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2G
|
||||
depends_on:
|
||||
- clickhouse
|
||||
- postgres
|
||||
|
||||
# ═══════════════════════════════════════════════
|
||||
# POSTGRES for Superset metadata
|
||||
# ═══════════════════════════════════════════════
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: postgres-superset
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- POSTGRES_DB=superset
|
||||
- POSTGRES_USER=superset
|
||||
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
|
||||
volumes:
|
||||
clickhouse-data:
|
||||
clickhouse-logs:
|
||||
trino-data:
|
||||
superset-data:
|
||||
postgres-data:
|
||||
@@ -1,25 +0,0 @@
|
||||
FROM python:3.14-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system deps
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl ca-certificates && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Python packages
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy scraper code
|
||||
COPY . .
|
||||
|
||||
# Create data directories
|
||||
RUN mkdir -p /data/output /data/checkpoints /data/sources
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV S3_ENDPOINT=https://s3.siax.io
|
||||
ENV OUTPUT_DIR=/data/output
|
||||
ENV CHECKPOINT_DIR=/data/checkpoints
|
||||
|
||||
CMD ["python", "-m", "scrapers.run_pipeline"]
|
||||
@@ -1,31 +0,0 @@
|
||||
# 0pt Scraper → b0ts
|
||||
|
||||
Scraper code has moved to the **b0ts** repo:
|
||||
https://git.siax.io/admin/b0ts
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
git clone https://git.siax.io/admin/b0ts.git
|
||||
cd b0ts
|
||||
python -m src.main
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
b0ts runs on port 4115:
|
||||
- `POST /pipeline/start` — Start continuous scraping
|
||||
- `GET /pipeline/status` — Check progress
|
||||
- `GET /analytics/quality` — Data quality metrics
|
||||
|
||||
## MCP
|
||||
|
||||
MCP tools available at `schemas/mcp/b0ts.mcp.json`.
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
# On SIAX Cloud Server 2
|
||||
ssh -i ~/.ssh/id_ed25519_siax root@188.40.230.69
|
||||
cd /opt/b0ts && python3 -m src.main
|
||||
```
|
||||
@@ -1 +0,0 @@
|
||||
# 0pt Scraper Engine v2 - Production scrapers package
|
||||
Binary file not shown.
Binary file not shown.
@@ -1,261 +0,0 @@
|
||||
"""
|
||||
ClickHouse Data Pipeline - Skickar scrapad data direkt till ClickHouse.
|
||||
|
||||
Ersätter:
|
||||
- BigQuery (används aldrig)
|
||||
- MinIO/S3 fil-lagring (mellanlagring)
|
||||
|
||||
Pipeline: Scraper -> Parquet -> ClickHouse (columnar OLAP)
|
||||
"""
|
||||
import os, json, time, gzip
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Any
|
||||
|
||||
CLICKHOUSE_HOST = os.getenv("CLICKHOUSE_HOST", "localhost")
|
||||
CLICKHOUSE_PORT = int(os.getenv("CLICKHOUSE_PORT", "8123"))
|
||||
CLICKHOUSE_USER = os.getenv("CLICKHOUSE_USER", "default")
|
||||
CLICKHOUSE_PASSWORD = os.getenv("CLICKHOUSE_PASSWORD", "")
|
||||
CLICKHOUSE_DB = os.getenv("CLICKHOUSE_DB", "opt_scraped")
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "https://s3.siax.io")
|
||||
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "")
|
||||
S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "")
|
||||
|
||||
|
||||
class ClickHouseWriter:
|
||||
"""Write scraped data to ClickHouse via HTTP interface."""
|
||||
|
||||
def __init__(self):
|
||||
self.client = None
|
||||
self._connect()
|
||||
|
||||
def _connect(self):
|
||||
from clickhouse_driver import Client
|
||||
self.client = Client(
|
||||
host=CLICKHOUSE_HOST,
|
||||
port=CLICKHOUSE_PORT,
|
||||
user=CLICKHOUSE_USER,
|
||||
password=CLICKHOUSE_PASSWORD,
|
||||
database=CLICKHOUSE_DB,
|
||||
settings={"max_memory_usage": 8000000000},
|
||||
)
|
||||
|
||||
def create_database(self):
|
||||
self.client.execute(f"CREATE DATABASE IF NOT EXISTS {CLICKHOUSE_DB}")
|
||||
|
||||
def ensure_tables(self):
|
||||
"""Create all required tables for scraped data."""
|
||||
|
||||
# Companies table (multi-country)
|
||||
self.client.execute("""
|
||||
CREATE TABLE IF NOT EXISTS companies (
|
||||
org_nr String,
|
||||
country FixedString(2),
|
||||
source String,
|
||||
name String,
|
||||
legal_form String DEFAULT '',
|
||||
status String DEFAULT '',
|
||||
address String DEFAULT '',
|
||||
postal_code String DEFAULT '',
|
||||
city String DEFAULT '',
|
||||
country_code String DEFAULT '',
|
||||
email String DEFAULT '',
|
||||
phone String DEFAULT '',
|
||||
website String DEFAULT '',
|
||||
employees UInt32 DEFAULT 0,
|
||||
founded_date Date DEFAULT '1970-01-01',
|
||||
registered_capital Float64 DEFAULT 0,
|
||||
currency FixedString(3) DEFAULT '',
|
||||
industry_code String DEFAULT '',
|
||||
industry_description String DEFAULT '',
|
||||
lei String DEFAULT '',
|
||||
last_updated DateTime DEFAULT now(),
|
||||
raw_json String DEFAULT '',
|
||||
scraped_at DateTime DEFAULT now()
|
||||
) ENGINE = ReplacingMergeTree(last_updated)
|
||||
ORDER BY (country, org_nr)
|
||||
PARTITION BY country
|
||||
SETTINGS index_granularity = 8192
|
||||
""")
|
||||
|
||||
# Contacts table (board members, executives)
|
||||
self.client.execute("""
|
||||
CREATE TABLE IF NOT EXISTS contacts (
|
||||
org_nr String,
|
||||
country FixedString(2),
|
||||
full_name String,
|
||||
role String DEFAULT '',
|
||||
email String DEFAULT '',
|
||||
phone String DEFAULT '',
|
||||
source String DEFAULT '',
|
||||
confidence Float32 DEFAULT 0,
|
||||
last_updated DateTime DEFAULT now(),
|
||||
scraped_at DateTime DEFAULT now()
|
||||
) ENGINE = ReplacingMergeTree(last_updated)
|
||||
ORDER BY (country, org_nr, full_name)
|
||||
PARTITION BY country
|
||||
""")
|
||||
|
||||
# Vehicles table (for DE vehicle data)
|
||||
self.client.execute("""
|
||||
CREATE TABLE IF NOT EXISTS vehicles (
|
||||
registration String,
|
||||
country FixedString(2),
|
||||
vin String DEFAULT '',
|
||||
make String DEFAULT '',
|
||||
model String DEFAULT '',
|
||||
year UInt16 DEFAULT 0,
|
||||
fuel_type String DEFAULT '',
|
||||
owner_name String DEFAULT '',
|
||||
owner_org_nr String DEFAULT '',
|
||||
last_updated DateTime DEFAULT now(),
|
||||
scraped_at DateTime DEFAULT now()
|
||||
) ENGINE = ReplacingMergeTree(last_updated)
|
||||
ORDER BY (country, registration)
|
||||
PARTITION BY country
|
||||
""")
|
||||
|
||||
# Raw scrape log (for debugging/audit)
|
||||
self.client.execute("""
|
||||
CREATE TABLE IF NOT EXISTS scrape_log (
|
||||
url String,
|
||||
source String,
|
||||
country FixedString(2),
|
||||
method String DEFAULT '',
|
||||
status UInt16 DEFAULT 0,
|
||||
elapsed_ms UInt32 DEFAULT 0,
|
||||
bytes UInt32 DEFAULT 0,
|
||||
cf_blocked UInt8 DEFAULT 0,
|
||||
error String DEFAULT '',
|
||||
scraped_at DateTime DEFAULT now()
|
||||
) ENGINE = MergeTree
|
||||
ORDER BY (country, scraped_at)
|
||||
PARTITION BY toYYYYMM(scraped_at)
|
||||
TTL scraped_at + INTERVAL 90 DAY
|
||||
""")
|
||||
|
||||
# Statistics for monitoring
|
||||
self.client.execute("""
|
||||
CREATE TABLE IF NOT EXISTS scrape_stats (
|
||||
country FixedString(2),
|
||||
source String,
|
||||
date Date DEFAULT today(),
|
||||
total_requests UInt64 DEFAULT 0,
|
||||
successful UInt64 DEFAULT 0,
|
||||
cf_blocked UInt64 DEFAULT 0,
|
||||
errors UInt64 DEFAULT 0,
|
||||
total_bytes UInt64 DEFAULT 0,
|
||||
avg_elapsed_ms Float64 DEFAULT 0,
|
||||
PRIMARY KEY (country, source, date)
|
||||
) ENGINE = SummingMergeTree
|
||||
ORDER BY (country, source, date)
|
||||
""")
|
||||
|
||||
def insert_company(self, org_nr: str, country: str, source: str, **fields):
|
||||
"""Insert or update a company record."""
|
||||
data = {
|
||||
"org_nr": org_nr,
|
||||
"country": country,
|
||||
"source": source,
|
||||
"name": fields.get("name", ""),
|
||||
"legal_form": fields.get("legal_form", ""),
|
||||
"status": fields.get("status", ""),
|
||||
"address": fields.get("address", ""),
|
||||
"postal_code": fields.get("postal_code", ""),
|
||||
"city": fields.get("city", ""),
|
||||
"country_code": fields.get("country_code", ""),
|
||||
"email": fields.get("email", ""),
|
||||
"phone": fields.get("phone", ""),
|
||||
"website": fields.get("website", ""),
|
||||
"employees": int(fields.get("employees", 0)),
|
||||
"founded_date": fields.get("founded_date", "1970-01-01"),
|
||||
"registered_capital": float(fields.get("registered_capital", 0)),
|
||||
"currency": fields.get("currency", ""),
|
||||
"industry_code": fields.get("industry_code", ""),
|
||||
"industry_description": fields.get("industry_description", ""),
|
||||
"lei": fields.get("lei", ""),
|
||||
"raw_json": json.dumps(fields.get("raw_data", {}), ensure_ascii=False),
|
||||
}
|
||||
self.client.execute(
|
||||
"INSERT INTO companies VALUES",
|
||||
[data],
|
||||
)
|
||||
|
||||
def insert_contact(self, org_nr: str, country: str, full_name: str, role: str = "",
|
||||
email: str = "", phone: str = "", source: str = "", confidence: float = 0):
|
||||
self.client.execute(
|
||||
"INSERT INTO contacts VALUES",
|
||||
[{
|
||||
"org_nr": org_nr, "country": country, "full_name": full_name,
|
||||
"role": role, "email": email, "phone": phone,
|
||||
"source": source, "confidence": confidence,
|
||||
}],
|
||||
)
|
||||
|
||||
def log_scrape(self, url: str, source: str, country: str, method: str,
|
||||
status: int, elapsed_ms: int, bytes_count: int,
|
||||
cf_blocked: bool = False, error: str = ""):
|
||||
self.client.execute(
|
||||
"INSERT INTO scrape_log VALUES",
|
||||
[{
|
||||
"url": url[:500], "source": source, "country": country,
|
||||
"method": method, "status": status, "elapsed_ms": elapsed_ms,
|
||||
"bytes": bytes_count, "cf_blocked": 1 if cf_blocked else 0, "error": error,
|
||||
}],
|
||||
)
|
||||
|
||||
def query(self, sql: str) -> list:
|
||||
"""Execute SQL query and return results."""
|
||||
return self.client.execute(sql)
|
||||
|
||||
def close(self):
|
||||
if self.client:
|
||||
self.client.disconnect()
|
||||
|
||||
|
||||
class DuckDBWriter:
|
||||
"""Alternative: DuckDB for local/embedded analytics."""
|
||||
|
||||
def __init__(self, db_path: str = "/data/opt_scraped.duckdb"):
|
||||
import duckdb
|
||||
self.db_path = db_path
|
||||
self.conn = duckdb.connect(db_path)
|
||||
self._ensure_tables()
|
||||
|
||||
def _ensure_tables(self):
|
||||
self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS companies AS
|
||||
SELECT * FROM read_parquet('/data/parquet/*.parquet') WHERE 1=0
|
||||
""")
|
||||
self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS scrape_log (
|
||||
url VARCHAR, source VARCHAR, country VARCHAR,
|
||||
method VARCHAR, status INTEGER, elapsed_ms INTEGER,
|
||||
bytes INTEGER, cf_blocked BOOLEAN, scraped_at TIMESTAMP
|
||||
)
|
||||
""")
|
||||
|
||||
def import_parquet(self, parquet_dir: str = "/data/parquet"):
|
||||
"""Import all parquet files into DuckDB."""
|
||||
import glob
|
||||
files = glob.glob(f"{parquet_dir}/*.parquet")
|
||||
if not files:
|
||||
return 0
|
||||
self.conn.execute(f"""
|
||||
INSERT INTO companies
|
||||
SELECT * FROM read_parquet({files!r})
|
||||
""")
|
||||
return self.conn.execute("SELECT count(*) FROM companies").fetchone()[0]
|
||||
|
||||
def query(self, sql: str) -> list:
|
||||
return self.conn.execute(sql).fetchall()
|
||||
|
||||
def to_parquet(self, table: str, output_path: str):
|
||||
"""Export table to parquet for archival."""
|
||||
self.conn.execute(f"""
|
||||
COPY (SELECT * FROM {table}) TO '{output_path}'
|
||||
(FORMAT PARQUET, CODEC ZSTD)
|
||||
""")
|
||||
|
||||
def close(self):
|
||||
self.conn.close()
|
||||
@@ -1,77 +0,0 @@
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
scraper-brreg-no:
|
||||
image: scraper-engine:latest
|
||||
environment:
|
||||
- COUNTRY=NO
|
||||
- SOURCE=brreg
|
||||
- MAX_CONCURRENCY=100
|
||||
- S3_ENDPOINT=${S3_ENDPOINT}
|
||||
- S3_ACCESS_KEY=${S3_ACCESS_KEY}
|
||||
- S3_SECRET_KEY=${S3_SECRET_KEY}
|
||||
volumes:
|
||||
- scraper-data:/data
|
||||
restart: unless-stopped
|
||||
command: ["python", "-m", "scrapers.run_pipeline", "--country", "NO", "--source", "brreg"]
|
||||
|
||||
scraper-brreg-under:
|
||||
image: scraper-engine:latest
|
||||
environment:
|
||||
- COUNTRY=NO
|
||||
- SOURCE=brreg_under
|
||||
- MAX_CONCURRENCY=100
|
||||
volumes:
|
||||
- scraper-data:/data
|
||||
restart: unless-stopped
|
||||
command: ["python", "-m", "scrapers.run_pipeline", "--country", "NO", "--source", "brreg_under"]
|
||||
|
||||
scraper-dk-cvrapi:
|
||||
image: scraper-engine:latest
|
||||
environment:
|
||||
- COUNTRY=DK
|
||||
- SOURCE=cvrapi
|
||||
- MAX_CONCURRENCY=30
|
||||
volumes:
|
||||
- scraper-data:/data
|
||||
restart: unless-stopped
|
||||
command: ["python", "-m", "scrapers.run_pipeline", "--country", "DK", "--source", "cvrapi"]
|
||||
|
||||
scraper-de-northdata:
|
||||
image: scraper-engine:latest
|
||||
environment:
|
||||
- COUNTRY=DE
|
||||
- SOURCE=northdata
|
||||
- MAX_CONCURRENCY=20
|
||||
volumes:
|
||||
- scraper-data:/data
|
||||
restart: unless-stopped
|
||||
command: ["python", "-m", "scrapers.run_pipeline", "--country", "DE", "--source", "northdata"]
|
||||
|
||||
scraper-de-handelsregister:
|
||||
image: scraper-engine:latest
|
||||
environment:
|
||||
- COUNTRY=DE
|
||||
- SOURCE=handelsregister
|
||||
- MAX_CONCURRENCY=20
|
||||
volumes:
|
||||
- scraper-data:/data
|
||||
restart: unless-stopped
|
||||
command: ["python", "-m", "scrapers.run_pipeline", "--country", "DE", "--source", "handelsregister"]
|
||||
|
||||
s3-sync:
|
||||
image: scraper-engine:latest
|
||||
depends_on:
|
||||
- scraper-brreg-no
|
||||
environment:
|
||||
- S3_ENDPOINT=${S3_ENDPOINT}
|
||||
- S3_ACCESS_KEY=${S3_ACCESS_KEY}
|
||||
- S3_SECRET_KEY=${S3_SECRET_KEY}
|
||||
- S3_BUCKET=opt-scraped-data
|
||||
volumes:
|
||||
- scraper-data:/data
|
||||
command: ["python", "-m", "scrapers.s3_sync"]
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
scraper-data:
|
||||
@@ -1,243 +0,0 @@
|
||||
"""
|
||||
0pt Scraper Engine v2
|
||||
=====================
|
||||
Universal scraping engine for DK/NO/DE company data.
|
||||
Uses multi-method bypass chain + header rotation for maximum speed.
|
||||
|
||||
Metoder (i prioritetsordning):
|
||||
1. curl_cffi chrome124 - TLS fingerprint impersonation
|
||||
2. curl_cffi edge99 - alternative fingerprint
|
||||
3. cloudscraper - basic CF bypass
|
||||
4. requests + header rotation - direct HTTP
|
||||
5. httpx async - high concurrency
|
||||
6. FlareSolverr - heavy CF bypass (kraver Docker)
|
||||
"""
|
||||
|
||||
import json, time, random, os, asyncio
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
import httpx
|
||||
from curl_cffi import requests as curl_requests
|
||||
import cloudscraper
|
||||
|
||||
# ── Config ──
|
||||
OUTPUT_DIR = Path(__file__).parent / "output"
|
||||
HEADER_POOL = {
|
||||
"user_agents": [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15",
|
||||
],
|
||||
"languages": {
|
||||
"DK": "da-DK,da;q=0.9,en;q=0.8,de;q=0.7",
|
||||
"NO": "nb-NO,nb;q=0.9,no;q=0.8,en;q=0.7,da;q=0.6",
|
||||
"DE": "de-DE,de;q=0.9,en;q=0.8,fr;q=0.7",
|
||||
"SE": "sv-SE,sv;q=0.9,en;q=0.8",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScrapeResult:
|
||||
source: str
|
||||
url: str
|
||||
method: str
|
||||
status: int
|
||||
elapsed: float
|
||||
bytes: int
|
||||
cf_blocked: bool = False
|
||||
data: Optional[dict] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
def rotate_headers(country="DE"):
|
||||
ua = random.choice(HEADER_POOL["user_agents"])
|
||||
lang = HEADER_POOL["languages"].get(country, "en-US,en;q=0.9")
|
||||
return {
|
||||
"User-Agent": ua,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
|
||||
"Accept-Language": lang,
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"Cache-Control": "no-cache",
|
||||
"Sec-Ch-Ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
|
||||
"Sec-Ch-Ua-Mobile": "?0",
|
||||
"Sec-Fetch-Dest": "document",
|
||||
"Sec-Fetch-Mode": "navigate",
|
||||
"Sec-Fetch-Site": "none",
|
||||
"Sec-Fetch-User": "?1",
|
||||
"Upgrade-Insecure-Requests": "1",
|
||||
}
|
||||
|
||||
|
||||
def check_cf_blocked(text: str) -> bool:
|
||||
indicators = [
|
||||
"cf-wrapper", "Just a moment", "Checking your browser",
|
||||
"Cloudflare", "__cf_chl_f_tk", "cf-browser-verification",
|
||||
"challenge-form", "Attention Required",
|
||||
]
|
||||
return any(i in text for i in indicators[:5000])
|
||||
|
||||
|
||||
# ── Method implementations ──
|
||||
|
||||
_methods_cache = {}
|
||||
|
||||
def _get_methods():
|
||||
if _methods_cache:
|
||||
return _methods_cache["methods"]
|
||||
|
||||
cloud = cloudscraper.create_scraper(
|
||||
browser={"browser": "chrome", "platform": "windows", "mobile": False}
|
||||
)
|
||||
|
||||
methods = [
|
||||
("curl_cffi chrome124", lambda url, c="DK": curl_requests.get(
|
||||
url, impersonate="chrome124", headers=rotate_headers(c), timeout=15)),
|
||||
("curl_cffi edge99", lambda url, c="DK": curl_requests.get(
|
||||
url, impersonate="edge99", headers=rotate_headers(c), timeout=15)),
|
||||
("curl_cffi chrome110", lambda url, c="DK": curl_requests.get(
|
||||
url, impersonate="chrome110", headers=rotate_headers(c), timeout=15)),
|
||||
("curl_cffi safari17", lambda url, c="DK": curl_requests.get(
|
||||
url, impersonate="safari17_0", headers=rotate_headers(c), timeout=15)),
|
||||
("cloudscraper", lambda url, c="DK": cloud.get(
|
||||
url, headers=rotate_headers(c), timeout=15)),
|
||||
("requests_rotated", lambda url, c="DK": requests.get(
|
||||
url, headers=rotate_headers(c), timeout=15)),
|
||||
("httpx_rotated", lambda url, c="DK": httpx.get(
|
||||
url, headers=rotate_headers(c), timeout=15)),
|
||||
]
|
||||
|
||||
_methods_cache["methods"] = methods
|
||||
return methods
|
||||
|
||||
|
||||
def scrape(url: str, country: str = "DK", method_index: int = -1) -> list[ScrapeResult]:
|
||||
"""Scrape a URL with all (or specific) methods. Returns list of results."""
|
||||
methods = _get_methods()
|
||||
if method_index >= 0:
|
||||
methods = [methods[method_index]]
|
||||
|
||||
results = []
|
||||
for name, fn in methods:
|
||||
for attempt in range(2):
|
||||
try:
|
||||
start = time.time()
|
||||
r = fn(url, country)
|
||||
elapsed = time.time() - start
|
||||
text_sample = str(r.content[:10000] if hasattr(r, "content") else "")
|
||||
blocked = check_cf_blocked(text_sample)
|
||||
|
||||
results.append(ScrapeResult(
|
||||
source=url,
|
||||
url=url,
|
||||
method=name,
|
||||
status=r.status_code,
|
||||
elapsed=round(elapsed, 3),
|
||||
bytes=len(r.content) if hasattr(r, "content") else 0,
|
||||
cf_blocked=blocked,
|
||||
))
|
||||
break
|
||||
except Exception as e:
|
||||
if attempt == 1:
|
||||
results.append(ScrapeResult(
|
||||
source=url, url=url, method=name,
|
||||
status=0, elapsed=0, bytes=0,
|
||||
error=str(e)[:200],
|
||||
))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def scrape_best(url: str, country: str = "DK") -> ScrapeResult:
|
||||
"""Try all methods, return the fastest successful result."""
|
||||
results = scrape(url, country)
|
||||
ok = [r for r in results if r.status == 200 and not r.cf_blocked]
|
||||
if ok:
|
||||
return min(ok, key=lambda r: r.elapsed)
|
||||
# If all failed, return the one with highest status
|
||||
return max(results, key=lambda r: r.status) if results else ScrapeResult(source=url, url=url, method="none", status=0, elapsed=0, bytes=0, error="No methods available")
|
||||
|
||||
|
||||
async def scrape_async(urls: list[str], country: str = "DK", concurrency: int = 20) -> list[ScrapeResult]:
|
||||
"""Scrape multiple URLs concurrently using async httpx."""
|
||||
from httpx import AsyncClient, Limits, Timeout
|
||||
limits = Limits(max_keepalive_connections=concurrency, max_connections=concurrency)
|
||||
timeout = Timeout(15.0)
|
||||
|
||||
async with AsyncClient(limits=limits, timeout=timeout) as client:
|
||||
async def fetch(url):
|
||||
start = time.time()
|
||||
try:
|
||||
r = await client.get(url, headers=rotate_headers(country))
|
||||
elapsed = time.time() - start
|
||||
text_sample = str(r.content[:10000])
|
||||
return ScrapeResult(
|
||||
source=url, url=url, method="async httpx",
|
||||
status=r.status_code, elapsed=round(elapsed, 3),
|
||||
bytes=len(r.content), cf_blocked=check_cf_blocked(text_sample),
|
||||
)
|
||||
except Exception as e:
|
||||
return ScrapeResult(
|
||||
source=url, url=url, method="async httpx",
|
||||
status=0, elapsed=time.time()-start, bytes=0, error=str(e)[:200],
|
||||
)
|
||||
|
||||
sem = asyncio.Semaphore(concurrency)
|
||||
async def bounded(url):
|
||||
async with sem:
|
||||
return await fetch(url)
|
||||
|
||||
tasks = [bounded(url) for url in urls]
|
||||
return await asyncio.gather(*tasks)
|
||||
|
||||
|
||||
def save_result(result: ScrapeResult, data: dict = None):
|
||||
"""Save scrape result to disk."""
|
||||
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
domain = result.url.split("/")[2].replace(".", "_")
|
||||
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
safe_name = f"{domain}_{result.method}_{ts}.json"
|
||||
path = OUTPUT_DIR / safe_name
|
||||
payload = {
|
||||
"timestamp": ts,
|
||||
"url": result.url,
|
||||
"method": result.method,
|
||||
"status": result.status,
|
||||
"elapsed": result.elapsed,
|
||||
"bytes": result.bytes,
|
||||
"cf_blocked": result.cf_blocked,
|
||||
"error": result.error,
|
||||
"data": data,
|
||||
}
|
||||
path.write_text(json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def benchmark_source(name: str, url: str, country: str = "DK"):
|
||||
"""Full benchmark of a single source with all methods."""
|
||||
print(f"\n {'='*60}")
|
||||
print(f" KALLA: {name}")
|
||||
print(f" URL: {url}")
|
||||
print(f" {'='*60}")
|
||||
|
||||
results = scrape(url, country)
|
||||
ok = [r for r in results if r.status == 200 and not r.cf_blocked]
|
||||
|
||||
for r in results:
|
||||
status_str = "OK" if r.status == 200 else f"ERR {r.status}"
|
||||
cf_str = " [CF]" if r.cf_blocked else ""
|
||||
err_str = f" - {r.error}" if r.error else ""
|
||||
print(f" {r.method:30s} -> {status_str:10s} {r.elapsed:6.3f}s {r.bytes:8d}b{cf_str}{err_str}")
|
||||
|
||||
if ok:
|
||||
best = min(ok, key=lambda r: r.elapsed)
|
||||
print(f"\n >>> BAST: {best.method} ({best.elapsed:.3f}s, {best.bytes}b)")
|
||||
return best
|
||||
return None
|
||||
@@ -1,256 +0,0 @@
|
||||
"""
|
||||
0pt Scraper Engine v2 - Production
|
||||
====================================
|
||||
Körs som Docker container på SIAX Cl0ud Server 2.
|
||||
Data lagras i MinIO S3 (s3.siax.io) för obegränsad lagring.
|
||||
|
||||
Stödjer:
|
||||
- Alla DK/NO/DE källor med auto-detection av bästa metod
|
||||
- Header rotation + curl_cffi TLS fingerprint bypass
|
||||
- Async concurrency för maxhastighet
|
||||
- Resume-kapabilitet (kraschtålig)
|
||||
- Batch processing med checkpointing
|
||||
"""
|
||||
import json, time, random, os, asyncio, sys, gzip
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
import requests
|
||||
import httpx
|
||||
from curl_cffi import requests as curl_requests
|
||||
import cloudscraper
|
||||
|
||||
# ── Config from env ──
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "https://s3.siax.io")
|
||||
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "")
|
||||
S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "")
|
||||
S3_BUCKET = os.getenv("S3_BUCKET", "opt-scraped-data")
|
||||
OUTPUT_DIR = Path(os.getenv("OUTPUT_DIR", "/data/output"))
|
||||
CHECKPOINT_DIR = Path(os.getenv("CHECKPOINT_DIR", "/data/checkpoints"))
|
||||
MAX_CONCURRENCY = int(os.getenv("MAX_CONCURRENCY", "50"))
|
||||
HEADER_POOL = {
|
||||
"user_agents": [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15",
|
||||
],
|
||||
"languages": {
|
||||
"DK": "da-DK,da;q=0.9,en;q=0.8,de;q=0.7",
|
||||
"NO": "nb-NO,nb;q=0.9,no;q=0.8,en;q=0.7,da;q=0.6",
|
||||
"DE": "de-DE,de;q=0.9,en;q=0.8,fr;q=0.7",
|
||||
},
|
||||
}
|
||||
# Pre-computed method rankings per source from benchmarks
|
||||
METHOD_RANKING = {
|
||||
"no-brreg-api": ["cloudscraper", "curl_cffi safari17", "curl_cffi chrome124"],
|
||||
"no-proff": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome110"],
|
||||
"no-1881": ["curl_cffi chrome124", "curl_cffi safari17", "cloudscraper"],
|
||||
"no-gulesider": ["curl_cffi safari17", "curl_cffi chrome124", "cloudscraper"],
|
||||
"no-flowfirma": ["curl_cffi chrome124", "curl_cffi safari17", "curl_cffi chrome110"],
|
||||
"dk-cvrapi": ["curl_cffi safari17", "curl_cffi chrome110", "cloudscraper"],
|
||||
"dk-proff": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome110"],
|
||||
"dk-krak": ["curl_cffi safari17", "curl_cffi chrome124", "cloudscraper"],
|
||||
"dk-118": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome124"],
|
||||
"dk-grundigt": ["curl_cffi safari17", "curl_cffi edge99", "curl_cffi chrome110"],
|
||||
"de-northdata": ["curl_cffi edge99", "curl_cffi chrome110", "curl_cffi safari17"],
|
||||
"de-handelsregister": ["curl_cffi edge99", "curl_cffi chrome110", "cloudscraper"],
|
||||
"de-moneyhouse": ["curl_cffi chrome110", "curl_cffi edge99", "curl_cffi safari17"],
|
||||
"de-unternehmensregister": ["curl_cffi safari17", "curl_cffi chrome110", "curl_cffi chrome124"],
|
||||
"de-creditreform": ["curl_cffi chrome110", "curl_cffi safari17", "curl_cffi edge99"],
|
||||
"default": ["curl_cffi chrome124", "curl_cffi safari17", "cloudscraper", "curl_cffi edge99", "requests"],
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScrapeResult:
|
||||
url: str
|
||||
method: str
|
||||
status: int
|
||||
elapsed: float
|
||||
bytes: int
|
||||
cf_blocked: bool = False
|
||||
data: Optional[dict] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
def rotate_headers(country="DE"):
|
||||
ua = random.choice(HEADER_POOL["user_agents"])
|
||||
lang = HEADER_POOL["languages"].get(country, "en-US,en;q=0.9")
|
||||
return {
|
||||
"User-Agent": ua,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
|
||||
"Accept-Language": lang,
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"Cache-Control": "no-cache",
|
||||
"Sec-Ch-Ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
|
||||
"Sec-Ch-Ua-Mobile": "?0",
|
||||
"Sec-Fetch-Dest": "document",
|
||||
"Sec-Fetch-Mode": "navigate",
|
||||
"Sec-Fetch-Site": "none",
|
||||
"Sec-Fetch-User": "?1",
|
||||
"Upgrade-Insecure-Requests": "1",
|
||||
}
|
||||
|
||||
|
||||
def check_cf_blocked(text: str) -> bool:
|
||||
indicators = ["cf-wrapper", "Just a moment", "Checking your browser",
|
||||
"Cloudflare", "__cf_chl_f_tk", "challenge-form"]
|
||||
return any(i in text[:5000] for i in indicators)
|
||||
|
||||
|
||||
class ScraperEngine:
|
||||
"""Main scraper engine - production ready."""
|
||||
|
||||
def __init__(self):
|
||||
self.cloud = cloudscraper.create_scraper(
|
||||
browser={"browser": "chrome", "platform": "windows", "mobile": False}
|
||||
)
|
||||
self.cache = {}
|
||||
self.stats = {"total": 0, "ok": 0, "cf_blocked": 0, "errors": 0, "total_bytes": 0}
|
||||
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
CHECKPOINT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _get_method(self, name: str):
|
||||
"""Get a scraping method by name."""
|
||||
if name == "curl_cffi chrome124":
|
||||
return lambda url, c="DE": curl_requests.get(url, impersonate="chrome124", headers=rotate_headers(c), timeout=15)
|
||||
elif name == "curl_cffi edge99":
|
||||
return lambda url, c="DE": curl_requests.get(url, impersonate="edge99", headers=rotate_headers(c), timeout=15)
|
||||
elif name == "curl_cffi chrome110":
|
||||
return lambda url, c="DE": curl_requests.get(url, impersonate="chrome110", headers=rotate_headers(c), timeout=15)
|
||||
elif name == "curl_cffi safari17":
|
||||
return lambda url, c="DE": curl_requests.get(url, impersonate="safari17_0", headers=rotate_headers(c), timeout=15)
|
||||
elif name == "cloudscraper":
|
||||
return lambda url, c="DE": self.cloud.get(url, headers=rotate_headers(c), timeout=15)
|
||||
elif name == "requests":
|
||||
return lambda url, c="DE": requests.get(url, headers=rotate_headers(c), timeout=15)
|
||||
return None
|
||||
|
||||
def scrape_url(self, url: str, country: str = "DE", method_rank_key: str = "default") -> ScrapeResult:
|
||||
"""Try methods in ranking order until one succeeds."""
|
||||
methods = METHOD_RANKING.get(method_rank_key, METHOD_RANKING["default"])
|
||||
|
||||
for method_name in methods:
|
||||
fn = self._get_method(method_name)
|
||||
if not fn:
|
||||
continue
|
||||
try:
|
||||
start = time.time()
|
||||
r = fn(url, country)
|
||||
elapsed = time.time() - start
|
||||
text_sample = str(r.content[:10000] if hasattr(r, "content") else "")
|
||||
blocked = check_cf_blocked(text_sample)
|
||||
|
||||
self.stats["total"] += 1
|
||||
if r.status_code == 200 and not blocked:
|
||||
self.stats["ok"] += 1
|
||||
self.stats["total_bytes"] += len(r.content)
|
||||
return ScrapeResult(url=url, method=method_name, status=r.status_code,
|
||||
elapsed=round(elapsed, 3), bytes=len(r.content), cf_blocked=blocked)
|
||||
elif blocked:
|
||||
self.stats["cf_blocked"] += 1
|
||||
else:
|
||||
self.stats["errors"] += 1
|
||||
except Exception as e:
|
||||
self.stats["errors"] += 1
|
||||
continue
|
||||
|
||||
return ScrapeResult(url=url, method="all_failed", status=0, elapsed=0, bytes=0, error="All methods failed")
|
||||
|
||||
def scrape_batch(self, urls: list[tuple], country: str = "DE", method_key: str = "default", max_workers: int = 10) -> list[ScrapeResult]:
|
||||
"""Scrape a batch of URLs concurrently using thread pool."""
|
||||
results = []
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as ex:
|
||||
futures = {ex.submit(self.scrape_url, url, country, method_key): url for url, _ in urls}
|
||||
for f in as_completed(futures):
|
||||
results.append(f.result())
|
||||
return results
|
||||
|
||||
async def scrape_async_batch(self, urls: list[str], country: str = "DE", concurrency: int = None) -> list[ScrapeResult]:
|
||||
"""Massively parallel async scraping."""
|
||||
concurrency = concurrency or MAX_CONCURRENCY
|
||||
from httpx import AsyncClient, Limits, Timeout
|
||||
limits = Limits(max_keepalive_connections=concurrency, max_connections=concurrency)
|
||||
timeout = Timeout(30.0)
|
||||
|
||||
async with AsyncClient(limits=limits, timeout=timeout) as client:
|
||||
async def fetch(url):
|
||||
start = time.time()
|
||||
try:
|
||||
r = await client.get(url, headers=rotate_headers(country), follow_redirects=True)
|
||||
elapsed = time.time() - start
|
||||
return ScrapeResult(url=url, method="async_httpx", status=r.status_code,
|
||||
elapsed=round(elapsed, 3), bytes=len(r.content),
|
||||
cf_blocked=check_cf_blocked(str(r.content[:10000])))
|
||||
except Exception as e:
|
||||
return ScrapeResult(url=url, method="async_httpx", status=0,
|
||||
elapsed=time.time()-start, bytes=0, error=str(e)[:200])
|
||||
|
||||
sem = asyncio.Semaphore(concurrency)
|
||||
async def bounded(url):
|
||||
async with sem:
|
||||
return await fetch(url)
|
||||
|
||||
tasks = [bounded(url) for url in urls]
|
||||
results = await asyncio.gather(*tasks)
|
||||
for r in results:
|
||||
self.stats["total"] += 1
|
||||
if r.status == 200:
|
||||
self.stats["ok"] += 1
|
||||
self.stats["total_bytes"] += r.bytes
|
||||
return results
|
||||
|
||||
def save_result(self, result: ScrapeResult, raw_data: str = None):
|
||||
"""Save result with compression to disk."""
|
||||
domain = result.url.split("/")[2].replace(".", "_")
|
||||
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
safe_name = f"{domain}_{result.method}_{ts}.json.gz"
|
||||
path = OUTPUT_DIR / safe_name
|
||||
payload = {
|
||||
"timestamp": ts, "url": result.url, "method": result.method,
|
||||
"status": result.status, "elapsed": result.elapsed, "bytes": result.bytes,
|
||||
"cf_blocked": result.cf_blocked, "error": result.error,
|
||||
"data_sample": raw_data[:10000] if raw_data else None,
|
||||
}
|
||||
with gzip.open(path, "wt", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False)
|
||||
return path
|
||||
|
||||
def save_checkpoint(self, batch_id: str, completed: list, total: int):
|
||||
"""Save progress checkpoint for resume."""
|
||||
path = CHECKPOINT_DIR / f"checkpoint_{batch_id}.json"
|
||||
data = {"batch_id": batch_id, "completed": len(completed), "total": total,
|
||||
"timestamp": datetime.now().isoformat(), "urls": completed}
|
||||
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
def load_checkpoint(self, batch_id: str) -> set:
|
||||
"""Load completed URLs from checkpoint."""
|
||||
path = CHECKPOINT_DIR / f"checkpoint_{batch_id}.json"
|
||||
if path.exists():
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
return set(data.get("urls", []))
|
||||
return set()
|
||||
|
||||
def print_stats(self):
|
||||
"""Print current scraping statistics."""
|
||||
total = self.stats["total"]
|
||||
ok = self.stats["ok"]
|
||||
rate = ok / (self.stats.get("start_time", 1) and
|
||||
max(1, (time.time() - self.stats.get("start_time", time.time()))))
|
||||
print(f" Stats: {ok}/{total} OK ({self.stats['cf_blocked']} CF, {self.stats['errors']} err)")
|
||||
print(f" Rate: {rate:.0f} req/s | Data: {self.stats['total_bytes']/1024/1024:.1f} MB")
|
||||
|
||||
|
||||
def print_header():
|
||||
print("""
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ 0PT SCRAPER ENGINE v2 - PRODUCTION ║
|
||||
║ DK/NO/DE Company Data Pipeline → MinIO S3 ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
""")
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,9 +0,0 @@
|
||||
# 0pt Scraper Engine - Production dependencies
|
||||
curl_cffi>=0.15.0
|
||||
cloudscraper>=1.2.71
|
||||
httpx>=0.28.0
|
||||
aiohttp>=3.10.0
|
||||
requests>=2.33.0
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=5.3.0
|
||||
boto3>=1.35.0 # MinIO S3 sync
|
||||
@@ -1,86 +0,0 @@
|
||||
"""
|
||||
0pt Scraper Benchmark - test alla källor for DK/NO/DE
|
||||
"""
|
||||
import sys, os, time, json
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
from engine import scrape_best, scrape_async, benchmark_source, OUTPUT_DIR
|
||||
from datetime import datetime
|
||||
|
||||
# === ALLA KALLOR TESTADE ===
|
||||
SOURCES = {
|
||||
"DK": [
|
||||
("CVR API (cvrapi.dk)", "https://cvrapi.dk/api?search=Novo+Nordisk&country=dk", "api"),
|
||||
("Proff.dk", "https://proff.dk", "agg"),
|
||||
("Krak.dk", "https://www.krak.dk/", "dir"),
|
||||
("118.dk", "https://118.dk/", "dir"),
|
||||
("De Gule Sider", "https://www.degulesider.dk/", "dir"),
|
||||
("Companydata.dk", "https://companydata.dk", "agg"),
|
||||
("Grundigt.dk", "https://grundigt.dk", "agg"),
|
||||
("Datafordeler.dk", "https://datafordeler.dk", "gov"),
|
||||
("CVRcheck.dk", "https://cvrcheck.dk", "api"),
|
||||
("Virk", "https://datacvr.virk.dk/", "gov"),
|
||||
],
|
||||
"NO": [
|
||||
("Brreg API (enhet)", "https://data.brreg.no/enhetsregisteret/api/enheter/915651232", "api"),
|
||||
("Brreg API (search)", "https://data.brreg.no/enhetsregisteret/api/enheter?size=10", "api"),
|
||||
("Brreg API (underenheter)", "https://data.brreg.no/enhetsregisteret/api/underenheter?size=10", "api"),
|
||||
("Proff.no", "https://proff.no", "agg"),
|
||||
("1881.no", "https://www.1881.no/", "dir"),
|
||||
("Gulesider.no", "https://www.gulesider.no/", "dir"),
|
||||
("Selskapssok.no", "https://selskapssok.no", "agg"),
|
||||
("Flowfirma.no", "https://flowfirma.no", "agg"),
|
||||
("Offentligdata.no", "https://offentligdata.no", "agg"),
|
||||
("Regnskapsbasen.no", "https://www.regnskapsbasen.no", "agg"),
|
||||
("Okonomi.no", "https://okonomi.no/bedrifter", "agg"),
|
||||
("Visverdi.no", "https://visverdi.no", "agg"),
|
||||
],
|
||||
"DE": [
|
||||
("Northdata.de (Bayer)", "https://www.northdata.de/Bayer+AG", "agg"),
|
||||
("Handelsregister.de", "https://www.handelsregister.de", "gov"),
|
||||
("Unternehmensregister.de", "https://www.unternehmensregister.de", "gov"),
|
||||
("Bundesanzeiger.de", "https://www.bundesanzeiger.de", "gov"),
|
||||
("Companyhouse.de", "https://www.companyhouse.de", "agg"),
|
||||
("Moneyhouse.de", "https://www.moneyhouse.de", "agg"),
|
||||
("Creditreform.de", "https://www.creditreform.de", "agg"),
|
||||
("Personensuche.de", "https://www.personensuche.de", "dir"),
|
||||
("Registercheck.de", "https://www.registercheck.de", "agg"),
|
||||
("Whitepages.de", "https://www.whitepages.de", "dir"),
|
||||
],
|
||||
}
|
||||
|
||||
def run():
|
||||
print("=" * 80)
|
||||
print(f"0PT SCRAPER BENCHMARK - {datetime.now().strftime('%Y-%m-%d %H:%M')}")
|
||||
print("=" * 80)
|
||||
|
||||
results_by_country = {}
|
||||
|
||||
for country, sources in SOURCES.items():
|
||||
print(f"\n{'#'*70}")
|
||||
print(f"# {country} - {len(sources)} källor")
|
||||
print(f"{'#'*70}")
|
||||
|
||||
country_results = []
|
||||
for name, url, src_type in sources:
|
||||
best = benchmark_source(name, url, country)
|
||||
if best:
|
||||
country_results.append({"name": name, "url": url, "method": best.method, "elapsed": best.elapsed, "status": best.status, "bytes": best.bytes})
|
||||
time.sleep(0.3)
|
||||
|
||||
results_by_country[country] = country_results
|
||||
|
||||
print(f"\n [{country}] SAMMANSTALLNING:")
|
||||
for r in sorted(country_results, key=lambda x: x["elapsed"]):
|
||||
print(f" {r['name']:30s} {r['method']:25s} {r['elapsed']:.3f}s {r['bytes']}b")
|
||||
|
||||
# Save report
|
||||
report = {
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"results": results_by_country,
|
||||
}
|
||||
path = OUTPUT_DIR / f"benchmark_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
|
||||
path.write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
print(f"\n Rapport sparad: {path}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -1,235 +0,0 @@
|
||||
"""
|
||||
0pt Scraper Pipeline - Production Batch Processor
|
||||
|
||||
Kör batch-scraping för angivet land/källa med checkpointing.
|
||||
Data sparas lokalt och synkas till MinIO S3.
|
||||
|
||||
Användning:
|
||||
python -m scrapers.run_pipeline --country NO --source brreg
|
||||
python -m scrapers.run_pipeline --country DK --source cvrapi --search "Novo Nordisk"
|
||||
python -m scrapers.run_pipeline --country DE --source northdata --hrb "HRB 123456"
|
||||
"""
|
||||
import os, sys, json, time, asyncio
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
from engine_v2 import ScraperEngine, ScrapeResult, print_header
|
||||
|
||||
# ── Source configurations ──
|
||||
SOURCES = {
|
||||
"NO": {
|
||||
"brreg": {
|
||||
"name": "Brreg Enhetsregisteret",
|
||||
"type": "api",
|
||||
"url_template": "https://data.brreg.no/enhetsregisteret/api/enheter/{}",
|
||||
"method_key": "no-brreg-api",
|
||||
"total_estimate": 1168208,
|
||||
"batch_size": 100,
|
||||
"pagination": True,
|
||||
"list_url": "https://data.brreg.no/enhetsregisteret/api/enheter?size=100&page={}",
|
||||
"fields": 38,
|
||||
},
|
||||
"brreg_under": {
|
||||
"name": "Brreg Underenheter",
|
||||
"type": "api",
|
||||
"method_key": "no-brreg-api",
|
||||
"total_estimate": 849423,
|
||||
"batch_size": 100,
|
||||
"pagination": True,
|
||||
"list_url": "https://data.brreg.no/enhetsregisteret/api/underenheter?size=100&page={}",
|
||||
},
|
||||
"proff": {
|
||||
"name": "Proff.no",
|
||||
"type": "web",
|
||||
"url_template": "https://proff.no/company/companydetails/{}",
|
||||
"method_key": "no-proff",
|
||||
},
|
||||
"1881": {
|
||||
"name": "1881.no",
|
||||
"type": "web",
|
||||
"url_template": "https://www.1881.no/sok/?q={}",
|
||||
"method_key": "no-1881",
|
||||
},
|
||||
"flowfirma": {
|
||||
"name": "Flowfirma.no",
|
||||
"type": "web",
|
||||
"url_template": "https://flowfirma.no/company/{}",
|
||||
"method_key": "no-flowfirma",
|
||||
},
|
||||
},
|
||||
"DK": {
|
||||
"cvrapi": {
|
||||
"name": "CVR API",
|
||||
"type": "api",
|
||||
"url_template": "https://cvrapi.dk/api?search={}&country=dk",
|
||||
"method_key": "dk-cvrapi",
|
||||
"total_estimate": 700000,
|
||||
},
|
||||
"proff": {
|
||||
"name": "Proff.dk",
|
||||
"type": "web",
|
||||
"url_template": "https://proff.dk/company/companydetails/{}",
|
||||
"method_key": "dk-proff",
|
||||
},
|
||||
"grundigt": {
|
||||
"name": "Grundigt.dk",
|
||||
"type": "web",
|
||||
"url_template": "https://grundigt.dk/virksomhed/{}",
|
||||
"method_key": "dk-grundigt",
|
||||
},
|
||||
},
|
||||
"DE": {
|
||||
"northdata": {
|
||||
"name": "Northdata.de",
|
||||
"type": "web",
|
||||
"url_template": "https://www.northdata.de/{}",
|
||||
"method_key": "de-northdata",
|
||||
"total_estimate": 3500000,
|
||||
},
|
||||
"handelsregister": {
|
||||
"name": "Handelsregister.de",
|
||||
"type": "web",
|
||||
"url_template": "https://www.handelsregister.de/{}",
|
||||
"method_key": "de-handelsregister",
|
||||
},
|
||||
"moneyhouse": {
|
||||
"name": "Moneyhouse.de",
|
||||
"type": "web",
|
||||
"url_template": "https://www.moneyhouse.de/{}",
|
||||
"method_key": "de-moneyhouse",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
async def scrape_brreg_bulk(engine: ScraperEngine, country: str, source_key: str):
|
||||
"""Scrape Brreg API with pagination - full company register."""
|
||||
source = SOURCES[country][source_key]
|
||||
total = source["total_estimate"]
|
||||
batch = source["batch_size"]
|
||||
list_url = source["list_url"]
|
||||
method_key = source["method_key"]
|
||||
|
||||
print(f"\n[{country}] {source['name']} - {total} companies")
|
||||
print(f" Method: {method_key} | Concurrency: {engine.MAX_CONCURRENCY}")
|
||||
|
||||
engine.stats["start_time"] = time.time()
|
||||
total_pages = (total // batch) + 1
|
||||
completed_urls = set()
|
||||
|
||||
# Check for checkpoint
|
||||
checkpoint_id = f"{country}_{source_key}"
|
||||
completed_urls = engine.load_checkpoint(checkpoint_id)
|
||||
print(f" Checkpoint: {len(completed_urls)}/{total_pages} pages completed")
|
||||
|
||||
for page in range(total_pages):
|
||||
if f"page_{page}" in completed_urls:
|
||||
continue
|
||||
|
||||
url = list_url.format(page)
|
||||
result = engine.scrape_url(url, country, method_key)
|
||||
|
||||
if result.status == 200 and not result.cf_blocked:
|
||||
import json as j
|
||||
try:
|
||||
data = j.loads(result.data) if result.data else {}
|
||||
items = data.get("_embedded", {}).get(
|
||||
"enheter" if source_key == "brreg" else "underenheter", []
|
||||
)
|
||||
for item in items:
|
||||
engine.save_result(ScrapeResult(
|
||||
url=url, method=result.method,
|
||||
status=200, elapsed=result.elapsed,
|
||||
bytes=len(j.dumps(item)),
|
||||
data=item
|
||||
), j.dumps(item))
|
||||
except:
|
||||
pass
|
||||
|
||||
# Track progress
|
||||
if (page + 1) % 10 == 0:
|
||||
elapsed = time.time() - engine.stats["start_time"]
|
||||
rate = (page + 1) / elapsed if elapsed > 0 else 0
|
||||
remaining = (total_pages - page) / rate if rate > 0 else 0
|
||||
print(f" Page {page+1}/{total_pages} ({rate:.0f} pages/s, ETA: {remaining:.0f}s)")
|
||||
engine.print_stats()
|
||||
|
||||
# Checkpoint every 100 pages
|
||||
if (page + 1) % 100 == 0:
|
||||
completed = [f"page_{p}" for p in range(page + 1)]
|
||||
engine.save_checkpoint(checkpoint_id, completed, total_pages)
|
||||
|
||||
|
||||
async def scrape_by_search(engine: ScraperEngine, country: str, source_key: str, search_terms: list[str]):
|
||||
"""Scrape by searching company names."""
|
||||
source = SOURCES[country][source_key]
|
||||
template = source["url_template"]
|
||||
method_key = source["method_key"]
|
||||
|
||||
print(f"\n[{country}] {source['name']} - {len(search_terms)} searches")
|
||||
engine.stats["start_time"] = time.time()
|
||||
|
||||
for i, term in enumerate(search_terms):
|
||||
url = template.format(term.replace(" ", "+"))
|
||||
result = engine.scrape_url(url, country, method_key)
|
||||
|
||||
if result.status == 200:
|
||||
engine.save_result(result)
|
||||
|
||||
if (i + 1) % 10 == 0:
|
||||
elapsed = time.time() - engine.stats["start_time"]
|
||||
rate = (i + 1) / elapsed if elapsed > 0 else 0
|
||||
remaining = (len(search_terms) - i) / rate if rate > 0 else 0
|
||||
print(f" {i+1}/{len(search_terms)} ({rate:.0f} req/s, ETA: {remaining:.0f}s)")
|
||||
engine.print_stats()
|
||||
|
||||
|
||||
def parse_args():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="0pt Scraper Pipeline")
|
||||
parser.add_argument("--country", choices=["NO", "DK", "DE"], required=True, help="Country code")
|
||||
parser.add_argument("--source", required=True, help="Source key (brreg, cvrapi, northdata, etc)")
|
||||
parser.add_argument("--search", nargs="+", help="Search terms for name-based sources")
|
||||
parser.add_argument("--concurrency", type=int, default=50, help="Max async concurrency")
|
||||
parser.add_argument("--pages", type=int, default=0, help="Number of pages (0=all)")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
print_header()
|
||||
args = parse_args()
|
||||
|
||||
country = args.country
|
||||
source_key = args.source
|
||||
max_conc = args.concurrency
|
||||
|
||||
if country not in SOURCES or source_key not in SOURCES[country]:
|
||||
print(f"Error: unknown source {country}/{source_key}")
|
||||
print(f"Available: {[f'{c}/{k}' for c, srcs in SOURCES.items() for k in srcs.keys()]}")
|
||||
return 1
|
||||
|
||||
engine = ScraperEngine()
|
||||
source = SOURCES[country][source_key]
|
||||
|
||||
print(f" Country: {country}")
|
||||
print(f" Source: {source['name']} ({source['type']})")
|
||||
print(f" Method: {source['method_key']}")
|
||||
print(f" Concurrency: {max_conc}")
|
||||
print()
|
||||
|
||||
if source.get("pagination"):
|
||||
asyncio.run(scrape_brreg_bulk(engine, country, source_key))
|
||||
elif args.search:
|
||||
asyncio.run(scrape_by_search(engine, country, source_key, args.search))
|
||||
else:
|
||||
print("Error: need --search for non-paginated sources, or use --search with search terms")
|
||||
return 1
|
||||
|
||||
print(f"\n Pipeline complete!")
|
||||
engine.print_stats()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,57 +0,0 @@
|
||||
"""
|
||||
S3 Sync - Upload scraped data to MinIO periodically.
|
||||
"""
|
||||
import os, time, json, gzip
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
OUTPUT_DIR = Path(os.getenv("OUTPUT_DIR", "/data/output"))
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "https://s3.siax.io")
|
||||
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "")
|
||||
S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "")
|
||||
S3_BUCKET = os.getenv("S3_BUCKET", "opt-scraped-data")
|
||||
SYNC_INTERVAL = int(os.getenv("SYNC_INTERVAL", "300")) # 5 min
|
||||
|
||||
def get_s3_client():
|
||||
import boto3
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=S3_ENDPOINT,
|
||||
aws_access_key_id=S3_ACCESS_KEY,
|
||||
aws_secret_access_key=S3_SECRET_KEY,
|
||||
)
|
||||
|
||||
def sync_to_s3():
|
||||
"""Upload all new files to MinIO S3."""
|
||||
s3 = get_s3_client()
|
||||
# Ensure bucket exists
|
||||
try:
|
||||
s3.head_bucket(Bucket=S3_BUCKET)
|
||||
except:
|
||||
s3.create_bucket(Bucket=S3_BUCKET)
|
||||
|
||||
uploaded = 0
|
||||
for fpath in OUTPUT_DIR.glob("*.json.gz"):
|
||||
key = f"raw/{fpath.name}"
|
||||
try:
|
||||
s3.upload_file(str(fpath), S3_BUCKET, key)
|
||||
uploaded += 1
|
||||
fpath.unlink() # Remove local after upload
|
||||
except Exception as e:
|
||||
print(f" S3 upload failed: {fpath.name} - {e}")
|
||||
|
||||
return uploaded
|
||||
|
||||
def run():
|
||||
print(f"S3 Sync started -> {S3_ENDPOINT}/{S3_BUCKET}")
|
||||
while True:
|
||||
try:
|
||||
uploaded = sync_to_s3()
|
||||
if uploaded:
|
||||
print(f" Uploaded {uploaded} files to S3")
|
||||
except Exception as e:
|
||||
print(f" Sync error: {e}")
|
||||
time.sleep(SYNC_INTERVAL)
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -1,316 +0,0 @@
|
||||
"""
|
||||
Proxy Rotator & WAF Bypass - ALLA gratis metoder.
|
||||
|
||||
Metoder:
|
||||
1. Header rotation (6 User-Agents x sprak x devices)
|
||||
2. curl_cffi TLS fingerprint (chrome124, chrome110, edge99, safari17)
|
||||
3. cloudscraper (basic CF bypass)
|
||||
4. FlareSolverr (heavy CF bypass, via Docker)
|
||||
5. Tor rotation (SOCKS5, gratis)
|
||||
6. Free proxy pool (scraped from 5+ sources)
|
||||
7. Playwright browser automation
|
||||
8. nodriver (undetected browser)
|
||||
9. Fake identity generation
|
||||
10. Session cookie reuse
|
||||
"""
|
||||
import os, json, time, random, socket, asyncio
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
from curl_cffi import requests as curl_requests
|
||||
import cloudscraper
|
||||
import requests
|
||||
|
||||
# ── Config ──
|
||||
FLARESOLVERR_URL = os.getenv("FLARESOLVERR_URL", "http://localhost:8191/v1")
|
||||
TOR_PROXY = os.getenv("TOR_PROXY", "socks5://localhost:9050")
|
||||
USE_FLARESOLVERR = os.getenv("USE_FLARESOLVERR", "false").lower() == "true"
|
||||
USE_TOR = os.getenv("USE_TOR", "false").lower() == "true"
|
||||
|
||||
# ── Header Pool ──
|
||||
USER_AGENTS = [
|
||||
# Chrome Windows
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
|
||||
# Safari Mac
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15",
|
||||
# Firefox
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0",
|
||||
# Edge
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0",
|
||||
# Linux
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
# Mobile
|
||||
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1",
|
||||
"Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36",
|
||||
]
|
||||
|
||||
ACCEPT_LANGUAGES = {
|
||||
"DK": "da-DK,da;q=0.9,en;q=0.8,de;q=0.7",
|
||||
"NO": "nb-NO,nb;q=0.9,no;q=0.8,en;q=0.7,da;q=0.6",
|
||||
"DE": "de-DE,de;q=0.9,en;q=0.8,fr;q=0.7",
|
||||
"SE": "sv-SE,sv;q=0.9,en;q=0.8",
|
||||
"GB": "en-GB,en;q=0.9,fr;q=0.8",
|
||||
"FI": "fi-FI,fi;q=0.9,en;q=0.8,sv;q=0.7",
|
||||
"FR": "fr-FR,fr;q=0.9,en;q=0.8",
|
||||
"NL": "nl-NL,nl;q=0.9,en;q=0.8",
|
||||
"PL": "pl-PL,pl;q=0.9,en;q=0.8",
|
||||
}
|
||||
|
||||
SEC_CH_UA = {
|
||||
"chrome124": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
|
||||
"chrome125": '"Chromium";v="125", "Google Chrome";v="125", "Not-A.Brand";v="99"',
|
||||
"edge124": '"Chromium";v="124", "Microsoft Edge";v="124", "Not-A.Brand";v="99"',
|
||||
}
|
||||
|
||||
ACCEPT_ENCODINGS = ["gzip, deflate, br", "gzip, deflate", "gzip, deflate, br, zstd"]
|
||||
|
||||
|
||||
def rotate_headers(country="DE", impersonate="chrome124"):
|
||||
ua = random.choice(USER_AGENTS)
|
||||
lang = ACCEPT_LANGUAGES.get(country, "en-US,en;q=0.9")
|
||||
enc = random.choice(ACCEPT_ENCODINGS)
|
||||
sec_ch = SEC_CH_UA.get(impersonate, SEC_CH_UA["chrome124"])
|
||||
|
||||
return {
|
||||
"User-Agent": ua,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
|
||||
"Accept-Language": lang,
|
||||
"Accept-Encoding": enc,
|
||||
"Cache-Control": "no-cache",
|
||||
"Pragma": "no-cache",
|
||||
"Sec-Ch-Ua": sec_ch,
|
||||
"Sec-Ch-Ua-Mobile": "?0",
|
||||
"Sec-Ch-Ua-Platform": '"Windows"' if "Windows" in ua else ('"macOS"' if "Mac" in ua else '"Linux"'),
|
||||
"Sec-Fetch-Dest": "document",
|
||||
"Sec-Fetch-Mode": "navigate",
|
||||
"Sec-Fetch-Site": "none",
|
||||
"Sec-Fetch-User": "?1",
|
||||
"Upgrade-Insecure-Requests": "1",
|
||||
"DNT": "1",
|
||||
}
|
||||
|
||||
|
||||
def check_cf_blocked(text: str) -> bool:
|
||||
"""Check if response contains Cloudflare challenge."""
|
||||
indicators = [
|
||||
"cf-wrapper", "Just a moment", "Checking your browser",
|
||||
"Cloudflare", "__cf_chl_f_tk", "cf-browser-verification",
|
||||
"challenge-form", "Attention Required", "cf-error-page",
|
||||
"cloudflare.com/cdn-cgi", "/cdn-cgi/",
|
||||
]
|
||||
if not text:
|
||||
return False
|
||||
lower = text[:10000].lower()
|
||||
for ind in indicators:
|
||||
if ind in lower:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class ProxyRotator:
|
||||
"""Free proxy pool with rotation."""
|
||||
|
||||
def __init__(self):
|
||||
self.proxies = []
|
||||
self.current = 0
|
||||
self._load_free_proxies()
|
||||
|
||||
def _load_free_proxies(self):
|
||||
"""Load proxies from free sources."""
|
||||
sources = [
|
||||
"https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt",
|
||||
"https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/http.txt",
|
||||
"https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list-raw.txt",
|
||||
]
|
||||
for url in sources:
|
||||
try:
|
||||
r = requests.get(url, timeout=5)
|
||||
for line in r.text.strip().split("\n"):
|
||||
line = line.strip()
|
||||
if ":" in line and not line.startswith("#"):
|
||||
parts = line.split(":")
|
||||
if len(parts) == 2:
|
||||
self.proxies.append({"http": f"http://{line}", "https": f"http://{line}"})
|
||||
except:
|
||||
pass
|
||||
# Deduplicate
|
||||
seen = set()
|
||||
unique = []
|
||||
for p in self.proxies:
|
||||
if p["http"] not in seen:
|
||||
seen.add(p["http"])
|
||||
unique.append(p)
|
||||
self.proxies = unique
|
||||
print(f" ProxyRotator: loaded {len(self.proxies)} free proxies")
|
||||
|
||||
def get_proxy(self) -> Optional[dict]:
|
||||
if not self.proxies:
|
||||
return None
|
||||
self.current = (self.current + 1) % len(self.proxies)
|
||||
return self.proxies[self.current]
|
||||
|
||||
def get_tor_proxy(self) -> dict:
|
||||
return {"http": TOR_PROXY, "https": TOR_PROXY}
|
||||
|
||||
|
||||
class WAFBypass:
|
||||
"""Multi-method WAF bypass engine."""
|
||||
|
||||
def __init__(self):
|
||||
self.cloudscraper = cloudscraper.create_scraper(
|
||||
browser={"browser": "chrome", "platform": "windows", "mobile": False}
|
||||
)
|
||||
self.proxy = ProxyRotator()
|
||||
self.session_pool = {}
|
||||
self.curl_methods = {
|
||||
"chrome124": lambda url, h: curl_requests.get(url, impersonate="chrome124", headers=h, timeout=15),
|
||||
"chrome110": lambda url, h: curl_requests.get(url, impersonate="chrome110", headers=h, timeout=15),
|
||||
"edge99": lambda url, h: curl_requests.get(url, impersonate="edge99", headers=h, timeout=15),
|
||||
"safari17": lambda url, h: curl_requests.get(url, impersonate="safari17_0", headers=h, timeout=15),
|
||||
}
|
||||
|
||||
def try_flaresolverr(self, url: str) -> Optional[str]:
|
||||
"""Bypass CF via FlareSolverr Docker container."""
|
||||
if not USE_FLARESOLVERR:
|
||||
return None
|
||||
try:
|
||||
payload = {
|
||||
"cmd": "request.get",
|
||||
"url": url,
|
||||
"maxTimeout": 60000,
|
||||
"session": f"session_{hash(url) % 10000}",
|
||||
}
|
||||
r = requests.post(FLARESOLVERR_URL, json=payload, timeout=65)
|
||||
data = r.json()
|
||||
if data.get("status") == "ok":
|
||||
solution = data.get("solution", {})
|
||||
return solution.get("response", "")
|
||||
except:
|
||||
pass
|
||||
return None
|
||||
|
||||
def try_tor(self, url: str, headers: dict) -> Optional[requests.Response]:
|
||||
"""Route through Tor network."""
|
||||
if not USE_TOR:
|
||||
return None
|
||||
try:
|
||||
proxies = self.proxy.get_tor_proxy()
|
||||
r = requests.get(url, headers=headers, proxies=proxies, timeout=30)
|
||||
return r
|
||||
except:
|
||||
return None
|
||||
|
||||
def try_curl(self, url: str, country: str = "DE") -> Optional[requests.Response]:
|
||||
"""Try all curl_cffi impersonations in order."""
|
||||
for name, fn in self.curl_methods.items():
|
||||
try:
|
||||
headers = rotate_headers(country, name.replace("curl_cffi_", "").split("_")[0] if "curl_" in name else "chrome124")
|
||||
r = fn(url, headers)
|
||||
if r.status_code == 200 and not check_cf_blocked(str(r.content[:5000])):
|
||||
return r
|
||||
except:
|
||||
continue
|
||||
return None
|
||||
|
||||
def try_cloudscraper(self, url: str, country: str = "DE") -> Optional[requests.Response]:
|
||||
"""Try cloudscraper with different configs."""
|
||||
for browser in [
|
||||
{"browser": "chrome", "platform": "windows", "mobile": False},
|
||||
{"browser": "firefox", "platform": "windows", "mobile": False},
|
||||
{"browser": "chrome", "platform": "macos", "mobile": False},
|
||||
]:
|
||||
try:
|
||||
sc = cloudscraper.create_scraper(browser=browser)
|
||||
sc.headers.update(rotate_headers(country))
|
||||
r = sc.get(url, timeout=15)
|
||||
if r.status_code == 200 and not check_cf_blocked(str(r.content[:5000])):
|
||||
return r
|
||||
except:
|
||||
continue
|
||||
return None
|
||||
|
||||
def try_playwright(self, url: str) -> Optional[str]:
|
||||
"""Use Playwright headless browser as last resort."""
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
context = browser.new_context(
|
||||
user_agent=random.choice(USER_AGENTS),
|
||||
locale="en-US",
|
||||
viewport={"width": 1920, "height": 1080},
|
||||
)
|
||||
page = context.new_page()
|
||||
page.goto(url, timeout=30000)
|
||||
content = page.content()
|
||||
browser.close()
|
||||
if not check_cf_blocked(content):
|
||||
return content
|
||||
except:
|
||||
pass
|
||||
return None
|
||||
|
||||
def bypass(self, url: str, country: str = "DE") -> tuple:
|
||||
"""
|
||||
Full bypass chain. Returns (content: str, method: str, elapsed: float).
|
||||
|
||||
Chain order:
|
||||
1. curl_cffi chrome124 (fastest, 0.1-0.3s)
|
||||
2. cloudscraper (basic CF, 0.3-0.5s)
|
||||
3. curl_cffi edge99 (alt fingerprint)
|
||||
4. curl_cffi safari17 (Eniro bypass)
|
||||
5. Tor proxy rotation (anonymous, 2-5s)
|
||||
6. FlareSolverr (heavy CF, 5-15s)
|
||||
7. Playwright (full browser, 10-30s)
|
||||
"""
|
||||
start = time.time()
|
||||
|
||||
# Level 1: curl_cffi TLS fingerprint
|
||||
r = self.try_curl(url, country)
|
||||
if r and r.status_code == 200:
|
||||
return (r.text, "curl_cffi", time.time() - start)
|
||||
|
||||
# Level 2: cloudscraper
|
||||
r = self.try_cloudscraper(url, country)
|
||||
if r and r.status_code == 200:
|
||||
return (r.text, "cloudscraper", time.time() - start)
|
||||
|
||||
# Level 3: Tor
|
||||
if USE_TOR:
|
||||
r = self.try_tor(url, rotate_headers(country))
|
||||
if r and r.status_code == 200:
|
||||
return (r.text, "tor", time.time() - start)
|
||||
|
||||
# Level 4: FlareSolverr
|
||||
content = self.try_flaresolverr(url)
|
||||
if content:
|
||||
return (content, "flaresolverr", time.time() - start)
|
||||
|
||||
# Level 5: Playwright (last resort)
|
||||
content = self.try_playwright(url)
|
||||
if content:
|
||||
return (content, "playwright", time.time() - start)
|
||||
|
||||
return (None, "all_failed", time.time() - start)
|
||||
|
||||
def bypass_with_proxy(self, url: str, country: str = "DE") -> tuple:
|
||||
"""Bypass with proxy rotation for each method."""
|
||||
start = time.time()
|
||||
proxy = self.proxy.get_proxy()
|
||||
headers = rotate_headers(country)
|
||||
|
||||
for name, fn in self.curl_methods.items():
|
||||
try:
|
||||
if proxy:
|
||||
r = fn(url, headers)
|
||||
else:
|
||||
r = fn(url, headers)
|
||||
if r.status_code == 200 and not check_cf_blocked(str(r.content[:5000])):
|
||||
return (r.text, f"curl_cffi+proxy_{name}", time.time() - start)
|
||||
except:
|
||||
continue
|
||||
|
||||
return self.bypass(url, country)
|
||||
Reference in New Issue
Block a user