c0py: initial scaffolding for website registry tool
CI (SIAX Cloud) / security (push) Successful in 12s
CI (SIAX Cloud) / contracts (push) Failing after 12s
CI (SIAX Cloud) / quality (push) Failing after 8s
Deploy to Coolify / deploy (push) Failing after 2s

This commit is contained in:
2026-09-16 13:15:21 +02:00
commit e6cc719f37
58 changed files with 1131 additions and 0 deletions
+13
View File
@@ -0,0 +1,13 @@
PORT=3000
NODE_ENV=development
CL0UD_BASE_URL=https://cl0ud.siax.io
ZITADEL_ISSUER=https://id-customers.siax.io
ZITADEL_AUDIENCE=api.c0py.siax.io
DATABASE_URL=postgresql://c0py:c0py@localhost:5432/c0py
AUD0_BASE_URL=
AUD0_API_KEY=
ST0RE_BASE_URL=
INF0_BASE_URL=
N0D_BASE_URL=
CRAWL_TIMEOUT_MS=30000
MAX_REGISTRY_PAGES=1000
+34
View File
@@ -0,0 +1,34 @@
# C0PY — Environment Variables
# All secrets via Infisical (vault.siax.io). Never commit real values.
# Server
PORT=3000
NODE_ENV=development
# CL0UD (Auth/RBAC/Tenant)
CL0UD_BASE_URL=https://cl0ud.siax.io
CL0UD_AUDIENCE=siax
# Zitadel
ZITADEL_ISSUER=https://id-customers.siax.io
ZITADEL_AUDIENCE=api.c0py.siax.io
# DATABASE
DATABASE_URL=postgresql://c0py:c0py@localhost:5432/c0py
# AUD0 (Audit/Evidence) — not yet live, fail-closed
AUD0_BASE_URL=
AUD0_API_KEY=
# ST0RE (Storage) — not yet live
ST0RE_BASE_URL=
# INF0 (AI analysis) — optional, for content analysis
INF0_BASE_URL=
# N0D (Async crawling jobs) — optional
N0D_BASE_URL=
# C0PY internal
CRAWL_TIMEOUT_MS=30000
MAX_REGISTRY_PAGES=1000
+43
View File
@@ -0,0 +1,43 @@
name: CI (SIAX Cloud)
on: [push, pull_request, workflow_dispatch]
jobs:
quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
- uses: pnpm/action-setup@v4
with:
version: 9.15.9
- name: Install
run: pnpm install --frozen-lockfile
- name: Typecheck
run: pnpm run typecheck
- name: Test
run: pnpm run test
- name: Build
run: pnpm run build
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Trivy
run: curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
- name: Scan
run: trivy fs --scanners vuln,secret --severity HIGH,CRITICAL --exit-code 0 .
contracts:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
- uses: pnpm/action-setup@v4
with:
version: 9.15.9
- name: Install
run: pnpm install --frozen-lockfile
- name: Check schema/contracts
run: pnpm --filter @siax/c0py-core test
+15
View File
@@ -0,0 +1,15 @@
name: Deploy to Coolify
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Trigger Coolify redeploy
env:
COOLIFY_TOKEN: ${{ secrets.COOLIFY_TOKEN }}
run: |
if [ -z "$COOLIFY_TOKEN" ]; then echo "::error::COOLIFY_TOKEN is not set"; exit 1; fi
curl -fsS -X POST "https://cloud.siax.io/api/v1/deploy?uuid=c0py&force=false" -H "Authorization: Bearer $COOLIFY_TOKEN" -o /dev/null -w 'coolify deploy HTTP %{http_code}\n'
+10
View File
@@ -0,0 +1,10 @@
name: Masterplan Lock
on:
schedule:
- cron: "0 4 * * *"
jobs:
lock:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: echo "masterplan lock placeholder"
+13
View File
@@ -0,0 +1,13 @@
node_modules/
dist/
build/
*.log
.env
.env.local
.env.*.local
coverage/
.pnpm-store/
**/node_modules
pnpm-lock.yaml
.DS_Store
tmp/
+1
View File
@@ -0,0 +1 @@
{"signers": [], "committers": [], "contracts": []}
+25
View File
@@ -0,0 +1,25 @@
# C0PY — AGENTS.md
## Repo-konventioner
- **Package manager**: pnpm (frozen-lockfile)
- **Framework**: Next.js (apps/web), Fastify (apps/api)
- **TypeScript**: strict mode, bundler module resolution
- **CI**: Gitea Actions — `.gitea/workflows/ci.yml`
- **Deploy**: Coolify via `.gitea/workflows/deploy.yml`
- **Secrets**: Infisical (`vault.siax.io`) — ALDRIG i repo
- **Testing**: vitest (apps/api, packages/*)
- **Lint**: eslint flat config
- **OTel**: bootstrap vid entry (services/entry.ts), ej FATAL — fail-closed via config
- **Tenant**: bound to session via CL0UD, NEVER from client input
## Kontrakt
- API-kontrakt: `contracts/openapi.yaml`
- Schema-versionshantering: semantisk versionering i `packages/types`
## Fällor
- `NEXT_PUBLIC_*` ska ALDRI innehålla hemligheter
- Alla writes till AUD0/ST0RE via klientskikt, aldrig direkt
- Crawling ska vara throttle-ad och respect robots.txt
+4
View File
@@ -0,0 +1,4 @@
Contributors to c0py follow the Developer Certificate of Origin (DCO).
See https://developercertificate.org/
By contributing to this project, you agree to the DCO.
+57
View File
@@ -0,0 +1,57 @@
# C0PY — Deployment
## SIAX Cloud Deploy
- **Config**: [`siax.config.json`](siax.config.json)
- **CI/CD**: `.gitea/workflows/ci.yml` — körs vid push/PR till main (Gitea Actions, git.siax.io)
- **Runtime**: Node 22, port 3000
- **Domain**: `c0py.siax.io`
## Deploy via Coolify
- **Coolify dashboard**: `cloud.siax.io`
- **Resource type**: Application (git source)
- **Git URL**: `git@github.com:sax3l/c0py.git`
- **Build pack**: Dockerfile
- **Domain**: `c0py.siax.io`
## Docker
```bash
docker build -t siax/c0py .
docker run -d -p 3000:3000 \
-e PORT=3000 \
-e NODE_ENV=production \
-e DATABASE_URL=postgresql://... \
-e CL0UD_BASE_URL=https://cl0ud.siax.io \
-e ZITADEL_ISSUER=https://id-customers.siax.io \
-e ZITADEL_AUDIENCE=api.c0py.siax.io \
-e AUD0_BASE_URL=https://aud0.siax.io \
-e AUD0_API_KEY=... \
-e ST0RE_BASE_URL=https://st0re.siax.io \
siax/c0py
```
## Miljövariabler
| Variable | Krävs | Default | Beskrivning |
|---|---|---|---|
| `PORT` | Nej | `3000` | HTTP-serverport |
| `NODE_ENV` | Nej | `development` | Läge |
| `CL0UD_BASE_URL` | Ja | — | CL0UD kontrollplan URL |
| `ZITADEL_ISSUER` | Ja | — | OIDC-utfärdare |
| `ZITADEL_AUDIENCE` | Ja | — | API-audience |
| `DATABASE_URL` | Ja | — | Postgres-anslutning |
| `AUD0_BASE_URL` | Nej | — | AUD0 evidens-URL |
| `AUD0_API_KEY` | Nej | — | AUD0 autentisering |
| `ST0RE_BASE_URL` | Nej | — | ST0RE lagrings-URL |
| `INF0_BASE_URL` | Nej | — | INF0 AI-URL |
| `N0D_BASE_URL` | Nej | — | N0D beräknings-URL |
## Hälsocheck
```
GET /health
```
Förväntad respons: `200 OK`
+39
View File
@@ -0,0 +1,39 @@
# C0PY API Server
FROM node:22-alpine AS base
WORKDIR /app
RUN corepack enable && corepack prepare pnpm@9.15.9 --activate
COPY package.json pnpm-workspace.yaml pnpm-lock.yaml ./
COPY packages ./packages
COPY apps ./apps
RUN pnpm install --frozen-lockfile
FROM node:22-alpine AS runtime
WORKDIR /app
RUN apk add --no-cache wget
COPY --from=base /app/node_modules ./node_modules
COPY --from=base /app/apps/api/package.json ./apps/api/package.json
COPY --from=base /app/apps/api/src ./apps/api/src
COPY --from=base /app/packages/c0py-core/package.json ./packages/c0py-core/package.json
COPY --from=base /app/packages/c0py-core/src ./packages/c0py-core/src
COPY --from=base /app/packages/config/package.json ./packages/config/package.json
COPY --from=base /app/packages/config/src ./packages/config/src
COPY --from=base /app/packages/types/package.json ./packages/types/package.json
COPY --from=base /app/packages/types/src ./packages/types/src
ENV NODE_ENV=production
ENV PORT=3000
ENV HOST=0.0.0.0
USER node
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -qO- http://127.0.0.1:3000/health >/dev/null 2>&1 || exit 1
CMD ["node", "--import", "tsx/esm", "apps/api/src/index.ts"]
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 SIAX Technology AB
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+37
View File
@@ -0,0 +1,37 @@
# C0PY
**Verktyg för att skapa fullständigt register över en hemsida ut alla aspekter.**
C0PY konsumerar en webbadress och producerar ett komplett register: sida-för-sida innehåll, struktur, prestanda, säkerhet, SEO, metadata och mer. Allt lagras som en tillförlitlig, återläsbar sammanställning.
## Snabbkommandon
```bash
pnpm install
pnpm --filter @siax/c0py-api dev
pnpm --filter @siax/c0py-web dev
```
## Arkitektur
```
apps/api/ → Fastify/Next.js API — /v1/c0py/* endpoints
apps/web/ → Next.js UI — registerhantering, dashboard, visualisering
packages/c0py-core → crawling, registry-uppbyggnad, analysmotor
packages/config → Zod-validerad env-laddare
packages/types → delade TypeScript-typer
```
## Konsumerade kapabiliteter
| Kapabilitet | Syfte | Status |
|---|---|---|
| CL0UD | Autentisering, RBAC, tenant | live |
| AUD0 | Regeleveranshändelser | (ej live) |
| ST0RE | Objektlagring av registerdata | (ej live) |
| INF0 | Innehållsanalys | (ej live) |
| N0D | Asynkrone crawling-jobb | (ej live) |
## Deploy
SIAX Cloud via Coolify. Se [`DEPLOY.md`](DEPLOY.md).
+4
View File
@@ -0,0 +1,4 @@
{
"version": "1.0",
"description": "C0PY security contract — deny-by-default, fail-closed, tenant-bound"
}
+26
View File
@@ -0,0 +1,26 @@
{
"name": "@siax/c0py-api",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"dev": "tsx watch src/index.ts",
"start": "NODE_ENV=production tsx src/index.ts",
"build": "tsc",
"typecheck": "tsc --noEmit",
"test": "vitest run"
},
"dependencies": {
"@siax/c0py-core": "workspace:*",
"@siax/c0py-config": "workspace:*",
"@siax/c0py-types": "workspace:*",
"fastify": "^5.12.0",
"zod": "^3.24.0"
},
"devDependencies": {
"@types/node": "^24.0.0",
"tsx": "^4.21.0",
"typescript": "^5.7.2",
"vitest": "^3.0.0"
}
}
+4
View File
@@ -0,0 +1,4 @@
import { envSchema } from "@siax/c0py-config";
export type Env = typeof envSchema;
export { envSchema };
+53
View File
@@ -0,0 +1,53 @@
import Fastify from "fastify";
import { envSchema } from "@siax/c0py-config";
async function main() {
const env = envSchema.parse(process.env);
const server = Fastify({ logger: true });
server.get("/health", async () => ({
status: "ok",
timestamp: new Date().toISOString(),
}));
server.get("/", async () => ({
name: "c0py",
version: "0.1.0",
status: "running",
}));
// Registry endpoints
server.get("/v1/c0py/registries", async () => {
return { registries: [] };
});
server.post("/v1/c0py/registries", async (request) => {
const body = (request.body as Record<string, unknown>) ?? {};
return { created: body };
});
server.get("/v1/c0py/registries/:id", async (request) => {
const { id } = request.params as Record<string, string>;
return { id };
});
// Scan endpoints
server.post("/v1/c0py/scans", async (request) => {
const body = (request.body as Record<string, unknown>) ?? {};
return { scanId: "scan-" + Date.now(), created: body };
});
server.get("/v1/c0py/scans/:id", async (request) => {
const { id } = request.params as Record<string, string>;
return { id };
});
await server.listen({ port: env.PORT, host: "0.0.0.0" });
console.log(`c0py API listening on port ${env.PORT}`);
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
+1
View File
@@ -0,0 +1 @@
testTimeout: 10000,
+24
View File
@@ -0,0 +1,24 @@
# C0PY API -- tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"baseUrl": ".",
"paths": {
"@siax/c0py-config": ["../packages/config/src/index.ts"],
"@siax/c0py-core": ["../packages/c0py-core/src/index.ts"],
"@siax/c0py-types": ["../packages/types/src/index.ts"]
}
},
"include": ["src/**/*"],
"exclude": ["node_modules"]
}
+1
View File
@@ -0,0 +1 @@
{"type": "vitest"}
+5
View File
@@ -0,0 +1,5 @@
next.config.js
node_modules
.next
out
dist
+8
View File
@@ -0,0 +1,8 @@
import type { NextConfig } from "next";
const nextConfig: NextConfig = {
output: "standalone",
images: { unoptimized: true },
};
export default nextConfig;
+25
View File
@@ -0,0 +1,25 @@
{
"name": "@siax/c0py-web",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"typecheck": "tsc --noEmit",
"test": "vitest run"
},
"dependencies": {
"next": "^15.0.0",
"react": "^19.0.0",
"react-dom": "^19.0.0"
},
"devDependencies": {
"@types/node": "^24.0.0",
"@types/react": "^19.0.0",
"@types/react-dom": "^19.0.0",
"typescript": "^5.7.2",
"vitest": "^3.0.0"
}
}
+4
View File
@@ -0,0 +1,4 @@
// Placeholder - Next.js app router entry
export default function Home() {
return <div>c0py</div>;
}
+5
View File
@@ -0,0 +1,5 @@
/// <reference types="next" />
/// <reference types="next/image-types/global" />
// NEXT_PUBLIC_* prefixed env vars are inlined at build time
// No secret values should use NEXT_PUBLIC_ prefix
+1
View File
@@ -0,0 +1 @@
{"compilerOptions": {"target": "ES2022", "module": "ESNext", "moduleResolution": "bundler", "jsx": "preserve", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "forceConsistentCasingInFileNames": true, "resolveJsonModule": true, "isolatedModules": true, "noEmit": true, "baseUrl": ".", "paths": {"@siax/c0py-core": ["../../packages/c0py-core/src/index.ts"], "@siax/c0py-config": ["../../packages/config/src/index.ts"], "@siax/c0py-types": ["../../packages/types/src/index.ts"]}}, "include": ["."], "exclude": ["node_modules", ".next"]}
+56
View File
@@ -0,0 +1,56 @@
# C0PY — Plattformsinfrastruktur
Det här dokumentet beskriver hur c0py ska sättas upp på SIAX Cloud.
## Steg 1: Skapa repo (Gitea)
```bash
# Repot skapas på git.siax.io som sax3l/c0py
# Pusha sedan lokalt repo
git push git@100.121.34.99:2222/sax3l/c0py.git main
```
## Steg 2: Coolify app
| Inställning | Värde |
|---|---|
| Resource type | Application |
| Git source | `git@100.121.34.99:2222/sax3l/c0py.git` |
| Build pack | Dockerfile |
| Domain | `c0py.siax.io` |
| Port | 3000 |
| Healthcheck path | `/health` |
## Steg 3: Postgres
Skapa dedikerad Postgres i Coolify för c0py (ej shared). Databasnamn: `c0py`.
## Steg 4: Infisical secrets
Projekt: `c0py` (eller nytt under `siax-platform`)
| Secret | Värde |
|---|---|
| `DATABASE_URL` | Postgres connection string |
| `CL0UD_BASE_URL` | `https://cl0ud.siax.io` |
| `ZITADEL_ISSUER` | `https://id-customers.siax.io` |
| `ZITADEL_AUDIENCE` | `api.c0py.siax.io` |
| `PORT` | `3000` |
## Steg 5: DNS
`c0py.siax.io` → A-record → server med Coolify appen (Cloudflare + PowerDNS dual-write).
## Steg 6: Zitadel projekt
Skapa OIDC-app under projekt `c0py` (eller motsvarande). Client credentials flow för API, PKCE för web.
## Steg 7: Deploy verification
1. `pnpm install --frozen-lockfile` (eller motsvarande i CI)
2. `pnpm run typecheck`
3. `pnpm run test`
4. `pnpm run build`
5. Pusha main → CI → Deploy via `.gitea/workflows/deploy.yml`
6. Verifiera `https://c0py.siax.io/health` → 200 OK
7. Verifiera fail-closed: `https://c0py.siax.io/v1/c0py/registries` → 401 utan token
+76
View File
@@ -0,0 +1,76 @@
# C0PY Architecture
## Översikt
C0PY är ett verktyg för att skapa fullständigt register över en hemsida ut alla aspekter. Det konsumerar en webbadress och producerar ett strukturerat register med information om sidor, struktur, innehåll, prestanda och mer.
## Komponenter
```
┌──────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ apps/web │────→│ apps/api │────→│ packages/ │
│ (Next.js) │◄────│ (Fastify/API) │◄────│ c0py-core │
└──────────────┘ └──────────────────┘ └─────────────────┘
┌───────┴───────┐
│ Integrations │
│ (CL0UD,ST0RE│
│ AUD0,INF0,N0D│
└───────────────┘
```
## Ytor
| Yta | Beskrivning |
|---|---|
| `apps/api/src/routes/` | REST-rutter under /v1/c0py/* |
| `apps/api/src/services/` | Tjocka tjänster, rent TS |
| `apps/web/src/app/` | Next.js App Router — registerhantering |
| `apps/web/src/components/` | Delade UI-komponenter |
| `packages/c0py-core/` | Crawling, registry-uppbyggnad, analys |
| `packages/config/` | Zod-env-validering |
| `packages/types/` | Gemensamma typer |
## Konsumerade kapabiliteter
- **CL0UD**: Autentisering (Zitadel OIDC), RBAC, tenant-isolering
- **AUD0**: Regeleveranshändelser (skrivs när tillgängligt, fail-closed annars)
- **ST0RE**: Objektlagring för registerdata
- **INF0**: AI-baserad innehållsanalys
- **N0D**: Asynkrone crawling-jobb för stora sajter
## Datamodell (initiell)
```
Registry
├── id: uuid
├── url: string (målhemsadress)
├── name: string
├── createdAt: timestamp
├── updatedAt: timestamp
├── Page[]
│ ├── id: uuid
│ ├── registryId: uuid
│ ├── url: string
│ ├── path: string
│ ├── title: string?
│ ├── description: string?
│ ├── status: number
│ ├── contentType: string?
│ ├── discoveredAt: timestamp
│ └── analysis: JSON?
└── ScanRun[]
├── id: uuid
├── registryId: uuid
├── status: pending|running|completed|failed
├── startedAt: timestamp
└── completedAt: timestamp?
```
## Säkerhet
- Tenant identifierad via CL0UD-session, ALDRIG från klientinput
- Alla writes fail-closed (inget vs fel config = nek)
- Inga hemligheter i repo
+14
View File
@@ -0,0 +1,14 @@
# SIAX toolchain-kontrakt v0.1.0 — Lefthook för c0py.
# Aktiveras med: npx lefthook install. Hook-skip ersätter aldrig CI-gates.
pre-commit:
parallel: true
commands:
check:
run: mise run check
pre-push:
parallel: false
commands:
verify:
run: mise run verify
+40
View File
@@ -0,0 +1,40 @@
# SIAX toolchain-kontrakt v0.1.0 (sax3l/siax-templates @ SHA-pinad i
# .siax/template-pin.json) — c0py-instans. Genererad ur toolchain-discovery
# (observerade data: packageManager + CI-enforced scripts, 2026-09-16).
# mise är task-ENTRY; gate-logiken bor i .gitea/workflows/ (repots egen CI).
[tools]
node = "22"
[tasks.setup]
description = "Installera beroenden (frozen lockfile)"
run = "set -eu; pnpm install --frozen-lockfile"
[tasks.check]
description = "Snabba deterministiska kontroller (spegler CI-enforced gates)"
run = '''
set -eu
pnpm run typecheck
'''
[tasks.verify]
description = "Full svit: test + build enligt repots egna scripts"
run = '''
set -eu
pnpm run test
pnpm run build
'''
[tasks.build]
description = "Bygg (repots build-script)"
run = "set -eu; pnpm run build"
[tasks.release]
description = "Verifiera innan release; deploy/release-vägen ägs av repots egna workflows"
run = '''
set -eu
pnpm run test
pnpm run build
echo "Release-vägen: .gitea/workflows/ äger deployen. Byt aldrig förbi approval." >&2
'''
+1
View File
@@ -0,0 +1 @@
{"name": "c0py", "version": "0.1.0", "private": true, "description": "C0PY — SIAX Cloud. Verktyg för att skapa fullständigt register över en hemsida ut alla aspekter.", "type": "module", "scripts": {"build": "pnpm -r build", "typecheck": "pnpm -r typecheck", "test": "pnpm -r test", "lint": "eslint .", "dev:api": "pnpm --filter @siax/c0py-api dev", "dev:web": "pnpm --filter @siax/c0py-web dev", "start:api": "pnpm --filter @siax/c0py-api start"}, "devDependencies": {"typescript": "^5.7.2", "@types/node": "^24.0.0", "@eslint/js": "^9.18.0", "eslint": "^9.18.0", "eslint-config-prettier": "^10.0.0", "prettier": "^3.4.2", "typescript-eslint": "^8.20.0"}, "engines": {"node": ">=20.17.0"}, "pnpm": {"overrides": {"nanoid@<5.1.16": "5.1.16"}}, "packageManager": "pnpm@9.15.9"}
+19
View File
@@ -0,0 +1,19 @@
{
"name": "@siax/c0py-core",
"version": "0.1.0",
"private": true,
"type": "module",
"main": "./src/index.ts",
"types": "./src/index.ts",
"scripts": {
"typecheck": "tsc --noEmit",
"test": "vitest run"
},
"dependencies": {
"@siax/c0py-types": "workspace:*"
},
"devDependencies": {
"typescript": "^5.7.2",
"vitest": "^3.0.0"
}
}
+94
View File
@@ -0,0 +1,94 @@
export class Crawler {
private maxPages: number;
private timeoutMs: number;
private visited: Set<string> = new Set();
private pages: CrawledPage[] = [];
private queue: string[] = [];
constructor(options: { maxPages: number; timeoutMs: number }) {
this.maxPages = options.maxPages;
this.timeoutMs = options.timeoutMs;
}
async crawl(startUrl: string): Promise<CrawledPage[]> {
this.visited.clear();
this.pages = [];
this.queue = [startUrl];
while (this.queue.length > 0 && this.pages.length < this.maxPages) {
const url = this.queue.shift()!;
if (this.visited.has(url)) continue;
this.visited.add(url);
try {
const page = await this.fetchPage(url);
this.pages.push(page);
for (const link of page.links) {
if (!this.visited.has(link)) {
this.queue.push(link);
}
}
} catch {
// Skip unreachable pages, continue crawl
}
}
return this.pages;
}
private async fetchPage(url: string): Promise<CrawledPage> {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), this.timeoutMs);
try {
const res = await fetch(url, {
signal: controller.signal,
headers: { "User-Agent": "c0py/0.1" },
redirect: "follow",
});
const body = await res.text();
const links = this.extractLinks(body, url);
return {
url,
path: new URL(url).pathname,
title: this.extractTag(body, "title"),
description: this.extractMetaDescription(body),
statusCode: res.status,
contentType: res.headers.get("content-type"),
links,
};
} finally {
clearTimeout(timeout);
}
}
private extractLinks(html: string, baseUrl: string): string[] {
const links: string[] = [];
const re = /href\s*=\s*["']([^"']+)["']/gi;
let m;
while ((m = re.exec(html)) !== null) {
try {
const abs = new URL(m[1], baseUrl).href;
if (!links.includes(abs)) links.push(abs);
} catch {
// Skip invalid URLs
}
}
return links;
}
private extractTag(html: string, tag: string): string | null {
const re = new RegExp(`<${tag}[^>]*>([\\s\\S]*?)<\\/${tag}>`, "i");
const m = html.match(re);
if (!m) return null;
return m[1].replace(/<[^>]+>/g, "").trim().slice(0, 500) || null;
}
private extractMetaDescription(html: string): string | null {
const re = /<meta[^>]*name=["']description["'][^>]*content=["']([^"']*)["']/i;
const m = html.match(re);
return m ? m[1].slice(0, 500) : null;
}
}
+12
View File
@@ -0,0 +1,12 @@
import type { Page, Registry, ScanRun } from "@siax/c0py-types";
export interface CrawlResult {
pages: Page[];
totalPages: number;
durationMs: number;
error: string | null;
}
export interface RegistryAnalyzer {
analyze(pages: Page[]): Record<string, unknown>;
}
+5
View File
@@ -0,0 +1,5 @@
export * from "./crawler/crawler";
export * from "./registry/registry";
export * from "./registry/analyzer";
export * from "./crawler/types";
export * from "./registry/types";
@@ -0,0 +1,39 @@
import type { Page } from "@siax/c0py-types";
export interface AnalysisResult {
totalPages: number;
uniqueDomains: number;
avgStatusCode: number;
statusCodes: Record<number, number>;
contentTypes: Record<string, number>;
hasBrokenLinks: boolean;
brokenLinks: number;
}
export function analyzePages(pages: Page[]): AnalysisResult {
const statusCodes: Record<number, number> = {};
const contentTypes: Record<string, number> = {};
let brokenLinks = 0;
for (const page of pages) {
statusCodes[page.statusCode] = (statusCodes[page.statusCode] ?? 0) + 1;
const ct = page.contentType?.split(";")[0].trim() ?? "unknown";
contentTypes[ct] = (contentTypes[ct] ?? 0) + 1;
if (page.statusCode >= 400) brokenLinks++;
}
const avgStatusCode =
pages.length > 0
? pages.reduce((sum, p) => sum + p.statusCode, 0) / pages.length
: 0;
return {
totalPages: pages.length,
uniqueDomains: new Set(pages.map((p) => new URL(p.url).host)).size,
avgStatusCode: Math.round(avgStatusCode * 100) / 100,
statusCodes,
contentTypes,
hasBrokenLinks: brokenLinks > 0,
brokenLinks,
};
}
@@ -0,0 +1,31 @@
import type { Page, Registry, ScanRun } from "@siax/c0py-types";
import type { CrawledPage } from "./types";
export function assembleRegistry(
url: string,
name: string,
crawled: CrawledPage[]
): { registry: Registry; pages: Page[] } {
const registry: Registry = {
id: crypto.randomUUID(),
url,
name,
createdAt: new Date(),
updatedAt: new Date(),
};
const pages: Page[] = crawled.map((c) => ({
id: crypto.randomUUID(),
registryId: registry.id,
url: c.url,
path: c.path,
title: c.title,
description: c.description,
statusCode: c.statusCode,
contentType: c.contentType,
discoveredAt: new Date(),
analysis: null,
}));
return { registry, pages };
}
+15
View File
@@ -0,0 +1,15 @@
export interface CrawlOptions {
url: string;
maxPages: number;
timeoutMs: number;
}
export interface CrawledPage {
url: string;
path: string;
title: string | null;
description: string | null;
statusCode: number;
contentType: string | null;
links: string[];
}
+65
View File
@@ -0,0 +1,65 @@
import { describe, it, expect } from "vitest";
import { analyzePages } from "../src/registry/analyzer";
import type { Page } from "@siax/c0py-types";
const pages: Page[] = [
{
id: "1",
registryId: "r1",
url: "https://example.com/",
path: "/",
title: "Home",
description: null,
statusCode: 200,
contentType: "text/html",
discoveredAt: new Date(),
analysis: null,
},
{
id: "2",
registryId: "r1",
url: "https://example.com/about",
path: "/about",
title: "About",
description: null,
statusCode: 200,
contentType: "text/html",
discoveredAt: new Date(),
analysis: null,
},
{
id: "3",
registryId: "r1",
url: "https://example.com/missing",
path: "/missing",
title: null,
description: null,
statusCode: 404,
contentType: "text/html",
discoveredAt: new Date(),
analysis: null,
},
];
describe("analyzePages", () => {
it("counts pages correctly", () => {
const result = analyzePages(pages);
expect(result.totalPages).toBe(3);
expect(result.brokenLinks).toBe(1);
expect(result.hasBrokenLinks).toBe(true);
expect(result.uniqueDomains).toBe(1);
});
it("aggregates status codes", () => {
const result = analyzePages(pages);
expect(result.statusCodes[200]).toBe(2);
expect(result.statusCodes[404]).toBe(1);
});
it("handles empty input", () => {
const result = analyzePages([]);
expect(result.totalPages).toBe(0);
expect(result.avgStatusCode).toBe(0);
expect(result.hasBrokenLinks).toBe(false);
});
});
+16
View File
@@ -0,0 +1,16 @@
import { describe, it, expect } from "vitest";
import { Crawler } from "../src/crawler/crawler";
describe("Crawler", () => {
it("instantiates with options", () => {
const c = new Crawler({ maxPages: 10, timeoutMs: 5000 });
expect(c).toBeDefined();
});
it("crawls a real page", async () => {
const c = new Crawler({ maxPages: 1, timeoutMs: 10000 });
const pages = await c.crawl("https://example.com");
expect(pages.length).toBeGreaterThanOrEqual(1);
expect(pages[0].statusCode).toBe(200);
}, 15000);
});
+24
View File
@@ -0,0 +1,24 @@
import { describe, it, expect } from "vitest";
import { assembleRegistry } from "../src/registry/registry";
describe("assembleRegistry", () => {
it("creates a registry with pages", () => {
const result = assembleRegistry("https://example.com", "Example", [
{
url: "https://example.com/",
path: "/",
title: "Home",
description: null,
statusCode: 200,
contentType: "text/html",
links: [],
},
]);
expect(result.registry.url).toBe("https://example.com");
expect(result.registry.name).toBe("Example");
expect(result.pages).toHaveLength(1);
expect(result.pages[0].id).toBeDefined();
expect(result.pages[0].registryId).toBe(result.registry.id);
});
});
+1
View File
@@ -0,0 +1 @@
testTimeout: 10000,
+22
View File
@@ -0,0 +1,22 @@
# C0PY Core Package -- tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"isolatedModules": true,
"declaration": true,
"outDir": "dist",
"baseUrl": ".",
"paths": {
"@siax/c0py-types": ["../types/src/index.ts"]
}
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
+1
View File
@@ -0,0 +1 @@
{"type": "vitest"}
+19
View File
@@ -0,0 +1,19 @@
{
"name": "@siax/c0py-config",
"version": "0.1.0",
"private": true,
"type": "module",
"main": "./src/index.ts",
"types": "./src/index.ts",
"scripts": {
"typecheck": "tsc --noEmit",
"test": "vitest run"
},
"dependencies": {
"zod": "^3.24.0"
},
"devDependencies": {
"typescript": "^5.7.2",
"vitest": "^3.0.0"
}
}
+4
View File
@@ -0,0 +1,4 @@
import { envSchema } from "@siax/c0py-config";
export type Env = typeof envSchema;
export { envSchema };
+19
View File
@@ -0,0 +1,19 @@
# C0PY Config Package -- tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"isolatedModules": true,
"declaration": true,
"outDir": "dist",
"baseUrl": "."
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
+1
View File
@@ -0,0 +1 @@
{"type": "vitest"}
+16
View File
@@ -0,0 +1,16 @@
{
"name": "@siax/c0py-types",
"version": "0.1.0",
"private": true,
"type": "module",
"main": "./src/index.ts",
"types": "./src/index.ts",
"scripts": {
"typecheck": "tsc --noEmit",
"test": "vitest run"
},
"devDependencies": {
"typescript": "^5.7.2",
"vitest": "^3.0.0"
}
}
+30
View File
@@ -0,0 +1,30 @@
export interface Page {
id: string;
registryId: string;
url: string;
path: string;
title: string | null;
description: string | null;
statusCode: number;
contentType: string | null;
discoveredAt: Date;
analysis: Record<string, unknown> | null;
}
export interface ScanRun {
id: string;
registryId: string;
url: string;
status: "pending" | "running" | "completed" | "failed";
startedAt: Date;
completedAt: Date | null;
error: string | null;
}
export interface Registry {
id: string;
url: string;
name: string;
createdAt: Date;
updatedAt: Date;
}
+19
View File
@@ -0,0 +1,19 @@
# C0PY Types Package -- tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"isolatedModules": true,
"declaration": true,
"outDir": "dist",
"baseUrl": "."
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
+1
View File
@@ -0,0 +1 @@
{"type": "vitest"}
+3
View File
@@ -0,0 +1,3 @@
packages:
- 'apps/*'
- 'packages/*'
+1
View File
@@ -0,0 +1 @@
{"$schema": "https://github.com/renovatebot/renovate-config/blob/master/renovate-schema.json", "extends": ["config:recommended"], "packageRules": [{"matchPackagePatterns": ["^@siax/"], "rangeStrategy": "bump"}, {"matchPackagePatterns": ["typescript"], "rangeStrategy": "bump"}, {"matchPackagePatterns": ["node"], "rangeStrategy": "update"}], "vulnerabilityReport": {"enabled": true}, "schedule": ["before 3am on Saturday"]}
+11
View File
@@ -0,0 +1,11 @@
import { mkdirSync, writeFileSync } from "node:fs";
mkdirSync(".siax", { recursive: true });
writeFileSync(
".siax/template-pin.json",
JSON.stringify({
version: "siax.template-pin/v1",
pinned: {
"siax-templates": { "toolchain": "main", "pin": "sha256-placeholder" },
},
}, null, 2),
);
+1
View File
@@ -0,0 +1 @@
{"project": "c0py", "framework": "node", "buildCommand": "pnpm run build", "installCommand": "pnpm install --frozen-lockfile", "nodeVersion": "22", "runtime": "node_server", "port": 3000, "healthCheckPath": "/health", "startCommand": "pnpm run start:api", "environments": {"preview": {"domainPattern": "{branch}.preview.c0py.siax.io"}, "production": {"domain": "c0py.siax.io"}}, "env": {"DATABASE_URL": "@infisical:c0py/DATABASE_URL", "CL0UD_BASE_URL": "@infisical:c0py/CL0UD_BASE_URL", "ZITADEL_ISSUER": "@infisical:c0py/ZITADEL_ISSUER", "ZITADEL_AUDIENCE": "@infisical:c0py/ZITADEL_AUDIENCE", "AUD0_BASE_URL": "@infisical:c0py/AUD0_BASE_URL", "AUD0_API_KEY": "@infisical:c0py/AUD0_API_KEY", "ST0RE_BASE_URL": "@infisical:c0py/ST0RE_BASE_URL", "INF0_BASE_URL": "@infisical:c0py/INF0_BASE_URL", "N0D_BASE_URL": "@infisical:c0py/N0D_BASE_URL", "PORT": "3000"}, "resourceLimits": {"cpu": "2.0", "memory": "1G"}}
+1
View File
@@ -0,0 +1 @@
{"$schema": "https://git.siax.io/sax3l/siax-standard/raw/branch/main/schema/siax.repo.schema.json", "repo": "c0py", "category": "product", "subcategory": "website-registry", "appId": "C0PY", "appIdJustification": "Produkt: webbplatsregistreringsverktyg. Kategori product per siax-standard. ej plattform (K1).", "owner": "SIAX Technology AB", "codeowners": ["@sax3l"], "lifecycle": "active", "operationalState": "dev", "gitHost": "git.siax.io", "node": null, "capabilityPrefix": "c0py", "packageName": "@siax/c0py", "domains": [], "consumes": ["CL0UD", "AUD0", "ST0RE", "INF0", "N0D"], "consumedBy": [], "children": [{"path": "apps/api", "purpose": "API server — /v1/c0py/* endpoints (registry, crawl, analysis)", "lifecycle": "active"}, {"path": "apps/web", "purpose": "Next.js UI — dashboard, registry management, visualization", "lifecycle": "active"}, {"path": "packages/c0py-core", "purpose": "Core domain: crawling, registry assembly, analysis engine", "lifecycle": "active"}, {"path": "packages/config", "purpose": "Zod-validated env loader shared by all surfaces", "lifecycle": "active"}, {"path": "packages/types", "purpose": "Shared TypeScript types across apps and packages", "lifecycle": "active"}], "replacesSaaS": [], "contracts": {"openapi": "contracts/openapi.yaml"}, "standardVersion": "1.0"}
+1
View File
@@ -0,0 +1 @@
{"compilerOptions": {"target": "ES2022", "module": "ESNext", "moduleResolution": "bundler", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "forceConsistentCasingInFileNames": true, "resolveJsonModule": true, "isolatedModules": true, "declaration": true, "declarationMap": true, "sourceMap": true, "outDir": "dist", "rootDir": ".", "baseUrl": ".", "paths": {"@siax/c0py-core": ["packages/c0py-core/src/index.ts"], "@siax/c0py-types": ["packages/types/src/index.ts"], "@siax/c0py-config": ["packages/config/src/index.ts"]}}, "include": ["apps/**/*", "packages/**/*"], "exclude": ["node_modules", "dist"]}