feat(agent): telemetry + CI-gate quick wins from the "AI systems that ship" audit (#677)
* feat(agent): telemetry completeness + durability, CI gates, commit_method provenance Quick wins from the "Building AI systems that ship" audit: - mcp.tool_called gains errorMessage (message_sv, truncated 500 chars) on all failure exits; new mcp.skill_loaded event on every gnubok_load_skill (all tiers) so atom usage is finally measurable - event_log: (event_type, created_at) index; cleanup cron keeps mcp.*/agent.* telemetry 180 days (delivery events stay 30) - CI: lint ratchet (npm run check:lint — 60 legacy errors baselined, fails only on NEW errors) and a pg-real coverage gate (migrations touching trigger/RPC/RLS/DEFERRABLE require a *.pg.test.ts change; escape hatch: -- pg-test: covered-by/skip) - journal_entries.commit_method CHECK widened with 'api_key'/'agent'; the MCP approve path records 'api_key' truthfully instead of 'user_accept' (agent_first_vision §8 P0-1). 'agent' is reserved — ALL MCP traffic (incl. claude.ai OAuth, whose access_token is a minted API key) authenticates as api_key today Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(import): derive opening balances from prior-year #UB when SIE lacks #IB (#675) SIE files exported without #IB 0 rows (only #UB -1) previously imported with zero opening balances. getEffectiveOpeningBalances() now derives IB from prior-year UB for balance-sheet accounts when explicit #IB is absent, surfaces the derivation as an info issue in the import preview, and excludes share-capital vouchers from opening-balance detection. Detection regexes are shared between parser and importer so the two checks cannot drift. 507 lib/import tests pass. (Authored in a parallel session in this checkout; included per request.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): address PR #677 bot findings — RoPA entry, execFileSync, gate scope note Triage of the compliance-swarm + Greptile findings: Applied: - .compliance/ropa.yaml: new mcp.telemetry processing activity declaring the 180-day mcp.*/agent.* retention, lawful basis, data categories, and the no-args/no-results minimisation (ISO A.8.10, GDPR Art.5(1)(c) — the retention split is now formally documented, referenced from the cron) - check-pg-test-coverage.mjs: execFileSync with argv array — no shell, so a hostile base-ref can't inject (ASVS V13.2.1); verified an injection attempt exits 2 without executing - check-pg-test-coverage.mjs: documented the PR-level (not per-migration) scope of the gate so reviewers know to check coverage per migration when a PR carries several risky migrations (Greptile P2) Acknowledged, no change: - errorMessage PII risk: messages are domain-mapped strings; event_log already persists far richer delivery payloads under the same RLS; now declared in ropa.yaml - cron error envelope: errorResponse maps to the canonical safe envelope and the endpoint is CRON_SECRET-gated - two-pass delete "partial state": TTL deletes are idempotent — the next daily run sweeps whatever a failed pass left behind - skill_loaded actorLabel/sessionId: mirrors the pre-existing mcp.tool_called payload; sessionId is the join key the analytics exist for Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
076bb169f8
commit
bc61862e76
@@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* CI gate: a PR that adds or changes a migration touching a trigger,
|
||||
* function/RPC, RLS policy, or DEFERRABLE constraint must also add or extend
|
||||
* a *.pg.test.ts.
|
||||
*
|
||||
* This enforces the rule documented in .claude/rules/database.md ("pg-real
|
||||
* tests: any PR touching a trigger/RPC/RLS/DEFERRABLE must include or extend
|
||||
* a *.pg.test.ts") — previously instruction-only, which means it got skipped.
|
||||
*
|
||||
* Escape hatch: a migration may declare, in a SQL comment, either
|
||||
* -- pg-test: covered-by tests/pg/<file>.pg.test.ts
|
||||
* -- pg-test: skip (<reason>)
|
||||
* Both are visible in review and greppable later. Use them sparingly —
|
||||
* "covered-by" when an existing test already exercises the changed object,
|
||||
* "skip" when the change is genuinely untestable (e.g. a NOTIFY-only fixup).
|
||||
*
|
||||
* Scope: the gate is PR-level, not per-migration — ANY *.pg.test.ts change
|
||||
* satisfies it. With multiple risky migrations in one PR, reviewers must
|
||||
* still confirm each one is actually covered (or carries an escape hatch);
|
||||
* mapping tests to migrations automatically would be guesswork.
|
||||
*
|
||||
* Usage: node scripts/check-pg-test-coverage.mjs
|
||||
* PG_GATE_BASE — git ref to diff against (default: origin/main)
|
||||
*/
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { existsSync, readFileSync } from 'node:fs'
|
||||
|
||||
const base = process.env.PG_GATE_BASE || 'origin/main'
|
||||
|
||||
let changed
|
||||
try {
|
||||
// Three-dot diff: changes on the PR side since the merge-base with `base`.
|
||||
// --diff-filter=ACMR skips deletions (a deleted migration has no content to
|
||||
// scan). execFileSync with an argv array — no shell, so a hostile base-ref
|
||||
// string can't inject (git just rejects an invalid rev via the catch below).
|
||||
changed = execFileSync('git', ['diff', '--name-only', '--diff-filter=ACMR', `${base}...HEAD`], {
|
||||
encoding: 'utf8',
|
||||
})
|
||||
.trim()
|
||||
.split('\n')
|
||||
.filter(Boolean)
|
||||
} catch (err) {
|
||||
console.error(`check-pg-test-coverage: failed to diff against "${base}".`)
|
||||
console.error('Set PG_GATE_BASE to a fetched ref (CI: origin/${{ github.base_ref }}).')
|
||||
console.error(String(err))
|
||||
process.exit(2)
|
||||
}
|
||||
|
||||
const migrations = changed.filter(
|
||||
(f) => f.startsWith('supabase/migrations/') && f.endsWith('.sql'),
|
||||
)
|
||||
const pgTests = changed.filter((f) => f.endsWith('.pg.test.ts'))
|
||||
|
||||
// DDL that the database.md rule classifies as requiring real-Postgres coverage.
|
||||
const RISKY_DDL = [
|
||||
{ kind: 'trigger', re: /\bCREATE\s+(OR\s+REPLACE\s+)?(CONSTRAINT\s+)?TRIGGER\b/i },
|
||||
{ kind: 'function/RPC', re: /\bCREATE\s+(OR\s+REPLACE\s+)?FUNCTION\b/i },
|
||||
{ kind: 'RLS policy', re: /\b(CREATE|ALTER|DROP)\s+POLICY\b/i },
|
||||
{ kind: 'RLS enable/disable', re: /\b(ENABLE|DISABLE)\s+ROW\s+LEVEL\s+SECURITY\b/i },
|
||||
{ kind: 'DEFERRABLE constraint', re: /\bDEFERRABLE\b/i },
|
||||
]
|
||||
|
||||
const ESCAPE_HATCH = /^\s*--\s*pg-test:\s*(covered-by\s+\S+|skip\b.*)$/im
|
||||
|
||||
const flagged = []
|
||||
for (const file of migrations) {
|
||||
if (!existsSync(file)) continue
|
||||
const raw = readFileSync(file, 'utf8')
|
||||
if (ESCAPE_HATCH.test(raw)) continue
|
||||
// Strip SQL line comments so prose mentioning "CREATE POLICY" doesn't trip the gate.
|
||||
const sql = raw.replace(/--.*$/gm, '')
|
||||
const kinds = RISKY_DDL.filter(({ re }) => re.test(sql)).map(({ kind }) => kind)
|
||||
if (kinds.length > 0) flagged.push({ file, kinds })
|
||||
}
|
||||
|
||||
if (flagged.length === 0) {
|
||||
console.log(
|
||||
migrations.length === 0
|
||||
? 'check-pg-test-coverage: no migrations in this diff.'
|
||||
: `check-pg-test-coverage: ${migrations.length} migration(s) changed, none touch trigger/RPC/RLS/DEFERRABLE.`,
|
||||
)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
if (pgTests.length > 0) {
|
||||
console.log(
|
||||
`check-pg-test-coverage: ${flagged.length} risky migration(s) accompanied by pg-real test change(s):`,
|
||||
)
|
||||
for (const t of pgTests) console.log(` test: ${t}`)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
console.error('check-pg-test-coverage: FAILED\n')
|
||||
console.error(
|
||||
'These migrations touch trigger/RPC/RLS/DEFERRABLE but the PR adds or extends no *.pg.test.ts:\n',
|
||||
)
|
||||
for (const { file, kinds } of flagged) {
|
||||
console.error(` ${file} (${kinds.join(', ')})`)
|
||||
}
|
||||
console.error(`
|
||||
The repo rule (.claude/rules/database.md) requires real-Postgres coverage for
|
||||
these objects — mocked Supabase tests cannot exercise them.
|
||||
|
||||
Fix one of:
|
||||
1. Add or extend a *.pg.test.ts covering the changed trigger/RPC/policy
|
||||
(helpers: tests/pg/setup.ts, tests/pg/fixtures.ts; run: npm run test:pg)
|
||||
2. If an existing pg test already covers it, annotate the migration:
|
||||
-- pg-test: covered-by tests/pg/<file>.pg.test.ts
|
||||
3. If genuinely untestable, annotate with a reason:
|
||||
-- pg-test: skip (<reason>)
|
||||
`)
|
||||
process.exit(1)
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"totalErrors": 60,
|
||||
"perRule": {
|
||||
"@next/next/no-assign-module-variable": 1,
|
||||
"@typescript-eslint/no-explicit-any": 15,
|
||||
"prefer-const": 3,
|
||||
"react-hooks/preserve-manual-memoization": 6,
|
||||
"react-hooks/purity": 1,
|
||||
"react-hooks/set-state-in-effect": 28,
|
||||
"react-hooks/static-components": 6
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Ratchet guard for ESLint errors (sibling of no-new-antipatterns.mjs).
|
||||
*
|
||||
* `npm run lint` was never wired into CI, so ~60 pre-existing errors
|
||||
* accumulated across the repo. Fixing them all in one PR is churn; gating raw
|
||||
* `eslint` would break every PR until then. So: ratchet. Error counts are
|
||||
* tracked per rule in a committed baseline and can only go DOWN, never up —
|
||||
* a PR introducing a NEW error of any rule fails CI, while legacy errors are
|
||||
* burned down independently.
|
||||
*
|
||||
* Warnings stay advisory (only `--quiet` errors are counted).
|
||||
*
|
||||
* Known tradeoff: counts are per-rule repo-wide, not per-location — a PR that
|
||||
* fixes one legacy error of a rule can absorb one NEW error of the same rule
|
||||
* without tripping the gate. Acceptable for a burn-down ratchet; tighten to
|
||||
* per-file fingerprints if that ever bites.
|
||||
*
|
||||
* Usage:
|
||||
* node scripts/checks/no-new-lint-errors.mjs # check (CI)
|
||||
* node scripts/checks/no-new-lint-errors.mjs --update # re-baseline after fixing legacy errors
|
||||
*
|
||||
* Exit code 1 if any rule's error count exceeds its baseline.
|
||||
*/
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..')
|
||||
const BASELINE_PATH = path.join(ROOT, 'scripts', 'checks', 'eslint-baseline.json')
|
||||
|
||||
function runEslint() {
|
||||
const result = spawnSync(
|
||||
'npx',
|
||||
['eslint', '.', '--quiet', '-f', 'json'],
|
||||
{ cwd: ROOT, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 },
|
||||
)
|
||||
// ESLint exits 1 when errors exist — that's expected; only treat a missing/
|
||||
// unparsable report as fatal.
|
||||
if (!result.stdout) {
|
||||
console.error('no-new-lint-errors: eslint produced no JSON output')
|
||||
console.error(result.stderr ?? '')
|
||||
process.exit(2)
|
||||
}
|
||||
try {
|
||||
return JSON.parse(result.stdout)
|
||||
} catch {
|
||||
console.error('no-new-lint-errors: failed to parse eslint JSON output')
|
||||
process.exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
function collectCounts(report) {
|
||||
/** @type {Record<string, number>} */
|
||||
const perRule = {}
|
||||
/** @type {Record<string, string[]>} */
|
||||
const locations = {}
|
||||
for (const file of report) {
|
||||
for (const msg of file.messages) {
|
||||
if (msg.severity !== 2) continue
|
||||
const rule = msg.ruleId ?? 'fatal'
|
||||
perRule[rule] = (perRule[rule] ?? 0) + 1
|
||||
const rel = path.relative(ROOT, file.filePath)
|
||||
;(locations[rule] ??= []).push(`${rel}:${msg.line}:${msg.column}`)
|
||||
}
|
||||
}
|
||||
return { perRule, locations }
|
||||
}
|
||||
|
||||
const { perRule, locations } = collectCounts(runEslint())
|
||||
const total = Object.values(perRule).reduce((a, b) => a + b, 0)
|
||||
|
||||
if (process.argv.includes('--update')) {
|
||||
const sorted = Object.fromEntries(Object.entries(perRule).sort(([a], [b]) => a.localeCompare(b)))
|
||||
fs.writeFileSync(
|
||||
BASELINE_PATH,
|
||||
JSON.stringify({ totalErrors: total, perRule: sorted }, null, 2) + '\n',
|
||||
)
|
||||
console.log(`no-new-lint-errors: baseline updated — ${total} error(s) across ${Object.keys(perRule).length} rule(s).`)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
if (!fs.existsSync(BASELINE_PATH)) {
|
||||
console.error(`no-new-lint-errors: baseline missing at ${path.relative(ROOT, BASELINE_PATH)}.`)
|
||||
console.error('Run: node scripts/checks/no-new-lint-errors.mjs --update')
|
||||
process.exit(2)
|
||||
}
|
||||
|
||||
const baseline = JSON.parse(fs.readFileSync(BASELINE_PATH, 'utf8'))
|
||||
const baselineRules = baseline.perRule ?? {}
|
||||
|
||||
const regressions = []
|
||||
for (const [rule, count] of Object.entries(perRule)) {
|
||||
const allowed = baselineRules[rule] ?? 0
|
||||
if (count > allowed) regressions.push({ rule, count, allowed })
|
||||
}
|
||||
|
||||
if (regressions.length > 0) {
|
||||
console.error('no-new-lint-errors: FAILED — new ESLint errors beyond the baseline:\n')
|
||||
for (const { rule, count, allowed } of regressions) {
|
||||
console.error(` ${rule}: ${count} (baseline ${allowed})`)
|
||||
for (const loc of (locations[rule] ?? []).slice(0, 10)) {
|
||||
console.error(` ${loc}`)
|
||||
}
|
||||
}
|
||||
console.error(`
|
||||
Fix the new error(s) — run \`npx eslint . --quiet\` locally to see them.
|
||||
(If you fixed MORE legacy errors than you added and the rule still trips,
|
||||
re-baseline with: node scripts/checks/no-new-lint-errors.mjs --update)
|
||||
`)
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
const improved = total < (baseline.totalErrors ?? 0)
|
||||
console.log(
|
||||
`no-new-lint-errors: OK — ${total} error(s), baseline ${baseline.totalErrors}.` +
|
||||
(improved
|
||||
? ' Count went DOWN — ratchet it: node scripts/checks/no-new-lint-errors.mjs --update'
|
||||
: ''),
|
||||
)
|
||||
Reference in New Issue
Block a user