fix(ci): make the required security gate able to fail + repair the integration guard that could not pass #45

Merged

admin merged 2 commits from ci/make-required-security-gate-real-20260908 into main 2026-09-08 20:48:40 +00:00

Conversation 0 Commits 2 Files Changed 3

+91 -22

admin commented 2026-09-07 23:26:10 +00:00

Owner

Copy Link

Copy Source

Premise check first

PR #11 is not blocked — it was merged on 2026-08-11 (merged: true, merge commit 60145ee, merged by admin). Nothing is waiting on CI for it. 0pt-motor#11 is likewise merged; no other 0pt-family repo has a PR #11. This PR therefore does not "unblock #11"; it fixes what a look at that CI actually turned up.

What branch protection requires

main requires exactly two contexts:

CI (SIAX Cloud) / security (push)
CI (SIAX Cloud) / quality (push)

Both already reported green on all six open PRs. quality was real; security could not fail.

1. The required security context was a gate that cannot fail

All three Trivy invocations ran --exit-code 0, so the scanner printed findings and then exited 0 regardless. Measured on main@4514d97, Trivy 0.73.0, same target, only the flag differing:

run	flag	tree	exit
A	--exit-code 0	clean	0
B	--exit-code 0	planted AWS secret key	0 — reported CRITICAL, still green
C	--exit-code 1	same planted secret	1
D	--exit-code 1	probe removed	0

B is the defect: the required context reported success while holding a CRITICAL finding. D is why this is safe to land now — the tree is clean today, so flipping the flag does not open the gate red. (PR #19 measured 19 HIGH findings on 2026-08-11; those dependencies have since been updated on main, 142 commits later.)

Same flag fixed in security.yml and nightly.yml. A nightly scan exists to catch CVEs published against code that did not change — with --exit-code 0 it could never raise that alarm.

quality was verified real and left alone: a planted type error fails typecheck (exit 2), a planted assertion fails test (exit 1, 945-passing baseline).

2. The integration guard could not pass

It grepped vitest's coloured output:

grep -qE "skipped" && ! grep -qE "Tests  [0-9]+ passed"

The real line is ESC[2m Tests ESC[22m ESC[1mESC[32m97 passedESC[39m… — ANSI escapes sit between Tests and the digits, so that pattern can never match, while skipped still does. The guard fired on any run containing one skipped test.

Measured here, run 13911 job 31183: 13 files / 97 tests passed, 1 file / 2 skipped → FATAL: integration suites skipped instead of running. That message was false. It survived review because redirecting vitest to a file disables colour, so it is correct locally and misfires only in CI.

Replaced with the JSON reporter — structured, colourless, fail-closed. A/B on real reports, both directions:

all skipped (0 passed / 99 pending) -> exit 1   protection preserved
945 passed + 184 skipped            -> exit 0   false red removed
report file absent                  -> exit 1   fail closed

Confirmed in CI on be5e598: integration: 97 passed, 2 skipped, 0 failed, 99 total — same run shape the old guard called FATAL, now correctly green.

Result on be5e598

REQUIRED  success  CI (SIAX Cloud) / security (push)     <- now able to fail
REQUIRED  success  CI (SIAX Cloud) / quality (push)
advisory  success  CI (SIAX Cloud) / integration (push)  <- false red removed
advisory  success  Security Scan (SIAX Cloud) / security (push)

Job names are unchanged, so both required context strings are preserved exactly.

Not fixed here — needs an owner decision

services/scrapers/requirements.txt pins nothing (>= ranges only). Trivy's pip scanner cannot resolve unpinned specifiers, so it omits the file from the scan entirely — it never appears as a target. Pinning those same floors with == surfaces 3 HIGH findings today: aiohttp CVE-2025-69223, aiohttp CVE-2026-69244, lxml CVE-2026-41066.

Deliberately not suppressed with --skip-dirs, which would turn a measurable blind spot into declared policy. PR #19 proposes exactly that --skip-dirs services/scrapers; run D shows it is unnecessary for green and it would permanently hide this tree.

Runner flakiness (infrastructure, not this PR)

The first run of CI / security failed on siax-runner-5-s10 with curl exit 35 (SSL/connect) while installing Trivy — before scanning anything. The identical step succeeded the same minute on siax-runner-3-s6 and on rerun on siax-runner-9-s5. So the required gate's outcome depends on which runner picks it up: it could go red when the scanner failed to install, but never when the scanner found something — failing on noise, passing on signal. Worth pinning the Trivy binary + checksum (as PR #19 does) and investigating egress on siax-runner-5-s10.

DO NOT MERGE — deploy path + protected files

• cd.yml fires on push to main and POSTs to https://cloud.siax.io/api/v1/deploy. Both COOLIFY_APP_ID (repo variable) and COOLIFY_TOKEN (repo secret) are set, so its guard passes and a merge to main genuinely deploys. The in-file "no Coolify app provisioned yet" comment is stale.
• Branch protection lists .gitea/workflows/** under protected file patterns, so this PR cannot merge without that being lifted. Every file here is a workflow file.
• sax3l/0pt push-mirrors to github.com/sax3l/0pt with sync_on_commit.

Not merged, not self-approved. Requires Simon's authorised merge.

Generated with Claude Code (https://claude.com/claude-code)

## Premise check first **PR #11 is not blocked — it was merged on 2026-08-11** (`merged: true`, merge commit `60145ee`, merged by `admin`). Nothing is waiting on CI for it. `0pt-motor#11` is likewise merged; no other `0pt`-family repo has a PR #11. This PR therefore does not "unblock #11"; it fixes what a look at that CI actually turned up. ## What branch protection requires `main` requires exactly two contexts: ``` CI (SIAX Cloud) / security (push) CI (SIAX Cloud) / quality (push) ``` Both already reported green on all six open PRs. **`quality` was real; `security` could not fail.** ## 1. The required `security` context was a gate that cannot fail All three Trivy invocations ran `--exit-code 0`, so the scanner printed findings and then exited 0 regardless. Measured on `main@4514d97`, Trivy 0.73.0, same target, only the flag differing: | run | flag | tree | exit | |---|---|---|---| | A | `--exit-code 0` | clean | 0 | | B | `--exit-code 0` | planted AWS secret key | **0** — reported CRITICAL, still green | | C | `--exit-code 1` | same planted secret | **1** | | D | `--exit-code 1` | probe removed | 0 | **B is the defect**: the required context reported success while holding a CRITICAL finding. **D is why this is safe to land now** — the tree is clean today, so flipping the flag does not open the gate red. (PR #19 measured 19 HIGH findings on 2026-08-11; those dependencies have since been updated on `main`, 142 commits later.) Same flag fixed in `security.yml` and `nightly.yml`. A nightly scan exists to catch CVEs published against code that did not change — with `--exit-code 0` it could never raise that alarm. `quality` was verified real and left alone: a planted type error fails `typecheck` (exit 2), a planted assertion fails `test` (exit 1, 945-passing baseline). ## 2. The `integration` guard could not pass It grepped vitest's **coloured** output: ``` grep -qE "skipped" && ! grep -qE "Tests [0-9]+ passed" ``` The real line is `ESC[2m Tests ESC[22m ESC[1mESC[32m97 passedESC[39m…` — ANSI escapes sit between `Tests ` and the digits, so that pattern can never match, while `skipped` still does. The guard fired on **any** run containing one skipped test. Measured here, run 13911 job 31183: **13 files / 97 tests passed**, 1 file / 2 skipped → `FATAL: integration suites skipped instead of running`. That message was false. It survived review because redirecting vitest to a file disables colour, so it is correct locally and misfires only in CI. Replaced with the JSON reporter — structured, colourless, fail-closed. A/B on real reports, **both directions**: ``` all skipped (0 passed / 99 pending) -> exit 1 protection preserved 945 passed + 184 skipped -> exit 0 false red removed report file absent -> exit 1 fail closed ``` Confirmed in CI on `be5e598`: `integration: 97 passed, 2 skipped, 0 failed, 99 total` — same run shape the old guard called FATAL, now correctly green. ## Result on `be5e598` ``` REQUIRED success CI (SIAX Cloud) / security (push) <- now able to fail REQUIRED success CI (SIAX Cloud) / quality (push) advisory success CI (SIAX Cloud) / integration (push) <- false red removed advisory success Security Scan (SIAX Cloud) / security (push) ``` Job names are unchanged, so both required context strings are preserved exactly. ## Not fixed here — needs an owner decision `services/scrapers/requirements.txt` pins nothing (`>=` ranges only). Trivy's pip scanner cannot resolve unpinned specifiers, so **it omits the file from the scan entirely** — it never appears as a target. Pinning those same floors with `==` surfaces 3 HIGH findings today: `aiohttp` CVE-2025-69223, `aiohttp` CVE-2026-69244, `lxml` CVE-2026-41066. Deliberately **not** suppressed with `--skip-dirs`, which would turn a measurable blind spot into declared policy. PR #19 proposes exactly that `--skip-dirs services/scrapers`; run D shows it is unnecessary for green and it would permanently hide this tree. ## Runner flakiness (infrastructure, not this PR) The first run of `CI / security` failed on **`siax-runner-5-s10`** with `curl` exit 35 (SSL/connect) while *installing* Trivy — before scanning anything. The identical step succeeded the same minute on `siax-runner-3-s6` and on rerun on `siax-runner-9-s5`. So the required gate's outcome depends on which runner picks it up: it could go red when the scanner **failed to install**, but never when the scanner **found something** — failing on noise, passing on signal. Worth pinning the Trivy binary + checksum (as PR #19 does) and investigating egress on `siax-runner-5-s10`. ## DO NOT MERGE — deploy path + protected files - `cd.yml` fires on push to `main` and POSTs to `https://cloud.siax.io/api/v1/deploy`. **Both `COOLIFY_APP_ID` (repo variable) and `COOLIFY_TOKEN` (repo secret) are set**, so its guard passes and a merge to `main` genuinely deploys. The in-file "no Coolify app provisioned yet" comment is stale. - Branch protection lists `.gitea/workflows/**` under protected file patterns, so this PR cannot merge without that being lifted. Every file here is a workflow file. - `sax3l/0pt` push-mirrors to `github.com/sax3l/0pt` with `sync_on_commit`. Not merged, not self-approved. Requires Simon's authorised merge. Generated with Claude Code (https://claude.com/claude-code)

admin added 2 commits 2026-09-07 23:26:11 +00:00

fix(ci): make the required security gate capable of failing ... 1322d26fac

Branch protection on `main` requires exactly two status contexts:

    CI (SIAX Cloud) / security (push)
    CI (SIAX Cloud) / quality (push)

The `security` half could not fail. All three Trivy invocations in this
repo ran with `--exit-code 0`, so the scanner reported findings to stdout
and then exited 0 regardless.

Measured on main @4514d97, 2026-09-08, same scanner (Trivy 0.73.0), same
target, only the flag differing:

  A  --exit-code 0, clean tree                        -> exit 0
  B  --exit-code 0, planted AWS secret access key     -> exit 0  (DETECTED
     as CRITICAL in the report, gate still green)
  C  --exit-code 1, same planted secret               -> exit 1
  D  --exit-code 1, probe removed                     -> exit 0

B is the defect: the required context reported success while the scanner
had a CRITICAL secret in hand. D is why this is safe to land now -- the
tree is clean today, so flipping the flag does not open the gate red.
(PR #19 measured 19 HIGH findings on 2026-08-11; those dependencies have
since been updated on main, 142 commits later.)

The other required context, `quality`, was verified real and left alone:
a planted type error fails `pnpm run typecheck` (exit 2) and a planted
failing assertion fails `pnpm run test` (exit 1, 945 passed baseline).

nightly.yml carried the same flag. A nightly scan exists to catch CVEs
published against code that did not change; with `--exit-code 0` it could
never raise that alarm.

Not fixed here, needs an owner decision: services/scrapers/requirements.txt
uses `>=` ranges only. Trivy's pip scanner cannot resolve unpinned
specifiers, so it omits the file from the scan entirely -- it never appears
as a target. Pinning those same floors with `==` surfaces 3 HIGH findings
(aiohttp CVE-2025-69223, aiohttp CVE-2026-69244, lxml CVE-2026-41066).
Deliberately NOT suppressed with `--skip-dirs`, which would turn a
measurable blind spot into declared policy.

Job names are unchanged, so both required context strings are preserved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fix(ci): replace the integration anti-skip guard that could not pass ... be5e598646

The guard grepped vitest's HUMAN output:

    grep -qE "skipped" && ! grep -qE "Tests  [0-9]+ passed"

vitest colours that summary. The real line emitted on these runners is

    ESC[2m      Tests ESC[22m ESC[1mESC[32m97 passedESC[39m | ...

so ANSI escapes sit between "Tests " and the digits and the
two-spaces-then-digits pattern can never match. The first grep still
matched "skipped", so the condition was true whenever a single test
skipped -- no matter how many passed.

Measured on run 13911 job 31183 (this branch, 2026-09-08): 13 files /
97 tests PASSED, 1 file / 2 tests skipped, and the step still printed
"FATAL: integration suites skipped instead of running" and exited 1.
That message was false; the suites ran.

It survived review because it is correct locally -- redirecting vitest
to a file disables its colour, so the pattern matches on a developer
machine and misfires only under CI.

Replaced with vitest's JSON reporter: structured, colourless, and
fail-closed if the report is missing. A/B measured on real reports in
BOTH directions, so the replacement is not merely green:

    all skipped      (0 passed / 99 pending)  -> exit 1  (protection kept)
    945 passed + 184 skipped                  -> exit 0  (false red gone)
    report file absent                        -> exit 1  (fail closed)

The anti-skip guarantee the original guard was written for is preserved:
a run where every suite skips still fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CODEOWNERS rules requested review from sax3l 2026-09-07 23:26:11 +00:00

admin merged commit into main 2026-09-08 20:48:40 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

sax3l

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

siax-bot siax-ci siax-healer (siax-healer (CI self-healing agent)) admin (Simon Axelsson)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: admin/0pt#45