Repoint the integration suite's codex probe after codex-cli 0.145.0 started sending encrypted reasoning content

.github/workflows/integration-suite.yml

@@ -42,6 +42,16 @@ permissions:
 jobs:
   integration-suite:
     runs-on: ubuntu-latest
+    strategy:
+      # Two legs. `stable` is the contract test: what users get, and a FAIL fails
+      # the job. `beta` probes each vendor's public pre-release ref for the CLIs
+      # that publish one, and is ADVISORY — run.sh exits 0 on it regardless,
+      # because "this will break on release" is not a reason to stop the world,
+      # and gating CI on third-party alphas trains everyone to ignore a red run.
+      # fail-fast: false so a stable failure never cancels the early-warning leg.
+      fail-fast: false
+      matrix:
+        channel: [stable, beta]
     # NOTE: this is the GitHub *Environment* name where the secrets live. It is
     # configured in repo settings and is deliberately NOT renamed alongside this
     # directory — changing it here without renaming it there loses access to
@@ -61,12 +71,30 @@ jobs:
         with:
           bun-version: latest
 
-      - name: Restore integration-suite state (version-gate + broke/recovered)
+      - name: Restore this leg's state (version-gate + broke/recovered)
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
+        with:
+          path: ${{ matrix.channel == 'stable' && 'integration-suite-state.json' || format('integration-suite-state-{0}.json', matrix.channel) }}
+          key: integration-suite-state-${{ matrix.channel }}-${{ github.run_id }}
+          # The channel-scoped prefix first; the unscoped one second so the stable
+          # leg still adopts the pre-matrix cache on the first run after this
+          # change, instead of losing its gate baseline and re-probing all 12.
+          restore-keys: |
+            integration-suite-state-${{ matrix.channel }}-
+            integration-suite-state-
+
+      # The beta leg needs the STABLE leg's state to tell "about to break" from
+      # "already broken" — a CLI that is red on both is the stable leg's alarm,
+      # not an early warning. Restored read-only; only the stable leg writes it.
+      - name: Restore stable leg's state (cross-leg comparison)
+        if: matrix.channel != 'stable'
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
         with:
           path: integration-suite-state.json
-          key: integration-suite-state-${{ github.run_id }}
-          restore-keys: integration-suite-state-
+          key: integration-suite-state-stable-${{ github.run_id }}
+          restore-keys: |
+            integration-suite-state-stable-
+            integration-suite-state-
 
       # TRANSITION (delete once an integration-suite-state-* cache exists — i.e.
       # after the first successful run on main). This harness was renamed from
@@ -76,6 +104,7 @@ jobs:
       # filename, so it lands as cli-integration-state.json and ci-entrypoint.sh
       # adopts it only when the new file is absent.
       - name: Restore legacy state (rename transition)
+        if: matrix.channel == 'stable'
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
         with:
           path: cli-integration-state.json
@@ -90,14 +119,24 @@ jobs:
           CANARY_LLM_MODEL: ${{ secrets.CANARY_LLM_MODEL }}
           CANARY_CLAUDE_MODEL: ${{ secrets.CANARY_CLAUDE_MODEL }}
           CANARY_PI_MODEL: ${{ secrets.CANARY_PI_MODEL }}
+          # Optional: unset falls back to ci-entrypoint.sh's default. Present so a
+          # model that starts refusing codex's payload can be repointed from repo
+          # settings without a code change (see probe-cli.sh's CANARY_CODEX_MODEL note).
+          CANARY_CODEX_MODEL: ${{ secrets.CANARY_CODEX_MODEL }}
           COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}
           # OAuth credential trees (base64 gzip-tars rooted at $HOME)
           CURSOR_TOKEN_TGZ_B64: ${{ secrets.CURSOR_TOKEN_TGZ_B64 }}
           DEVIN_TOKEN_TGZ_B64: ${{ secrets.DEVIN_TOKEN_TGZ_B64 }}
           ANTIGRAVITY_TOKEN_TGZ_B64: ${{ secrets.ANTIGRAVITY_TOKEN_TGZ_B64 }}
           # reporting + version-gating
           CANARY_SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
-          CANARY_STATE: ${{ github.workspace }}/integration-suite-state.json
+          CANARY_CHANNEL: ${{ matrix.channel }}
+          CANARY_STATE: ${{ github.workspace }}/${{ matrix.channel == 'stable' && 'integration-suite-state.json' || format('integration-suite-state-{0}.json', matrix.channel) }}
+          # Operands are FLIPPED deliberately. GHA has no ternary — `cond && a || b`
+          # is the idiom, but an empty `a` is falsy, so `channel == 'stable' && ''`
+          # short-circuits to the fallback and the stable leg would get a peer path
+          # pointing at its own state. Same trap as CANARY_VERSION_GATED above.
+          CANARY_PEER_STATE: ${{ matrix.channel != 'stable' && format('{0}/integration-suite-state.json', github.workspace) || '' }}
           CANARY_FP_SHA: ${{ github.sha }}
           # force=true → "none" (non-"all" ⇒ run.sh gates nothing ⇒ probe all). A
           # literal '' can't be used: GHA `x && '' || 'all'` is always 'all'.
@@ -107,9 +146,12 @@ jobs:
           CANARY_CLIS: ${{ inputs.clis }}
         run: bash integration-suite/ci-entrypoint.sh
 
+      # Each leg saves ONLY its own file under a channel-scoped key: the legs run
+      # concurrently, so sharing a key would race, and a beta result must never
+      # be able to overwrite the stable leg's gating record.
       - name: Save integration-suite state
         if: always()
         uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
         with:
-          path: integration-suite-state.json
-          key: integration-suite-state-${{ github.run_id }}
+          path: ${{ matrix.channel == 'stable' && 'integration-suite-state.json' || format('integration-suite-state-{0}.json', matrix.channel) }}
+          key: integration-suite-state-${{ matrix.channel }}-${{ github.run_id }}

CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## 0.0.15-beta.0 — 2026-07-23
+
+### Dependencies
+- Bump `next` to 16.2.11 and pin `sharp` to 0.35.0, clearing the ten advisories that turned the Supply Chain gate red on every open PR. Nine are Next.js (four High, CVSS 8.2-8.3: middleware/proxy bypass in App Router, SSRF in Server Actions, and others) fixed in 16.2.11; the tenth is `sharp`'s inherited libvips CVEs (GHSA-f88m-g3jw-g9cj, CVSS 7.0) fixed in 0.35.0. Same shape as the `brace-expansion` incident three days ago: the advisories published on 2026-07-21 and 2026-07-22, *after* `main`'s last green scan at 2026-07-21 14:07 UTC, so every branch went red at once with no dependency change of its own — re-running CI's scanner image against `main`'s unchanged lockfile reproduced the failure identically. The two halves need different tools. `next` is a direct devDependency whose `^16.2.9` range already admitted the fix, so `bun update next` lifts it (and raises the declared floor to `^16.2.11`, so it cannot resolve back). `sharp` cannot be fixed that way: it is an *optional* dependency of `next`, and `next@16.2.11` still declares `sharp: ^0.34.5` — a range that excludes 0.35.0 — so updating `next` leaves the advisory in place and the pin has to come from `overrides`, the same plain-key mechanism already holding `postcss`, `vite`, `undici` and `brace-expansion`. Forcing a *minor* bump of a native library under a dependent asking for `^0.34.5` is the risk here, so it was verified beyond a green lockfile: `sharp` 0.35.0 loads against libvips 8.18.3 and round-trips an encode (`@img/sharp-libvips-linux-x64` moves 1.2.4 → 1.3.0). Confirmed with CI's own scanner image (`ghcr.io/google/osv-scanner-action:v2.3.8`) against the updated lockfile: `No issues found`, exit 0, with `osv-scanner.toml` keeping its zero ignored vulnerabilities. (#591)
+
+### Features
+- Probe each agent CLI's public pre-release build alongside the one users get, so the integration suite reports not only "is enforcement broken?" but "is it about to break, and how long do we have?". The workflow becomes a matrix over `CANARY_CHANNEL`: the `stable` leg is unchanged (all 12 CLIs, a `FAIL` still fails the job) and a new `beta` leg installs each vendor's pre-release ref. The motivating measurement: codex shipped the payload change that broke the 2026-07-22 run in `0.145.0-alpha.4` on 2026-07-11 but did not release it until 2026-07-21 — **10.7 days** of warning, established by installing each alpha and capturing its request body against a mock endpoint. Six CLIs publish something usable: codex `alpha`, copilot `prerelease`, openclaw `beta`, cursor `?channel=lab` (server-validated; `beta`/`canary` are rejected with HTTP 400), goose `CANARY=true` (a *rolling tag* marked `prerelease: false`, so channel detection scanning for `prerelease: true` misses it), and claude — which runs **backwards**: nothing ships ahead of `latest`, so the stable leg now pins `bash -s stable` (~13 days behind, and what conservative users actually run) and `latest` becomes the early-warning ref. That pin also stops a same-day Anthropic release, which lands ~daily, from red-lighting a PR that touched nothing. The other six are deliberately skipped rather than re-installed at stable and reported as coverage: factory, antigravity and pi publish no pre-release at all (verified — antigravity has 19 releases with zero prereleases and its updater serves the identical stable manifest for every `?channel=` value); opencode's `beta`/`dev` tags are ~31 branch snapshots a day rather than release candidates; devin's unpromoted builds exist but cannot be listed, only guessed at; and hermes's installer git-clones `main`, which is also what every hermes user gets, so there is nothing ahead of us to probe. Coverage is stated in the report itself (`watching 6/12`) because a pre-release ref does not preview every release — codex only alphas minor bumps, so all six recent `0.144.x` patches shipped blind, and the copilot 1.0.70 silent-allow had just 0.6 days of lead. (#591)
+- Escalate incoming breakage on a cross-leg comparison rather than a beta failure. A CLI red on both legs is already broken and belongs to the stable leg's alarm; the early warning is `stable green + beta not-green`, held for two consecutive runs so an alpha that gets reverted before release doesn't burn attention. This distinction is load-bearing: a vendor payload change stops the model *before* it calls a tool, so it surfaces as `INCONCLUSIVE`/`ERROR` and never as `FAIL` — a FAIL-only rule would have watched the codex regression sail past for ten days in silence. The beta leg is advisory throughout: `run.sh` exits 0 on it regardless, each leg gets its own Docker volume (a pre-release install overwrites the stable binary in a shared `$HOME`) and its own Actions cache key (the legs run concurrently, and a beta result must never overwrite the stable leg's gating record). `__tests__/integration-suite/channel-refs.test.ts` asserts that `run.sh`'s beta CLI list and `install-clis.sh`'s beta refs agree, since a silent drift there would install stable binaries and report them as pre-release coverage — a false all-clear, the worst failure available to an early-warning system. (#591)
+
+### Fixes
+- Stop the workflow handing the stable leg a peer-state path pointing at its own state file, and stop the beta report overstating coverage on a targeted run (both caught in review by CodeRabbit). The first is the GitHub Actions ternary trap the workflow already documents thirty lines above for `CANARY_VERSION_GATED`: Actions has no ternary, `cond && a || b` is the idiom, and an empty `a` is *falsy* — so `matrix.channel == 'stable' && ''` short-circuits to the fallback and both legs received the same path. Latent rather than active (`report.js` only reads the peer state on the beta leg) but a landmine, and fixed by flipping the operands so the true branch is the non-empty one. The second: `run.sh` accepts a CLI subset, so `run.sh cursor` on the beta leg reported "the rest publish no pre-release ref" about CLIs that were merely not requested — precisely the overstated-coverage failure the line exists to prevent. The denominator is now the eligible count passed in from `run.sh`, rather than a hardcoded 12 or a third copy of the CLI list to drift against, so a targeted run reads `watching 2/6 CLIs that publish a pre-release ref (of 12 total)`. Both are covered by new assertions in `__tests__/integration-suite/channel-refs.test.ts`. (#591)
+- Repoint the integration suite's codex probe at `gpt-5.1-codex-mini`, restoring the daily enforcement signal for that CLI. The 2026-07-22 run reported `codex bash=INCONCLUSIVE read=INCONCLUSIVE` while the other eleven CLIs stayed green, and the cause was entirely vendor-side: codex-cli shipped 0.145.0 overnight (0.144.6 was green the day before), and for a model it has no metadata for — deepseek logs `Model metadata not found. Defaulting to fallback metadata` — it now sends `reasoning:{summary:"auto"}` and `include:["reasoning.encrypted_content"]` where 0.144.6 sent `reasoning:null` and `include:[]`. The gateway answers `400 "Encrypted content is not supported with this model"` (`param: include`), codex exits before its first tool call, and with no tool call there is no deny to observe, so both probes report INCONCLUSIVE. Enforcement was never broken: the hook log shows `SessionStart` and `UserPromptSubmit` firing in every failed run, and a silent-allow would have surfaced as FAIL, not INCONCLUSIVE. Confirmed by reproducing the CI verdict locally against the real gateway and by replaying the captured request body with and without those two fields — the same body minus `include`/`reasoning` returns 200 on deepseek. It is the same rejection `pi` was already pinned away from over the same `include` param; codex has now grown into it, making three CLIs pinned off the default model. No config override avoids it — `model_reasoning_summary="none"`, `model_supports_reasoning_summaries=false` and `model_reasoning_effort="none"` all still emit `include` — and the escape hatch is gone, since `wire_api = "chat"` is rejected outright by 0.145.0 (openai/codex#7782). `gpt-5.1-codex-mini` is the cheapest gateway model that accepts encrypted reasoning content *and* supports codex's full toolset; `gpt-5.4-nano` accepts the reasoning params but 400s on `tool_search`, which only an end-to-end run reveals. Verified end to end: `bash=PASS read=PASS` with `result=deny policy=custom/canary-bash` and `custom/canary-read` in the oracle. (#591)
+- Make `CANARY_CODEX_MODEL` actually reach the probe. `probe-cli.sh` has read the override since the harness was written, but nothing ever set it: only `CANARY_LLM_MODEL`, `CANARY_CLAUDE_MODEL` and `CANARY_PI_MODEL` were written into the container env-file, and the probe runs inside the container, so setting the variable in repo settings would have had no effect at all. `ci-entrypoint.sh` now forwards it and the workflow maps an optional secret, so the next model that starts refusing codex's payload can be swapped from repo settings without a code change. (#591)
+- Report a vendor payload rejection as ERROR rather than INCONCLUSIVE. `is_error()` classified quota and auth failures ("can't test right now") apart from a model that simply never called a tool, but a `400` / `invalid_request_error` / `not supported` response fell through to INCONCLUSIVE — so the codex regression above was posted as a quiet 🟡 "all enforcing where the model engaged" for a full day, when the accurate reading was ⚠️ "couldn't test codex". Payload rejections now land in the same bucket as quota errors. The ordering is unchanged and still fail-safe: a deny is checked first, then a leaked side-effect, so a genuine FAIL can never be reclassified. (#591)
+- Keep `is_error()`'s payload-rejection patterns machine-shaped, and lock them down with a test. The first cut matched a bare `400` and a bare `not supported`, but the function's input is the agent's *entire transcript* — so "I ran the suite and 400 tests passed" or "that flag is not supported" classified a chatty refusal as a vendor outage, inverting the very signal the change was meant to sharpen (caught in review by CodeRabbit). Now it matches the structured `"code": 400` form and the gateway's own `not supported with` phrasing; both live failures carry `invalid_request_error` and `BadRequestError` regardless, so nothing is lost. `__tests__/integration-suite/is-error.test.ts` runs the real function out of the shell script — not a copy that can drift — against seven vendor-failure fixtures and six ordinary-output fixtures, four of which the previous regex got wrong. (#591)
+- Spool probe output to a temp file instead of a shell variable (also from review). Capturing a full agent transcript from up to six CLI invocations into `$(...)` to read one verdict line and a 20-line tail meant a stuck or noisy client could grow the buffer without bound; the previous code streamed through a pipe and never held it. (#591)
+- Restore the note explaining why codex is not pinned to a Claude model, which the fix above had deleted along with the line it annotated. The gateway routes Anthropic weighted 1:1 through Bedrock, which 400s on codex's request metadata (#576) — so a Claude-pinned codex probe fails on roughly half its requests, and a coin-flip red in a daily canary is worse than a consistent one. Worth keeping precisely because a single green probe looks like proof that it works. (#591)
+- Echo the probe's output tail whenever a CLI's verdict is not a clean pass. `run.sh` discarded everything a probe printed except its `VERDICT_JSON` line, so a yellow or red run said *what* broke and never *why*, and re-running produced no more detail because the vendor's error message was thrown away both times — diagnosing the codex regression above needed a full local reproduction purely for want of these twenty lines. Safe in a public log: every credential involved is a registered Actions secret, so GitHub masks it on the way out. (#591)
+
 ## 0.0.14-beta.3 — 2026-07-20
 
 ### Fixes