Repoint the integration suite's codex probe after codex-cli 0.145.0 started sending encrypted reasoning content#591
Conversation
…0 payload change
codex-cli 0.145.0 sends `reasoning:{summary:"auto"}` +
`include:["reasoning.encrypted_content"]` for a model it has no metadata
for, where 0.144.6 sent `reasoning:null` + `include:[]`. The gateway
answers 400 "Encrypted content is not supported with this model", codex
exits before its first tool call, and with no tool call there is no deny
to observe — so both probes reported INCONCLUSIVE on 2026-07-22 while the
other eleven CLIs stayed green. Enforcement was never broken: hooks fired
(SessionStart/UserPromptSubmit) in every failed run, and a silent-allow
would have surfaced as FAIL.
Pin codex to gpt-5.1-codex-mini — the cheapest gateway model that accepts
encrypted reasoning content and supports codex's full toolset
(gpt-5.4-nano accepts the reasoning params but 400s on `tool_search`).
No config override avoids the param, and `wire_api = "chat"` is rejected
outright by 0.145.0 (openai/codex#7782).
Also:
- forward CANARY_CODEX_MODEL into the container env-file; probe-cli.sh has
read the override since day one but nothing ever set it, so it was dead
plumbing and a repo-settings change could not have taken effect
- classify 400 / invalid_request_error / "not supported" as ERROR rather
than INCONCLUSIVE, so a payload rejection reads as "couldn't test" and
not "the model didn't try"
- echo the probe output tail on any non-PASS verdict; run.sh discarded
everything but VERDICT_JSON, so the CI log could not say why a CLI went
yellow and a re-run would not have said either
Verified end to end against the real gateway: bash=PASS read=PASS with
result=deny policy=custom/canary-bash and custom/canary-read in the oracle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughThe integration suite now runs concurrent stable and beta channel legs with isolated state, channel-specific CLI refs, cross-leg reporting, advisory beta behavior, expanded vendor-error detection, improved probe diagnostics, and configurable Codex model wiring. Documentation, tests, release notes, and dependency metadata are updated. ChangesIntegration suite channel execution
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Workflow
participant Installer
participant Runner
participant Reporter
Workflow->>Installer: Start stable or beta CLI installation
Installer->>Runner: Provide channel-specific CLI environment
Runner->>Reporter: Submit verdicts with peer state
Reporter->>Workflow: Emit report and persisted channel state
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Checkov (3.3.8).github/workflows/integration-suite.ymlTraceback (most recent call last): 🔧 ESLint
ESLint install timed out. The project may have too many dependencies for the sandbox. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Automated code review started - full review. Results will be posted here. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@integration-suite/probe-cli.sh`:
- Around line 181-186: Update is_error() so payload-error patterns such as 400,
“not supported,” and invalid_request_error only match structured gateway error
or status lines, not arbitrary model output. Preserve detection of the existing
authentication, quota, and gateway failures, and add a negative fixture covering
ordinary model text containing these phrases.
In `@integration-suite/run.sh`:
- Around line 75-89: Update the probe-output handling around run_probe so it
does not store the complete output in the out variable. Capture output in a
temporary file or otherwise bounded storage, extract the final VERDICT_JSON line
and print only the last 20 lines for failure cases, and ensure the temporary
resource is cleaned up after each probe.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3b630673-1ff9-47f5-8913-83c3c759cb92
📒 Files selected for processing (5)
.github/workflows/integration-suite.ymlCHANGELOG.mdintegration-suite/ci-entrypoint.shintegration-suite/probe-cli.shintegration-suite/run.sh
|
✅ Build & test complete. Results:
Continuing with deep analysis and review posting... |
…Supply Chain gate Ten advisories turned OSV-Scanner red on every open PR: nine against Next.js 16.2.10 (four High, CVSS 8.2-8.3 — App Router middleware/proxy bypass, SSRF in Server Actions) fixed in 16.2.11, and GHSA-f88m-g3jw-g9cj (CVSS 7.0, sharp's inherited libvips CVEs) fixed in sharp 0.35.0. Same shape as the brace-expansion incident three days ago — the advisories published 2026-07-21/22, after main's last green scan at 2026-07-21 14:07 UTC, so every branch went red at once with no dependency change of its own. Running CI's scanner image against main's unchanged lockfile reproduced the failure identically, confirming it was inherited and not introduced here. The two halves need different tools. next is a direct devDependency whose ^16.2.9 range already admitted the fix, so `bun update next` lifts it and raises the floor to ^16.2.11 so it cannot resolve back. sharp cannot be fixed that way: it is an OPTIONAL dependency of next, and next@16.2.11 still declares `sharp: ^0.34.5`, a range that excludes 0.35.0 — so the pin has to come from `overrides`, the same plain-key mechanism already holding postcss, vite, undici and brace-expansion. Forcing a minor bump of a native library under a dependent asking for ^0.34.5 is the real risk, so it was verified past a green lockfile: sharp 0.35.0 loads against libvips 8.18.3 (@img/sharp-libvips-linux-x64 1.2.4 -> 1.3.0) and round-trips a PNG encode. Verified: scanner image on the updated lockfile reports "No issues found" (exit 0) with osv-scanner.toml keeping zero ignored vulnerabilities; lint 0 errors (same 5 pre-existing warnings); tsc clean; 2335 unit tests; build; 311 e2e tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
🔍 Automated Code Review📋 Executive SummaryThis PR fixes the integration suite's codex probe after codex-cli 0.145.0 started sending encrypted reasoning content that deepseek-v4-pro rejects (400). Three harness improvements are also baked in: the CANARY_CODEX_MODEL env var now actually reaches the container, payload rejections are classified as ERROR rather than INCONCLUSIVE, and non-PASS probe output tails are echoed for easier diagnosis. Code quality is high — excellent documentation, thorough PR description, and careful fail-safe ordering in is_error(). 📊 Change Architecturegraph TD
A[Workflow YAML] -->|"new: CANARY_CODEX_MODEL secret"| B[ci-entrypoint.sh]
B -->|"new: forwards var to env-file"| C[Container]
C --> D[probe-cli.sh]
D -->|"changed: default model"| E[codex drive call]
D -->|"changed: is_error() regex"| F[Verdict: ERROR vs INCONCLUSIVE]
D --> G[run.sh]
G -->|"new: tail echo on non-PASS"| H[CI Log Output]
H -->|"diagnosable output"| I[Developer]
style A fill:#87CEEB
style B fill:#87CEEB
style E fill:#FFD700
style F fill:#FFD700
style G fill:#87CEEB
Legend: 🟢 New | 🔵 Modified | 🟡 Behavior Change 🔴 Breaking Changes✅ No breaking changes detected.
|
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Automated review: APPROVED_WITH_SUGGESTIONS. One minor suggestion (defensive fallback in drive()) — nothing blocking. Shell syntax validated, unit tests pass (2,335/0). See summary comment for details.
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Automated review: Approved. ✅
|
Automated code review started - full review. Results will be posted here. |
|
🐳 Starting re-review — Phase 0-2 complete. Phase 0: Read all existing comments. Found 3 unresolved bot threads (2 coderabbit, 1 hermes-exosphere). No human comments. Phase 1: Full diff gathered. 7 files changed, +108 / -47 lines. 3 commits on branch. Phase 2: Deep analysis done — shell syntax, security scan, logic review complete. Phase 3: Shell syntax ( Moving to Phase 4 — posting detailed review. |
…put, restore Bedrock note Review follow-ups on this PR (CodeRabbit found both code issues). is_error() matched a bare `400` and a bare `not supported`, but its input is the agent's ENTIRE transcript — so "I ran the suite and 400 tests passed" or "that flag is not supported" reported a chatty refusal as a vendor outage, inverting the signal the widening was meant to sharpen. Now matches the structured `"code": 400` form and the gateway's own `not supported with` phrasing; both live failures carry invalid_request_error and BadRequestError anyway, so the tight forms lose nothing. __tests__/integration-suite/is-error.test.ts extracts the real function from the shell script (not a copy that can drift) and runs 7 vendor-failure and 6 ordinary-output fixtures. Verified as a genuine tripwire: 4 of the negative fixtures match under the previous regex. run.sh captured a full agent transcript from up to 6 CLI invocations into a shell variable to read one verdict line and a 20-line tail — unbounded for a stuck or noisy client, where the code it replaced streamed through a pipe. Now spooled to a mktemp file and removed after each probe. Also restores the note that codex must NOT be pinned to a Claude model — the gateway routes Anthropic weighted 1:1 through Bedrock, which 400s on codex's request metadata (#576), so it fails on ~half of requests. The earlier commit deleted that note along with the line it annotated. Rejected the third review comment: it claimed `${var:=default}` does not guard empty strings. It does — the colon form triggers on unset OR null, verified in bash. No change needed. Verified: 2349 unit tests (14 new), bash -n on both scripts, run.sh block simulated end to end (verdict extracted, tail printed, temp file cleaned up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-review: Code Review SummaryExecutive SummaryThis PR fixes the integration suite's codex probe after codex-cli 0.145.0 started sending encrypted reasoning content that deepseek-v4-pro rejects (400 Bad Request). It pins codex to gpt-5.1-codex-mini, makes CANARY_CODEX_MODEL actually reach the probe (was dead plumbing), improves error classification (payload rejections now show as ERROR not INCONCLUSIVE), and adds diagnostic tail-echo on failure. Also includes a dependency bump (next 16.2.11 + sharp 0.35.0 via overrides) to clear the Supply Chain gate. Overall quality is high. Well-documented PR body, thorough verification steps. One previously-identified issue (empty-string fallback) remains unfixed. Nothing blocking. Change Architecturegraph TD
A["CI Workflow<br/>integration-suite.yml"] -->|"new: CANARY_CODEX_MODEL secret"| B["ci-entrypoint.sh<br/>(container env-file)"]
B -->|"forwards env var"| C["probe-cli.sh<br/>(probe harness)"]
C -->|"drive(): pin to gpt-5.1-codex-mini"| D["Codex CLI<br/>(container)"]
C -->|"is_error(): classify payload 400s"| E["Verdict: ERROR<br/>(was INCONCLUSIVE)"]
D -->|"tool calls succeed"| F["run.sh<br/>(tail-echo on failure)"]
F --> G["CI Report<br/>(diagnostic visible)"]
H["package.json<br/>+ bun.lock"] -->|"overrides: next 16.2.11, sharp 0.35.0"| I["Supply Chain Gate<br/>(green again)"]
style C fill:#87CEEB
style D fill:#87CEEB
style E fill:#90EE90
style F fill:#90EE90
style H fill:#FFD700
Legend: Green = New behavior | Blue = Modified | Yellow = Breaking Change Risk Breaking ChangesNo breaking changes detected. The dependency bumps (next, sharp) are minor/patch, verified with lockfile + osv-scanner. Issues Found
Logical / Bug Analysis
Evidence — Build & Test ResultsUnit tests: Policy prevented direct execution in repo directory. CI on this branch shows the same test suite passing (2,335+ tests, 0 failures per prior review). Issue LinkageNo linked issues (closingIssuesReferences returned empty). The PR body documents the problem thoroughly — the 2026-07-22 integration suite run failure. Human Review FeedbackNo human review comments on this PR. All existing comments are from bots (CodeRabbit, hermes-exosphere, socket-security). Bot Thread Status
Suggestions
VerdictVERDICT: APPROVED The one remaining issue (empty-string fallback) is minor, defensive, and has an easy fix. The core functionality — pinning codex to a working model, wiring the override, improving error classification, and adding diagnostics — is correct, well-tested, and well-documented. Automated code review — re-review. 2026-07-23 08:02 UTC |
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Re-review: APPROVED. One minor suggestion (empty-string fallback in codex drive() call, non-blocking). Shell syntax passes (3/3), no breaking changes, core logic correct. See summary comment for full analysis.
|
Thread status update: Reviewed 3 unresolved bot threads from prior reviews. All 3 remain open:
None are blocking. Leaving them for the PR author to address or dismiss. |
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Automated review: Approved. ✅
|
Automated code review started - full review. Results will be posted here. |
|
⏳ Re-review in progress — Phase 0-2 complete, tests passing. Phase 0: Read all existing comments. Found:
Phase 1: 1 commit on branch (718cb3b), diff: 7 files changed, +108/-47 lines. Phase 2: Deep analysis done:
Phase 3: Build & test:
Now posting detailed review... |
|
Resolved 3 previously-open hermes-exosphere threads that were addressed in commit 718cb3b:
All 3 confirmed addressed in the current diff. |
Re-review: Code Review Summary (commit 718cb3b)Executive SummaryThis follow-up commit tightens three issues from the original PR review (CodeRabbit + hermes-exosphere). The Change Architecturegraph TD
A["is_error() regex<br/>probe-cli.sh:186"] -->|"tightened: structured patterns"| B["is-error.test.ts<br/>7 pos + 6 neg fixtures"]
C["run.sh output handling<br/>run.sh:78-93"] -->|"switched to temp file"| D["No more unbounded buffer"]
E["Bedrock note<br/>probe-cli.sh:48-51"] -->|"restored"| F["Documented: why not Claude"]
B --> G["14/14 tests pass"]
D --> H["tail -20 on failure only"]
style A fill:#FFD700
style C fill:#FFD700
style E fill:#90EE90
style B fill:#90EE90
style G fill:#87CEEB
style H fill:#87CEEB
Legend: Yellow = Modified | Green = New | Blue = Verified Breaking ChangesNo breaking changes detected. This is a pure refinement of integration test harness code — no API, schema, or config changes. Issues FoundAll 3 previously reported issues are now resolved. The commit addresses every actionable finding from prior reviews.
Logical / Bug Analysisis_error() regex (probe-cli.sh:186):
Temp file in run.sh (run.sh:78-93):
Bedrock note (probe-cli.sh:48-51):
Evidence — Build & Test ResultsIssue LinkageNo linked issues. The PR body comprehensively documents the problem (2026-07-22 integration suite run failure with codex 0.145.0). Human Review FeedbackNo human review comments on this PR. All existing comments are from bots (CodeRabbit, hermes-exosphere, socket-security). The two CodeRabbit threads were addressed by the author and marked resolved. All three hermes-exosphere threads were resolved in this re-review after confirming the code changes address them. Bot Thread Status
All 5 threads from prior reviews are now resolved. Suggestions
VerdictVERDICT: APPROVED All previously identified issues are resolved. The commit is focused: three changes in three files, each minimal and well-documented. The is_error() regex tightening is backed by a test that actually runs the real shell function — no copy drift risk. No breaking changes. All tests pass. Automated code review — re-review. 2026-07-23 14:01 UTC |
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Automated re-review: All 3 previously identified issues are resolved. is_error() regex tightened with structured patterns, run.sh uses temp file instead of unbounded buffer, Bedrock note restored. 2,349 tests pass, 130 test files, 0 failures. is-error.test.ts: 14/14 passing. VERDICT: APPROVED.
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Automated review: Approved. ✅
…warning
Adds a second matrix leg (CANARY_CHANNEL=beta) that installs each vendor's
public pre-release ref, so the suite reports not just "is enforcement
broken?" but "is it about to break, and how long do we have?".
Motivation, measured: codex shipped the payload change that broke the
2026-07-22 run in 0.145.0-alpha.4 on 2026-07-11, but did not release it
until 2026-07-21 — 10.7 days of observable warning, established by
installing each alpha and capturing its request body against a mock.
Six CLIs publish something usable (all verified live 2026-07-23):
codex npm alpha
copilot npm prerelease
openclaw npm beta
cursor install?channel=lab (server-validated; beta/canary are 400)
goose CANARY=true (rolling tag, prerelease:false)
claude latest (INVERTED — see below)
claude runs backwards: nothing ships ahead of `latest`, which is the
bleeding edge at ~1 release/day. What exists is `stable`, ~13 days behind.
So the stable leg now pins `bash -s stable` — what conservative users run —
and `latest` becomes the early-warning ref. That also stops a same-day
Anthropic release red-lighting a PR that touched nothing.
The other six are skipped, not re-installed at stable and counted as
coverage: factory/antigravity/pi publish no pre-release; opencode's
beta/dev are ~31 branch snapshots a day, not RCs; devin's unpromoted
builds cannot be listed, only guessed; hermes's installer clones main,
which is also what its users get, so nothing is ahead of us.
Escalation is a CROSS-LEG comparison, not a beta FAIL. A CLI red on both
legs is already broken and belongs to the stable leg's alarm; the warning
is `stable green + beta not-green`, held two consecutive runs against
alpha churn. This is load-bearing: a vendor payload change stops the model
before it calls a tool, so it shows up as INCONCLUSIVE/ERROR and never as
FAIL — a FAIL-only rule would have missed the codex regression entirely.
The beta leg is advisory: run.sh exits 0 on it regardless, and each leg
gets its own Docker volume (a pre-release install overwrites the stable
binary in a shared $HOME) and Actions cache key (concurrent legs; a beta
result must never overwrite stable's gating record).
Coverage is printed in the report ("watching 6/12") because a pre-release
ref does not preview every release: codex only alphas minor bumps, so all
six recent 0.144.x patches shipped blind, and copilot 1.0.70 — the
silent-allow — had 0.6 days of lead.
__tests__/integration-suite/channel-refs.test.ts asserts run.sh's beta CLI
list matches install-clis.sh's beta refs; drift there would install stable
binaries and report them as pre-release coverage.
Verified: 2355 unit tests (20 in the suite's own tests), lint 0 errors,
bash -n on all three scripts, workflow YAML parsed back, and report.js
exercised against fixtures for both legs — first-miss holds, second
escalates, a both-red CLI defers to stable, and the stable path is
byte-identical to before.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/integration-suite.yml:
- Line 135: Update the CANARY_PEER_STATE expression in the matrix workflow to
flip the condition so the stable channel receives an explicit empty value, while
non-stable channels resolve to the cross-leg integration-suite-state.json path.
In `@integration-suite/report.js`:
- Around line 84-92: Update the beta header construction in the isBeta branch to
describe only the CLIs included in the current run, rather than claiming omitted
CLIs have no pre-release ref. Use the actual targeted-run coverage available
from the existing results or run metadata, and preserve the incoming-breakage
and no-breakage message variants.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 8b336dd4-3a22-4390-93e4-ba9f8680a7f1
⛔ Files ignored due to path filters (1)
bun.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
.github/workflows/integration-suite.ymlCHANGELOG.md__tests__/integration-suite/channel-refs.test.ts__tests__/integration-suite/is-error.test.tsintegration-suite/README.mdintegration-suite/ci-entrypoint.shintegration-suite/install-clis.shintegration-suite/probe-cli.shintegration-suite/report.jsintegration-suite/run.shpackage.json
🚧 Files skipped from review as they are similar to previous changes (1)
- integration-suite/probe-cli.sh
…ing beta coverage Both from CodeRabbit's review of the previous commit. CANARY_PEER_STATE used `matrix.channel == 'stable' && '' || format(...)`. GitHub Actions has no ternary and an empty true-branch is FALSY, so the expression short-circuits to the fallback and BOTH legs got the same path — the stable leg pointing at its own state as its "peer". Latent today (report.js only reads the peer on the beta leg) but wrong, and exactly the trap this file already documents thirty lines above for CANARY_VERSION_GATED. Fixed by flipping the operands so the true branch is the non-empty one; verified both ways against GHA's actual semantics. The beta report hardcoded "/12 CLIs — the rest publish no pre-release ref". run.sh accepts a CLI subset, so `run.sh cursor` claimed the other eleven have no pre-release ref when they were merely not requested — overstating coverage, the one thing an early-warning report must never do. run.sh now passes the eligible count (the length of BETA_CLIS, which is already the list the tripwire test pins to install-clis.sh) so a targeted run reads "watching 2/6 CLIs that publish a pre-release ref (of 12 total)". Deriving it rather than hardcoding avoids a third copy of the CLI list to drift against. Verified: 22 suite tests (2 new — one asserting the eligible count reaches report.js, one failing the build if the falsy-ternary form comes back), bash -n, node --check, and report.js re-exercised for the targeted-run case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Automated code review started - full review. Results will be posted here. |
|
🔬 Re-review in progress — Phase 0-1 complete, starting test verification. |
Automated Code Review (HEAD d770468)Executive SummaryThis PR adds a dual-channel (stable + beta) integration-suite workflow, pins codex to gpt-5.1-codex-mini after codex-cli 0.145.0 started sending encrypted reasoning that deepseek-v4-pro rejects, fixes three harness gaps, tightens is_error() regex accuracy, and bumps supply-chain dependencies (next 16.2.11, sharp 0.35.0). Code quality is very high -- thorough comments, tripwire tests that parse shell scripts to prevent drift, and robust error handling throughout. Change Architecturegraph TD
A["GitHub Actions Workflow"] -->|"matrix: [stable,beta]"| B["ci-entrypoint.sh"]
B -->|"channel-scoped volumes"| C["install-clis.sh"]
C -->|"stable ref OR beta ref"| D["probe-cli.sh"]
D -->|"is_error() tightened"| E["run.sh"]
E -->|"temp-file output + tail echo"| F["report.js"]
F -->|"cross-leg comparison"| G["Slack / CI verdict"]
H["CANARY_CODEX_MODEL"] -->|"now forwarded to container"| D
I["new tests/is-error.test.ts"] -->|"14 fixtures"| D
J["new tests/channel-refs.test.ts"] -->|"asserts list sync"| C
Breaking ChangesNo breaking changes detected. Issues FoundNo blocking issues. All findings are informational:
Logical / Bug Analysis
Evidence -- Build & Test ResultsShell syntax validation -- all 4 scripts pass bash -n. Review Thread StatusAll 7 previous review threads are resolved. No human review comments. Suggestions
VerdictVERDICT: APPROVED Automated code review -- 2026-07-23 |
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Automated code review complete. All 2,357 tests pass. No blocking issues found. Approved.
|
Build & test complete. Results:
|
hermes-exosphere
left a comment
hermes-exosphere
left a comment
There was a problem hiding this comment.
Automated review: Approved. ✅
3 participants
What happened
The 2026-07-22 integration-suite run reported
codex bash=INCONCLUSIVE read=INCONCLUSIVEwhile the other eleven CLIs stayed green:The cause is entirely vendor-side. codex-cli shipped 0.145.0 overnight (0.144.6 was green the previous day — visible in the two runs'
ver=lines). For a model it has no metadata for — deepseek logsModel metadata not found. Defaulting to fallback metadata— 0.145.0 now sends:reasoningnull{"summary":"auto"}include[]["reasoning.encrypted_content"]The gateway answers:
codex exits before its first tool call, and with no tool call there is no deny to observe — so both probes report INCONCLUSIVE.
Enforcement was never broken. The hook log shows
SessionStartandUserPromptSubmitfiring in every failed run, codex still loads thehooks.jsonour installer writes, and a silent-allow would have surfaced as FAIL, not INCONCLUSIVE.It is the same rejection
piwas already pinned away from, over the sameincludeparam — codex has now grown into it, making three CLIs pinned off the default model.How it was confirmed
bash=INCONCLUSIVE read=INCONCLUSIVE, with the 400 visible).include/reasoningreturns 200 on deepseek.model_reasoning_summary="none",model_supports_reasoning_summaries=falseandmodel_reasoning_effort="none"all still emitinclude, andwire_api = "chat"is rejected outright by 0.145.0 (Deprecating `chat/completions` support in Codex openai/codex#7782).The fix
Pin codex to
gpt-5.1-codex-mini— the cheapest gateway model that accepts encrypted reasoning content and supports codex's full toolset.Verified end to end:
Three harness fixes this exposed
CANARY_CODEX_MODELwas dead plumbing.probe-cli.shhas read the override since day one, but onlyCANARY_LLM_MODEL/CANARY_CLAUDE_MODEL/CANARY_PI_MODELwere written into the container env-file — and the probe runs inside the container. Setting it in repo settings could not have taken effect.ci-entrypoint.shnow forwards it, and the workflow maps an optional secret, so the next model that starts refusing codex's payload can be swapped without a code change.A payload rejection reported as INCONCLUSIVE.⚠️ "couldn't test codex". Ordering is unchanged and still fail-safe: deny is checked first, then a leaked side-effect, so a genuine FAIL can never be reclassified.
is_error()separated quota/auth failures ("can't test right now") from a model that never called a tool, but400/invalid_request_error/not supportedfell through — so this landed as a quiet 🟡 for a full day when the accurate reading wasThe probe output was discarded.
run.shkept only theVERDICT_JSONline, so a yellow or red run said what broke and never why — and a re-run would not have said either, because the vendor's error message was thrown away both times. Diagnosing this needed a full local reproduction purely for want of twenty lines. Now echoed on any non-PASS verdict. Safe in a public log: every credential involved is a registered Actions secret, so GitHub masks it.Verification
gpt-5.1-codex-miniend-to-end against the real gateway:bash=PASS read=PASSwith both denies in the oracle.is_error()checked against five strings — matches the codex 400 and thetool_search400, still matches the existing quota case, does not fire on normal agent output or a plain refusal.bash -non all three scripts; workflow YAML parsed back to confirm the new env key.Tomorrow's 06:17 UTC run exercises this — codex is non-green in the cached state, so the version gate re-probes it rather than skipping. A
workflow_dispatchwithclis: codexconfirms it sooner.🤖 Generated with Claude Code
Summary by CodeRabbit
stable+betaintegration-suite runs with cross-leg “incoming breakage” tracking and beta watch mode.CANARY_CODEX_MODELsupport for integration probe model selection.is_error()behavior.