(0.1.5) scan agent still creates no draft artifacts on GitHub-hosted runners

[PV-Bhat]

Summary

(follow-up to Issues #181, #73)
0.1.5 does not fix the GitHub Actions path. It replaces the opaque seal error with a much better message, but a CI scan still authenticates, performs billed model work, and then produces nothing:

Could not save the Codex Security scan: Scan agent did not create required draft
artifacts: scan-manifest.json, findings.json, coverage.json. Check that the scan
agent can run shell commands and write to the scan directory before retrying.
Partial output was kept at /home/runner/work/_temp/scan-results.

Exit code 2. The output directory is empty, so "partial output was kept" points at nothing (same observation as #181).

This is a follow-up to #181 (closed without comment ~2 minutes before 0.1.5 published) and #73 (still open). I am filing separately because the new diagnostic changes what is actionable: the tool now names the precondition it needs ("the scan agent can run shell commands and write to the scan directory"), which makes the local-vs-CI asymmetry below the useful signal.

The asymmetry: identical version, local succeeds, CI fails

	Local	GitHub Actions
Version	0.1.5	0.1.5
Auth	--auth chatgpt (ChatGPT subscription)	--auth default, OPENAI_API_KEY from repo secret
OS	Windows 11, Node 22	ubuntu-latest, Node 22 via actions/setup-node
Target	scan . --diff origin/main	scan . --diff origin/<base>
Agent wrote draft artifacts?	Yes	No
Result	threat model, worker reviews, 5 validated findings, per-finding validation reports	empty directory, exit 2

The successful local run wrote exactly the files the CI run is missing, plus the scaffolding it used to produce them:

threat_model.md
write_threat_model.py            <- agent-authored helper scripts
assemble_discovery.py
assemble_validation.py
append_validation_receipt.py
artifacts/01_context/threat_model.md
artifacts/02_discovery/raw_candidates.jsonl
artifacts/02_discovery/worker_results/*.json
artifacts/04_reconciliation/deduped_candidates.jsonl
artifacts/05_findings/<finding-id>/validation_report.md
artifacts/05_findings/validation_summary.md

So the scan agent's method is: author Python/shell helpers into the scan directory, execute them, and assemble artifacts from their output. That works under subscription auth on a developer machine and does not work on a GitHub-hosted runner. Given the maintainer note on #73 (a macOS reproduction where the inner agent hit sandbox-exec: sandbox_apply: Operation not permitted and could therefore produce no drafts), the natural hypothesis is that the agent's shell/write capability is unavailable or silently denied in the Actions environment — but the CLI does not verify that capability before the first billed turn.

Two concrete questions, since the answer determines whether users can work around this today:

1. Does the scan agent's execution sandbox behave differently under --auth api-key than under --auth chatgpt? That is the largest uncontrolled variable between the two runs above.
2. Is there a supported way to inspect or relax the agent's shell/filesystem capability in CI (e.g. a --codex TOML key such as sandbox_mode)? scan --help documents --codex for arbitrary TOML key/value pairs but the CI guide does not mention any sandbox configuration, and https://learn.chatgpt.com/docs/security/cli/ci lists only Node 22+ and Python 3.10+ as runner requirements.

Environment

• @openai/codex-security@0.1.5 (also reproduced on 0.1.4), installed per the CI guide: npm install --prefix "$RUNNER_TEMP/codex-security" --ignore-scripts --no-audit --no-fund
• ubuntu-latest, Node 22 via actions/setup-node, python3 3.12 present on PATH
• Private repository (Python + TypeScript monorepo), same-repo PR, --auth default with OPENAI_API_KEY from a repository secret
• --output-dir "$RUNNER_TEMP/scan-results" (outside the repo, as the CLI requires), --max-cost 2, --fail-on-severity high, six --knowledge-base files
• No outer sandbox: a plain Actions runner, not a Codex desktop session
• Cost: three CI runs, ~$4.50 total, zero reports

Why

Merge-path gating is the CLI's headline use case, and the documented recipe is the path most teams will adopt first. In CI nobody is watching the log, so the current behaviour converts a tool defect directly into silent recurring spend — every push to every open PR bills and yields nothing. #181 measured $3.30 before giving up; we added ~$4.50 independently.

Requests

1. Verify the agent's shell/write capability before the first billed turn. The new error message already knows what the precondition is; checking it during preflight would turn this class of failure from "$1.50 and no result" into "$0 and a clear message". --dry-run currently does not appear to cover it.
2. Document the CI limitation while it stands, on https://learn.chatgpt.com/docs/security/cli/ci. Right now the guide presents a recipe that cannot complete, which is what sends people through the paid-debugging loop.
3. Only print Partial output was kept at <dir> when that directory is non-empty (also raised in Documented GitHub Actions CI recipe cannot complete a scan on 0.1.4: every run bills, seals nothing, and its SARIF upload step 403s on private repos #181; still present in 0.1.5).
4. State whether api-key vs ChatGPT auth affects agent capability, so teams know whether "run it locally on a subscription" is the supported interim posture. Empirically it is the only one that works for us.

Happy to run further instrumented CI reproductions if there is a diagnostic build or an env var that logs the agent's shell attempts — though I'd rather not keep paying per experiment to characterize it.