Skip to content

ci(eval): persist eval reports via auto-merge PR#6

Merged
jasonodoom merged 1 commit into
mainfrom
fix/eval-report-via-pr
Jun 9, 2026
Merged

ci(eval): persist eval reports via auto-merge PR#6
jasonodoom merged 1 commit into
mainfrom
fix/eval-report-via-pr

Conversation

@jasonodoom

Copy link
Copy Markdown
Contributor

The weekly and monthly eval jobs generate a report, then push it to main. main is protected (requires the test check), so the bot push is rejected with GH006 after all the compute is done. The monthly job spends real money on vision scoring before hitting this, and the report is then lost with the ephemeral runner.

Change

Replace the direct push to main with a branch-and-PR flow:

  • Push the generated report to a dated bot/ branch (bot/weekly-eval-, bot/monthly-vision-eval-).
  • Open a PR into main and enable auto-merge (squash). ci.yml runs the test check on the PR; once it passes, the PR merges and the branch is deleted.
  • Checkout uses RELEASE_PAT instead of GITHUB_TOKEN. PRs opened with GITHUB_TOKEN do not start workflow runs, so the required test check would stay pending and auto-merge would never fire. RELEASE_PAT (already used by the release workflow for the same reason) makes the PR trigger CI.

Branch protection stays intact: reports still pass through the test gate before landing on main. The smoke path on the monthly workflow continues to skip the PR entirely.

Notes

  • Reuses the existing RELEASE_PAT secret; no new secret required. Assumes it carries pull-request write (classic repo scope or fine-grained contents+PR). If a run reports a permissions error opening the PR, the PAT scope needs widening.
  • No change to the eval compute steps, which already pass.

main is protected (requires the test check), so the eval bots could
not push reports directly: the run did all the work, then died at
GH006 protected-branch-update-failed.

Push the generated report to a dated bot/ branch and open a PR that
auto-merges once ci.yml passes. Checkout uses RELEASE_PAT rather than
GITHUB_TOKEN so the PR actually triggers the test check; PRs opened by
GITHUB_TOKEN do not start workflow runs, which would leave the
required check pending and auto-merge stuck.
@jasonodoom jasonodoom enabled auto-merge (squash) June 9, 2026 05:47
@jasonodoom jasonodoom merged commit d641dce into main Jun 9, 2026
3 checks passed
@jasonodoom jasonodoom deleted the fix/eval-report-via-pr branch June 9, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant