OS check CI fix - #11036
Open
Frauschi wants to merge 3 commits into
Open
Conversation
GitHub caps a single `run:` step at 21000 characters. os-check.yml embedded its 109-entry Linux config list as a heredoc inside that step, and commit c00e726 ("Add AES-GCM DEM, CryptoCb support, and devId threading to ECIES") pushed it from 20662 to 21813 characters. Since that merge on 2026-07-24 GitHub has refused to load the file at all: every run of the workflow ends in failure within 0s with zero jobs, on master and on every PR branch. The failure is easy to miss. The run registers under the literal path `.github/workflows/os-check.yml` rather than its `name:` field, its check suite carries no check runs so there are no logs or annotations, and because GitHub cannot read the file it cannot apply the `on:` filters either - hence the master push runs for a workflow whose push trigger is restricted to release/**. Meanwhile `gh pr checks` still reports hundreds of green checks from the other workflows. Move the config lists to checked-in JSON under .github/configs/; parallel-make-check.py already accepts the JSON path as its positional argument. Splitting the step in two would not have been enough: the heredoc alone was 21387 characters once de-indented. os-check-linux.json 109 configs os-check-macos.json 7 configs pq-all.json 31 configs multi-arch.json 23 configs smoke-test.json 10 configs The os-check lists are the fix; the other three are preventive - pq-all and multi-arch were the next largest run steps at 14076 and 11011 characters. The largest remaining run step is now 6285 characters. Each list was compared object-for-object against the version it replaces, so this is a pure relocation with no coverage change.
A workflow file GitHub cannot load does not fail loudly. Its runs end within 0s with zero jobs, no logs, no annotations and no check runs, and the workflow re-registers under its bare path instead of its `name:` field. Among the few hundred checks on a PR that reads as unrelated flake, so the coverage just disappears: os-check.yml was in this state on master for ten days in July 2026 before anyone noticed, and no open PR reported a problem the whole time. Add two guards. Pre-merge, check-workflows.py measures every `run:` step against GitHub's 21000 character cap and fails the build past it, with a warning from 18000 so a growing step is noticed while there is still runway. Sizes come from the parsed YAML, which is what the Actions service evaluates, so block-scalar indentation needs no guessing. It runs from check-source-text.yml over every workflow and composite action rather than only PR-changed files: the cap applies per file, the whole sweep takes well under a second, and a file can be pushed over the line by a change elsewhere in the PR. Note that this cap is enforced by the service and not by the workflow schema, so neither a YAML validator nor actionlint reports it. Post-merge, workflow-health.yml runs check-workflow-health.py daily and looks for the symptom rather than any particular cause, so a workflow that stops loading for a reason nobody anticipated is still caught. Two signals: an active workflow whose registered name equals its path, and a completed run that failed with zero jobs (prefiltered on created_at == updated_at, so only a handful need a jobs lookup). Against the live repository the first signal flags os-check.yml and nothing else across 107 workflows, and reports clean on wolfTPM and wolfMQTT. It exits 1 on a finding and 2 when the check could not be carried out at all, because a missing token and a broken workflow call for different responses. Findings go into a single reused issue rather than another red check that would blend into the noise: the body is rewritten on each run, a comment is posted only when the set of affected workflows changes, and the issue closes itself once everything loads again. Finding that issue reliably turned out to be the fiddly part, and the approach here is the one that survived testing against a live repository. The issue is identified by both a dedicated label and its title, and looked up through the REST issues endpoint. Both halves of that identity matter: the label alone is a normal repository label that anyone can apply, and an adopted issue has its body overwritten and is then closed, so matching on the label alone would destroy a mislabelled issue. Searching by title instead is unusable, because search ignores --state and returns closed issues, which had the monitor re-closing an already closed issue on every clean run. `gh issue list` reads a GraphQL replica that can lag. The REST endpoint lags too, by about 2.4s for a newly created issue, so the lookup re-checks a few times before concluding nothing is open - without that, consecutive runs each open a duplicate, and a clean run right after an outage fails to close the issue it just opened. Verified against the commit that caused the outage: check-workflows.py fails on acff4d6 (21813 characters) and passes on its parent f5ace71, which it flags at 20662 - already inside the warning band, 338 characters short of breaking. The full issue lifecycle (open, repeat with no comment, comment on change, close, stay closed, reopen a fresh issue for a new outage) was exercised end to end against a live repository.
The ccache that keeps this workflow fast is written only by the weekday seed job, which runs on a schedule against the default branch. Nothing had seeded it since 2026-07-24, because the workflow itself was failing to load for that whole stretch, and the Actions cache evicts entries untouched for seven days. The first run after the fix therefore reported "Cache not found for input keys" on all four shards. That cold run measured ~102 thread-minutes per shard, or 24-27 minutes of wall including checkout, deps and autogen, against a 30 minute timeout. Three minutes of headroom on the slowest shard is not enough, and a shard killed by the timeout presents as a test failure rather than as a cold cache. Raise it to 40. The comment above it claimed ~68 thread-minutes and ~20 minutes of wall, which the measurement above contradicts, so replace it with the measured figures. macOS and Windows are left alone: they came in at 11.6 minutes against 45 and 2.0 against 6. Add workflow_dispatch so the seed can be run on demand rather than waiting up to a day for the next cron, which matters exactly in the situation above, where every PR run stays cold until something refills the cache. Adding the trigger alone would not have been enough. The seed behaviour hangs off `github.event_name == 'schedule'` in five places (CCACHE_RECACHE, --build-only on linux and macOS, the cache save, and skipping the Windows job), so a manual run would have gone through the full test path and saved nothing. All five now treat a dispatch as a seed as well. The condition is written against `github.event.inputs.seed` rather than `inputs.seed`, because the `inputs` context is only documented as available on workflow_dispatch and workflow_call, whereas `github.event` always exists. That yields strings, so it is compared explicitly rather than for truthiness, where the string 'false' would read as true. It tests `!= 'false'` and not `== 'true'` so that a dispatch which sends no input at all still seeds: a declared default is not reliably reflected into `github.event.inputs`, and keying on the positive would have made `gh workflow run` quietly skip the seeding it was invoked to do. Comments are brought in line with all of this, including one that was already wrong before the trigger existed: the macOS ccache step is read-only purely on pull_request, so every non-PR run writes that cache, where the note claimed only the seed did. The two platforms seed differently in a second way as well - linux sets CCACHE_RECACHE and so rebuilds from scratch, macOS never does and only accumulates deltas. Neither behaviour is changed here, but both are now written down at the top of the file rather than left to be rediscovered from a surprising cache.
Frauschi
force-pushed
the
fix-os-check-run-length
branch
from
August 3, 2026 15:05
3b989bd to
91769d9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
.github/workflows/os-check.ymlhas not run at all since 2026-07-24. Every one of its runs since then, on master and on every PR branch, has ended infailurewithin 0s with zero jobs. That is roughly ten days during which no merge to master was covered by 109 Linux configs, 7 macOS configs, or the Windows build job, and no PR reported a problem.Root cause
GitHub caps a single
run:step at 21000 characters ("Exceeded max expression length 21000").make_check_linuxembedded its whole config list as a<<'EOF'heredoc inside that step. Commit c00e726 added three ECIES configs and pushed the step from 20662 to 21813 characters. Past the cap GitHub refuses to load the entire file, so no job is ever created.run:stepmake_check_linuxbuild step after c00e726The cap is enforced by the Actions service rather than by the workflow schema, so neither a YAML validator nor
actionlintreports it. The file parses fine locally.Why nobody noticed
A workflow that fails to load looks nothing like a workflow that fails:
.github/workflows/os-check.yml, instead of itsname:field (Ubuntu-Macos-Windows Tests), so it does not look like the workflow you know.on:filters either. Runs therefore appear for events the file excludes, e.g.pushon master althoughpush:is restricted torelease/**. That extra noise is a symptom, not the trigger.gh pr checksstill looks healthy at a glance.Changes
1. Fix the outage (a6bd8c0)
Move the config lists out of the workflow files into checked-in JSON under
.github/configs/.parallel-make-check.pyalready accepts the JSON path as its positional argument, so this is a pure relocation..github/configs/os-check-linux.json.github/configs/os-check-macos.json.github/configs/pq-all.json.github/configs/multi-arch.json.github/configs/smoke-test.jsonThe os-check lists are the fix. The other three are preventive: pq-all and multi-arch were the next largest
run:steps at 14076 and 11011 characters. The largest remainingrun:step in the repository is now 6285 characters.Note that splitting the step in two would not have been enough. The heredoc alone was 21387 characters once de-indented, still over the cap on its own. The content has to leave the workflow file.
2. Make this class of failure visible (641c39d)
Pre-merge:
.github/scripts/check-workflows.pymeasures everyrun:step against the cap and fails the build past it, with a warning from 18000 so a growing step is noticed while there is still runway. Sizes come from the parsed YAML, which is what the Actions service evaluates, so block-scalar indentation needs no guessing. It runs fromcheck-source-text.ymlacross every workflow and composite action, not just PR-changed files, since the cap applies per file and a file can be pushed over the line by a change elsewhere in the PR.Post-merge:
.github/workflows/workflow-health.ymlruns.github/scripts/check-workflow-health.pydaily. It looks for the symptom rather than any particular cause, so a workflow that stops loading for a reason nobody anticipated is still caught. Two signals: an active workflow whose registered name equals its path, and a completed run that failed with zero jobs. Findings go into a single reused GitHub issue rather than another red check that would blend into the noise. The body is rewritten each run, a comment is posted only when the set of affected workflows changes, and the issue closes itself once everything loads again.python3-yamlis added to the apt package list and to theubuntu-24.04-fulldeb bundle for the linter.3. Survive the cold cache the outage left behind (91769d9)
The ccache that keeps this workflow fast is written only by its own seed job, which runs on a schedule against the default branch. Nothing had seeded it since 2026-07-24, and the Actions cache evicts entries untouched for seven days, so the first run on this branch reported
Cache not found for input keyson all four shards.That cold run took 24-27 minutes of wall per shard against a 30 minute timeout. Raise the linux timeout to 40: three minutes of headroom on the slowest shard is not enough, and a shard killed by the timeout presents as a test failure rather than as a cold cache. macOS and Windows are left alone, at 11.6 minutes against 45 and 2.0 against 6.
Add
workflow_dispatchso the seed can be run on demand instead of waiting up to a day for the next cron, which is what the situation above calls for. The trigger alone would not have been enough: the seed behaviour hangs offgithub.event_name == 'schedule'in five places, so a manual run would have taken the full test path and saved nothing. All five now treat a dispatch as a seed too.Testing
Each config list was parsed out of the pre-change workflow and compared object-for-object against its new JSON file. All five match exactly, so coverage is unchanged.
Builds run from the new files in a clean worktree with each workflow's real flags:
no-sys-ca-certs(macOS),no-examples-no-mallocandopensslextra-no-filesystem-no-bio(Linux),cryptonlyanddefault(smoke,-Werror),pkcs7-mldsa-onlyandmldsa-verify-only-dynamic-keys(pq-all). Sharding still balances: os-check 28/27/27/27, pq-all 8/8/8/7.The linter was checked against the commit that caused the outage:
run:stepThe parent already sat inside the warning band, 338 characters short of breaking, so the guard would have fired before the breaking change was written.
This branch's own os-check run is the direct evidence that the workflow loads again: it registers under the workflow's real name and creates 9 jobs (4 linux shards, macOS, 4 Windows builds), all passing, while master's run minutes earlier still registers as
.github/workflows/os-check.ymlwith zero jobs. The linter,multi-archandpq-allall pass in CI, and the linter's CI output matches its local run exactly (checked 112 files; largest run: step is 6285 characters).The monitor flags os-check.yml and nothing else across 107 workflows, reports clean on wolfTPM and wolfMQTT, and had its full issue lifecycle and its failure paths exercised against a live repository.
Notes for reviewers
workflow-health.yml.workflow_dispatchon os-check, and the scheduledWorkflow Healthjob. Neither has CI evidence here.CCACHE_RECACHE, and writes its cache on every non-PR run rather than only on seeds). No behaviour is changed here; the difference is now written down at the top of os-check.yml instead of being latent.