ci(build): split allTests into parallel per-target jobs to end OOM flakes#853
Closed
michalharakal wants to merge 1 commit into
Closed
ci(build): split allTests into parallel per-target jobs to end OOM flakes#853michalharakal wants to merge 1 commit into
michalharakal wants to merge 1 commit into
Conversation
…akes The single build-job ran `assemble allTests` in one ubuntu-latest job. On a 7 GB runner the combined peak memory (Gradle JVM + Kotlin daemon + per-target test forks + Node for JS/Wasm + native toolchain, all at once) intermittently exceeded physical RAM. The runner was OOM-killed and GitHub reported "The operation was canceled" after ~15 min with no BUILD FAILED — the flaky ~20-minute red builds (seen on PRs #850, #852). Changes: - Split the test run into a matrix of three legs (jvm / js-wasm / native), each running one target family so no single job holds the whole footprint. Legs run in parallel, so wall-clock is the slowest leg instead of the sum. - Run `assemble` as its own memory-light job, preserving the build/package guarantee for all targets. - Add Gradle dependency/wrapper caching to each job (same pattern as docs.yml) so parallel legs don't cold-download the dependency graph. - Capture the memory snapshot on `cancelled()` as well as `failure()` — an OOM-killed runner reports cancelled, not failed, so the existing diagnostic never fired on the runs that needed it. - Keep an aggregate `build-job` gate with the same name so the existing required status check keeps working without re-pointing branch protection. No production code changed; allTests passes locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
build-jobintermittently fails after ~15 min with "The operation was canceled" and noBUILD FAILED— the flaky ~20-minute reds (e.g. PRs #850, #852). Investigation shows this is runner memory exhaustion, not a code defect:OutOfMemoryError, no failing test.build.ymlranassemble allTestsin a singleubuntu-latest(7 GB) job.allTestsfans out the Gradle JVM (-Xmx4g) + Kotlin daemon (1 GB) + per-target test forks + Node (JS/Wasm) + the native C toolchain concurrently; peak RSS sometimes tips over 7 GB → OOM-kill. Sometimes it just fits (which is why ci(docs): publish on release and stop fork PRs failing on comment #850 went green on retry). It passes locally because dev machines have more RAM.Memory infodiagnostic usedif: failure(), but an OOM-killed runner reports cancelled, not failed — so it never captured data on exactly these runs.Fix
jvm/js-wasm/native— each running one target family (jvmTest/jsTest wasmJsTest wasmWasiTest/linuxX64Test). No single job holds the whole footprint, and the legs run in parallel, so wall-clock is the slowest leg instead of the sum. Directly targets both the OOM flakiness and the ~20-min duration.assemblebecomes its own memory-light job, preserving the build/package guarantee for all targets (incl. those without tests).docs.yml) so parallel legs don't cold-download the dependency graph.cancelled()too, so future OOMs are actually captured.build-jobgate keeps the exact same check name, so the existing required status check keeps working with no branch-protection change.Coverage / verification
allTestsran (Apple targets are macOS-only and were already skipped on Linux).linuxX64is a declared target;jvmTest jsTest wasmJsTest wasmWasiTestdry-run resolves cleanly.allTestspasses locally (BUILD SUCCESSFUL); no production code changed here — workflow only.Follow-up
If the
js-wasmleg ever approaches the limit, it can be split further (js vs wasm) — and the memory snapshot will now be captured to confirm.