Skip to content

ci(build): split allTests into parallel per-target jobs to end OOM flakes#853

Closed
michalharakal wants to merge 1 commit into
developfrom
ci/split-alltests-parallel-jobs
Closed

ci(build): split allTests into parallel per-target jobs to end OOM flakes#853
michalharakal wants to merge 1 commit into
developfrom
ci/split-alltests-parallel-jobs

Conversation

@michalharakal

Copy link
Copy Markdown
Contributor

Problem

build-job intermittently fails after ~15 min with "The operation was canceled" and no BUILD FAILED — the flaky ~20-minute reds (e.g. PRs #850, #852). Investigation shows this is runner memory exhaustion, not a code defect:

  • The job log goes silent minutes before the cancel (runner becomes unresponsive), then GitHub reaps it — the classic system-OOM signature. No OutOfMemoryError, no failing test.
  • build.yml ran assemble allTests in a single ubuntu-latest (7 GB) job. allTests fans out the Gradle JVM (-Xmx4g) + Kotlin daemon (1 GB) + per-target test forks + Node (JS/Wasm) + the native C toolchain concurrently; peak RSS sometimes tips over 7 GB → OOM-kill. Sometimes it just fits (which is why ci(docs): publish on release and stop fork PRs failing on comment #850 went green on retry). It passes locally because dev machines have more RAM.
  • Bonus: the Memory info diagnostic used if: failure(), but an OOM-killed runner reports cancelled, not failed — so it never captured data on exactly these runs.

Fix

  • Split the test run into a 3-leg matrixjvm / js-wasm / native — each running one target family (jvmTest / jsTest wasmJsTest wasmWasiTest / linuxX64Test). No single job holds the whole footprint, and the legs run in parallel, so wall-clock is the slowest leg instead of the sum. Directly targets both the OOM flakiness and the ~20-min duration.
  • assemble becomes its own memory-light job, preserving the build/package guarantee for all targets (incl. those without tests).
  • Gradle caching per job (same pattern/pin as docs.yml) so parallel legs don't cold-download the dependency graph.
  • Memory diagnostic now fires on cancelled() too, so future OOMs are actually captured.
  • Aggregate build-job gate keeps the exact same check name, so the existing required status check keeps working with no branch-protection change.

Coverage / verification

  • The three legs cover every runnable Linux test target that allTests ran (Apple targets are macOS-only and were already skipped on Linux).
  • Task names verified: linuxX64 is a declared target; jvmTest jsTest wasmJsTest wasmWasiTest dry-run resolves cleanly.
  • allTests passes locally (BUILD SUCCESSFUL); no production code changed here — workflow only.

Follow-up

If the js-wasm leg ever approaches the limit, it can be split further (js vs wasm) — and the memory snapshot will now be captured to confirm.

…akes

The single build-job ran `assemble allTests` in one ubuntu-latest job.
On a 7 GB runner the combined peak memory (Gradle JVM + Kotlin daemon +
per-target test forks + Node for JS/Wasm + native toolchain, all at
once) intermittently exceeded physical RAM. The runner was OOM-killed
and GitHub reported "The operation was canceled" after ~15 min with no
BUILD FAILED — the flaky ~20-minute red builds (seen on PRs #850, #852).

Changes:
- Split the test run into a matrix of three legs (jvm / js-wasm /
  native), each running one target family so no single job holds the
  whole footprint. Legs run in parallel, so wall-clock is the slowest
  leg instead of the sum.
- Run `assemble` as its own memory-light job, preserving the
  build/package guarantee for all targets.
- Add Gradle dependency/wrapper caching to each job (same pattern as
  docs.yml) so parallel legs don't cold-download the dependency graph.
- Capture the memory snapshot on `cancelled()` as well as `failure()`
  — an OOM-killed runner reports cancelled, not failed, so the existing
  diagnostic never fired on the runs that needed it.
- Keep an aggregate `build-job` gate with the same name so the existing
  required status check keeps working without re-pointing branch
  protection.

No production code changed; allTests passes locally.
@michalharakal
michalharakal deleted the ci/split-alltests-parallel-jobs branch July 21, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant