.github/workflows: feed hive-eest fixtures from cache instead of per-run CDN download - #21882
Merged
Merged
Conversation
…N download test-hive-eest's eels sims download the fixtures tarball from the GitHub release CDN inside the image build, and with --docker.nocache that repeats on every run — slow and exposed to transient 504s (and bypassing the hardened tools/test-fixtures.sh entirely). Restore the EEST tarballs from the base-branch cache warmed by cache-warming-eest-fixtures.yml (restore-only; the warmer owns saving), fall back to the hardened tools/test-fixtures.sh on a miss, stage the matrix tarball into the eels sim build context, and pass --sim.buildarg fixtures=/fixtures so the sim consumes the local fixtures instead of downloading. Requires the local-fixtures eels Dockerfiles from erigontech/hive#2; the hive ref is pinned to that branch's HEAD for end-to-end CI validation.
…-fixtures erigontech/hive#2 merged into yperbasis/client-pool; pin to that merge commit (9afe5f15) instead of the now-merged feature branch HEAD.
…-eest-local-fixtures
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the test-hive-eest GitHub Actions workflow to stop downloading EEST fixture tarballs from the GitHub release CDN inside the eels simulator image build, and instead provide fixtures locally from a warmed GitHub Actions cache with a hardened fallback download path.
Changes:
- Pin Hive checkout to a commit that supports local-fixtures for the eels simulators.
- Restore EEST fixture tarballs from a base-branch-warmed cache (with a fallback to
tools/test-fixtures.shon cache miss). - Stage the fixture tarball into the simulator build context and switch the simulator build arg from a CDN URL to
/fixtures.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
yperbasis
approved these changes
Jun 18, 2026
yperbasis
left a comment
yperbasis
left a comment
Member
There was a problem hiding this comment.
Approving. Minor / non-blocking:
- Combined cache restores all four tarballs (~2.4 GB) per shard though each uses only one (
eest_stable/eest_devnet);eest_benchmark/eest_zkevmare never used by this matrix. Per-tarball cache keys would cut restore reads across the ~11 jobs (~26 GB → ~5 GB) — warmer-side follow-up at most. - Stale warmer doc:
cache-warming-eest-fixtures.yml's header still says it serves onlytest-eest-spec.yml; this PR adds a second consumer. - Exact cache hit skips sha256 verification (only the miss/download path verifies via
test-fixtures.sh) — fine given the warmer verifies before save and write-scoping is restricted, just a slight asymmetry. - Pre-existing: the
Read pinned Hive refstep'soutputs.refis unused (the ref is hardcoded inline).
…ion) On a cache miss we only need the .tar.gz to stage into the eels sim build context — the eels Dockerfile extracts it. Pass --download-only to tools/test-fixtures.sh so it fetches + sha256-verifies the tarball without extracting the multi-GB corpus. (Addresses Copilot review on #21882.)
Member
|
@lystopad can you please upstream this to Hive? Otherwise this change will not be applied once we switch back the Hive EEST jobs to use |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
test-hive-eest'seelssimulators download the fixtures tarball from the GitHub release CDN inside the image build (consume cache --input <url>). With hive's--docker.nocache, that download repeats on every run — slow, exposed to transient release-CDN 504s, and bypassing the hardenedtools/test-fixtures.shentirely.Change
Feed the fixtures from cache + a hardened fetch, and hand them to the sim locally:
cache-warming-eest-fixtures.yml— restore-only (the warmer owns saving, so no per-ref duplicates).tools/test-fixtures.sh(5-min exponential backoff).--sim.buildarg fixtures=/fixtures, so the sim'sconsumereads a local directory instead of downloading.The
branchbuild-arg (devnetconsume/execution-specs selection) is unchanged.Hive dependency (now merged)
The local-fixtures eels Dockerfiles landed in erigontech/hive#2 (merged into
yperbasis/client-pool). The hiverefis pinned to that merge commit (9afe5f15).Validation
End-to-end verified on a real runner via
workflow_dispatch(run 27152648786): exact-key cache hit (no CDN download), staged tarball picked up by the eels build, andconsume engineran the fullparis+shanghaishard — 3573 tests, 0 failed. The fixtures-directory layout was also confirmed againstexecution-specs/consumelocally (it accepts the extracted dir and recursively builds its index).