ci(windows): hash overlay/triplet/requirements into vcpkg cache key, plus related fixes#5934
Merged
Conversation
The Windows vcpkg cache key was only keyed by matrix.vcpkg-version, so changes to: - thirdparty/vcpkg/ports/** (MeshLib's overlay ports) - thirdparty/vcpkg/triplets/** - requirements/windows.txt - thirdparty/install.bat didn't invalidate the cache. A pre-built C:\vcpkg\installed\ tree was restored verbatim and install.bat's "vcpkg install ..." saw every port as "already installed" (total install time: 3.57 ms in the last PR run), so overlay ports never got consulted. Concrete symptom observed on PR #5932 (zlib → zlib-ng-compat overlay): - Linux-vcpkg (which rebuilds vcpkg inside a Docker image every time `prepare-image / linux-vcpkg-build-upload` runs) picked up zlib-ng; compression tests showed -0.3% zip size and -41 to -45% wall time on Release builds. - Windows vcpkg served the cached pre-zlib-ng tree; zip sizes and test times were indistinguishable from master. Fix: include a hashFiles() of the overlay/triplet/requirements/ install-script content in the cache key. First run after merging this takes the full ~20 min vcpkg install; subsequent runs hit the cache again as before.
Fedr
added a commit
that referenced
this pull request
Apr 20, 2026
Observed in run 24638998286: on Windows, a fresh install of our zlib overlay (triggered by PR #5934 busting the vcpkg cache key) failed with: error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst ... 404 from every msys2 mirror Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on Windows invokes vcpkg_acquire_msys to download the pinned mingw-w64-pkgconf package from msys2 mirrors. That particular version has been pruned from the live mirrors; every one of the six fallback URLs 404s. Deterministic failure that re-runs won't fix. The call is not needed by our consumers. Every MeshLib/vcpkg consumer that depends on zlib finds it via the CMake target `ZLIB::ZLIB` exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat mode - it just doesn't get post-processed. Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup for ZLIB, copyright install) are unaffected.
Fedr
added a commit
that referenced
this pull request
Apr 20, 2026
One-line change to .github/workflows/build-test-windows.yml: append a unique suffix to the vcpkg cache key so the existing cached C:\vcpkg\installed\ tree is not restored. vcpkg reinstalls every port from scratch; upstream blosc's vcpkg_fixup_pkgconfig() tries to fetch mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst from msys2 mirrors, gets 404 on all six, and fails the Windows build. This is the same failure first seen on PR #5932 after PR #5934 (which adds the path-based cache busting in a principled way) exposed it. Master itself doesn't fail because its Windows vcpkg cache is a hot hit every run. Purely a diagnostic PR. Do not merge.
MaxRayskiy
pushed a commit
that referenced
this pull request
Apr 21, 2026
Observed in run 24638998286: on Windows, a fresh install of our zlib overlay (triggered by PR #5934 busting the vcpkg cache key) failed with: error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst ... 404 from every msys2 mirror Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on Windows invokes vcpkg_acquire_msys to download the pinned mingw-w64-pkgconf package from msys2 mirrors. That particular version has been pruned from the live mirrors; every one of the six fallback URLs 404s. Deterministic failure that re-runs won't fix. The call is not needed by our consumers. Every MeshLib/vcpkg consumer that depends on zlib finds it via the CMake target `ZLIB::ZLIB` exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat mode - it just doesn't get post-processed. Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup for ZLIB, copyright install) are unaffected.
MaxRayskiy
pushed a commit
that referenced
this pull request
Apr 21, 2026
Observed in run 24638998286: on Windows, a fresh install of our zlib overlay (triggered by PR #5934 busting the vcpkg cache key) failed with: error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst ... 404 from every msys2 mirror Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on Windows invokes vcpkg_acquire_msys to download the pinned mingw-w64-pkgconf package from msys2 mirrors. That particular version has been pruned from the live mirrors; every one of the six fallback URLs 404s. Deterministic failure that re-runs won't fix. The call is not needed by our consumers. Every MeshLib/vcpkg consumer that depends on zlib finds it via the CMake target `ZLIB::ZLIB` exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat mode - it just doesn't get post-processed. Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup for ZLIB, copyright install) are unaffected.
Fedr
added a commit
that referenced
this pull request
Apr 23, 2026
Resolves conflicts from master's recent work (MRZlib move MRIOExtras→ MRMesh, #5948 adds MRZlibTests.cpp, #5933+#5960 overhaul CompressZip tests, #5957 CUDA cache-key fix, #5722 triplet-keyed vcpkg cache). Conflict resolutions: - .github/workflows/build-test-windows.yml — combined both sides' improvements in the cache key: master-side per-triplet partition (for the x64-windows-meshlib-iterator-debug triplet from #5722) AND branch-side hashFiles(overlay ports / triplets / requirements / install.bat) (from #5934, needed so overlay port changes on this PR bust the cache). Key is now vcpkg-cache-<VERSION>-<TRIPLET>-<HASH>. - source/MRMesh/MRZlib.h — accept master's version (master re-homed the file in #5944; branch had deleted it when it lived briefly in MRIOExtras). - source/MRIOExtras/MRZlib.{h,cpp} — follow master's layout: keep MRIOExtras/MRZlib.h as a forwarding header (#include <MRMesh/MRZlib.h>) and delete MRIOExtras/MRZlib.cpp (the old implementation body, now in MRMesh). - source/MRTest/MRTest.vcxproj — rename ClCompile entry from MRZlib.cpp to MRZlibTests.cpp to match master's #5948 test-file rename. Also drops the duplicate MRZlib.cpp entry. - source/MRTest/MRZlib.cpp — deleted, master renamed to MRZlibTests.cpp in #5948. - source/MRTest/MRZipCompressTests.cpp — accept master's version, which includes the level-0..9 loop (#5960) and the round-trip decompress check (#5958) on top of the branch's earlier simpler version. Branch-side novelty was just the sphere test itself, which is already on master. - source/MRTest/MRZlibTests.cpp — accept master's version, engine-agnostic assertions (tolerance-based compressedSize checks that I relaxed in the zlib-ng PR work). Branch's unique contributions — the actual point of this PR — are preserved: - thirdparty/vcpkg/ports/zlib/{portfile.cmake,vcpkg.json} -- vcpkg overlay port that builds zlib-ng in ZLIB_COMPAT=ON as a drop-in zlib replacement for every port that depends on zlib. - thirdparty/install.bat -- --overlay-ports flag added alongside the existing --overlay-triplets. - thirdparty/vcpkg/downloads/msys2-*.pkg.tar.zst -- pre-staged msys2 packages so vcpkg's pinned 2024.10.21 index doesn't hit stale msys2 mirrors during vcpkg_fixup_pkgconfig().
Fedr
added a commit
that referenced
this pull request
Apr 23, 2026
The master-merge earlier on this branch brought in PR #5722's cache-key format (vcpkg-cache-<VERSION>-<TRIPLET>) but dropped the hashFiles() appendix from #5934. On this PR that matters: the last run (24830925083) had a cache hit on the older key and reused a pre- overlay vcpkg installed tree, so "vcpkg install" saw every port already-installed and built zero of this PR's overlay zlib port. Effect: MRMesh.dll and libzip ended up linked against stock zlib, and the CompressSphereToZip numbers on Windows matched master's stock-zlib baseline (1172-1355 ms Release) instead of the zlib-ng speedup we see on Linux-vcpkg legs in the same run (~450 ms Release). Re-add the hashFiles() over thirdparty/vcpkg/{ports,triplets}/, requirements/windows.txt, and thirdparty/install.bat so a change to any of them busts the cache and forces a fresh install. On this PR specifically, that ensures the new thirdparty/vcpkg/ports/zlib overlay actually gets built and replaces stock zlib with zlib-ng- compat in the cached tree. The fix for the unit-test failure (MRMesh.ZlibCompressStats pinned stats.compressedSize against stock-zlib reference byte counts) came in automatically with the master merge -- master's version of source/MRTest/MRZlibTests.cpp uses engine-agnostic assertions.
Hash only the Windows triplet (x64-windows-meshlib.cmake) instead of every file under thirdparty/vcpkg/triplets/**, since install.bat only ever uses that one triplet on Windows.
…che-key # Conflicts: # .github/workflows/build-test-windows.yml
So the hashFiles(...) expression isn't duplicated between the cache restore and the diagnostic step.
Hoists the format(...) triplet path into a job-level env var so the hashFiles() call references env.VCPKG_TRIPLET_FILE instead of computing the path inline. The Compute Vcpkg Cache Key step now also logs the resolved path and fails fast if the file is missing — a misconfigured matrix.vcpkg_triplet would otherwise silently feed an empty string into hashFiles().
…che-key # Conflicts: # .github/workflows/build-test-windows.yml
Grantim
approved these changes
Apr 30, 2026
MaxRayskiy
approved these changes
Apr 30, 2026
The new cache key (action.yml: hashFiles over overlay ports / triplet file / requirements.txt / install.bat) means the very first run of each (vcpkg-version, triplet, port-content) combination misses actions/cache and falls back to the S3 binary cache. For the msvc-2019 + Release CMake + x64-windows-meshlib slot specifically, the `Enable VS2019 toolset for vcpkg` step modifies the triplet in-place — that changes vcpkg's port-ABI hash, so the S3 cache subspace it queries is sparsely populated and many ports fall through to source builds. On run 25170701978 that cold path was still inside Set up vcpkg at the 70-minute limit. The 150-minute ceiling covers the worst-case: ~75-90 min for a fully cold install.bat + ~55 min for the rest of the job (CUDA cache, MRBind, MeshLib build, tests, archives) measured from the matching msvc-2022 Release CMake sibling. Steady state once S3 has the binaries: ~65 min total, well under both limits.
…out to 70 min Removing the `Enable VS2019 toolset for vcpkg` step. The in-place edit of x64-windows-meshlib.cmake gave msvc-2019 jobs a distinct port-ABI hash from msvc-2022 jobs and produced an S3 binary-cache subspace that's only sparsely populated — surfacing as 60+ minutes of source builds whenever Set up vcpkg runs cold (e.g. first run after this PR's new cache key shape lands). With the step gone, all four matrix slots share one ABI hash and reuse the same S3 binaries. Reverts the timeout bump from eeb3cc9 — no longer needed once the slow cold path is gone. The `#vs2019toolset#` markers in thirdparty/vcpkg/triplets/x64-windows-meshlib.cmake are left in place; msvc-2019 jobs will now build vcpkg ports with the default toolset (v143 from the runner image) and link MeshLib itself against v142.
Reverts part of f59a3b9c. The `Enable VS2019 toolset for vcpkg` step in build-test-windows.yml has to come back, otherwise msvc-2019 jobs compute port-ABI hashes off the unmodified x64-windows-meshlib.cmake and find nothing in S3 (master populates entries with the v142-toolset edit applied). prepare-images.yml had two issues that left S3 sparse for the (2024.10.21, x64-windows-meshlib) combination, surfacing as ~40 min of opencascade source builds and a 70-min timeout on msvc-2019 Release CMake jobs: 1. Missing matrix row: vs2019 only built the iterator-debug triplet, not x64-windows-meshlib. msvc-2019 Debug/Release CMake jobs in build-test-windows.yml use x64-windows-meshlib, so their lookup space had no S3 entries to hit. 2. No toolset edit: even when prepare-image touches x64-windows-meshlib, it built ports against the unmodified triplet, computing v143-derived ABI hashes — which don't match what msvc-2019 build-test-windows jobs compute (v142-derived after the toolset uncomment). This commit: - Adds a third prepare-image row: vs2019 + x64-windows-meshlib. - Adds the same `Enable VS2019 toolset for vcpkg` step inside prepare-image, gated on `matrix.vs == 'vs2019'`, so both vs2019 rows build with the v142 toolset and produce S3 entries with ABI hashes that match what build-test-windows actually queries.
MaxRayskiy
approved these changes
Apr 30, 2026
Previously the `Enable VS2019 toolset for vcpkg` step lived inline in
build-test-windows.yml, gated on `matrix.cxx_compiler == 'msvc-2019'`.
Move it inside the composite action behind a new `enable-vs2019-toolset`
input (string boolean). The step runs *before* `Compute Vcpkg Cache
Key` so the modified triplet content is included in the cache-key hash,
preserving the same actions/cache and vcpkg-port-ABI semantics as the
inline version.
Caller becomes a one-liner: `enable-vs2019-toolset: ${{ matrix.cxx_compiler == 'msvc-2019' }}`.
prepare-images.yml and pip-build.yml don't pass the flag — default
'false' preserves existing behavior there. (prepare-images.yml has its
own inline copy of this step for the workflows that don't use the
action.)
Skipping bootstrap-vcpkg.bat (the effect of reuse-vcpkg: 'true') relies on the runner-image's pre-installed vcpkg.exe being ABI-compatible with the checked-out tag. The windows-2025 image's vcpkg.exe is now newer than tag 2024.10.21 expects: it tries to read scripts/vcpkg-tools.json (added to vcpkg post-2024.10.21) and exits 1 because the older source tree doesn't ship that file. Run the full setup — clean, pull, bootstrap-vcpkg.bat, integrate — so vcpkg.exe is rebuilt from the same 2024.10.21 source tree it then reads config from. ~30s slower per run, only fires on the release path.
- Remove the `reuse-vcpkg` input. After the previous commit dropped it from update-win-version.yml's call site, no caller used it. The action now always runs clean/fetch/checkout/bootstrap/integrate. - Skip `git fetch` entirely when the requested vcpkg-version tag is already in the runner's local vcpkg clone (typical case — the windows-2025 runner image ships a full microsoft/vcpkg checkout including all tags up to image-build time). Saves the ~13 s spent fetching master + new tags that we don't checkout anyway. - When the tag isn't local, fetch only that ref with `--depth=1` instead of fetching everything. - `--quiet` on git clean / fetch / checkout. Stops git from spamming the log with hundreds of "Updating files: NN%" lines during the big master→2024.10.21 working-tree rewrite. update-win-version.yml: pass `enable-vs2019-toolset: 'true'`. The job uses vcpkg-version 2024.10.21 + x64-windows-meshlib, and the matching S3 binary-cache subspace is populated by `prepare-image` runs that apply the v142 toolset edit. Without this flag, vcpkg computed v143-derived port-ABI hashes that didn't match S3, falling through to ~40-min source builds (e.g. opencascade) and hitting the 60-min timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
setup-vcpkg-windowscomposite action keys the vcpkg cache only onvcpkg-version+vcpkg-triplet:Changes to any of these don't invalidate the cache:
thirdparty/vcpkg/ports/**— MeshLib's overlay ports (customopencascade-minimal,laz-perf,openctm,clip, …)thirdparty/vcpkg/triplets/<triplet>.cmake— the active custom triplet passed via--overlay-tripletsrequirements/windows.txt— the package listinstall.batfeeds to vcpkgthirdparty/install.bat— changes to how vcpkg is invokedA pre-built
C:\vcpkg\installed\tree gets restored verbatim;install.bat'svcpkg install ...step sees every transitively-required port as already installed and completes in a few milliseconds without consulting overlay ports. Any PR that introduces a new overlay port, edits an existing one, or tweaks the triplet won't actually exercise its change on Windows until the cache happens to be rebuilt for an unrelated reason.Linux-vcpkg rebuilds vcpkg from scratch inside the
prepare-image / linux-vcpkg-build-uploadDocker image each time, so it doesn't hit this cache-key problem. Windows did.Fix
In
setup-vcpkg-windows/action.yml, add aCompute Vcpkg Cache Keystep that hashes the relevant paths and exposes the result asVCPKG_CACHE_KEY:Only the active triplet file is hashed (via
format()), not all files underthirdparty/vcpkg/triplets/— changes to a triplet that this job doesn't use should not bust its cache. The compute step alsotest -fs the resolved triplet path and fails fast with a::error::annotation if missing — a misconfiguredvcpkg-tripletwould otherwise silently feed an empty string intohashFiles.All three callers of
setup-vcpkg-windows(build-test-windows.yml,pip-build.ymlwindows-pip-build,update-win-version.yml) inherit the new behavior automatically. Steady-state unchanged: the same branch run twice in a row still hits the cache.Also in this PR
While auditing the cache flow a few related issues surfaced and got fixed in the same change:
build-test-windows.yml(as a separate step run before the action call). Now anenable-vs2019-toolsetinput on the action; gated onmatrix.cxx_compiler == 'msvc-2019'from the build-test-windows caller and pinned to'true'fromupdate-win-version.yml. Keeps the v142-marker uncomment hashed into the cache key.prepare-imagematrix expanded. Thewindows-vcpkg-build-uploadmatrix only builtvs2019 + x64-windows-meshlib-iterator-debug, leavings3://vcpkg-export/2024-10-18/x64-windows-meshlib/empty for builds against the regular triplet. Added a third row (vs2019 + x64-windows-meshlib) and the matchingEnable VS2019 toolset for vcpkgstep (gated onmatrix.vs == 'vs2019'), so msvc-2019 jobs now find pre-built v142 ABI binaries in S3 instead of source-building opencascade for ~40 min on every cold lookup.reuse-vcpkginput removed. Was a no-op for every caller afterupdate-win-version.ymlstopped passing it. Action now always runs clean → fetch → checkout → bootstrap → integrate. Skipping bootstrap was unsound: the runner image's pre-installedvcpkg.exeis newer than tag2024.10.21and readsscripts/vcpkg-tools.json, a file that the older source tree doesn't ship — causedread_contents failed with 2 (no such file or directory)on theupdate-win-versionpath.git fetch+ cleaner log. Skipgit fetchentirely when the requested vcpkg ref is already in the local clone (the windows-2025 runner image ships microsoft/vcpkg with all old tags); fall back togit fetch --depth=1 origin <tag>otherwise — only the requested ref, not all of master + new tags. Adds--quietto git clean / fetch / checkout to suppress the ~900 lines ofUpdating files: NN%progress on themaster → 2024.10.21working-tree rewrite.Cost
First run after this lands does a one-time vcpkg rebuild (~20 min per
(vcpkg-version, triplet, hashFiles)combination, runs in parallel) to populateactions/cacheunder the new key shape. Subsequent runs hit the cache as before. Every future overlay-affecting PR pays a one-time rebuild on first push, which is the correct behavior.Set up vcpkgstep on a steady-state cache-hit run, comparing equivalent matrix slots against master:Three of four slots improve by ~25–40 s; the fourth is within normal CI variance for cache-restore + S3 throughput.
Not in this PR
Same problem may or may not exist on other cached steps (CUDA cache, test-data cache). Only the vcpkg cache is audited here; no similar-shaped problems found on other caches in passing, but no exhaustive review.
Rollback
The change is contained to four files:
.github/actions/setup-vcpkg-windows/action.yml,.github/workflows/build-test-windows.yml,.github/workflows/prepare-images.yml,.github/workflows/update-win-version.yml. Revert restores the old key shape; cache entries under the new key become orphans and expire naturally.