Skip to content

ci(windows): hash overlay/triplet/requirements into vcpkg cache key, plus related fixes#5934

Merged
Fedr merged 13 commits into
masterfrom
ci/windows-vcpkg-cache-key
May 1, 2026
Merged

ci(windows): hash overlay/triplet/requirements into vcpkg cache key, plus related fixes#5934
Fedr merged 13 commits into
masterfrom
ci/windows-vcpkg-cache-key

Conversation

@Fedr
Copy link
Copy Markdown
Contributor

@Fedr Fedr commented Apr 19, 2026

Problem

The setup-vcpkg-windows composite action keys the vcpkg cache only on vcpkg-version + vcpkg-triplet:

key: vcpkg-cache-${{ inputs.vcpkg-version }}-${{ inputs.vcpkg-triplet }}

Changes to any of these don't invalidate the cache:

  • thirdparty/vcpkg/ports/** — MeshLib's overlay ports (custom opencascade-minimal, laz-perf, openctm, clip, …)
  • thirdparty/vcpkg/triplets/<triplet>.cmake — the active custom triplet passed via --overlay-triplets
  • requirements/windows.txt — the package list install.bat feeds to vcpkg
  • thirdparty/install.bat — changes to how vcpkg is invoked

A pre-built C:\vcpkg\installed\ tree gets restored verbatim; install.bat's vcpkg install ... step sees every transitively-required port as already installed and completes in a few milliseconds without consulting overlay ports. Any PR that introduces a new overlay port, edits an existing one, or tweaks the triplet won't actually exercise its change on Windows until the cache happens to be rebuilt for an unrelated reason.

Linux-vcpkg rebuilds vcpkg from scratch inside the prepare-image / linux-vcpkg-build-upload Docker image each time, so it doesn't hit this cache-key problem. Windows did.

Fix

In setup-vcpkg-windows/action.yml, add a Compute Vcpkg Cache Key step that hashes the relevant paths and exposes the result as VCPKG_CACHE_KEY:

- name: Compute Vcpkg Cache Key
  shell: bash
  run: |
    triplet_file="thirdparty/vcpkg/triplets/${{ inputs.vcpkg-triplet }}.cmake"
    echo "Triplet file: $triplet_file"
    test -f "$triplet_file" || { echo "::error::Triplet file not found: $triplet_file"; exit 1; }
    echo "VCPKG_CACHE_KEY=vcpkg-cache-${{ inputs.vcpkg-version }}-${{ inputs.vcpkg-triplet }}-${{ hashFiles('thirdparty/vcpkg/ports/**', format('thirdparty/vcpkg/triplets/{0}.cmake', inputs.vcpkg-triplet), 'requirements/windows.txt', 'thirdparty/install.bat') }}" >> "$GITHUB_ENV"

- name: Restore Vcpkg Cache
  uses: actions/cache@v5
  with:
    key: ${{ env.VCPKG_CACHE_KEY }}
    ...

Only the active triplet file is hashed (via format()), not all files under thirdparty/vcpkg/triplets/ — changes to a triplet that this job doesn't use should not bust its cache. The compute step also test -fs the resolved triplet path and fails fast with a ::error:: annotation if missing — a misconfigured vcpkg-triplet would otherwise silently feed an empty string into hashFiles.

All three callers of setup-vcpkg-windows (build-test-windows.yml, pip-build.yml windows-pip-build, update-win-version.yml) inherit the new behavior automatically. Steady-state unchanged: the same branch run twice in a row still hits the cache.

Also in this PR

While auditing the cache flow a few related issues surfaced and got fixed in the same change:

  • VS2019 toolset edit moved into the action. Previously inline in build-test-windows.yml (as a separate step run before the action call). Now an enable-vs2019-toolset input on the action; gated on matrix.cxx_compiler == 'msvc-2019' from the build-test-windows caller and pinned to 'true' from update-win-version.yml. Keeps the v142-marker uncomment hashed into the cache key.
  • prepare-image matrix expanded. The windows-vcpkg-build-upload matrix only built vs2019 + x64-windows-meshlib-iterator-debug, leaving s3://vcpkg-export/2024-10-18/x64-windows-meshlib/ empty for builds against the regular triplet. Added a third row (vs2019 + x64-windows-meshlib) and the matching Enable VS2019 toolset for vcpkg step (gated on matrix.vs == 'vs2019'), so msvc-2019 jobs now find pre-built v142 ABI binaries in S3 instead of source-building opencascade for ~40 min on every cold lookup.
  • reuse-vcpkg input removed. Was a no-op for every caller after update-win-version.yml stopped passing it. Action now always runs clean → fetch → checkout → bootstrap → integrate. Skipping bootstrap was unsound: the runner image's pre-installed vcpkg.exe is newer than tag 2024.10.21 and reads scripts/vcpkg-tools.json, a file that the older source tree doesn't ship — caused read_contents failed with 2 (no such file or directory) on the update-win-version path.
  • Faster git fetch + cleaner log. Skip git fetch entirely when the requested vcpkg ref is already in the local clone (the windows-2025 runner image ships microsoft/vcpkg with all old tags); fall back to git fetch --depth=1 origin <tag> otherwise — only the requested ref, not all of master + new tags. Adds --quiet to git clean / fetch / checkout to suppress the ~900 lines of Updating files: NN% progress on the master → 2024.10.21 working-tree rewrite.

Cost

First run after this lands does a one-time vcpkg rebuild (~20 min per (vcpkg-version, triplet, hashFiles) combination, runs in parallel) to populate actions/cache under the new key shape. Subsequent runs hit the cache as before. Every future overlay-affecting PR pays a one-time rebuild on first push, which is the correct behavior.

Set up vcpkg step on a steady-state cache-hit run, comparing equivalent matrix slots against master:

matrix slot master this PR Δ
msvc-2019 Release CMake, x64-windows-meshlib 3:29 2:48 −0:41
msvc-2019 Debug CMake, iterator-debug 2:49 2:23 −0:26
msvc-2022 Debug MSBuild 3:00 2:28 −0:32
msvc-2022 Release CMake 2:38 3:06 +0:28

Three of four slots improve by ~25–40 s; the fourth is within normal CI variance for cache-restore + S3 throughput.

Not in this PR

Same problem may or may not exist on other cached steps (CUDA cache, test-data cache). Only the vcpkg cache is audited here; no similar-shaped problems found on other caches in passing, but no exhaustive review.

Rollback

The change is contained to four files: .github/actions/setup-vcpkg-windows/action.yml, .github/workflows/build-test-windows.yml, .github/workflows/prepare-images.yml, .github/workflows/update-win-version.yml. Revert restores the old key shape; cache entries under the new key become orphans and expire naturally.

The Windows vcpkg cache key was only keyed by matrix.vcpkg-version,
so changes to:
  - thirdparty/vcpkg/ports/**   (MeshLib's overlay ports)
  - thirdparty/vcpkg/triplets/**
  - requirements/windows.txt
  - thirdparty/install.bat
didn't invalidate the cache. A pre-built C:\vcpkg\installed\ tree
was restored verbatim and install.bat's "vcpkg install ..." saw
every port as "already installed" (total install time: 3.57 ms in
the last PR run), so overlay ports never got consulted.

Concrete symptom observed on PR #5932 (zlib → zlib-ng-compat overlay):
  - Linux-vcpkg (which rebuilds vcpkg inside a Docker image every
    time `prepare-image / linux-vcpkg-build-upload` runs) picked up
    zlib-ng; compression tests showed -0.3% zip size and -41 to -45%
    wall time on Release builds.
  - Windows vcpkg served the cached pre-zlib-ng tree; zip sizes and
    test times were indistinguishable from master.

Fix: include a hashFiles() of the overlay/triplet/requirements/
install-script content in the cache key. First run after merging
this takes the full ~20 min vcpkg install; subsequent runs hit the
cache again as before.
Fedr added a commit that referenced this pull request Apr 20, 2026
Observed in run 24638998286: on Windows, a fresh install of our
zlib overlay (triggered by PR #5934 busting the vcpkg cache key)
failed with:

  error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED
  Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst
  ... 404 from every msys2 mirror

Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on
Windows invokes vcpkg_acquire_msys to download the pinned
mingw-w64-pkgconf package from msys2 mirrors. That particular version
has been pruned from the live mirrors; every one of the six fallback
URLs 404s. Deterministic failure that re-runs won't fix.

The call is not needed by our consumers. Every MeshLib/vcpkg consumer
that depends on zlib finds it via the CMake target `ZLIB::ZLIB`
exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads
lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated
by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat
mode - it just doesn't get post-processed.

Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat
build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup
for ZLIB, copyright install) are unaffected.
Fedr added a commit that referenced this pull request Apr 20, 2026
One-line change to .github/workflows/build-test-windows.yml: append a
unique suffix to the vcpkg cache key so the existing cached
C:\vcpkg\installed\ tree is not restored. vcpkg reinstalls every
port from scratch; upstream blosc's vcpkg_fixup_pkgconfig() tries to
fetch mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst from msys2
mirrors, gets 404 on all six, and fails the Windows build.

This is the same failure first seen on PR #5932 after PR #5934 (which
adds the path-based cache busting in a principled way) exposed it.
Master itself doesn't fail because its Windows vcpkg cache is a hot
hit every run.

Purely a diagnostic PR. Do not merge.
MaxRayskiy pushed a commit that referenced this pull request Apr 21, 2026
Observed in run 24638998286: on Windows, a fresh install of our
zlib overlay (triggered by PR #5934 busting the vcpkg cache key)
failed with:

  error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED
  Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst
  ... 404 from every msys2 mirror

Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on
Windows invokes vcpkg_acquire_msys to download the pinned
mingw-w64-pkgconf package from msys2 mirrors. That particular version
has been pruned from the live mirrors; every one of the six fallback
URLs 404s. Deterministic failure that re-runs won't fix.

The call is not needed by our consumers. Every MeshLib/vcpkg consumer
that depends on zlib finds it via the CMake target `ZLIB::ZLIB`
exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads
lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated
by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat
mode - it just doesn't get post-processed.

Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat
build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup
for ZLIB, copyright install) are unaffected.
MaxRayskiy pushed a commit that referenced this pull request Apr 21, 2026
Observed in run 24638998286: on Windows, a fresh install of our
zlib overlay (triggered by PR #5934 busting the vcpkg cache key)
failed with:

  error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED
  Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst
  ... 404 from every msys2 mirror

Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on
Windows invokes vcpkg_acquire_msys to download the pinned
mingw-w64-pkgconf package from msys2 mirrors. That particular version
has been pruned from the live mirrors; every one of the six fallback
URLs 404s. Deterministic failure that re-runs won't fix.

The call is not needed by our consumers. Every MeshLib/vcpkg consumer
that depends on zlib finds it via the CMake target `ZLIB::ZLIB`
exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads
lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated
by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat
mode - it just doesn't get post-processed.

Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat
build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup
for ZLIB, copyright install) are unaffected.
Fedr added a commit that referenced this pull request Apr 23, 2026
Resolves conflicts from master's recent work (MRZlib move MRIOExtras→
MRMesh, #5948 adds MRZlibTests.cpp, #5933+#5960 overhaul CompressZip
tests, #5957 CUDA cache-key fix, #5722 triplet-keyed vcpkg cache).

Conflict resolutions:

  - .github/workflows/build-test-windows.yml — combined both sides'
    improvements in the cache key: master-side per-triplet partition
    (for the x64-windows-meshlib-iterator-debug triplet from #5722)
    AND branch-side hashFiles(overlay ports / triplets / requirements
    / install.bat) (from #5934, needed so overlay port changes on this
    PR bust the cache). Key is now
    vcpkg-cache-<VERSION>-<TRIPLET>-<HASH>.

  - source/MRMesh/MRZlib.h — accept master's version (master re-homed
    the file in #5944; branch had deleted it when it lived briefly in
    MRIOExtras).

  - source/MRIOExtras/MRZlib.{h,cpp} — follow master's layout: keep
    MRIOExtras/MRZlib.h as a forwarding header (#include
    <MRMesh/MRZlib.h>) and delete MRIOExtras/MRZlib.cpp (the old
    implementation body, now in MRMesh).

  - source/MRTest/MRTest.vcxproj — rename ClCompile entry from
    MRZlib.cpp to MRZlibTests.cpp to match master's #5948 test-file
    rename. Also drops the duplicate MRZlib.cpp entry.

  - source/MRTest/MRZlib.cpp — deleted, master renamed to
    MRZlibTests.cpp in #5948.

  - source/MRTest/MRZipCompressTests.cpp — accept master's version,
    which includes the level-0..9 loop (#5960) and the round-trip
    decompress check (#5958) on top of the branch's earlier simpler
    version. Branch-side novelty was just the sphere test itself,
    which is already on master.

  - source/MRTest/MRZlibTests.cpp — accept master's version,
    engine-agnostic assertions (tolerance-based compressedSize checks
    that I relaxed in the zlib-ng PR work).

Branch's unique contributions — the actual point of this PR — are
preserved:

  - thirdparty/vcpkg/ports/zlib/{portfile.cmake,vcpkg.json}  -- vcpkg
    overlay port that builds zlib-ng in ZLIB_COMPAT=ON as a drop-in
    zlib replacement for every port that depends on zlib.
  - thirdparty/install.bat  -- --overlay-ports flag added alongside
    the existing --overlay-triplets.
  - thirdparty/vcpkg/downloads/msys2-*.pkg.tar.zst  -- pre-staged
    msys2 packages so vcpkg's pinned 2024.10.21 index doesn't hit
    stale msys2 mirrors during vcpkg_fixup_pkgconfig().
Fedr added a commit that referenced this pull request Apr 23, 2026
The master-merge earlier on this branch brought in PR #5722's
cache-key format (vcpkg-cache-<VERSION>-<TRIPLET>) but dropped the
hashFiles() appendix from #5934. On this PR that matters: the last
run (24830925083) had a cache hit on the older key and reused a pre-
overlay vcpkg installed tree, so "vcpkg install" saw every port
already-installed and built zero of this PR's overlay zlib port.
Effect: MRMesh.dll and libzip ended up linked against stock zlib,
and the CompressSphereToZip numbers on Windows matched master's
stock-zlib baseline (1172-1355 ms Release) instead of the zlib-ng
speedup we see on Linux-vcpkg legs in the same run (~450 ms Release).

Re-add the hashFiles() over thirdparty/vcpkg/{ports,triplets}/,
requirements/windows.txt, and thirdparty/install.bat so a change to
any of them busts the cache and forces a fresh install. On this PR
specifically, that ensures the new thirdparty/vcpkg/ports/zlib
overlay actually gets built and replaces stock zlib with zlib-ng-
compat in the cached tree.

The fix for the unit-test failure (MRMesh.ZlibCompressStats pinned
stats.compressedSize against stock-zlib reference byte counts) came
in automatically with the master merge -- master's version of
source/MRTest/MRZlibTests.cpp uses engine-agnostic assertions.
Hash only the Windows triplet (x64-windows-meshlib.cmake) instead of
every file under thirdparty/vcpkg/triplets/**, since install.bat only
ever uses that one triplet on Windows.
Fedr added 4 commits April 29, 2026 23:46
…che-key

# Conflicts:
#	.github/workflows/build-test-windows.yml
So the hashFiles(...) expression isn't duplicated between the cache
restore and the diagnostic step.
Hoists the format(...) triplet path into a job-level env var so the
hashFiles() call references env.VCPKG_TRIPLET_FILE instead of computing
the path inline. The Compute Vcpkg Cache Key step now also logs the
resolved path and fails fast if the file is missing — a misconfigured
matrix.vcpkg_triplet would otherwise silently feed an empty string into
hashFiles().
…che-key

# Conflicts:
#	.github/workflows/build-test-windows.yml
@Fedr Fedr changed the title ci(windows): bust vcpkg cache on overlay/triplet/requirements changes ci(windows): bust vcpkg cache when overlay ports / triplet / requirements change Apr 30, 2026
@Fedr Fedr requested a review from Grantim April 30, 2026 14:36
Fedr added 3 commits April 30, 2026 21:33
The new cache key (action.yml: hashFiles over overlay ports / triplet
file / requirements.txt / install.bat) means the very first run of
each (vcpkg-version, triplet, port-content) combination misses
actions/cache and falls back to the S3 binary cache. For the
msvc-2019 + Release CMake + x64-windows-meshlib slot specifically,
the `Enable VS2019 toolset for vcpkg` step modifies the triplet
in-place — that changes vcpkg's port-ABI hash, so the S3 cache
subspace it queries is sparsely populated and many ports fall
through to source builds. On run 25170701978 that cold path was
still inside Set up vcpkg at the 70-minute limit.

The 150-minute ceiling covers the worst-case: ~75-90 min for a fully
cold install.bat + ~55 min for the rest of the job (CUDA cache,
MRBind, MeshLib build, tests, archives) measured from the
matching msvc-2022 Release CMake sibling. Steady state once S3 has
the binaries: ~65 min total, well under both limits.
…out to 70 min

Removing the `Enable VS2019 toolset for vcpkg` step. The in-place edit
of x64-windows-meshlib.cmake gave msvc-2019 jobs a distinct port-ABI
hash from msvc-2022 jobs and produced an S3 binary-cache subspace
that's only sparsely populated — surfacing as 60+ minutes of source
builds whenever Set up vcpkg runs cold (e.g. first run after this PR's
new cache key shape lands). With the step gone, all four matrix slots
share one ABI hash and reuse the same S3 binaries.

Reverts the timeout bump from eeb3cc9 — no longer needed once
the slow cold path is gone.

The `#vs2019toolset#` markers in thirdparty/vcpkg/triplets/x64-windows-meshlib.cmake
are left in place; msvc-2019 jobs will now build vcpkg ports with the
default toolset (v143 from the runner image) and link MeshLib itself
against v142.
Reverts part of f59a3b9c. The `Enable VS2019 toolset for vcpkg` step
in build-test-windows.yml has to come back, otherwise msvc-2019 jobs
compute port-ABI hashes off the unmodified x64-windows-meshlib.cmake
and find nothing in S3 (master populates entries with the v142-toolset
edit applied).

prepare-images.yml had two issues that left S3 sparse for the
(2024.10.21, x64-windows-meshlib) combination, surfacing as ~40 min of
opencascade source builds and a 70-min timeout on msvc-2019 Release
CMake jobs:

1. Missing matrix row: vs2019 only built the iterator-debug triplet,
   not x64-windows-meshlib. msvc-2019 Debug/Release CMake jobs in
   build-test-windows.yml use x64-windows-meshlib, so their lookup
   space had no S3 entries to hit.

2. No toolset edit: even when prepare-image touches x64-windows-meshlib,
   it built ports against the unmodified triplet, computing v143-derived
   ABI hashes — which don't match what msvc-2019 build-test-windows
   jobs compute (v142-derived after the toolset uncomment).

This commit:
- Adds a third prepare-image row: vs2019 + x64-windows-meshlib.
- Adds the same `Enable VS2019 toolset for vcpkg` step inside
  prepare-image, gated on `matrix.vs == 'vs2019'`, so both vs2019 rows
  build with the v142 toolset and produce S3 entries with ABI hashes
  that match what build-test-windows actually queries.
@Fedr Fedr added full-ci run all steps test-pip-build Build Python wheels (and discard them) labels May 1, 2026
Fedr added 2 commits May 1, 2026 10:22
Previously the `Enable VS2019 toolset for vcpkg` step lived inline in
build-test-windows.yml, gated on `matrix.cxx_compiler == 'msvc-2019'`.
Move it inside the composite action behind a new `enable-vs2019-toolset`
input (string boolean). The step runs *before* `Compute Vcpkg Cache
Key` so the modified triplet content is included in the cache-key hash,
preserving the same actions/cache and vcpkg-port-ABI semantics as the
inline version.

Caller becomes a one-liner: `enable-vs2019-toolset: ${{ matrix.cxx_compiler == 'msvc-2019' }}`.

prepare-images.yml and pip-build.yml don't pass the flag — default
'false' preserves existing behavior there. (prepare-images.yml has its
own inline copy of this step for the workflows that don't use the
action.)
@Fedr Fedr removed the bump-vcpkg label May 1, 2026
Fedr added 2 commits May 1, 2026 17:56
Skipping bootstrap-vcpkg.bat (the effect of reuse-vcpkg: 'true')
relies on the runner-image's pre-installed vcpkg.exe being
ABI-compatible with the checked-out tag. The windows-2025 image's
vcpkg.exe is now newer than tag 2024.10.21 expects: it tries to read
scripts/vcpkg-tools.json (added to vcpkg post-2024.10.21) and exits 1
because the older source tree doesn't ship that file.

Run the full setup — clean, pull, bootstrap-vcpkg.bat, integrate —
so vcpkg.exe is rebuilt from the same 2024.10.21 source tree it then
reads config from. ~30s slower per run, only fires on the release
path.
- Remove the `reuse-vcpkg` input. After the previous commit dropped
  it from update-win-version.yml's call site, no caller used it. The
  action now always runs clean/fetch/checkout/bootstrap/integrate.
- Skip `git fetch` entirely when the requested vcpkg-version tag is
  already in the runner's local vcpkg clone (typical case — the
  windows-2025 runner image ships a full microsoft/vcpkg checkout
  including all tags up to image-build time). Saves the ~13 s spent
  fetching master + new tags that we don't checkout anyway.
- When the tag isn't local, fetch only that ref with `--depth=1`
  instead of fetching everything.
- `--quiet` on git clean / fetch / checkout. Stops git from spamming
  the log with hundreds of "Updating files: NN%" lines during the big
  master→2024.10.21 working-tree rewrite.

update-win-version.yml: pass `enable-vs2019-toolset: 'true'`. The job
uses vcpkg-version 2024.10.21 + x64-windows-meshlib, and the matching
S3 binary-cache subspace is populated by `prepare-image` runs that
apply the v142 toolset edit. Without this flag, vcpkg computed
v143-derived port-ABI hashes that didn't match S3, falling through to
~40-min source builds (e.g. opencascade) and hitting the 60-min
timeout.
@Fedr Fedr changed the title ci(windows): bust vcpkg cache when overlay ports / triplet / requirements change ci(windows): hash overlay/triplet/requirements into vcpkg cache key, plus related fixes May 1, 2026
@Fedr Fedr merged commit 5f0ded7 into master May 1, 2026
78 checks passed
@Fedr Fedr deleted the ci/windows-vcpkg-cache-key branch May 1, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants