Repository navigation
compile-cache-e2e dead-peers 'never-accepting' timing ceiling overruns on Windows release runners #1739
Description
compile-cache-e2e's "dead peers" case fails intermittently on the Windows release legs. Its never-accepting timing ceiling is exceeded by 0.6 to 1.1 s:
| Run | Branch | Leg | Reported | Ceiling |
|---|---|---|---|---|
| 39862992304 (job 120118169777) | master | Windows-cl-release, clang-cl | 8132 ms | 7000 ms |
| 40753758521 (job 123047155713) | feature/1728-fleet-compile-queue (#1737) | Windows-cl-release, cl | 7602 ms | 7000 ms |
The ceiling is (2N - 1/2) x deadline for N=2 exchanges (scripts/compile-cache-e2e.sh ~line 900–950). A rerun passed both times. The PR run had passed on that branch's two previous heads, which do not touch the cache dial.
Measured: two overruns, both on Windows GitHub runners, both on the never-accepting peer. Each reported 2 traced exchanges over 2 connections.
Inferred, not measured: a slow runner charges the time to the exchange deadline. In the master run, ran out of budget on the manifest fetch suggests that the per-exchange total, not the connect, is what stretches. Whether that is timer granularity, scheduler delay or a real overrun in the budget accounting is undetermined.
To decide:
- Does the launcher overrun its own configured deadline, which would be a defect?
- Or is the ceiling too tight for a loaded Windows runner? Per the testing rules, the fix is then to judge against something a slow host cannot fake, not to raise the number.
Per testing.md, run scripts/flake-rate.sh before deciding.
Activity
christianparpart commentedon Oct 28, 2026
Analysis against
scripts/compile-cache-e2e.sh§8b, the comment abovedead_connect_ms:(2N - 1/2) x TOTAL= 7000 ms. It tolerates a host stall of up to 3000 ms past the floor of N x TOTAL = 4000 ms.So this is the case the rules describe: a time bound that measures the HOST, as rounds 8 and 9 already found on the other legs. The other legs moved from time to a COUNT for this reason.
Raising the 7000 is not the fix. It would stop catching a doubled TOTAL. Candidate directions, each still needs a decision:
direct-ms + cache-ms. A stall between exchanges would then not count against either one.Not a regression from #1737: it fails on master (run 39862992304), and #1737 does not touch the cache dial.