Skip to content

test(e2e): the dead-peers ceiling asks a starvation witness before blaming the launcher - #1740

Merged
christianparpart merged 1 commit into
masterfrom
fix/1739-dead-peer-ceiling
Oct 29, 2026
Merged

christianparpart merged 1 commit into
masterfrom
fix/1739-dead-peer-ceiling

Conversation

@christianparpart

Copy link
Copy Markdown
Member

Closes #1739.

Why

The dead-peers never-accepting leg holds the launcher's reported cache time to a ceiling of (2N - 1/2) x TOTAL, which is 7000 ms. On loaded Windows runners it reported 7602 ms (in #1737) and 8132 ms (on master). 8132 is past the 8000 ms that a launcher which doubled its deadline reports, so time alone cannot tell a stalled host from a real regression. Raising the ceiling would stop it catching a doubled deadline.

What

  • A starvation witness runs beside the leg in both fixtures that hold this ceiling:
    • compile-cache-e2e.sh (POSIX) uses a perl ticker;
    • run-launcher-e2e.ps1 (which is what ctest runs on Windows) uses a Stopwatch ticker on its own runspace.
  • The witness counts only stalls. It requests 20 ms sleeps and sums only the part of each wake-up that was more than 50 ms late. Raw lateness measures the timer instead: on an idle host, Git Bash's perl is late by 2.5 s over 8 s and pwsh by about 3 s, either of which would excuse a doubled deadline. The stalled figure measured 0 under perl, WSL and pwsh.
  • It watches only the cache phase. The witness stops at the launcher's fall-back line, so a stall during the compile that follows is not counted.
  • The verdict is a pure function with three outcomes, e2e_dead_ceiling_verdict in bash and Get-E2EDeadCeilingVerdict in pwsh:
    • pass: the reported time is within the ceiling;
    • inconclusive: the host stalled at least as long as the overrun. This is printed, raised as a ::warning:: annotation, and repeated at the end of the run. The run does not fail;
    • fail: the host did not stall that long, or the witness left no reading. A missing reading fails closed.
  • Self-tests: the bash verdict has cases in e2e-helpers-selftest, and the pwsh verdict and witness have the new launcher-e2e-starvation-witness-selftest. A pwsh case asserts that the 20 ms and 50 ms constants match across the two languages.

Blind spots (stated in both files)

  • A host that starves the launcher but not the witness fails OPEN.
  • Starvation in slices shorter than 50 ms counts nothing, so it fails CLOSED. The noise is reduced, not eliminated.
  • ctest shows output only on failure, so an inconclusive warning on a passing run reaches GitHub only from the jobs that run the fixtures directly.

Tested

  • Self-tests:
    • bash: check-e2e-helpers.sh ran 1147 checks, 0 failed (WSL);
    • pwsh: the witness self-test ran 32 cases, 0 failed, and -SelfTestPorts ran 47.
  • Each guard bites: each was disabled in turn, and only its own case went red.
  • End-to-end: run-launcher-e2e.ps1 ran on Windows (cl and clang-cl). The never-accepting leg passed with 0 ms stalled, and the witness closed at the end of the cache phase.

…aming the launcher (#1739)

A never-accepting leg over its (2N - 1/2) x TOTAL ceiling cannot say whether the
host stalled or the launcher doubled its deadline: 8132 ms was reported on a loaded
runner, past the 8000 ms a doubled deadline reports. The ceiling stays as it is.

Both fixtures that hold that ceiling now run a starvation witness beside the leg:
compile-cache-e2e.sh a perl ticker (e2e_starvation_witness in e2e-common.sh), and
run-launcher-e2e.ps1 a Stopwatch ticker on a runspace of its own
(scripts/lib/E2EStarvationWitness.psm1). Each requests 20 ms sleeps and sums only
how far a wake-up was late by MORE than a 50 ms stall threshold. Raw lateness would
measure the timer instead: on an idle host Git Bash's perl is late 2546-2598 ms
over 8 s and pwsh 2924-3221 ms, either of which would excuse a doubled deadline
outright; the stalled figure is 0 under perl, WSL and pwsh. The witness stops once
the launcher prints its fall-back line, so its window is the cache phase rather
than the compile after it.

The pure verdict (e2e_dead_ceiling_verdict, Get-E2EDeadCeilingVerdict) answers one
of three outcomes: pass (within the ceiling), inconclusive (stalled >= overrun:
printed, annotated on CI, repeated at the run's end, not failed) and fail (a host
that did not stall that long, or a witness that left no reading, which fails
closed). The PowerShell verdict and witness have their own cases, registered as
launcher-e2e-starvation-witness-selftest.

Closes #1739.

Signed-off-by: Christian Parpart <c.parpart@lastrada.net>
@christianparpart christianparpart added the type/test Coverage gaps, flaky or unbuilt test targets, e2e fixtures label Oct 29, 2026
@github-actions github-actions Bot added area/launcher apps/fastcache-cc/ - the compiler launcher and its cache key area/compile-cache CompileCache/, the 0xFC wire, the executor, PathCanon os/windows Specific to Windows (MSVC, clang-cl, Winsock, IOCP) labels Oct 29, 2026
@christianparpart
christianparpart merged commit 53b0e53 into master Oct 29, 2026
29 of 32 checks passed
@christianparpart
christianparpart deleted the fix/1739-dead-peer-ceiling branch October 29, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compile-cache CompileCache/, the 0xFC wire, the executor, PathCanon area/launcher apps/fastcache-cc/ - the compiler launcher and its cache key os/windows Specific to Windows (MSVC, clang-cl, Winsock, IOCP) type/test Coverage gaps, flaky or unbuilt test targets, e2e fixtures

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compile-cache-e2e dead-peers 'never-accepting' timing ceiling overruns on Windows release runners

1 participant