dist-compile-e2e: worker and launcher once disagreed on g++'s toolchain fingerprint #1727

dist-compile-e2e failed once in a local gate run on the gate-gcc-release leg, at the toolchain-fingerprint agreement check, before any compile case ran. It did not reproduce afterwards. Filed so a recurrence has somewhere to land, and because a fixture or a node that occasionally disagrees about a compiler's identity is worth understanding, not retrying past.

What happened (measured)

• Local gate, Linux (Fedora 44, GCC 16.2.1), branch fix/clang-macro-prefix-map-on-preprocessed-input (fix(pace-cc): apply -fmacro-prefix-map on clang workers compiling preprocessed input #1726) on master 16fb5b7, 2026-10-22 between 20:57 and 21:14 CEST. gate-clang-debug: 6929/6929 passed. gate-gcc-release: 6927/6928; the one failure was dist-compile-e2e, after 1.56 s:

  dist-compile E2E FAILED: worker and launcher disagree on the toolchain fingerprint: '89774c206d5cea5704051af91f89689d' vs 'a069fc8108e5fb3adf1fd3be2f672f0c'
  

  The first value is the WORKER's: the first serving <compiler> as <fp> line in its log (scripts/dist-compile-e2e.sh, around line 903). The second is the LAUNCHER's: pace-cc --print-toolchain-fingerprint "$compiler" (around line 765). The compiler is the tree's CMAKE_CXX_COMPILER, /usr/bin/g++.
• An immediate rerun on the same binaries (unchanged mtimes) failed the same way.
• Every later run passed: 6 of 6 direct runs (2 on an idle host, 4 under 2×nproc busy loops), plus a full gate rerun at 22:18 that ended LOCAL GATE PASSED.
• Afterwards, every pace-cc on the machine (both gate trees, a clang-debug dev tree, /usr/bin) and a manually started gate pace-node give a069fc81… for /usr/bin/g++. No compiler that was tried (c++, gcc, cc, clang++, clang, x86_64-redhat-linux-g++) produces 89774c20….
• The branch under test changes no fingerprint input: no banner, DriverGrammarName or schema change. Its only DriverSpec change is a new carrier column.
• The only package transaction near the window was a reinstall of the pace RPM at 21:26 (dnf5.log), after the gate ended. Whether the failing rerun happened before or after it was not recorded.
• The failed run's work directory (/tmp/tmp.Sf11jVypOy, the worker log included) was cleaned up with the fixture, so the worker's full log is gone. The surviving ctest log shows the worker reaching compile node ready, scheduling for the fleet and the lease-signature line, each in the first poll, before the check failed.

Not established (inferred or open)

• Where 89774c20… came from. Nothing on the machine reproduces it.
• Possible lead, not checked: the worker re-logs serving {} as {} whenever its toolchain set is re-resolved (NodeToolchains.cpp, the refresh path around line 744), and the fixture takes the FIRST such line. If a worker can log one fingerprint and then a different one, a first-line read compares a superseded identity. That would make this a fixture defect, not a fingerprint defect. The other reading, two derivations of the same compiler genuinely disagreeing (environment, a stamp, the include-root walk), would be a fingerprint defect. The two need telling apart before anything is changed.
• Whether the 21:26 reinstall of /usr/bin/pace-* mattered. A gate run uses its own build-tree binaries, so a link is not expected, but it was not ruled out by timing.

Asked of whoever picks this up

1. Make the failure self-describing first: on this mismatch the fixture should print every serving … as … line from the worker log and the worker log's head before it fails, so a recurrence carries its own evidence.
2. Decide whether a worker may legitimately log more than one fingerprint for one --toolchain. If yes, the fixture should compare the latest (or the set). If no, a second line is the bug.
3. Only then judge whether this is a fingerprint-derivation defect.