Cache and distribute clang-tidy runs, not only compiles #1725

Problem

clang-tidy is now the slowest per-file step in our builds. Linting one translation unit costs several times what compiling it costs, it runs on every changed file in each merge-request pipeline (a diff-only gate), and developers run the same gate locally before pushing. All of that runs on the machine that started it, while the fleet pace-node built sits idle.

pace-cc already caches and dispatches compiles. clang-tidy has the same shape: one invocation per translation unit, with a compile command and a deterministic output. It would benefit from the same two things, in this order:

1. Caching. An unchanged translation unit with an unchanged configuration should not be analysed twice, whether in CI, on another developer's machine, or across rebases.
2. Distribution. A miss should be able to run on a worker.

Why the compile path doesn't carry over as is

Distributed compilation ships preprocessed text (the distcc non-pump model, distributed-compilation.md §3). That is enough for a compiler, but not for clang-tidy:

• Macro-aware checks need the preprocessor callbacks; on expanded text they see nothing or misfire. Examples: bugprone-macro-parentheses, cppcoreguidelines-macro-usage, modernize-macro-to-enum, and any check that skips code expanded from macros.
• // NOLINT / NOLINTNEXTLINE / NOLINTBEGIN are comments. Preprocessing drops them unless -C is used, and they also live inside headers.
• HeaderFilterRegex / --header-filter, --line-filter and the system-header classification depend on real file entries, not #line-presumed locations.
• The effective configuration is file-based. .clang-tidy files are resolved per source file up the directory tree (with InheritParentConfig). That is input the compile key never covers.
• --export-fixes / --fix produce replacements against real file paths and offsets.

Proposal

Phase 1: a clang-tidy result cache in the launcher (no distribution)

Add a pace-cc mode, or a sibling pace-tidy, that wraps clang-tidy <file> -- <compile args> / -p <compile_commands.json>.

Cache key. It is derived from:

• the comment-preserving preprocessed text of the TU (-E -C, without line markers, as the compile key does);
• the clang-tidy toolchain fingerprint: binary identity and version, plus loaded plugins (-load);
• the effective configuration after resolution, e.g. clang-tidy --dump-config for that file, normalised. That covers Checks, CheckOptions, WarningsAsErrors, HeaderFilterRegex, SystemHeaders and InheritParentConfig chains;
• the clang-tidy-relevant command-line options: --checks, --warnings-as-errors, --header-filter, --line-filter, --extra-arg*, --fix*, --export-fixes, --use-color, --quiet;
• the compile arguments that affect analysis.

Cached value. The diagnostics output (stdout/stderr), the exit status, and any exported fixes. Paths are stored canonicalised (PathCanon) and rewritten on a hit, exactly as the object and dependency records are today, so entries stay portable across checkout paths.

The same --show-stats surface reports hit and miss counts separately from compiles.

Phase 1 alone removes most CI cost, because most translation units in a diff-gated run are unchanged between pipelines of the same branch.

Phase 2: distribute misses

A tidy miss can't ship preprocessed text, so it ships the input closure instead (distcc's pump model, restricted to what the local preprocess pass already opened):

• the source file and every header the client's own preprocess pass opened, system headers included, plus the .clang-tidy files on the resolution path of each of them;
• each file sent content-addressed: the worker keeps a CAS of blobs and asks only for the ones it lacks, so the MSVC STL and Windows SDK headers move once per worker, not once per job;
• on the worker, a sandbox tree reconstructed at canonicalised paths, with clang-tidy run there against the worker's own toolchain matched by fingerprint (the existing rule: a job names a fingerprint, never a program);
• diagnostics and fixes mapped back to the client's paths, with the reply digest check (§6) applied as it is for objects.

Lease accounting. A tidy job is heavier than a compile, so it should carry a weight (or its own slot class) in the capacity model. Otherwise one slot per tidy job oversubscribes workers.

Acceptance

• Phase 1 runs a diff-only clang-tidy gate twice on an unchanged tree: the second run is all hits, with byte-identical diagnostics and the same exit status.
• Changing a .clang-tidy file, or a header the TU includes (including a NOLINT comment in it), invalidates exactly the affected entries.
• Cached output carries the paths of the checkout that reads it, not the one that stored it, and two checkouts at different depths share entries.
• Phase 2 matches local runs exactly, for diagnostics and fixes, on a corpus that exercises macro checks, NOLINT* inside headers, HeaderFilterRegex, and InheritParentConfig.
• Refusals (no-worker, no-capacity, …) fall back to running locally, as compiles do.

Open questions

• A sibling binary (pace-tidy) or a pace-cc mode keyed on argv[0] / the wrapped executable name?
• Should run-clang-tidy-style drivers that pass -p be supported directly, or only the per-file -- <args> form?
• Is there a size bound per input closure before a job should refuse and run locally? The first job per worker ships the whole STL/SDK closure.