[christianparpart]

Closes #1728.
Closes #1736.

Why

Several builds side by side on one workstation run hundreds of compiles at once. The kernel's OOM killer then takes down unrelated builds: 18 kills in 90 minutes, as measured in #1728. Every one of those compiles already goes through pace-cc, but until now pace could only say "compile it yourself".

What

Every compile that misses the cache now waits its turn in one queue on the scheduling leader. The leader admits work against each machine's slot and memory budget:

• Leader (Distributed::AdmissionQueue, AdmissionResponder):
  • First come, first served, with backfill: a later compile that fits may start ahead of a large one waiting at the head of the queue.
  • Head reservation: memory freed while the head waits is set aside for it, so it cannot be starved.
  • Over-budget compiles run alone.
  • Dedup by STORE: concurrent misses of one key on one machine compile once, and the waiters FETCH the stored result.
  • The queued request is held open and pulses Status::Progress, so the wait is never silent.
  • The connection is the lease: a killed client releases its charge when its socket closes.
  • Failover: a new leader rebuilds its ledger from each machine's report, using max(charged, held).
• Every non-leader node (follower or learner) relays Admit/Settle to the leader inline (AdmitForwarder), and never to itself.
• Every node sends a MachineBudget report each round: budget, held charges and measured peaks. A node with --slots=0 sends one too, and that is not a worker registration. New flags:
  • --machine-slots
  • --machine-memory-budget (default 75% of RAM minus the cache tier)
  • --compile-estimate-default (default 2 GiB)
• Launcher (pace-cc, MissFlow / AdmitSession):
  • Every miss is admitted at the local node before compiling locally or dispatching. The admission connection stays open on its own thread for the whole compile.
  • STORE happens before SETTLE.
  • A worker refusal re-queues the same ticket on the same connection.
  • Falls back to today's behaviour when no node answers, and skips the admission dial once a FETCH has already found the node unreachable.
• Peak memory is measured per compile, children included (wait4 on POSIX, a job object on Windows), and feeds the estimates. Estimates are keyed by cache key, then by (source path, compiler).
• Observability:
  • pace-cc -s shows the queue wait and the peak memory.
  • /metrics gets admission counters, a real wait _sum/_count, and per-machine budget and in-use gauges.
  • pacectl live-stats fleet, /fleet.json and the fleet page show the queue depth and the clients (machine, build tree, running, queued).
• 0xFC wire v16: new Admit, Settle and MachineBudget verbs. Status::Progress is allowed on {Compile, Admit}.
• core-cpp 0.6.0 → 0.11.0.
  • 0.10.0 adds core::ranges::Stride, because libc++ has no std::views::stride (ranges: core::ranges::Stride, because libc++ has no std::views::stride contour-terminal/core-cpp#80). The ranges-seam check now derives views:: selections.
  • 0.11.0 adds ISocket::armKeepAlive (net: arm TCP keepalive on an accepted socket contour-terminal/core-cpp#82).
• TCP keepalive at both ends of a held admission connection. The launcher and relay dials already armed it. Now the leader arms it on the accepted connection once that connection holds a grant (IConnectionHold::KeepAlive(), once per connection). A caller whose host vanishes without a FIN is noticed in about 16 s (30 s on Windows), not after the one-hour lease sweep. Every pace socket decorator forwards armKeepAlive.
• Rulebook: a new "compile queue" section in .agent/rules/distributed-compilation.md, tripwires in AGENT.md, operator docs for pace-cc, pace-node and pacectl, and the protocol grammar.

Decisions made along the way (recorded in the rules)

• A dispatchable compile tries the workers with room, then runs locally if this machine has room, so cores never idle while capacity exists.
• Admission always goes to the local node (PACE_ADDR). PACE_SCHEDULER no longer redirects it, and dispatch planning no longer requires it.
• An admission Worker grant files its lease per ticket. Dedup is the queue's job.
• A Local grant of a dispatchable compile carries its reason, so "no worker serves this toolchain" is still reported and counted.
• After MaxWorkerRefusals (3) the compile runs locally, still charged. A worker failure settles NotStored, never an abandon, so estimates aren't poisoned.

Compatibility

0xFC v15 ↔ v16 do not interoperate (MinSupportedVersion = 16). Raft (v5) and the on-disk state are unchanged, so an upgraded node keeps its identity and cluster state. In a fleet mixing 0.6.0 and this version, registration, dispatch and pacectl fail between old and new machines; every machine still compiles locally and uses its own cache. Upgrade every member together.

Known limitations (each filed)

• Dedup is per machine. Dedup across machines that share a cache is Compile queue: deduplicate one cache key across machines that share a cache #1730.
• Deferred test gaps are listed in Compile queue: close the test gaps the #1728 reviews deferred #1731.

Tested

• Local gate (scripts/local-gate.sh, WSL Ubuntu 24.04) on the head commit:
  • clang-debug (pinned clang-format and clang-tidy, ASan+UBSan): 7157/7157.
  • gcc-release: 7156/7156.
• ThreadSanitizer (scripts/tsan-gate.sh): clean on PaceTest, pace-node-tests, pace-cc-tests and pacectl-tests, with the canary live.
• Windows cl-debug: the full default set passes, except host-only failures that also fail on master and are now ticketed: FileTrust test hard-codes the English name of LOCAL SERVICE and fails on non-English Windows #1733, Tests that run python3 fail on Windows where the Store app-execution alias shadows a real Python #1734 and Link-using tests assume a created symlink/junction can be followed; on Windows a non-admin link may not open (error 448) #1735.
• End-to-end (FleetAdmission_test.cpp), each shown red with its rule removed:
  • six identical misses compile once;
  • a learner is queued through its own node;
  • kill -9 does not shrink the budget;
  • with the node stopped, behaviour is as today;
  • the leader's ledger never exceeds the budget under 40 mixed compiles.

Still to be done by the owner after merge: the acceptance run on the 32-core workstation (six concurrent cmake --build, zero OOM kills).