Fleet operator analytics, Part A: launchers report why compiles were (not) distributed, and a toolchain census #1723

Why

An operator investigating the compile fleet today has to ask developers to run pace-cc --show-stats, pace-cc --version or look inside their build tree. That interrupts them every time, and the operator still cannot see the fleet as a whole. A real investigation on 2026-10-20 hit three walls in one hour:

• The leases table does not say who asked, so the operator could not tell how much of the fleet their own build was using.
• A node (172.22.23.91) counted 6 cache requests refused for an unsupported wire version (pace_node_cache_requests_refused_unsupported_version_total), and nothing on the fleet said which client sent them or at which version.
• Every reason a compile was not distributed — cache hit, refused by the launcher, declined by the fleet and why, unreachable — lives only in each developer's local invocations.log (pace-cc --show-stats), on their own machine.

Premise this design rests on: every developer machine runs pace-node (installed by the official MSI); using the fleet without participating in it is not allowed by design. So every pace-cc has a node on its own machine, and every node already reports to the leader.

This is Part A of the fleet operator analytics design. Part B (keyboard navigation in pacectl live-stats fleet and a per-machine detail view) shipped in #1722. Operator actions (tier management, machine labels) are #1720.

Goals — the operator's questions

Answered in pacectl live-stats fleet (a terminal is enough; see Non-goals), without asking any developer to do anything:

1. Why is a build (mine or a developer's) not distributed through the cluster?
2. Who provides which compiler toolchain versions — whom should I ping to update theirs?
3. Which toolchains are currently in use, and by whom?
4. How often has each toolchain been invoked?

Lower priority, also wanted:

5. Out-of-date launchers: which machines run a pace-cc the fleet does not speak with.
6. Unserved toolchains: toolchains that are requested while no worker offers them.

Non-goals

• The web fleet page and /fleet.json — the terminal is enough for now.
• Any operator action (tier management, labels) — Operator actions: manage node cache tiers, and label machines with a human-readable name #1720.
• Fleet-wide aggregation of source files, paths, or per-translation-unit detail.
• Alerting or notification.

Anti-goals — failure even if it technically works

• A compile slowed down or failed by reporting. Reporting is off the compile path. The launcher already appends to its local statistics log per compile; shipping tallies happens separately, and a node that is unreachable means the report waits or is dropped. The build never notices.
• File names or paths leaving the machine. Only counts, outcome/reason enumerators, toolchain identity and the launcher version travel.
• A confident wrong signal (AGENT.md / .agent/rules/metrics-and-observability.md): a machine that sent no report renders as absent, never as zero; a reason the leader does not know renders as unknown, never folded into a neighbouring reason.

Design

A1. One source: launcher tallies, relayed by the launcher's own node (questions 1, 3, 4)

Decision: per-machine analytics come from ONE source, the launchers — not from the leader counting leases. Leader-side counting was considered and rejected:

• the leader's counts live in its memory and vanish on failover — a new leader would render zeros, a confident wrong signal;
• followers never see leases;
• attributing a lease to a machine is harder than it looks (a launcher dialling the scheduler does not prove a node identity; the caller's address is not a machine).

The launcher already knows every outcome, including the fleet's decline reason, so nothing is lost.

What exists: pace-cc classifies each compile into DispatchOutcome (src/apps/pace-cc/Stats.hpp: Unknown, NotConfigured, NotAttempted, Refused, Declined, Unreachable, …) and, for a decline, DeclineCause (src/apps/pace-cc/Dispatch.hpp; the human sentences are the DeclineReasonRow table in src/apps/pace-cc/Stats.cpp — "no worker serves this toolchain", "matching workers had withdrawn their slots", "a worker would not take an argument of this compile", …). It records each compile, with cache hit/miss, in invocations.log.

Change:

• pace-cc sends cumulative tallies to the pace-node on its own machine: per toolchain, per outcome and per decline reason, plus cache hits and misses, plus its own launcher version.
• It finds that node the way it already does for MINT-TICKET (TicketSourceFor, src/apps/pace-cc/TicketCredentials.hpp) — independent of PACE_ADDR, which may name a remote cache (a real setup here points it at a separate pace-kv on 172.22.23.11).
• The node aggregates what its machine's launchers send and forwards it to the leader on NODE-ANNOUNCE, the carrier it already uses for machine history (src/apps/pace-node/NodeAnnounce.hpp, NextHistoryBatch). The rulebook requires exactly ONE carrier for a handed-over batch (two callers of NextHistoryBatch would split one cursor), so this rides that verb rather than a new one.
• The leader files the figures under the node id, never an IP.
• Failover: because each node holds cumulative counters, a new leader has the whole picture after one announce round. Failover costs a round, never the history.

Toolchain identity in the report: the toolchain fingerprint and the display label the launcher already derives from the compiler's banner (toolchainLabel in src/apps/pace-cc/Dispatch.hpp, e.g. cl 19.44.35207), so a toolchain no worker offers still has a readable name. The label is display-only; nothing keys on it.

A2. Launchers too old to report (question 5)

A launcher too old for the report verb cannot send one. Its own machine's node sees it anyway: in the 172.22.23.91 case it was a local client refused for an unsupported wire version. Change: a node records the wire versions it refused from clients on its own machine (version → count) and forwards that on the same NODE-ANNOUNCE. The fleet then shows "this machine has a launcher speaking version N" without the launcher's cooperation. Today only the total is counted (pace_node_cache_requests_refused_unsupported_version_total, src/Pace/Metrics/MetricsCatalog.hpp); the refused version itself is not kept.

Optional, low priority: the leases table gains a requesting-machine column where the launcher's verified machine ticket names the machine; absent otherwise — never guessed from an IP.

A3. Toolchain census (questions 2 and 6)

Derived on the leader from data it already has — the worker registry (compiler label per endpoint, the workers section) — plus A1's per-machine usage. For each compiler family:

• which machines offer which version;
• which machines use which version;
• which machines are behind the newest version the fleet offers in that family (compare within a family only: cl with cl, clang-cl with clang-cl);
• which toolchains are used but offered by nobody (question 6).

The exact "behind" comparison is the implementer's call, stated in code.

A4. How it reaches the terminal

The fleet live stream already carries the leader's whole fleet document every tick (src/apps/pacectl/LiveSubscriber.cpp → ReadFleetSample → FleetDocument, one table per Distributed::FleetSection). New data arrives as new sections/columns of that document — new rows in FleetSectionTable (src/Pace/Distributed/FleetView.hpp) — so pacectl's renderer picks them up as data. Proposed (names may be refined):

• a toolchains section — A3's census: offered-by, used-by, behind, unserved;
• a clients section — per machine: launcher version(s), compiles, hits, dispatched, the top not-distributed reasons (A1), refused launcher versions (A2).

Both must also appear in the per-machine detail view from #1722 (DetailItems in src/apps/pacectl/DashboardPanel.cpp), which shows every section whose rows name a machine — add a row for each new section to MachineColumnOf's table (src/apps/pacectl/FleetFocus.hpp), whose test checks each column against the leader's FleetColumnNames.

Constraints

• pace-cc does not link the Pace library. Anything it speaks lives in the header-only, dependency-free src/Pace/Protocol/CompileCacheWire.hpp (.agent/rules/wire-and-protocol.md).
• The report verb is local-only: the node accepts it only from its own machine, like the cache tier (CacheResponder; locality is a property of the VERB — .agent/rules/distributed-compilation.md).
• A new verb is not one dispatch: read the "A new verb is not one dispatch" tripwire in AGENT.md (daemon switch arm, RelocatedVerbs row, documentation row per counter, live-stats layout pin).
• Counters are tallies (zero is a reading); a machine with no report is absent, and the absence is modelled in the snapshot (.agent/rules/metrics-and-observability.md).
• Text a peer sent (the toolchain label) must pass the UTF-8 gate where it enters, or one bad byte makes /fleet.json unparseable for the whole fleet.
• A wire grammar change bumps its version. Backwards compatibility is not owed yet (AGENT.md), but a mismatch must be detected, never silently misread.
• Reporting must never delay or fail a compile (Anti-goals).
• The usual project rules: data-driven tables, dependency injection for clock/network, std::expected, Catch2 tests beside the code, bash scripts/local-gate.sh before pushing.

Ordering

1. A1 — wire record + launcher sender + node aggregation + NODE-ANNOUNCE relay + leader storage, with the clients section reading it.
2. A2 — node-side refused-version record on the same relay.
3. A3 — the toolchains census section.
4. A4 grows with each: sections in the fleet document, rows in MachineColumnOf, the detail view.

Left to the implementer

• Section and column names; how many "top reasons" per machine to show.
• How the launcher batches and ships its tallies (it may derive them from its existing statistics log).
• Retention: cumulative-since-start, or windowed in the existing fleet history. If windowed, the history rules apply: raw counters stored, rate taken at render, a restart is a gap and not a spike (.agent/rules/metrics-and-observability.md).
• The exact "behind" comparison for compiler versions.

Acceptance

• With every node on a build that has this change, pacectl live-stats fleet shows for each machine — with no command run on that machine — its launcher version, its compiles, hits and dispatched counts, and the reasons its compiles were not distributed, per toolchain.
• A toolchains view lists each toolchain with the machines that offer it, the machines that use it, the machines behind the newest version in its family, and toolchains used but offered by nobody.
• A machine whose launcher is too old to report shows the refused launcher wire version.
• A machine that has sent no report renders as absent, not zero; an unknown reason renders as unknown.
• After a leader failover, the analytics are complete again within one announce round.
• A compile's duration and exit status are unchanged whether or not the local node is reachable (a test proves the report path is off the compile path).
• No source path or file name appears in any wire record or fleet section added here.