device/telemetry: pinger aborts all probing when the epoch fetch fails

[elitegreg]

Tracked by #4130.

Problem

Pinger.Tick fetches the current epoch before doing anything else and returns early if that
fails, so a single unreachable ledger RPC endpoint stops all TWAMP probing:

// controlplane/telemetry/internal/telemetry/pinger.go:56
func (p *Pinger) Tick(ctx context.Context) {
	epoch, err := p.getCurrentEpoch(ctx)
	if err != nil {
		p.log.Error("failed to get current epoch", "error", err)
		return          // line 59 — no probes at all this tick
	}
	peers := p.cfg.Peers.GetPeers()

Probing is pure UDP and needs no ledger access. The epoch is used only to build the
PartitionKey for the sample buffer. Losing it should not cost us the measurement.

Everything downstream then goes quiet with no additional signal: no probes means an empty buffer,
and Submitter.Tick returns at submitter.go:177 without logging when there are no partitions.

Note that peer discovery is not affected, because CachingFetcher serves stale data on RPC
error (internal/serviceability/cache.go:83-89). The epoch is the only call in the probe path
with no cache in front of it.

Impact

During the 2026-07-29 ledger RPC outage this cost us up to 19 hours of latency samples per device
across 23 devices on mainnet-beta. Measured against the 15,844 samples a full epoch holds:

Device	Missing samples	Equivalent
nyc002-dz002	6,844	19.0h
dz-ny7-sw01	6,825	19.0h
dfw001-dz002	6,820	18.9h
chi001-dz001	6,161	17.1h
(19 more)	68–1,803	11m–5h

The probes themselves would have succeeded the entire time.

Timing detail worth knowing: getCurrentEpoch (pinger.go:149-165) wraps the call in 3 tries with
exponential backoff, and each underlying call can burn ~43s (4 jsonrpc retries at a 10s client
timeout plus backoff). A failing Tick therefore takes ~130s, and the 10s ticker coalesces. So the
error at line 58 appears roughly every two minutes rather than every tick, which is sparse enough
that it read as "no obvious errors" during triage.

Proposed change

Cache the last known epoch on the Pinger and fall back to it. Refuse to probe only when there is
no cached epoch at all (agent just started and has never reached the ledger).

type Pinger struct {
	log *slog.Logger
	cfg *PingerConfig

	mu         sync.Mutex
	lastEpoch  uint64
	haveEpoch  bool
	epochStale time.Time   // when we started serving a cached value
}

In Tick:

epoch, err := p.getCurrentEpoch(ctx)
if err != nil {
	cached, ok := p.cachedEpoch()
	if !ok {
		p.log.Error("no epoch available and none cached, skipping probe tick", "error", err)
		return
	}
	epoch = cached   // probe anyway; samples buffer and flush when RPC returns
}

On success, store the value and clear the stale marker.

Two follow-on considerations for whoever picks this up:

1. Epoch boundary risk. Probing with a stale epoch across a rollover writes samples into the
   previous epoch's partition. Submitter.Tick already removes partitions whose epoch is behind
   the current one (submitter.go:205-208), so those samples would be discarded rather than
   corrupt anything. That is still better than not probing, but it caps how long a stale epoch
   should be trusted. A bound of roughly one epoch seems right; beyond that, stop probing and say
   so.
2. Alternative worth considering. The epoch is derivable locally from slot height and epoch
   schedule, both of which change predictably. Deriving it would remove the RPC dependency from
   the probe path entirely rather than making it best-effort. Larger change; noting it as an
   option.

Acceptance

• Probing continues through a total ledger RPC outage as long as the agent has previously
  fetched an epoch.
• Samples buffer during the outage and flush once RPC recovers.
• A test that fails GetCurrentEpoch for N ticks asserts that samples are still added to the
  buffer with the last known epoch.
• Refusal to probe only when no epoch has ever been fetched, and that case logs once, not per tick.

Context: incident analysis in the 2026-07-29 telemetry outage. Related: #4098, #4100.