[TRACKER] Telemetry outage 2026-07-29 — ledger RPC wedge and agent resilience

[elitegreg]

Track the fixes coming out of the device telemetry outage of 2026-07-29 → 2026-07-30 on
mainnet-beta.

Summary: a one-hour upstream network outage became a 22-hour telemetry outage, because a shared
Go HTTP transport has HTTP/2 enabled without health checks, and because the telemetry agent's probe
loop has a hard dependency on the RPC endpoint that failed. 23 devices lost between 11 minutes and
19 hours of latency samples. The last three required manually bouncing the agent, roughly 7 hours
after the upstream endpoint had fully recovered.

Timeline

Time (UTC)	Event
Jul 29 ~19:15	TeraSwitch (TSW) outage begins. Path to DZ Ledger LB 64.130.56.79 (Pittsburgh) blackholes.
Jul 29 19:09–19:14	nyc002-dz002, dz-ny7-sw01, dfw001-dz002 write their last epoch-193 samples.
Jul 29 19:15:55	nyc002-dz002 logs Submission failed after all retries (5 attempts, 4 samples, epoch 193).
Jul 29 19:20	DZ Ledger LB begins refusing a subset of connections. Geo-DNS keeps returning it; their health checks kept passing because the node was partially reachable.
Jul 29 19:26–19:45	Lake indexer records 503s from the ledger RPC across four activities.
Jul 29 21:03	chi001-dz001 stops writing.
Jul 30 ~09:00–14:00	Lake indexer ingestion lag grows to 304 minutes. Lake Indexer: Errors alerts fire.
Jul 30 11:37	DZ Ledger LB fully recovers. Wedged agents do not recover.
Jul 30 14:10:37	Epoch 194 begins. 20 devices appear to recover simultaneously (artifact, see below).
Jul 30 17:29 / 17:42 / 18:52	nyc002-dz002, dfw001-dz002, dz-ny7-sw01 resume after the telemetry agent is bounced.

Root cause

1. Trigger. TSW blackholes the path to one of five DZ Ledger load balancers. DZ Ledger's geo-DNS keeps
pointing at it because the node stayed partially reachable and their health checks passed. The other
four LBs were healthy throughout and were never used.

2. Why it outlived the LB recovery. tools/solana/pkg/rpc/retry.go:132 sets
ForceAttemptHTTP2: true, and nothing in the repo sets ReadIdleTimeout or PingTimeout (zero
occurrences). Two compounding failures follow:

• A 503 is a valid HTTP response, so the HTTP/2 client considers the connection healthy and never
  re-dials or re-resolves DNS. The process stays pinned to the dead IP.
• A blackholed path produces no FIN and no RST, so the connection is silently dead. Without HTTP/2
  keepalive pings nothing detects that. IdleConnTimeout: 5m (line 122) only evicts idle
  connections, and a loop polling every 10s never leaves it idle that long.

The connection becomes a zombie pinned for the lifetime of the process, clearable only by restart.

Upstream references: golang/go#39750,
golang/net#55,
merged commit,
why net/http does not expose it.

3. What that does to the agent. Pinger.Tick fetches the epoch first and returns early if it
fails (pinger.go:56-60), so one unreachable RPC endpoint stops all TWAMP probing. Probing is pure
UDP and needs no ledger access; the epoch is used only to build the partition key. The epoch is the
only call in the probe path with no cache in front of it. Peer discovery is not the failure:
CachingFetcher serves stale data on error (internal/serviceability/cache.go:83-89).

Silence then follows mechanically. No probes means an empty buffer, and Submitter.Tick returns at
submitter.go:177 with no log when there are no partitions.

4. Why it looked like "no errors." getCurrentEpoch wraps the call in 3 tries with exponential
backoff (pinger.go:149-165), and each underlying call can burn ~43s (4 jsonrpc retries at a 10s
client timeout plus backoff). A failing Tick takes ~130s and the 10s ticker coalesces, so the error
appears roughly every two minutes rather than every tick.

5. Blast radius. The lake indexer hit the same endpoint. From log_ingestion_runs:

RefreshTelemetryLatency  error  2026-07-29T19:26:38Z → 19:45:45Z
  "failed to get epoch info: RPCError{Code: 503, Message: "Service unavailable"}"

Same for RefreshServiceability, RefreshGeolocation, RefreshPermissionEvents. Any long-lived Go
process sharing the RPC constructor was exposed.

Measured damage

Epoch-193 completeness against the 15,844 samples a full epoch holds (epoch 193 ran
Jul 28 18:09:52 → Jul 30 14:10:37 at 10s sampling). 23 devices affected, a continuous spectrum from
11 minutes to 19 hours, which is the shape of a shared upstream dependency degrading differently per
device rather than independent faults.

Full per-device table

Device	Missing samples	Equivalent
nyc002-dz002	6,844	19.0h
dz-ny7-sw01	6,825	19.0h
dfw001-dz002	6,820	18.9h
chi001-dz001	6,161	17.1h
dz100a-ewr1-tsw	1,803	5.0h
was001-dz001	1,798	5.0h
nyc001-dz002	1,792	5.0h
chi001-dz002	1,775	4.9h
dz-ch2-sw01	1,059	2.9h
dz-chi-sw01	511	1.4h
au1c-dz01	492	1.4h
tyo001-dz001	371	1.0h
sjc001-dz002	215	36m
dz100a-sea1-tsw	211	35m
lax001-dz002	205	34m
sao001-dz001	197	33m
dzd-tok-01	172	29m
nyc001-dz001	162	27m
dz100a-iad1-tsw	149	25m
was001-dz002	134	22m
dz100a-lax1-tsw	74	12m
dz100a-slc1-tsw	72	12m
sjc001-dz001	68	11m

Caution when reading lake data for this incident. event_ts is derived as
start_timestamp_us + sample_index × 10s, so missing writes do not leave a hole. They compress the
derived timeline and the whole deficit accumulates at the tail of the epoch. Epoch 194 began at
14:10:37Z, which is exactly when 20 devices appeared to recover within a 9-second window. Nothing
recovered; the derived timelines realigned. "Device X stopped at 09:10" actually means "device X is
5 hours short for epoch 193," with the losses spread across the outage.

Child issues (this batch)

• device/telemetry: pinger aborts all probing when the epoch fetch fails #4125 device/telemetry: pinger aborts all probing when the epoch fetch fails
• device/telemetry: samples silently dropped when submission fails at buffer capacity #4126 device/telemetry: samples silently dropped when submission fails at buffer capacity
• device/telemetry: account-full path reports success and logs the wrong count #4127 device/telemetry: account-full path reports success and logs the wrong count
• device/telemetry: peer cache cleared before fallible work in refresh() #4128 device/telemetry: peer cache cleared before fallible work in refresh()
• device/telemetry: log RPC endpoint identity, collapse repeated failures into transitions #4129 device/telemetry: log RPC endpoint identity, collapse repeated failures into transitions
• Possibly https://github.com/malbeclabs/infra/issues/1703

#4125 is the one that would have prevented the data loss and should go first. #4126 and #4127 share
a new samples_dropped_total{reason=...} counter and should land together.

Deferred (not yet filed)

• tools/solana: HTTP/2 transport has no health checks, dead connections never evicted. The
  highest-severity finding overall and the reason the outage lasted 22 hours instead of 1. Not filed
  because it sits outside the telemetry agent, but nothing in the batch above prevents a recurrence
  without it. Fix is http2.ConfigureTransports with ReadIdleTimeout (~30s) and PingTimeout
  (~15s) in newHTTPTransport (retry.go:116-135). Reaches the controller, lake indexer,
  doublezerod, sdk/shreds, sdk/revdist, the internet latency collector, and the agent's
  non-namespaced path at once.
• sdk/telemetry: waitForSignatureVisible false negatives cause duplicate sample writes.
  executor.go:114-145. A landed transaction is reported as "dropped or rejected before cluster saw
  it" when the confirming endpoint lags, and the submitter then rewrites the batch up to 5 times
  with a fresh transaction each time. This is the exact error logged at 19:15:55.
• device/telemetry: agent metrics are not scraped on mainnet-beta.
  doublezero_device_telemetry_agent_* exists for exactly four devnet hosts (chi-dn-dzd1…dzd4).
  submitter_retries_exhausted was incrementing on the affected hosts the whole time and nobody
  could see it. This also blocks most of the value of device/telemetry: log RPC endpoint identity, collapse repeated failures into transitions #4129.
• monitoring: "No Samples" alert fires for deleted devices and measures derived time.
  dz-mad-01 was deleted onchain at 2026-07-27T06:10:20Z and has been firing 6 instances since.
  Separately, the derived-time issue above means the rule conflates "offline now" with "lost samples
  earlier this epoch."
• Investigation: why agents stayed wedged after the endpoint recovered. See open questions.

Drafts for all five exist; ask if you want them filed.

Open questions

Which HTTP path do mainnet devices use? cmd/telemetry/main.go:269-278 picks between two very
different clients:

if *managementNamespace != "" {
    // netns.SingleThreadTransport — HTTP/1.1, fresh TCP per request, no pooling
} else {
    rpcClient = dzrpc.NewWithRetries(*ledgerRPCURL, nil)  // pooled, ForceAttemptHTTP2
}

The HTTP/2 explanation only applies if mainnet devices run without --management-namespace. The
namespaced path dials fresh per request and structurally cannot hold a zombie connection. Given #743
is closed, devices may well be namespaced, in which case something else pinned those three and the
fix changes. Settle it with ps auxww | grep telemetry on dz-ny7-sw01.

Testnet impact. Not verified. Same code and same DNS, so assume the same effect until checked.

DZ Ledger root cause. Their health checks kept an LB marked healthy while it was partially
reachable, and geo-DNS never failed over to the other four. They are looking into terminating connections to problematic load balancers.

Exit criteria

• All five child issues merged.
• A total ledger RPC outage no longer stops TWAMP probing; samples buffer and flush on recovery.
• Every path that discards samples has both a log line and a counter.
• A 22-hour RPC outage produces single-digit log lines per component, not hundreds.
• The namespace question above is answered, and the wedge mechanism is confirmed rather than
  inferred.
• Check whether the duplicate-write path actually produced duplicate samples during the outage.

Related but separate

• Go JSON-RPC retry classification is broken; Go and Rust disagree on what's retryable #4098 / tools/solana: fix JSON-RPC retry classification and consolidate RPC construction #4100 — JSON-RPC retry classification, the 2026-07-28 incident in the same subsystem.
  tools/solana: fix JSON-RPC retry classification and consolidate RPC construction #4100 consolidated RPC construction into tools/solana/pkg/rpc.New, which is why the deferred
  HTTP/2 fix now reaches every Go ledger reader from one place.
• sdk/rs: retry classification misses JSON-RPC response errors (Rust parity with #4100) #4101 — Rust parity for the same retry classification.
• Duplicate Grafana rule drift: Lake Indexer: Errors, Lake Indexer: Down, and
  Network Component Service Down / Component: Service Down each exist in both
  folder-doublezero and folder-doublezero-managed and fire in parallel, but only the managed
  copies route to Slack. Looks like a half-finished migration to provisioned rules.

[nikw9944]

Possibly related: https://github.com/malbeclabs/infra/issues/1703