Prevent Redis outages from amplifying latency, source load, and logs

[lan17]

Priority

P1 — High — resolve before broad production adoption or large-scale use.

Problem

A failed remote read can impose client timeout/queue latency, log, run fallback, then attempt and log a separate failed write. A rejected lazy client factory is immediately eligible for another attempt. There is no DialCache-level cooldown, backoff, failure budget, or log suppression.

Evidence:

• Read failure path
• Fallback then write failure path
• Factory retry behavior

Open decision

Define which timeouts belong to DialCache versus the underlying client, then compare cooldown/circuit-breaker or equivalent policies.

Acceptance criteria

• Bound Redis attempts and log volume during sustained high-cardinality failure.
• Preserve fail-open reads/writes and surfaced invalidateRemote failures.
• Avoid retry stampedes during recovery.
• Add stable bypass, state, probe, recovery, and failure metrics.
• Test concurrent factory failure, sustained outage, partial recovery, and full recovery.
• Document the operational failure budget and recommended client timeouts.