Skip to content

fix(failover): heal on a running quorum instead of requiring all pods - #37

Merged
binhnguyenduc merged 1 commit into
masterfrom
feat/heal-while-pods-pending
Jul 24, 2026
Merged

fix(failover): heal on a running quorum instead of requiring all pods#37
binhnguyenduc merged 1 commit into
masterfrom
feat/heal-while-pods-pending

Conversation

@binhnguyenduc

Copy link
Copy Markdown

What

Relaxes the CheckAndHeal gate: the operator now proceeds to heal when a quorum (majority) of redis and sentinel pods are Running, instead of requiring the full set.

Why

CheckAndHeal returned early unless every redis and sentinel pod was Running. A single pod stuck Pending — unschedulable anti-affinity, an availability-zone loss taking a node down — blocked all healing: no master election, no sentinel monitor reconfiguration. The cluster could not recover on its own.

Why it's safe

The downstream heal path already operates only on running/reachable pods (GetRedisesIPs filters to PodRunning; SetMasterOnAll/CheckAllSlavesFromMaster skip unreachable pods), and the operator's force-election stays guarded by CheckSentinelQuorum + CheckIfMasterLocalhost. So only the early-return gate needed to change — no downstream behaviour changes.

  • Quorum = replicas/2 + 1 (2-of-3 proceeds; 1-of-3 still waits, so a mostly-down cluster is not auto-healed).
  • The normal all-running case behaves identically.
  • Bootstrap mode still waits for the full set (unchanged).

Changes

  • check.go: AreQuorumRunning helper + IsRedisRunningQuorum/IsSentinelRunningQuorum predicates (interface + mock).
  • checker.go: the CheckAndHeal gate uses the quorum predicates.
  • Tests: AreQuorumRunning table test; existing CheckAndHeal gate test routed to the quorum predicates.

Upstream references

Test

go build ./..., go test ./..., gofmt clean.

CheckAndHeal returned early unless every redis and sentinel pod was
Running, so a single pod stuck Pending (unschedulable affinity, availability
zone loss) blocked master election and sentinel reconfiguration for the
survivors, leaving the cluster unable to recover.

Gate on a quorum (majority) of pods running instead. The downstream heal
logic already operates only on the running/reachable pods and the
force-election stays guarded by the sentinel quorum check, so only the
early-return gate changes. Bootstrap mode still waits for the full set.

Refs upstream spotahome#535, spotahome#552, spotahome#601.
@binhnguyenduc
binhnguyenduc merged commit c6d321c into master Jul 24, 2026
7 checks passed
@binhnguyenduc
binhnguyenduc deleted the feat/heal-while-pods-pending branch July 24, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant