feat(failover): elect the highest-offset replica as master - #36
Merged
Conversation
When re-electing a master with no clear existing one (bootstrap or all-down recovery), the operator promoted the oldest pod, which could be a replica that lags behind and discard the data held by more up-to-date replicas. Elect by replication offset instead: query master_repl_offset from each pod and promote the one with the most data, keeping the oldest pod as a deterministic tie-breaker and as the fallback when offsets can't be read. Unreachable pods sort last so they are never elected. Adds redis.Client.GetReplicationOffset (+ mock). Refs upstream spotahome#385.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Changes the fallback master election (used when there is no clear existing master — bootstrap or all-down recovery) to promote the replica with the highest replication offset instead of the oldest pod.
Why
The oldest pod is an arbitrary choice: it may be a replica that has fallen behind, so promoting it silently discards the writes held by more up-to-date replicas. Electing by
master_repl_offsetpromotes the instance with the most data, minimizing data loss on failover.Behaviour
master_repl_offset(viaINFO replication) for each pod once, then order by offset desc, creationTimestamp asc.Design note
The interface method keeps its name
SetOldestAsMaster(oldest is now the tie-breaker/fallback) to bound the blast radius on this failover-critical path — the doc comment describes the offset-first behaviour.Changes
service/redis/client.go:GetReplicationOffset+ regex + metric; interface + mock.operator/redisfailover/service/heal.go: offset-based ordering in the election.TestSetOldestAsMasterPrefersHighestOffsetproves highest-offset wins.Upstream references
Test
go build ./...,go test ./...,gofmtclean.