Skip to content

feat(failover): elect the highest-offset replica as master - #36

Merged
binhnguyenduc merged 1 commit into
masterfrom
feat/master-by-offset
Jul 24, 2026
Merged

feat(failover): elect the highest-offset replica as master#36
binhnguyenduc merged 1 commit into
masterfrom
feat/master-by-offset

Conversation

@binhnguyenduc

Copy link
Copy Markdown

What

Changes the fallback master election (used when there is no clear existing master — bootstrap or all-down recovery) to promote the replica with the highest replication offset instead of the oldest pod.

Why

The oldest pod is an arbitrary choice: it may be a replica that has fallen behind, so promoting it silently discards the writes held by more up-to-date replicas. Electing by master_repl_offset promotes the instance with the most data, minimizing data loss on failover.

Behaviour

  • Query master_repl_offset (via INFO replication) for each pod once, then order by offset desc, creationTimestamp asc.
  • The oldest pod remains the tie-breaker (equal offsets) and the fallback when offsets can't be read — so on a fresh/idle cluster the result is identical to before.
  • Unreachable pods sort last (offset treated as -1) and are never elected (they'd fail promotion anyway).

Design note

The interface method keeps its name SetOldestAsMaster (oldest is now the tie-breaker/fallback) to bound the blast radius on this failover-critical path — the doc comment describes the offset-first behaviour.

Changes

  • service/redis/client.go: GetReplicationOffset + regex + metric; interface + mock.
  • operator/redisfailover/service/heal.go: offset-based ordering in the election.
  • Tests: existing election tests get equal-offset mocks (oldest tie-breaker preserved); new TestSetOldestAsMasterPrefersHighestOffset proves highest-offset wins.

Upstream references

Test

go build ./..., go test ./..., gofmt clean.

When re-electing a master with no clear existing one (bootstrap or
all-down recovery), the operator promoted the oldest pod, which could be
a replica that lags behind and discard the data held by more up-to-date
replicas.

Elect by replication offset instead: query master_repl_offset from each
pod and promote the one with the most data, keeping the oldest pod as a
deterministic tie-breaker and as the fallback when offsets can't be read.
Unreachable pods sort last so they are never elected.

Adds redis.Client.GetReplicationOffset (+ mock).

Refs upstream spotahome#385.
@binhnguyenduc
binhnguyenduc merged commit bab2ae3 into master Jul 24, 2026
7 checks passed
@binhnguyenduc
binhnguyenduc deleted the feat/master-by-offset branch July 24, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant