Skip to content

feat(heal): optionally protect the redis master from autoscaler eviction - #35

Merged
binhnguyenduc merged 1 commit into
masterfrom
feat/autoscaler-skip-master-eviction
Jul 24, 2026
Merged

feat(heal): optionally protect the redis master from autoscaler eviction#35
binhnguyenduc merged 1 commit into
masterfrom
feat/autoscaler-skip-master-eviction

Conversation

@binhnguyenduc

Copy link
Copy Markdown

What

Adds an opt-in redis.preventMasterEviction flag. When enabled, the operator annotates the current master pod with cluster-autoscaler.kubernetes.io/safe-to-evict: "false" and marks slaves "true", so the Kubernetes cluster-autoscaler won't drain the node running the master (which would force an avoidable failover). The annotation follows the master as the role moves — applied in both the checker and healer relabel paths.

Why

Cluster-autoscaler scale-down can drain the node hosting the redis master, triggering an unnecessary failover and a brief write outage. Pinning only the master (slaves stay evictable) lets the autoscaler still consolidate the rest of the pool.

Design

  • Opt-in (default ): always-on would silently pin a node from scale-down for every user. Off = annotations are never touched.
  • New Pod.UpdatePodAnnotations service method uses a JSON merge patch (creates the annotations map if absent), mirroring the existing targeted UpdatePodLabels patch rather than a full-object update.
  • The annotation helper reads desired state off the already-fetched pod and skips the patch when already correct, so it's safe to run every reconcile without extra API writes and self-corrects if the flag is toggled on after the master exists.

Note on the mock

Adding the interface method requires regenerating mocks/service/k8s. The committed mocks are mockery v2.20.0, which panics against the current Go 1.25 toolchain, so the new mock method is hand-written in the exact v2.20.0 format (a future correct regen is diff-free). Mocks are not part of the verify-generated CI check.

Changes

  • api/.../types.go: RedisSettings.PreventMasterEviction (+ regenerated CRD).
  • service/k8s/pod.go: UpdatePodAnnotations + mock.
  • check.go / heal.go: shared applyMasterEvictionAnnotation wired into both relabel paths.
  • Tests for the annotation logic; README section.

Upstream references

Test

go build ./..., go test ./..., gofmt clean; CRD regenerated via pinned codegen image.

Add an opt-in redis.preventMasterEviction flag. When enabled, the
operator annotates the current master pod with
cluster-autoscaler.kubernetes.io/safe-to-evict=false and marks slaves
true, so the cluster autoscaler will not drain the node running the
master and force an avoidable failover. The annotation follows the
master as the role moves, in both the checker and healer relabel paths.

Adds a Pod.UpdatePodAnnotations service method (JSON merge patch) and the
matching mock. Regenerated the CRD for the new field.

Refs upstream spotahome#689.
@binhnguyenduc
binhnguyenduc merged commit 85bc2f3 into master Jul 24, 2026
8 checks passed
@binhnguyenduc
binhnguyenduc deleted the feat/autoscaler-skip-master-eviction branch July 24, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant