fix(operator): place live backup workers with their RWO volume before disabling saves #717

Live Restic backups can schedule on a different node from the game that already mounts their ReadWriteOnce volume. With attachable RWO CSI disks, that can leave the worker waiting for attachment after the operator has disabled game saving. Backup worker preparation and a separate quiesce budget should prevent a scheduling failure from extending the save-disabled window.

Evidence comes from static inspection of master 04e41075cafc4c1a99bcb3723b0a8745ee810957; the multi-node failure has not been reproduced in this review. Node-local PV affinity can prevent the wrong-node placement, and storage-driver behavior matters.

Trigger and impact

Run a game on node A with an attachable RWO data PVC. Start a quiesced backup while node B has more spare capacity. The backup Pod has no placement constraint tying it to A, so the scheduler can place it on B and attachment can stall. For Minecraft, the game continues running with autosave disabled while the Job waits. On a cluster hosting 100 or more busy games, simultaneous backup preparation also competes for CPU, memory, and disk capacity.

The existing Job deadline bounds eventual Job failure, but its default is 24 hours and includes scheduling, startup, lock waiting, and copying. It is not a suitable default budget for disabling saves.

Pinned evidence

• Primary PVCs use ReadWriteOnce.
• Quiesce precedes Job creation.
• The backup Pod mounts the live PVC without placement or container resource settings.
• Job defaults and resolution; Restic can also wait ten minutes for a repository lock.
• Minecraft disables saves and flushes before backup.

Recommended implementation

Resolve and pin the source pod identity under lifecycle ownership. For a live copy, require same-node affinity to that exact workload on kubernetes.io/hostname, propagate required tolerations, and detect source migration before starting the copy. Handle stopped targets through a separate drained-workload path.

Introduce a preparation phase before quiesce: schedule a waiting worker, mount the PVC, pull its image, and validate repository access. Require a generation-bound start token before it reads the source. Add configurable worker resource requests/limits and a bounded preparation deadline.

Record a separate maximum quiesce duration. On expiry, mark the copy invalid, cancel it, drain its workers, and retry unquiesce with a visible failure condition. Document the recovery path when worker drainage or the agent is unavailable. Keep the full Job deadline as a separate backstop. A snapshot-then-upload strategy is an alternative where CSI support permits a shorter quiesce interval.

Acceptance criteria

• A live backup shares the game's node on a two-node attachable-RWO test fixture.
• Scheduling, image-pull, mount, and repository-preparation failures occur before save-off.
• Source pod replacement or relocation cannot start a copy using stale readiness.
• Configured worker resources and both preparation/quiesce deadlines reach the generated objects or orchestration state.
• CI verifies cancellation, cleanup, and operator restart without weakening existing worker drainage.

Preserve the configurable retry/deadline behavior from #573 and UID-bound ownership and cleanup intent from #590.