operator: stage OCI template upgrades before rolling all referencing game servers #731

A Module update replaces a shared GameTemplate, and the operator reconciles each referencing GameServer against the new pod template. A catalog update can therefore become a fleet rollout without per-server maintenance approval or a staged adoption policy.

Priority: Medium; production-readiness enhancement. Evidence: static inspection of upstream 04e41075cafc4c1a99bcb3723b0a8745ee810957 (2026-10-16). The failure scenarios below have not been reproduced in a running cluster.

Evidence and limits

• operator/internal/controller/module_controller.go:101 resolves an empty Module version to the catalog's latest version.
• operator/internal/controller/module_controller.go:296 verifies template ownership, then replaces the existing template spec.
• operator/internal/controller/gameserver_controller.go:489 maps template changes to referencing servers.
• operator/internal/controller/gameserver_controller.go:1510 rewrites the StatefulSet pod template.
• operator/internal/controller/gameserver_controller.go:551 assigns the primary PVC spec only on creation and retains its GameServer ownership.

The concern is disruption and application/save compatibility, not an assertion that a normal template upgrade deletes PVCs. Configured signature checks, digest handling, minimum-operator checks, ownership validation, and retained extra/mod PVCs are useful safeguards.

Recommended fix

1. Separate installing a verified template revision from adopting it on running servers. Give each server an explicit effective revision/digest and a pending-update condition.
2. Add an opt-in rollout policy with maintenance windows, bounded concurrency, canaries, and pause/resume controls. Preserve a documented policy for existing installations rather than changing their behavior silently.
3. Route rollout through the same lifecycle exclusion used by backups/restores/wipes and through the game's graceful stop contract. Do not replace a quiesced process during a protected copy.
4. Pin production Module versions until staged adoption exists. Treat version pinning as an immediate operational control, not a complete rollback mechanism.
5. Record the image/template revision and relevant configuration with backups. Explain that save-format changes may require restoring data along with an older image.
6. Document PVC retention for updates separately from GameServer deletion; deletion can garbage-collect GameServer-owned claims.

Acceptance criteria

• Installing a revision can leave current games running until explicitly admitted by rollout policy.
• A canary failure stops further adoption; concurrency limits hold across operator restart/leader changes.
• Active backups/restores prevent rollout and admission resumes after cleanup.
• Upgrade tests verify retained PVC identity and contents, and a representative game's save compatibility.
• Rollback guidance distinguishes reverting the runtime from restoring application data.
• Shared-template changes do not bypass per-server maintenance policy.