fix(operator): back up stopped targets without retaining an unreachable-agent quiesce guard #716

A scheduled backup of an idle-asleep or manually stopped GameServer can fail before creating its copy Job and then block later quiesced backups until the server wakes. This concerns installs with the agent client enabled and spec.quiesce=true. Inline backup policies set that flag to true.

Evidence comes from static inspection of master 04e41075cafc4c1a99bcb3723b0a8745ee810957; this report does not claim a runtime reproduction.

Trigger and impact

Let a server enter idle sleep, then let its backup schedule fire. The operator claims backup ownership, records pending quiesce cleanup, and calls the absent agent. It records the transport failure as a Failed Backup before creating a Restic Job. Terminal reconciliation keeps calling the absent agent to unquiesce and retains ownership. A later protected backup waits for the first owner, so the failed attempt can turn into a missed-backup chain. Manually waking the server permits cleanup but does not rerun the failed copy.

Pinned evidence

• Inline schedules enable quiesce.
• Quiesce intent, agent call, and failure handling; Job creation follows quiesce.
• Terminal cleanup retries and releases only after success; another owner blocks acquisition.
• Agent transport errors remain errors. The gone-target escape belongs to deletion cleanup, rather than normal terminal reconciliation.

Recommended implementation

Add an explicit stopped-target preparation path. Acquire durable target ownership before checking whether the game has stopped, then use uncached reads to confirm the owned StatefulSet has observed zero replicas and that game/writer pods have drained. Record an immutable preparation result such as already-stopped, bound to the GameServer UID and workload generation, and skip both quiesce and unquiesce for that copy. Hold the workload stopped until its copy workers finish; leave the user's suspend/idle intent intact.

For cleanup from existing failed attempts, distinguish a confirmed absent original process from an unreachable running process. Bind quiesce cleanup to the affected pod UID so a replacement cannot inherit an ambiguous cleanup obligation. An HTTP timeout alone must not authorize releasing protection. Coordinate the stopped-workload fence with the shared lifecycle exclusion follow-up.

Acceptance criteria

• Idle-asleep and manually stopped targets produce a restorable backup without waking the game.
• Subsequent scheduled copies proceed after cleanup, without deleting the Failed Backup to unblock them.
• A live but unreachable agent still prevents an unprotected copy.
• Concurrent wake, operator restart, pod replacement, and delayed cleanup preserve the target identity and workload fence.
• CI covers the new state transitions and retains existing quiesce serialization, worker-drain, and finalizer cases; no local test run accompanies this report.

Preserve the cleanup-before-release behavior from #590 and terminal retry/finalizer handling from #454.