fix(operator): make PlanetScale setup-phase applies recoverable after crash

[morgo]

A PlanetScale apply moves through engine setup-phase states — preparing_branch, applying_branch_changes, validating_branch, creating_deploy_request, validating_deploy_request — before per-table work begins. These states matched no clause of the operator claim queries: not pending, not in the claimable set, not terminal. So a worker crash mid-setup left the apply non-terminal and permanently unclaimable, with no operator able to recover it.

This adds the setup-phase states to claimableApplyStates, so a stale-heartbeat apply stuck in setup is reclaimed for recovery just like the other in-flight states. A healthy worker keeps its heartbeat fresh, so an in-progress setup is never claimed out from under it; terminal states stay excluded. Both the apply-level (FindNextApply, ClaimApplyByID) and operation-level (FindNextApplyOperation) claim paths share this set and now treat setup-phase applies consistently.

worker crash during setup
  apply.state = applying_branch_changes (non-terminal)
  heartbeat goes stale
    before: matches no claim clause -> wedged forever
    after:  matches stale-active clause -> operator reclaims and resumes

Merge order: This should land after #265 (resume with persisted engine state). On main today, a claimed setup-phase Vitess apply resumes through the path that re-runs engine setup without persisted state, which can duplicate branch/deploy-request work. Merging this before #265 converts "wedged after crash" into "recovered wrongly," which is worse. The deleted negative test is correct only once #265 is in.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes a recovery gap in the operator claim logic where PlanetScale engine setup-phase applies (e.g., preparing_branch, applying_branch_changes, validating_branch, creating_deploy_request, validating_deploy_request) could become permanently unclaimable after a worker crash, leaving them stuck non-terminal without any operator being able to resume them.

Changes:

• Adds PlanetScale setup-phase states to the shared claimableApplyStates set so stale-heartbeat setup applies can be reclaimed.
• Adds unit test coverage for reclaiming stale setup-phase applies via both apply-level and operation-level claim paths.
• Updates integration claimability expectations to treat setup-phase PlanetScale states as claimable (when stale) and removes the prior “not claimed” integration test.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
pkg/storage/mysqlstore/applies.go	Expands claimableApplyStates() to include PlanetScale setup-phase states and updates rationale comment.
pkg/storage/mysqlstore/applies_test.go	Adds tests ensuring stale setup-phase applies are reclaimable via FindNextApply and ClaimApplyByID.
pkg/storage/mysqlstore/apply_operations_test.go	Adds an operation-claim-path test verifying recovery can lease a stale operation and then claim its setup-phase parent apply.
integration/operator_test.go	Updates claimability matrix to expect setup-phase PlanetScale states to be claimable; removes obsolete “not claimed” test.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aparajon]

The state set is exactly the five setup-phase states (terminal, failed_retryable, and stopped correctly excluded — each has its own path), all three claim paths share the same predicate, and the heartbeat-before-engine-apply ordering means a healthy worker can't be claimed out from under during long branch waits. The fixture engine/db-type mismatch from earlier review was properly fixed.

One sequencing concern that should gate the merge: this should land after #265 (resume with persisted engine state). On main today, a claimed setup-phase Vitess apply resumes through the path that re-runs engine setup without persisted state — which can duplicate branch/deploy-request work. The deleted negative test was the guard for exactly that scenario; deleting it is correct once #265 is in, but merging this first converts "wedged after crash" into "recovered wrongly," which is worse. Suggest noting the dependency in the PR body (or stacking the branches) so merge order is explicit.

Smaller: the tests prove claimability at both levels, but nothing exercises the full claim → resume → engine-reattach flow for a setup-phase crash (branch exists, no deploy request yet). With #265's tests adjacent this is acceptable; a combined integration test would close the loop.

Verdict: LGTM contingent on landing after #265.

---

🤖 This review was generated by Claude Code (claude-fable-5) with maintainer approval.

[aparajon]

Approving with an explicit sequencing condition: land this only after #265 (resume with persisted engine state) merges with its save-error fix. On main today, a claimed setup-phase Vitess apply would resume through the path that can duplicate engine work — the deleted negative test was the guard for exactly that. The claimable state set itself is verified correct.

🤖 Approval issued by Claude Code (claude-fable-5) with maintainer approval.