Skip to content

Heal production after cancelled merge-group deploys #2621

Description

@devantler

🤖 Generated by the Codex Daily AI Engineer

Problem

The merge-queue production heal added for #2026 only runs when needs.deploy-prod.result == 'failure'. Cancelling a merge-group workflow after its deploy composite has started yields cancelled, so the heal job is cancelled/skipped even though workload push may already have moved the mutable latest artifact to an unmerged synthetic ref.

Live evidence: Platform merge-group run 29263580947 was cancelled after its 🚀 Deploy to Prod job had been in progress. The PR had already been dequeued, but 🩹 Heal Prod (restore main on merge-group failure) concluded cancelled with zero steps. There is therefore no proof that main was restored, and the current #2620 merge-group deploy must be allowed to finish naturally to heal or supersede the speculative artifact.

Expected behavior

Any merge-group run that may have entered the deployment composite but does not complete successfully must restore the current tip of main before the production lane is considered clean. Operator cancellation must not bypass that safety invariant.

Proposed direction

  • Make the heal path cover cancelled deployments as well as failed deployments.
  • Preserve the shared prod-deploy concurrency lock and current-main checkout.
  • Add a structural/behavioral workflow regression test for both failure and cancelled results.
  • Document the safe operator sequence for a stuck merge-group deploy so cancellation cannot strand a speculative artifact.

Acceptance criteria

  • A failed merge-group deploy triggers the current heal path.
  • A cancelled merge-group deploy that may have started publishing also schedules a current-main heal.
  • A successful merge-group deploy never runs the heal job.
  • The heal cannot race another production deployment.
  • Tests cover failure, cancellation, and success conditions.
  • Existing actionlint/workflow validation remains green.

Rough size

Small-to-medium: workflow condition hardening, regression coverage, and operator guidance.

Related: #2026.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    ✅ Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions