release: automate safe nightly macOS beta qualification, promotion, and health monitoring

[Git-on-my-level]

Goal

Make macOS beta releases fully automated and safe on a nightly cadence: protected main candidate → signed/notarized immutable artifact → trusted macOS qualification → automatic beta-only pointer advance → one-hour health monitor/rollback response.

Stable and shared backend production deployment remain explicitly manual and outside this scope.

Current state

The repository already has most pipeline stages:

• .github/workflows/desktop_auto_release.yml creates one daily candidate;
• Codemagic builds, signs, notarizes, smokes, and publishes the candidate;
• .github/workflows/desktop_qualify_beta.yml runs exact-tag qualification on [self-hosted, macos, omi-desktop-qualification];
• .github/workflows/desktop_promote_beta.yml validates canonical qualification evidence and advances the beta pointer.

The operational gaps exposed by the .85 incident:

1. Codemagic qualification dispatch failed with HTTP 403, leaving a published candidate unqualified.
2. The trusted qualification runner was unavailable; local Docker qualification was unreliable.
3. Normal automated beta promotion is gated by the human-reviewed prod GitHub environment even though it changes only the beta control-plane pointer.
4. The merged source for audited emergency beta rollback needs a separately reviewed control-plane deployment before it can be relied on.

Deliverables

A. Dedicated trusted qualification runner

• Provision an always-on managed Mac and register it as a GitHub self-hosted runner with labels:
  self-hosted, macos, omi-desktop-qualification.
• Manage the runner via launchd so it survives reboot/network interruption.
• Install and continuously health-check Docker Desktop/OrbStack, Xcode/Swift, Python, gh, and required caches/disk headroom.
• Restrict the runner to this repository/release runner group; never run pull-request or arbitrary-ref code there.
• Add runner readiness monitoring and alert when it is offline, busy beyond an SLA, Docker-unhealthy, or low on disk.

B. Reliable automatic qualification dispatch

• Remove the single point of failure on Codemagic workflow dispatch.
• Preferred implementation: a GitHub scheduled reconciliation workflow (every 5–10 minutes) that discovers the newest published v*-macos candidate, validates published smoke/release metadata, and dispatches exact-tag qualification iff canonical qualifiedBeta evidence is absent.
• Ensure idempotency and a shared concurrency group; never qualify stale or superseded candidates automatically.
• Alternatively/follow-up, repair Codemagic's narrowly scoped actions:write dispatch credential.

C. Automatic normal beta promotion

• Create a separate desktop-beta-control GitHub environment for normal fully qualified automatic beta promotion.
• Use a least-privilege credential that can only move platform=macos, channel=beta through the control-plane API.
• Do not give it backend deployment, Stable-promotion, or broad admin capabilities.
• Keep server-side exact tag/SHA, artifact digest, signed-smoke, Tier-2/fault evidence, expected-current pointer/generation CAS, audit, and Stable non-impact checks mandatory.
• Keep manual/exceptional promotion under a human-reviewed environment; do not weaken Stable/backend controls.

D. Nightly cadence and health monitoring

• Schedule one nightly candidate from protected main, only when releasable desktop changes exist.
• On successful beta pointer advancement, run independent one-hour monitoring for:
  • beta pointer/appcast and expected artifact digest;
  • Stable pointer unchanged;
  • release-specific crash/startup health;
  • update/download availability.
• Persist an auditable health report and alert on regression.

E. Rollback readiness

• Separately deploy and verify the already merged macOS-beta-only emergency rollback control-plane capability before enabling automated rollback action.
• Default health response should alert/recommend rollback; automated rollback must be narrowly scoped to the prior verified beta pointer with CAS/audit and an explicit policy.

Acceptance criteria

1. With a healthy runner, a newly published nightly candidate reaches beta without a human GitHub approval after all normal qualification evidence passes.
2. No production backend or Stable action occurs in the automated path.
3. A failed Codemagic dispatch cannot strand an eligible candidate; reconciliation dispatches it automatically.
4. The qualification runner rejects PR/arbitrary-ref execution and reports health/offline status.
5. Automatic promotion cannot advance an unqualified, stale, mismatched-digest, non-macOS, or non-beta candidate.
6. A full dry run with a non-live candidate demonstrates candidate → qualification → beta-control request → health-monitor report.
7. The post-promotion monitor proves Stable remains unchanged and records a one-hour outcome.
8. Rollback readiness is tested before any policy allows an automatic rollback.

Related

• release: add audited macOS beta emergency forward-promotion lane #10063 — audited macOS beta emergency forward-promotion lane
• fix: add audited macOS beta emergency rollback #10051 — merged source for audited macOS beta emergency rollback
• .github/workflows/desktop_auto_release.yml
• .github/workflows/desktop_qualify_beta.yml
• .github/workflows/desktop_promote_beta.yml