A transient umami 500 fails Deploy to Production and evicts PRs from the merge queue

[devantler]

🤖 Generated by the Daily AI Engineer

Problem

A transient HTTP 500 from umami — the analytics app — fails the entire 🚀 Deploy to Production step, which fails the merge_group run, which evicts the PR from the merge queue. An unrelated dependency bump therefore cannot merge because an analytics tenant-provisioning call blipped.

Observed 2026-07-14 on renovate PR platform#2611 (run 29329060647):

Error: could not authenticate to umami after retries; last login failure: HTTP 500
Error: list teams failed: 500
    at ensureTeam ([eval]:54:20)

The deploy was otherwise fine — the kubeconfig worked throughout (it was reading live pod logs), and neighbouring merge_group deploys succeeded (pr-2612 10:45Z, pr-2608 09:03Z, pr-2630 08:40Z).

It really was transient. The umami-provision-tenants CronJob failed during the deploy window and then completed successfully ~12 minutes later, in 11 seconds. umami itself is healthy (umami-umami-primary 2/2 Running, CNPG umami-db healthy, 3 instances). But there is a recurring pattern of intermittent failures in that CronJob's history (failures at 7d, 5d and 10h ago, interleaved with successes).

Why it matters

The blast radius is disproportionate to the fault:

• A best-effort, idempotent tenant-provisioning step for an analytics app is a hard gate on prod delivery.
• Because platform merges go through a merge queue, the failure doesn't just fail a check — it evicts the PR, so the merge lane silently stalls and every queued change waits on an unrelated 500.
• It is indistinguishable at a glance from a real deploy failure, which costs a root-cause pass every time (this one did).

Proposed direction

Make the umami tenant-provisioning step non-gating for the deploy, while still surfacing its failure:

1. Retry with backoff around the umami login / list teams calls (it already retries once — the window here outlasted it), and
2. let the step fail soft — report the failure (annotation / alert) without failing Deploy to Production. Provisioning is idempotent and the CronJob re-runs on schedule, which is exactly what already self-healed it here.

Alternatively, move tenant provisioning out of the deploy path entirely and let the CronJob own it, with an alert if it stays failed across N runs.

Either way the principle is the same: an analytics-app blip should not be able to evict PRs from the merge queue.

Rough size

S–M — one workflow/script step plus an alert.

Found while root-causing platform#2611's eviction (re-queued; the transient had cleared).

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• fix: harden merge-queue deploy-prod verification against Cilium mutual-auth first-packet drop (queue wedges) #2337

🔗 Related PRs

#2317 - ci: migrate deploy-prod Flux diagnostics to shared diagnose-flux composite [merged]
#2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
#2339 - docs(agents): record merge-queue mechanics + root-cause-before-requeue [merged]
#2404 - fix: tenant artifact verification rejects the dual signing identities from #2399 [merged]
#2488 - feat(tenants): prune-protect the ascoachingogvaner skeleton for the KRO handover [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Daily AI Engineer

Closing — I filed this on a wrong diagnosis

The premise of this issue is false, and I would rather retract it than leave it to send someone down the same false trail.

I read could not authenticate to umami ... HTTP 500 / list teams failed: 500 in the failed deploy log and concluded a transient umami blip had failed Deploy to Production and evicted platform#2611. Those lines were dumped diagnostics — the deploy prints the logs of pre-existing failed Job pods on failure — not the failing assertion.

The actual failure was:

✗ infrastructure-controllers  HealthCheckFailed
    timeout waiting for HelmRelease/ksail-operator status: 'InProgress'

The ksail-operator cannot upgrade because the Talos nodes cannot pull any new ksail image (stale GHCR credential — every ReplicaSet since v7.168.0 is scaled to 0 while prod runs v7.167.0). Full evidence on platform#2611. That is tracked under #2613.

What remains true but is much narrower than this issue claims: the umami-provision-tenants CronJob does fail intermittently (failed runs 7d / 5d / 10h ago, interleaved with successes, and one completed in 11s during this window). That is worth looking at on its own merits — but it is a CronJob flake, it is not gating prod deploys, and it did not evict anything.

Closing as invalid rather than editing it, because the headline claim — "an analytics blip evicts PRs from the merge queue" — is simply not what happened. If the umami CronJob flake deserves tracking, it should be filed as what it is.

Lesson recorded for myself: read the failing assertion, not the diagnostic dump the failure handler prints.