feat(reliability): add PodDisruptionBudget and pod anti-affinity for Temporal workers

[kiwueke2]

Description

Adds a PodDisruptionBudget (maxUnavailable: 1) for both Temporal worker deployments — the Temporal server worker role and the nv-config-manager workflow executor so node drains can't evict all replicas at once.

Also activates the pod anti-affinity rules that were already in values.yaml but commented out. With preferredDuringSchedulingIgnoredDuringExecution on kubernetes.io/hostname, the scheduler will spread the 3 replicas across different nodes rather than potentially stacking them on one. Added the app.kubernetes.io/component: temporal-worker label to the worker pod template so the PDB selector and anti-affinity rule actually land on the right pods.

Validation

Verified locally with helm template command that both PDBs render with correct selectors, the new label appears on the worker pod template, and the anti-affinity block is present in the Deployment spec.

helm template kiwi deploy/helm \
  --values deploy/helm/values.yaml \
  --values deploy/helm/values-local.yaml \
  --set secrets.method=kubernetes \
  --set externalServices.nautobot.server=dummy.local \
  --set externalServices.nautobot.local=false \
  --set externalServices.redis.host=dummy.local \
  --set externalServices.nats.server=dummy.local \
  --namespace kiwi-qa \
  2>&1 | grep -A 20 "kind: PodDisruptionBudget"

• Standard CI passes.
• Kind integration passes, or this PR explains why it was not run.

The kind integration test is manual due to taking ~30 min to complete. When the PR is ready for review,
run Actions -> Kind Integration -> Run workflow against the copy-pr-bot generated
pull-request/<PR_NUMBER> branch. Use the default test_path for the full suite,
or narrow it only while debugging.

Passing Kind Integration run:

Checklist

• I am familiar with the contributing guidelines in CONTRIBUTING.md.
• Commits are signed off for DCO compliance.
• New or existing tests cover these changes, or the PR explains why tests are not needed.
• Documentation is updated for user-facing behavior changes.
• Generated artifacts are updated when applicable, such as OpenAPI specs,
  docs screenshots, or Helm/rendered outputs.

Summary by CodeRabbit

• New Features
  • Added PodDisruptionBudgets (maxUnavailable: 1) for Temporal server workers and nv-config-manager workflow-executor workers.
• Bug Fixes
  • Updated worker labeling and PodMonitor selection so monitoring and scheduling/disruption rules apply to the correct worker roles.
• Documentation
  • Added “Worker Reliability” guidance for voluntary node disruptions, including eviction expectations, inspecting PDBs, and validating pod spread with preferred pod anti-affinity.
• Refactor
  • Enabled preferred pod anti-affinity for both worker types to spread replicas across nodes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b925641d-96e5-4647-af0c-554dc9faacc9

📥 Commits

Reviewing files that changed from the base of the PR and between 6389177 and 989cda7.

📒 Files selected for processing (3)

• deploy/helm/templates/monitoring.yaml
• deploy/helm/templates/temporal.yaml
• docs/temporal/temporal-deployment.mdx

✅ Files skipped from review due to trivial changes (1)

• docs/temporal/temporal-deployment.mdx

🚧 Files skipped from review as they are similar to previous changes (1)

• deploy/helm/templates/temporal.yaml

---

📝 Walkthrough

Walkthrough

The Helm chart updates Temporal worker labels, adds PodDisruptionBudgets, enables preferred anti-affinity for both worker deployments, updates monitoring selection, and documents the resulting reliability and verification behavior.

Changes

Temporal Worker Reliability

Layer / File(s)	Summary
Worker labels and PDBs deploy/helm/templates/temporal.yaml, deploy/helm/templates/monitoring.yaml	The shared Temporal deployment template adds a worker-only component label, adds a worker label to the config manager worker pod template, updates the worker PodMonitor selector, and defines two worker PodDisruptionBudgets with maxUnavailable: 1.
Worker anti-affinity deploy/helm/values.yaml	Temporal worker and configManagerWorker values switch to active preferred anti-affinity rules using the component label selectors.
Worker reliability docs docs/temporal/temporal-deployment.mdx	The Temporal deployment guide adds Worker Reliability guidance for PodDisruptionBudgets, anti-affinity, and verification commands.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main reliability changes: PodDisruptionBudgets and pod anti-affinity for Temporal workers.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/temporal-worker-reliability

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/helm/templates/temporal.yaml`:
- Around line 778-794: The PodDisruptionBudget selector is too broad and matches
both worker pods via the shared app/component labels. Update the
nv-config-manager worker pod template to add a unique identifying label, then
change this PDB’s selector in the temporal.yaml template to match that label
instead of only app: {{ $temporalName }} and component: worker. Use the existing
nv-config-manager.temporalWorkerName resource and its pod template labels to
keep the selector scoped to the intended worker only.

In `@deploy/helm/values.yaml`:
- Around line 1731-1742: The anti-affinity settings are attached at the wrong
level, so `configManagerWorker.affinity` remains empty. Move the `affinity`
block to be nested under `temporal.configManagerWorker` in the Helm values,
keeping the existing `podAntiAffinity` structure intact so the worker Deployment
reads `.Values.temporal.configManagerWorker.affinity` correctly.

In `@docs/temporal/temporal-deployment.mdx`:
- Around line 60-69: The verification example only checks the Temporal server
worker pods, so it does not actually cover both worker Deployments described in
this section. Update the verification guidance near the pod anti-affinity
explanation to include the nv-config-manager workflow-executor pods as well,
either by documenting its selector explicitly or by using the new unique label
once it exists, so the command sequence verifies spread for both worker sets.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4ae18d1e-0b29-4fd8-afc8-dea22ff0e7c0

📥 Commits

Reviewing files that changed from the base of the PR and between b3ff88d and 51d3f99.

📒 Files selected for processing (3)

• deploy/helm/templates/temporal.yaml
• deploy/helm/values.yaml
• docs/temporal/temporal-deployment.mdx

[kiwueke2]

Verifications on local deployment

1. Both PDBs are active.

kiwueke@K3W96WM0GH ~ % kubectl get pdb -n nv-config-manager
NAME                                                      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
nv-config-manager-temporal-nv-config-manager-worker-pdb   N/A             1                 1                     41m
nv-config-manager-temporal-worker-pdb                     N/A             1                 1                     41m
kiwueke@K3W96WM0GH ~ %

2. The configManagerWorker PDB is working correctly. Current: 2 — it found 2 pods matching selector app=nv-config-manager-temporal, component=worker

kiwueke@K3W96WM0GH ~ % kubectl describe pdb nv-config-manager-temporal-nv-config-manager-worker-pdb -n nv-config-manager
Name:             nv-config-manager-temporal-nv-config-manager-worker-pdb
Namespace:        nv-config-manager
Max unavailable:  1
Selector:         app=nv-config-manager-temporal,component=worker
Status:
    Allowed disruptions:  1
    Current:              2
    Desired:              1
    Total:                2
Events:
  Type    Reason  Age                From               Message
  ----    ------  ----               ----               -------
  Normal  NoPods  41m (x2 over 41m)  controllermanager  No matching pods found

3. Confirm the temporal-server-worker pod carries the expected label app.kubernetes.io/component=temporal-worker.

kiwueke@K3W96WM0GH ~ % kubectl get pod -n nv-config-manager -l app.kubernetes.io/component=temporal-worker \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels}{"\n"}{end}'
nv-config-manager-temporal-worker-857644c64f-2r96l	{"app":"nv-config-manager-temporal","app.kubernetes.io/component":"temporal-worker","component":"worker","pod-template-hash":"857644c64f"}

4. Anti-affinity is live in the temporal-server-worker deployment spec. The policy: prefer to schedule away from nodes already running a pod with app.kubernetes.io/component=temporal-worker, topology key kubernetes.io/hostname. In production with 3 replicas across 3+ nodes, this distributes each replica to a distinct node.

kiwueke@K3W96WM0GH ~ % kubectl get deployment nv-config-manager-temporal-worker -n nv-config-manager \
  -o jsonpath='{.spec.template.spec.affinity}' | python3 -m json.tool
{
    "podAntiAffinity": {
        "preferredDuringSchedulingIgnoredDuringExecution": [
            {
                "podAffinityTerm": {
                    "labelSelector": {
                        "matchExpressions": [
                            {
                                "key": "app.kubernetes.io/component",
                                "operator": "In",
                                "values": [
                                    "temporal-worker"
                                ]
                            }
                        ]
                    },
                    "topologyKey": "kubernetes.io/hostname"
                },
                "weight": 100
            }
        ]
    }
}

5. Same confirmation for the configManagerWorker. Policy is identical in structure but uses component=worker as the label key, which is the actual label stamped on these pods.

kiwueke@K3W96WM0GH ~ % kubectl get deployment nv-config-manager-temporal-nv-config-manager-worker -n nv-config-manager \
  -o jsonpath='{.spec.template.spec.affinity}' | python3 -m json.tool
{
    "podAntiAffinity": {
        "preferredDuringSchedulingIgnoredDuringExecution": [
            {
                "podAffinityTerm": {
                    "labelSelector": {
                        "matchExpressions": [
                            {
                                "key": "component",
                                "operator": "In",
                                "values": [
                                    "worker"
                                ]
                            }
                        ]
                    },
                    "topologyKey": "kubernetes.io/hostname"
                },
                "weight": 100
            }
        ]
    }
}

[kiwueke2]

/ok to test 6389177