Skip to content

Prevent KubernetesExecutor from launching stale workloads#69762

Open
wolvery wants to merge 1 commit into
apache:mainfrom
wolvery:fix-kubernetes-stale-workload-race
Open

Prevent KubernetesExecutor from launching stale workloads#69762
wolvery wants to merge 1 commit into
apache:mainfrom
wolvery:fix-kubernetes-stale-workload-race

Conversation

@wolvery

@wolvery wolvery commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Fixes #69760

This adds a final DB preflight before KubernetesExecutor creates a worker pod for an Airflow 3 ExecuteTask workload. A workload can sit in the executor queue while Kubernetes pod creation is delayed, and in HA scheduler deployments the task instance can be retried, cleared, or otherwise moved on before the pod is eventually created.

The executor now verifies that the workload still matches the current queued task instance before calling run_next:

  • the immutable TaskInstance id still exists
  • the TaskInstance is still QUEUED
  • try_number still matches the workload
  • queued_by_job_id still belongs to this scheduler job

If the workload is stale, it is dropped before pod creation and executor bookkeeping for that launch is cleared.

Tests added:

  • KubernetesExecutor drops a stale ExecuteTask workload before pod creation when the DB task state has moved on

Local verification:

  • python3 -m compileall providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py
  • AIRFLOW_HOME=/var/folders/_l/8htlzsws7gjg8_p6ht_hl0tc0000gp/T/opencode/airflow-test-home uv run pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_sync_drops_stale_execute_task_workload_before_pod_creation -q --with-db-init
  • AIRFLOW_HOME=/var/folders/_l/8htlzsws7gjg8_p6ht_hl0tc0000gp/T/opencode/airflow-test-home uv run pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_run_next_exception_requeue providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_skip_pod_creation_on_create_pods_after -q
  • uv run ruff check providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py
  • uv run ruff format --check providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py

Gen-AI disclosure

  • Yes: GPT was used to assist in authoring this PR. The changes were reviewed and validated by the contributor before submission.

Generated-by: GPT following the guidelines: https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions

@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Jul 11, 2026
@shahar1 shahar1 requested a review from amoghrajesh July 11, 2026 16:27
@wolvery wolvery marked this pull request as draft July 11, 2026 21:07
@wolvery wolvery marked this pull request as ready for review July 12, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KubernetesExecutor can start stale worker pods for obsolete TaskInstance UUIDs under HA schedulers

1 participant