[v3-1-test] Fix dag processor callback cleanup for versioned bundle files (#66484)#69771
Open
Vamsi-klu wants to merge 2 commits into
Open
[v3-1-test] Fix dag processor callback cleanup for versioned bundle files (#66484)#69771Vamsi-klu wants to merge 2 commits into
Vamsi-klu wants to merge 2 commits into
Conversation
…#66484) * Fix callback orphan cleanup for versioned bundle files closes: apache#66483 * Remove newsfragment for callback orphan cleanup fix * Fix versioned dag file presence checks * Preserve public signatures for versioned dag file checks * Preserve manager cleanup extension points
The backport already fixes the versioned bundle callback cleanup path, but the follow-up keeps the regression coverage deterministic and easier to maintain before release-manager review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #66484 ("Fix dag processor callback cleanup for versioned bundle files") to
v3-1-test, addressing #63374 where Dag-levelon_failure_callbackcould be dropped for versioned/git bundles before it was processed.Problem
The Dag File Processor queued callback files with a
DagFileInfowhosebundle_versionparticipates in equality. The periodic scan recreated the same file withbundle_version=None, so cleanup treated the queued versioned callback file as missing and purged it before processing.What Changed
presence_keyhandling from Fix dag processor callback cleanup for versioned bundle files #66484.v3-1-testStats.gaugespelling while resolving the cherry-pick context conflict.processed_recently()helper and tightens the new regression tests with deterministic time and specced/autospecced mocks.Why The Follow-Up Commit Exists
The review feedback was correct: the backport carried a now-unused helper and a few tests were too loose for release-branch review. The follow-up does not change the fix behavior; it makes the backport smaller and the test evidence stronger.
Impact
The backport should only affect cleanup/termination decisions for versioned callback files. Normal Dag file processing and unversioned callback cleanup continue to use the same presence logic.
No new newsfragment was added because the original user-facing changelog entry already exists for #66484 on
main.related: #63374
Testing
AIRFLOW_HOME=/tmp/airflow-home-69771 UV_CACHE_DIR=/tmp/uv-cache-69771 uv run --project airflow-core pytest airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_handle_removed_files_uses_public_extension_points airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_purge_removed_files_keeps_versioned_callback_file_when_unversioned_file_is_present airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_terminate_orphan_processes_keeps_versioned_callback_processor_when_unversioned_file_is_present airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_terminate_orphan_processes_kills_processor_when_file_is_truly_absent airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_add_new_files_to_queue_skips_versioned_files_already_represented airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_prepare_file_queue_skips_file_when_versioned_processor_is_in_progress airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_prepare_file_queue_skips_recently_processed_file_with_versioned_stats --with-db-init -xvs7 passed, 1 warningWas generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-5) following the guidelines
Drafted-by: Codex (GPT-5); reviewed by @Vamsi-klu before posting