Add AssetAndTimeSchedule timetable#58543
Conversation
c752a20 to
760844c
Compare
AssetAndTimeScheduleAssetAndTimeSchedule
|
There was a problem hiding this comment.
Pull request overview
Adds an AssetAndTimeSchedule timetable to support time-based scheduling where scheduled DagRuns only start once an asset condition is satisfied (i.e., “time AND assets”), without creating additional asset-triggered DagRuns.
Changes:
- Introduces
AssetAndTimeSchedulein both the Task SDK (authoring) and Airflow core (runtime) with serialization/deserialization support. - Prevents
AssetAndTimeScheduledags from being treated as asset-triggered inDagModel.dags_needing_dagruns, and adds scheduler logic to start queued runs only when asset conditions are met. - Adds unit tests and documentation describing the new schedule behavior and usage.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| task-sdk/src/airflow/sdk/definitions/timetables/assets.py | Adds SDK AssetAndTimeSchedule wrapper timetable for DAG authoring. |
| task-sdk/src/airflow/sdk/init.pyi | Exposes AssetAndTimeSchedule in the SDK typing surface. |
| task-sdk/src/airflow/sdk/init.py | Exposes AssetAndTimeSchedule at runtime via SDK exports. |
| task-sdk/docs/api.rst | Documents the new SDK class in generated API docs. |
| airflow-core/src/airflow/timetables/assets.py | Adds core AssetAndTimeSchedule timetable implementation + (de)serialization. |
| airflow-core/src/airflow/serialization/encoders.py | Registers AssetAndTimeSchedule as a built-in timetable and encodes it. |
| airflow-core/src/airflow/models/dag.py | Filters out non-asset-trigger timetables from triggered_date_by_dag calculation. |
| airflow-core/src/airflow/jobs/scheduler_job_runner.py | Defers starting queued runs for AssetAndTimeSchedule until assets are ready; consumes ADRQ rows on start. |
| airflow-core/tests/unit/timetables/test_assets_timetable.py | Adds serialization/deserialization and delegation tests for the new timetable. |
| airflow-core/tests/unit/jobs/test_scheduler_job.py | Adds scheduler tests for asset gating, ADRQ consumption behavior, and timeout behavior. |
| airflow-core/docs/authoring-and-scheduling/timetable.rst | Adds docs comparing AssetOrTimeSchedule vs AssetAndTimeSchedule and usage examples. |
| airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst | Adds a dedicated section explaining “gate scheduled runs on asset updates”. |
|
Copilot's feedback are helpful, I've fixed both issues.
I've updated the link in the PR description to a permalink.
I think that would require significant changes across the API and frontend. I'd prefer to keep this PR focused on the core functionality and address the UI improvements in a follow-up if needed. |
d5612a8 to
a94354b
Compare
7c8452e to
0d9487f
Compare
Lee-W
left a comment
There was a problem hiding this comment.
one last question, but mostly good
|
It's not a feature I would ever use, but I'm good with it, and it seems there's an actual use case. Would like to know how @uranusjr thinks. Thanks! |
b590149 to
9e42573
Compare
…with asset conditions
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
…il all required assets are queued
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
The annotation on AssetAndTimeSchedule.__init__ was migrated to Collection[SerializedAsset] | SerializedAssetBase, leaving the TYPE_CHECKING-only Asset import unreferenced and breaking mypy on two test files that passed SDK Assets to the core class. The scheduler tests now import AssetAndTimeSchedule (and the matching CronTriggerTimetable) from airflow.sdk so the types align with how DAG authors use it; the timetable test fixture mirrors core_asset_timetable by constructing SerializedAsset directly.
9e42573 to
9b94c1b
Compare
related: #58056
Why
Use case: #58056
How
1. Implement
AssetAndTimeScheduleclassBasically almost the same as AssetOrTimeSchedule, but this class inherits from
Timetableand delegatesrun_idgeneration to the wrapped timetable.2. Assign
AssetAndTimeScheduleto thenon_asset_dagssetModifying DagModel.dags_needing_dagruns to prevent
AssetAndTimeSchedulefrom being assigned totriggered_date_by_dag.3. Add starting logic for
AssetAndTimeSchedulein the scheduler.Updating the DAG status from
QUEUEDtoRUNNINGwhen both the timetable and asset conditions are meet.What
Here are my test DAGs:
upstream_asset_producerdownstream_asset_and_time_consumerFrom the screenshots, the downstream DAG runs as expected: it runs only after both the timetable and the assets are ready.
Upstream dag
Asset production time
Downstream,
AssetAndTimeScheduleDiscussion
Since I added this logic in
dag.pyto guarantee that onlyAssetTriggeredTimetablecan be added totriggered_date_by_dag.Should I remove the extra guard logic in
SchedulerJobRunner._create_dag_runs_asset_triggered?airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py
Lines 1800 to 1805 in 9e7b36e