Skip to content

Add AssetAndTimeSchedule timetable#58543

Open
nailo2c wants to merge 14 commits into
apache:mainfrom
nailo2c:feat-58056-asset_and_time_schedule
Open

Add AssetAndTimeSchedule timetable#58543
nailo2c wants to merge 14 commits into
apache:mainfrom
nailo2c:feat-58056-asset_and_time_schedule

Conversation

@nailo2c
Copy link
Copy Markdown
Contributor

@nailo2c nailo2c commented Nov 20, 2025

related: #58056

Why

Use case: #58056

How

1. Implement AssetAndTimeSchedule class

Basically almost the same as AssetOrTimeSchedule, but this class inherits from Timetable and delegates run_id generation to the wrapped timetable.

2. Assign AssetAndTimeSchedule to the non_asset_dags set

Modifying ‎DagModel.dags_needing_dagruns‎ to prevent AssetAndTimeSchedule from being assigned to triggered_date_by_dag.

3. Add starting logic for AssetAndTimeSchedule in the scheduler.

Updating the DAG status from QUEUED to RUNNING when both the timetable and asset conditions are meet.

What

Here are my test DAGs:

upstream_asset_producer

import pendulum
from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator
from airflow.sdk.definitions.asset import Asset

my_asset = Asset("/my/example/asset")

with DAG(
    dag_id="upstream_asset_producer",
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    schedule="0 * * * *",
    catchup=False,
    tags=["example", "upstream"],
) as upstream_dag:
    BashOperator(
        task_id="produce_asset",
        bash_command="echo 'Asset produced!'",
        outlets=[my_asset],
    )

downstream_asset_and_time_consumer

import pendulum
from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator
from airflow.sdk.definitions.asset import Asset
from airflow.timetables.trigger import CronTriggerTimetable
from airflow.timetables.assets import AssetAndTimeSchedule

my_asset = Asset("/my/example/asset")

with DAG(
    dag_id="downstream_asset_and_time_consumer",
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    schedule=AssetAndTimeSchedule(
        timetable=CronTriggerTimetable("4 * * * *", timezone="UTC"),
        assets=[my_asset],
    ),
    catchup=False,
    max_active_runs=1,
    tags=["example", "downstream", "asset-and-time"],
) as downstream_dag:
    BashOperator(
        task_id="consume_asset",
        bash_command="echo 'Asset condition and time condition were both met!'",
    )

From the screenshots, the downstream DAG runs as expected: it runs only after both the timetable and the assets are ready.

  1. Upstream dag

    AssetAndTimeSchedule_example_1_upstream

  2. Asset production time

    AssetAndTimeSchedule_example_2_asset

  3. Downstream, AssetAndTimeSchedule

    AssetAndTimeSchedule_example_3_downstream

Discussion

Since I added this logic in dag.py to guarantee that only AssetTriggeredTimetable can be added to triggered_date_by_dag.

if not isinstance(timetable, AssetTriggeredTimetable):
    del adrq_by_dag[dag_id]
    continue

Should I remove the extra guard logic in SchedulerJobRunner._create_dag_runs_asset_triggered?

if not isinstance(dag.timetable, AssetTriggeredTimetable):
self.log.error(
"DAG '%s' was asset-scheduled, but didn't have an AssetTriggeredTimetable!",
dag_model.dag_id,
)
continue

@boring-cyborg boring-cyborg Bot added the area:Scheduler including HA (high availability) scheduler label Nov 20, 2025
@nailo2c nailo2c requested a review from Lee-W as a code owner March 22, 2026 04:49
@nailo2c nailo2c force-pushed the feat-58056-asset_and_time_schedule branch from c752a20 to 760844c Compare March 22, 2026 05:01
Comment thread airflow-core/src/airflow/timetables/assets.py
Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py Outdated
Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py Outdated
Comment thread airflow-core/src/airflow/models/dag.py
@eladkal eladkal changed the title feat-58056: Implement AssetAndTimeSchedule Add AssetAndTimeSchedule Mar 24, 2026
@eladkal eladkal added this to the Airflow 3.3.0 milestone Mar 24, 2026
@eladkal eladkal added kind:feature Feature Requests type:new-feature Changelog: New Features and removed kind:feature Feature Requests labels Mar 24, 2026
@eladkal eladkal changed the title Add AssetAndTimeSchedule Add AssetAndTimeSchedule timetable Mar 24, 2026
@Lee-W
Copy link
Copy Markdown
Member

Lee-W commented Apr 7, 2026

  1. Thanks! yep the permalink was probably the thing that made me confused 🤦‍♂️
  2. Sounds good to me!
  3. I'm thinking about whether we have a way to reduce the confusion. If I were to be the user, I might be confused. 🤔 Do you think there's an easy way to change how the schedule box displays based on the current status? (I "guess" not, but worth trying)

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an AssetAndTimeSchedule timetable to support time-based scheduling where scheduled DagRuns only start once an asset condition is satisfied (i.e., “time AND assets”), without creating additional asset-triggered DagRuns.

Changes:

  • Introduces AssetAndTimeSchedule in both the Task SDK (authoring) and Airflow core (runtime) with serialization/deserialization support.
  • Prevents AssetAndTimeSchedule dags from being treated as asset-triggered in DagModel.dags_needing_dagruns, and adds scheduler logic to start queued runs only when asset conditions are met.
  • Adds unit tests and documentation describing the new schedule behavior and usage.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
task-sdk/src/airflow/sdk/definitions/timetables/assets.py Adds SDK AssetAndTimeSchedule wrapper timetable for DAG authoring.
task-sdk/src/airflow/sdk/init.pyi Exposes AssetAndTimeSchedule in the SDK typing surface.
task-sdk/src/airflow/sdk/init.py Exposes AssetAndTimeSchedule at runtime via SDK exports.
task-sdk/docs/api.rst Documents the new SDK class in generated API docs.
airflow-core/src/airflow/timetables/assets.py Adds core AssetAndTimeSchedule timetable implementation + (de)serialization.
airflow-core/src/airflow/serialization/encoders.py Registers AssetAndTimeSchedule as a built-in timetable and encodes it.
airflow-core/src/airflow/models/dag.py Filters out non-asset-trigger timetables from triggered_date_by_dag calculation.
airflow-core/src/airflow/jobs/scheduler_job_runner.py Defers starting queued runs for AssetAndTimeSchedule until assets are ready; consumes ADRQ rows on start.
airflow-core/tests/unit/timetables/test_assets_timetable.py Adds serialization/deserialization and delegation tests for the new timetable.
airflow-core/tests/unit/jobs/test_scheduler_job.py Adds scheduler tests for asset gating, ADRQ consumption behavior, and timeout behavior.
airflow-core/docs/authoring-and-scheduling/timetable.rst Adds docs comparing AssetOrTimeSchedule vs AssetAndTimeSchedule and usage examples.
airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst Adds a dedicated section explaining “gate scheduled runs on asset updates”.

Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment thread airflow-core/docs/authoring-and-scheduling/timetable.rst
@nailo2c
Copy link
Copy Markdown
Contributor Author

nailo2c commented Apr 10, 2026

Copilot's feedback are helpful, I've fixed both issues.

  1. Thanks! yep the permalink was probably the thing that made me confused 🤦‍♂️

I've updated the link in the PR description to a permalink.

  1. I'm thinking about whether we have a way to reduce the confusion. If I were to be the user, I might be confused. 🤔 Do you think there's an easy way to change how the schedule box displays based on the current status? (I "guess" not, but worth trying)

I think that would require significant changes across the API and frontend. I'd prefer to keep this PR focused on the core functionality and address the UI improvements in a follow-up if needed.

@nailo2c nailo2c force-pushed the feat-58056-asset_and_time_schedule branch from d5612a8 to a94354b Compare April 11, 2026 01:20
Copy link
Copy Markdown
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly good, 2 questions

Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py Outdated
Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py
@nailo2c nailo2c force-pushed the feat-58056-asset_and_time_schedule branch from 7c8452e to 0d9487f Compare April 28, 2026 20:17
Copy link
Copy Markdown
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one last question, but mostly good

Comment thread airflow-core/src/airflow/timetables/assets.py Outdated
@Lee-W
Copy link
Copy Markdown
Member

Lee-W commented Apr 30, 2026

It's not a feature I would ever use, but I'm good with it, and it seems there's an actual use case. Would like to know how @uranusjr thinks. Thanks!

@nailo2c nailo2c force-pushed the feat-58056-asset_and_time_schedule branch 5 times, most recently from b590149 to 9e42573 Compare May 6, 2026 16:13
nailo2c and others added 14 commits May 6, 2026 13:45
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
The annotation on AssetAndTimeSchedule.__init__ was migrated to
Collection[SerializedAsset] | SerializedAssetBase, leaving the
TYPE_CHECKING-only Asset import unreferenced and breaking mypy on
two test files that passed SDK Assets to the core class. The
scheduler tests now import AssetAndTimeSchedule (and the matching
CronTriggerTimetable) from airflow.sdk so the types align with how
DAG authors use it; the timetable test fixture mirrors
core_asset_timetable by constructing SerializedAsset directly.
@nailo2c nailo2c force-pushed the feat-58056-asset_and_time_schedule branch from 9e42573 to 9b94c1b Compare May 6, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler ready for maintainer review Set after triaging when all criteria pass. type:new-feature Changelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants