Support Backfills for a subset of Tasks

[whitleykeith]

Description

We want the ability to support backfilling only a subset of tasks. Ideally this would be done via a task regex parameter or something similar

Use case/motivation

We have DAGs that write multiple datasets out, and we frequently have cases where we wish to backfill on a specific dataset (therefore a specific task or task(s))

Another example is sensors for historical data, where the external sensors may depend on data that is no longer there (i.e. task sensors after the metadata db has had historical data deleted for performance etc.)

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Fury0508]

Hi, I would be interested in working on this once the expected semantics are agreed.

From an initial review, I think there are two possible interfaces:

1. Accept an explicit list of task IDs in the backfill API.
2. Provide a CLI/UI regex or pattern that resolves to that list of task IDs.

Before implementing it, I would like to clarify how dependencies and DAG-run state should behave.

My proposed initial semantics would be:

• Only the selected tasks are scheduled.
• Upstream and downstream tasks are not automatically included.
• Dependencies on unselected upstream tasks are ignored for this backfill.
• The request is rejected when the selection matches no tasks.
• The backfill/DAG run is considered successful based only on the selected tasks.
• The resolved task IDs are persisted with the backfill so that its behaviour remains reproducible if the DAG later changes.

Alternatively, we could automatically include upstream dependencies, but that may not address the sensor use case described in the issue.

Would this general direction be acceptable? If so, I would be happy to prepare a PR, beginning with the API/model and tests before adding the CLI and UI support.