Skip to content

Multi Team: Automatically create and assign team default pools#69768

Draft
SameerMesiah97 wants to merge 1 commit into
apache:mainfrom
SameerMesiah97:Multi-Team-Default-Team-Pool
Draft

Multi Team: Automatically create and assign team default pools#69768
SameerMesiah97 wants to merge 1 commit into
apache:mainfrom
SameerMesiah97:Multi-Team-Default-Team-Pool

Conversation

@SameerMesiah97

Copy link
Copy Markdown
Contributor

Description

This change introduces automatic default pool management for multi-team deployments.

With this implementation, a corresponding default pool named default_pool_<team_name> is automatically created on team creation. During DAG parsing, tasks that do not explicitly specify a pool are assigned to their team's default pool based on the DAG bundle association, while tasks with explicitly configured pools are left unchanged.

Rationale

In multi-team deployments, tasks without an explicit pool are currently assigned to the global default_pool, allowing one team's workload to consume capacity shared across all teams. Avoiding this requires users to manually create and configure team-specific pools and update every task to reference them.

By automatically provisioning a per-team default pool and applying it during DAG parsing, Airflow provides sensible defaults while preserving explicit pool assignments. This reduces operational overhead and helps isolate execution capacity between teams without requiring changes to existing DAGs.

Tests

Added unit tests verifying that:

  • Tasks using default_pool are assigned to the appropriate team default pool during DAG parsing.
  • Tasks with explicitly configured pools are not modified.
  • DAGs without an associated team continue to use default_pool.
  • Creating a team automatically provisions its corresponding default pool.
  • Team default pool names are generated correctly.

Documentation

Updated the multi-team documentation to describe automatic creation of team default pools and how they are applied during DAG parsing.

Backwards Compatibility

Existing behaviour is preserved when multi-team mode is disabled. Tasks with explicitly configured pools continue to use their configured pool without modification.

Closes: #69170

assign tasks without an explicit pool to the team's default pool during DAG
parsing. Add unit tests covering pool assignment and team creation, and
update the multi-team documentation to describe the new behaviour.
@SameerMesiah97 SameerMesiah97 force-pushed the Multi-Team-Default-Team-Pool branch from 4c57afb to 9691842 Compare July 12, 2026 15:03
Comment on lines +184 to +186
for task in dag.tasks:
if task.pool == Pool.DEFAULT_POOL_NAME:
task.pool = Pool.get_default_team_pool_name(dag_team_name)

@justinpakzad justinpakzad Jul 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The addition to the team create command would only create the default pools for new teams going forward. That would mean for already existing teams (without explicitly assigned pools), their tasks would be assigned a pool that doesn't actually exist. Users would need to manually create the team default pools. I know multi-team is still "experimental" and relatively new, so this might not be an issue, but worth flagging.

Another potential edge case is if a user explicitly assigns the global default pool to a task, this would actually overwrite it with the default team pool. Probably not very common, but worth mentioning as this would effectively mean that the default global pool would be unreachable for team Dags.

@SameerMesiah97 SameerMesiah97 marked this pull request as draft July 12, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi Team: Add team default pools for tasks

2 participants