Multi Team: Automatically create and assign team default pools#69768
Multi Team: Automatically create and assign team default pools#69768SameerMesiah97 wants to merge 1 commit into
Conversation
assign tasks without an explicit pool to the team's default pool during DAG parsing. Add unit tests covering pool assignment and team creation, and update the multi-team documentation to describe the new behaviour.
4c57afb to
9691842
Compare
| for task in dag.tasks: | ||
| if task.pool == Pool.DEFAULT_POOL_NAME: | ||
| task.pool = Pool.get_default_team_pool_name(dag_team_name) |
There was a problem hiding this comment.
The addition to the team create command would only create the default pools for new teams going forward. That would mean for already existing teams (without explicitly assigned pools), their tasks would be assigned a pool that doesn't actually exist. Users would need to manually create the team default pools. I know multi-team is still "experimental" and relatively new, so this might not be an issue, but worth flagging.
Another potential edge case is if a user explicitly assigns the global default pool to a task, this would actually overwrite it with the default team pool. Probably not very common, but worth mentioning as this would effectively mean that the default global pool would be unreachable for team Dags.
Description
This change introduces automatic default pool management for multi-team deployments.
With this implementation, a corresponding default pool named
default_pool_<team_name>is automatically created on team creation. During DAG parsing, tasks that do not explicitly specify a pool are assigned to their team's default pool based on the DAG bundle association, while tasks with explicitly configured pools are left unchanged.Rationale
In multi-team deployments, tasks without an explicit pool are currently assigned to the global
default_pool, allowing one team's workload to consume capacity shared across all teams. Avoiding this requires users to manually create and configure team-specific pools and update every task to reference them.By automatically provisioning a per-team default pool and applying it during DAG parsing, Airflow provides sensible defaults while preserving explicit pool assignments. This reduces operational overhead and helps isolate execution capacity between teams without requiring changes to existing DAGs.
Tests
Added unit tests verifying that:
default_poolare assigned to the appropriate team default pool during DAG parsing.default_pool.Documentation
Updated the multi-team documentation to describe automatic creation of team default pools and how they are applied during DAG parsing.
Backwards Compatibility
Existing behaviour is preserved when multi-team mode is disabled. Tasks with explicitly configured pools continue to use their configured pool without modification.
Closes: #69170