Skip to content

Add deferrable mode to SnowparkContainerJobOperator#70103

Open
justinpakzad wants to merge 2 commits into
apache:mainfrom
justinpakzad:feature/snowpark-container-job-deferrable
Open

Add deferrable mode to SnowparkContainerJobOperator#70103
justinpakzad wants to merge 2 commits into
apache:mainfrom
justinpakzad:feature/snowpark-container-job-deferrable

Conversation

@justinpakzad

Copy link
Copy Markdown
Contributor

Description

Follow-up to #68259 adding deferrable mode to SnowparkContainerJobOperator. Snowpark Container jobs are often long-running (ML training, batch processing, other containerized workloads that run for hours), which makes them a good fit for deferrable mode.

The SnowflakeHook is synchronous, so this follows a similar approach the Anthropic provider took. The trigger wraps each blocking hook call in asyncio.to_thread so a single poll doesn't stall the shared triggerer event loop. A timeout parameter is introduced for both modes. Cleanup is consistent across both modes, on timeout the service is dropped to stop billing on a job past its deadline. On poll errors and terminal failures it's left in place for inspection.

The status Enum and terminal/non-terminal frozensets were moved to the trigger file so the operator can import them without a circular dependency.

Testing

  • Unit tests for the trigger (serialization, terminal/unexpected/timeout/error events, cleanup) and the operator's defer and execute_complete paths.
  • Manually tested end-to-end against a real Snowflake account: deferred success (job completes, logs retrieved, service dropped), terminal failure, both timeout types, on_kill, and the drop toggles.
Was generative AI tooling used to co-author this PR?
  • Yes (Claude Opus 4.6)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers provider:snowflake Issues related to Snowflake provider labels Jul 19, 2026
@justinpakzad
justinpakzad force-pushed the feature/snowpark-container-job-deferrable branch from a61e341 to dcc37d5 Compare July 19, 2026 22:49
@justinpakzad
justinpakzad marked this pull request as ready for review July 20, 2026 23:08
@justinpakzad
justinpakzad requested a review from potiuk as a code owner July 20, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:snowflake Issues related to Snowflake provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant