Skip to content

Add BUILDKITE_GIT_CHECKOUT_TIMEOUT timeout to add upper duration on the checkout hook#3794

Open
brandonnovick wants to merge 1 commit intobuildkite:mainfrom
brandonnovick:add_git_checkout_timeout
Open

Add BUILDKITE_GIT_CHECKOUT_TIMEOUT timeout to add upper duration on the checkout hook#3794
brandonnovick wants to merge 1 commit intobuildkite:mainfrom
brandonnovick:add_git_checkout_timeout

Conversation

@brandonnovick
Copy link
Copy Markdown

@brandonnovick brandonnovick commented Apr 6, 2026

Description

Adds a new --git-checkout-timeout flag / BUILDKITE_GIT_CHECKOUT_TIMEOUT environment variable that allows configuring a timeout (in seconds) for the entire git checkout phase. When the timeout is exceeded, the job fails with exit code 95. When unset or set to 0, checkout runs with no timeout (existing behavior).

Context

Long-running or hung git operations (clone, fetch, checkout, submodule update) can cause jobs to hang indefinitely. This is particularly bad in this modern era of GitHub where conditions have been suffering recently. This provides a mechanism to fail fast with a clear exit code when checkout takes too long. This is useful for CI environments where retries are leaned upon heavily , even in the face of needed infrastructure improvements (potential network throttling, etc. )

Changes

  • Added GitCheckoutTimeout config field to ExecutorConfig with
    env:"BUILDKITE_GIT_CHECKOUT_TIMEOUT" tag
  • Added --git-checkout-timeout CLI flag (default 0, meaning no timeout) in global.go
  • Wired the flag through BootstrapConfig, AgentStartConfig, and AgentConfiguration
  • Added BUILDKITE_GIT_CHECKOUT_TIMEOUT to the protected env vars list
  • Added checkoutWithTimeout() wrapper in checkout.go that applies context.WithTimeout around
    the checkout call; returns shell.ExitError{Code: 95} on timeout
  • Job runner sets BUILDKITE_GIT_CHECKOUT_TIMEOUT env var for child processes when configured
  • Added reflect.Int support to ExecutorConfig.ReadFromEnvironment so int-typed env-tagged
    fields are loaded correctly

Testing

  • Tests have run locally (with go test ./...). Buildkite employees may check this if the
    pipeline has run automatically.
  • Code is formatted (with go tool gofumpt -extra -w .)

Contributions

This PR was largely generated by Claude code, but reviewed manually and tweaked by a human

@brandonnovick brandonnovick changed the title Add checkout timeout Add BUILDKITE_GIT_CHECKOUT_TIMEOUT timeout to add upper duration on the checkout hook Apr 6, 2026
@brandonnovick brandonnovick marked this pull request as ready for review April 6, 2026 13:18
@brandonnovick brandonnovick requested review from a team as code owners April 6, 2026 13:18
Copy link
Copy Markdown
Contributor

@zhming0 zhming0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for raising this PR @brandonnovick , we agree with the general direction but we think it's best if we be putting the timeout per attempt.

Please let us know you want to give it an another shot, or alternatively raising a feature request to us.

Lastly, for our curiosity, how often is such git hanging issue happening for you?

Comment thread internal/job/checkout.go
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a global timeout to checkout is something that we are a bit hesitant about, given there is an retry mechanism underneath the checkout.

However, we think it's very reasonable to add a timeout setting for each retry attempt, it would:

  • Kill a hung individual git operation quickly
  • Still allow the retry loop to kick in and try again

Unblocking CI is our ultimate goal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants