fix(integration): stop GetChangedTargets test timing out flakily#219
Open
xytan0056 wants to merge 1 commit into
Open
fix(integration): stop GetChangedTargets test timing out flakily#219xytan0056 wants to merge 1 commit into
xytan0056 wants to merge 1 commit into
Conversation
xytan0056
force-pushed
the
fix/integration-test-timeout-flake
branch
2 times, most recently
from
July 18, 2026 00:48
3f55a0a to
a22c2bf
Compare
The nested `bazel query` intermittently wedged during Bazel's loading phase
("Loading: 0 packages loaded" until the timeout fired), which surfaced as
"context deadline exceeded". It was a hang, not slowness: workers are pooled and
reused across requests, and each reuse does a `git checkout` that swaps the
source tree out from under a long-lived Bazel server still watching it, which
can deadlock the next query's loading phase. Raising timeouts only moves the
wall — the second GetChangedTargets subtest reliably hit it.
- Run the nested query in Bazel batch mode (--batch) in the integration test,
via a new config.RepositoryConfig.BazelStartupOptions knob plumbed through the
native graph runner as Bazel startup flags. Batch mode spawns a fresh,
short-lived Bazel per query, so no server state persists across checkouts and
there is nothing to wedge. Verified locally: integration_test PASSED in 81s.
- Raise requestTimeout to 20m (was equal to query_timeout at 10m, leaving zero
headroom) and keep query_timeout at 600s as a generous per-query bound.
- Persist Bazel's disk cache (.bazelrc --disk_cache) across CI runs to speed the
outer build.
xytan0056
force-pushed
the
fix/integration-test-timeout-flake
branch
from
July 18, 2026 00:59
a22c2bf to
ebf9656
Compare
sywhang
approved these changes
Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TestIntegration_GetChangedTargetsflakes withcontext deadline exceeded(run).Cause
A Bazel server hang, not slowness — the failing nested query sits at
Loading: 0 packages loadedfor the full timeout. Workers are pooled and reused; each reuse does agit checkoutthat swaps the tree under a long-lived Bazel server, intermittently wedging the next query's loading phase. Raisingquery_timeoutonly moves the wall.Fix (test/CI-only)
--batchmode (fresh server per query, no state across checkouts), via a newRepositoryConfig.BazelStartupOptionsknob — defaults empty, no prod change.requestTimeoutto 20m (was equal toquery_timeout).~/.cache/bazel-disk-cacheacross runs.Verified: local
PASSED in 81s; CI Integration Tests green in 6m45s.