Fix OpenCodeEnv client timeout and retry transient sandbox-API polls#1005
Open
sergiopaniego wants to merge 2 commits into
Open
Fix OpenCodeEnv client timeout and retry transient sandbox-API polls#1005sergiopaniego wants to merge 2 commits into
sergiopaniego wants to merge 2 commits into
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 017dedc. Configure here.
OpenCodeEnv inherited MCPToolClient's 60s message_timeout_s default, but a rollout runs up to 900s server-side, so run_rollout timed out client-side mid-rollout. Add __init__ defaulting message_timeout_s to 1800s, matching PiEnv. HFBgJob.wait() polls Sandbox.processes(), which occasionally drops a single poll with httpx.TransportError (RemoteProtocolError, read timeout). Retry on the next tick and give up only after a run of consecutive failures, so a transient disconnect no longer aborts the rollout.
sergiopaniego
force-pushed
the
fix-hf-sandbox-client-timeout-and-retry
branch
from
July 23, 2026 13:57
017dedc to
cf2e876
Compare
HFBgJob.wait() retried httpx.TransportError from processes() without checking the deadline, so a finite timeout could be exceeded by up to _MAX_TRANSIENT_POLL_ERRORS * _WAIT_POLL_INTERVAL_S and a blip near the limit raised TransportError instead of the TimeoutError callers rely on. Fall through to the single deadline check + sleep on both the normal poll and the retry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #998, which landed
HFSandboxBackend. Two small robustness fixes surfaced while validating loop-owning training on remote HF sandboxes.1. OpenCodeEnv client timeout
OpenCodeEnvinheritedMCPToolClient's 60smessage_timeout_sdefault, but a rollout runs up to 900s server-side (_RUN_ROLLOUT_TIMEOUT_S). Sorun_rollout()timed out client-side mid-rollout. Adds an__init__defaultingmessage_timeout_sto 1800s, matching the fix already inPiEnv(#999).2. Retry transient sandbox-API polls
HFBgJob.wait()pollsSandbox.processes(), which occasionally drops a single poll withhttpx.TransportError(observedRemoteProtocolError: Server disconnected without sending a response, also read timeouts). Before this, one blip aborted the whole rollout. Now it retries on the next poll tick and gives up only after a run of consecutive failures, so a transient disconnect no longer kills a training step.wait()lives inopencode_env/sandbox/hf.py, shared by both opencode_env and pi_env, so this covers both.Tests
Two new unit tests in
test_opencode_hf_sandbox.py: retry recovers from a few transient poll errors, and a persistent outage re-raises instead of looping forever. Full suite green.Validated end-to-end: loop-owning AsyncGRPO training on HF Jobs (h200x2, opencode on remote HF sandboxes) completes cleanly with the transient poll errors now absorbed.
Note
Low Risk
Targeted client timeout and poll-retry logic with unit tests; no auth, data, or API contract changes.
Overview
Improves robustness for long OpenCode rollouts and Hugging Face sandbox background-job polling.
OpenCodeEnvnow defaultsmessage_timeout_sto 1800s (via a new__init__) so MCPrun_rolloutcalls are not cut off by the inherited 60s client timeout while the server allows runs up to ~900s.HFBgJob.wait()in the shared HF sandbox backend retrieshttpx.TransportErroronSandbox.processes()polls instead of failing on a single API blip; it stops after 10 consecutive transient errors and still respects finitetimeoutdeadlines. Unit tests cover recovery, persistent failure, and timeout-vs-transient behavior.Reviewed by Cursor Bugbot for commit 94a5ed5. Bugbot is set up for automated code reviews on this repo. Configure here.