Skip to content

test(cloudxr): fix flaky terminate race in test_background.py - #1327

Merged
nv-jakob merged 1 commit into
mainfrom
jakob/fix-race-in-pid-test
Oct 23, 2026
Merged

nv-jakob merged 1 commit into
mainfrom
jakob/fix-race-in-pid-test

Conversation

@nv-jakob

@nv-jakob nv-jakob commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

test_sigterm_and_clears_the_pid_file intermittently failed with "assert False is True" from terminate(). Its stand-in for the service, python -m time run 30, never slept: a builtin module has no code object, so the child exited within milliseconds. The test passed only if terminate() read /proc//cmdline before then; a short stall on a loaded CI runner found a zombie with an empty cmdline instead, so read_pid() returned None. Unrelated to the CloudXR runtime, which this code path never loads.

Add a stand_in_service fixture that runs a module sleeping for its last argument, and use it in the three tests that relied on -m time. Move the cmdline poll from #1209 into _poll_read_pid() and wait on it before terminate(), since a fresh child's cmdline can still be empty when Popen() returns. Record both pitfalls in tests/AGENTS.md.

Reproduced by stalling 50 ms after spawn(): the old test fails as on CI, the new one passes.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Testing

Checklist

  • I have read and understood the contribution guidelines
  • I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix/feature works (or explained why not)
  • I have signed off all my commits (git commit -s) per the DCO

Summary by CodeRabbit

  • Tests
    • Updated background-service checks to use a persistent child process and wait for it to be recognized before proceeding. Coverage includes service recognition, detached spawning, log appending, and termination. These changes affect automated test behavior; no end-user-facing changes are noted.

test_sigterm_and_clears_the_pid_file intermittently failed with
"assert False is True" from terminate(). Its stand-in for the service,
`python -m time run 30`, never slept: a builtin module has no code
object, so the child exited within milliseconds. The test passed only
if terminate() read /proc/<pid>/cmdline before then; a short stall on
a loaded CI runner found a zombie with an empty cmdline instead, so
read_pid() returned None. Unrelated to the CloudXR runtime, which this
code path never loads.

Add a stand_in_service fixture that runs a module sleeping for its last
argument, and use it in the three tests that relied on `-m time`. Move
the cmdline poll from #1209 into _poll_read_pid() and wait on it before
terminate(), since a fresh child's cmdline can still be empty when
Popen() returns. Record both pitfalls in tests/AGENTS.md.

Reproduced by stalling 50 ms after spawn(): the old test fails as on
CI, the new one passes.

Signed-off-by: Jakob Bornecrantz <tbornecrantz@nvidia.com>
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 8b031d1 and 5c930dc.

📒 Files selected for processing: 2
  • tests/AGENTS.md
  • tests/python/core/cloudxr/test_background.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Walkthrough

Background-service tests now use a sleeping stand-in module instead of Python’s time module. A shared helper polls read_pid() for up to two seconds. The tests use it to confirm that the child process is recognized before checking process behavior, including termination. Test guidance now calls for stand-in child processes to persist independently and for immediate /proc/<pid>/cmdline checks to poll for a nonempty command line.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 5c930

The test changes address process startup races with a persistent stand-in service and PID polling.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 1 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly identifies the CloudXR test fix and the flaky termination race addressed by the changes.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 1 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor

🤖 Agent-generated (Claude Code · claude-sonnet-5-5)

Summary

The fix is correct and well scoped. The old stand-in (python -m time run 30) exited within milliseconds because a builtin module has no code object, so the test only passed if terminate() read /proc/<pid>/cmdline before the child died. The new stand_in_service fixture keeps the child alive, and _poll_read_pid() covers the empty-cmdline window after Popen(). I checked this against background.py on main: spawn() passes run 30 so sys.argv[-1] is "30", read_pid() matches on the monkeypatched _MODULE, and terminate() returns True once the SIGTERMed child is a zombie with an empty cmdline. The tests/AGENTS.md note is in the right section. No blockers. I did not run the tests, so the 50 ms stall reproduction is unverified on my side.

Legend: 🚫 Blocker · 💡 Suggestion · 🔍 Nit

Finding
💡 PR description, ## Testing — still the empty template placeholder. The 50 ms stall reproduction is in the Description; move it here and add the pytest command and platform.
💡 tests/python/core/cloudxr/test_background.py, stand_in_service — monkeypatch.setenv("PYTHONPATH", ...) overwrites any existing PYTHONPATH instead of prepending. Harmless for a stdlib-only child, but prepending is safer if the stand-in ever imports more.
💡 tests/python/core/cloudxr/test_background.py, test_sigterm_and_clears_the_pid_file — if _poll_read_pid or terminate raises, the 30 s child is never killed. Sibling tests use try/finally; a finally that kills it is cheap insurance.
🔍 _poll_read_pid — the 2.0 s deadline could be a named constant.
🔍 Commit trailer — no Co-Authored-By line; only needed if an agent did the substantive work.
🔍 PR description — "Move the cmdline poll from #1209" narrates history; the description can state just the current shape of the change.

Actionables (for bots — copy-paste-ready for AI)

Agent-generated suggestions, not human-vetted obligations. Skip anything wrong, already addressed, or not worth the churn.

  • PR description — fill in ## Testing with the reproduction and test command.
  • tests/python/core/cloudxr/test_background.py (test_sigterm_and_clears_the_pid_file) — wrap the body in try/finally that kills pid if it is still alive.
  • tests/python/core/cloudxr/test_background.py (stand_in_service) — prepend tmp_path to the existing PYTHONPATH rather than replacing it.

@gareth-morgan-nv gareth-morgan-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM see agent review above

@nv-jakob
nv-jakob merged commit ba15610 into main Oct 23, 2026
37 checks passed
@nv-jakob
nv-jakob deleted the jakob/fix-race-in-pid-test branch October 23, 2026 13:50

This branch was successfully deployed

1 active deployment
dev — 5c930dc0 Deployed Oct 23, 2026 by nv-jakob via publish-wheel #5957
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants