[CI] Add retry if timeout

[xyao-nv]

Summary

Kit itself intermittently deadlocks during startup — not because of orphans, but because Kit's internal thread synchronization fails on low-CPU runners. No orphans present when it happens.

Detailed description

• when Kit deadlocks, the 900s timeout kills it, and the retry launches a fresh Kit process which almost certainly succeeds (since the deadlock is rare and random).

• Without the retry, a single Kit deadlock fails the whole CI run. With the retry, it costs ~15 extra minutes but the test passes.

[isaaclab-review-bot]

Good approach — retrying on Kit's intermittent startup deadlock is the right call for CI reliability. The refactoring into _run_once is clean and the retry-only-on-timeout logic is correct (non-zero exits still fail fast). A couple of things to consider:

Orphan cleanup between retries: The docstring says "orphaned Kit processes are cleaned up" but I don't see any cleanup code. When subprocess.run hits a timeout, it calls process.kill() which sends SIGKILL to the child PID only — not the whole process group. Since start_new_session=True puts the child in its own group, the grandchildren (Kit shader compiler, GPU workers, etc.) survive as orphans. Before retrying, you'd want to kill the entire process group, e.g.:

except subprocess.TimeoutExpired as exc:
    # Kill the entire process group, not just the leader
    try:
        os.killpg(os.getpgid(exc.cmd_process.pid), signal.SIGKILL)
    except (ProcessLookupError, OSError):
        pass

Since subprocess.run doesn't expose the Popen object after timeout, you may need to switch to manual Popen management in _run_once to get access to the PID for group cleanup. Without this, retrying with orphaned GPU-holding processes still running could cause the retry to fail for a different reason.

Minor: The docstring could be updated to remove the "orphan cleanup" mention until that's actually implemented, to keep docs honest.

[isaaclab-review-bot]

Orphan process leak on timeout: subprocess.run calls process.kill() on timeout, which sends SIGKILL to the child PID only. With start_new_session=True, the child's process group (Kit shader compiler, GPU workers, etc.) won't be killed — they'll survive as orphans holding GPU resources.

Consider switching to manual Popen so you can kill the whole process group on timeout:

def _run_once(cmd, env, timeout_sec, capture_output):
    kwargs = dict(env=env, start_new_session=True)
    if capture_output:
        kwargs.update(stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    proc = subprocess.Popen(cmd, **kwargs)
    try:
        stdout, stderr = proc.communicate(timeout=timeout_sec)
    except subprocess.TimeoutExpired:
        # Kill the entire process group, not just the leader
        try:
            os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
        except (ProcessLookupError, OSError):
            pass
        proc.wait()
        raise
    if capture_output:
        return subprocess.CompletedProcess(cmd, proc.returncode, stdout, stderr)
    return subprocess.CompletedProcess(cmd, proc.returncode)

This is especially important since the whole point of the retry is to get a clean slate — retrying with orphaned GPU processes still running might not help.

[isaaclab-review-bot]

nit: The docstring says "orphaned Kit processes are cleaned up" but I don't see cleanup code in the implementation. Consider removing that claim until it's implemented, or adding the process group kill between retries.