[0z5a]

Required human contributor answers

1. What outcome are you trying to achieve?

Fix the abort we hit while iterating: the process aborted during CUDA graph capture.

2. What is the highest-leverage variable for that outcome?

Whether the Python garbage collector can run, and free CUDA state, in the middle of a CUDA graph capture. (Answers 2 and 3 were drafted by the agent at the contributor's explicit request.)

3. How will this work move that variable rather than create busywork?

One gc.collect() before each capture closes that window, so failure no longer depends on when the collector happens to fire. Captures are rare and bounded by the graph cache, so replay pays nothing. The new test forces the collection inside a capture, so the bug is reproduced deterministically instead of relying on GC timing.

4. Who is the customer or user, and what do they actually need?

Users running on SM90 GPUs. They need CUDA graph capture not to abort or be invalidated.

PR necessity and bugfix evidence

The human contributor decided on 2026-10-22 to fix this in its own PR against main rather than inside #130. Observed failure: a full GPU regression run of #130 at 9874287 aborted (Fatal Python error: Aborted, Python stack in Garbage-collecting under vllm_rlt/worker/cuda_graph.py::_capture) during test_mixed_execution; see the #130 reply. It was reproduced on a GH200, intermittently in the full suite and deterministically with the new test below.

Purpose

torch.cuda.graph no longer runs gc.collect() before capture; since PyTorch made it opt-in, it collects only when torch.compiler.config.force_cudagraph_gc (TORCH_CUDAGRAPH_GC) is set, which defaults to False. Reference cycles left by an earlier engine, holding CUDA graphs, streams, CUDA tensors and pinned UvaBuffers, then survive into a later capture. When an allocation inside that capture triggers the collector, the CUDA state is freed mid-capture: the capture is invalidated (cudaErrorStreamCaptureInvalidated) or the process aborts from a destructor.

_capture now calls gc.collect() after the warm-up runs and before the capture. Captures are rare (bounded by the graph cache), so the collection is not on the replay path.

How it was found: forcing one collection inside every capture, across test_attention*, test_speculative, test_async_speculative, test_cuda_graph, test_kv_cache, test_split_prefill and test_mixed_execution, gave 13 failures (12 cudaErrorStreamCaptureInvalidated, 1 invalid argument). At that point up to 995 unreachable objects were pending, including 2–4 CUDAGraphs and up to 85 CUDA tensors. With the fix, every one of 80 probed captures found 0 unreachable objects, and all 255 tests passed. Which freed object issues the illegal CUDA call (graph destruction, private-pool release, pinned-host free or stream destruction) was not isolated; the fix does not depend on it.

Test Plan

vllm-rlt Version / Commit: 45565cc on main 4a35034.

• New GPU test tests/test_cuda_graph.py::test_capture_survives_collecting_an_earlier_engines_garbage. With the collector disabled, it runs one engine with CUDA graphs and drops it, leaving its cycles. A second engine then collects garbage inside its first capture; its outputs must match the first engine's.
• Core GPU regressions: python -m pytest --run-gpu -q -rA -k 'not flash_attn' tests/test_{attention,attention_64bit,speculative,async_speculative,cuda_graph,kv_cache,split_prefill,mixed_execution,huginn,input_preparation,backend_selection}.py
• CPU: python -m pytest -q -m "not gpu"; Ruff check and format.

Test Result

One GH200 (SM90), torch 2.14.0+cu130, Triton 3.8.0, transformers 4.55.0, Python 3.12. The GPU runs used a tree whose two changed files match 45565cc by SHA256.

• New test on main 4a35034: aborts in 2 of 2 runs (Fatal Python error: Aborted, Garbage-collecting inside _capture). With the fix: passes in 3 of 3 runs (one on the exact tree; two on an earlier copy that lacked only the three comment lines).
• Core GPU regressions with the fix: 327 passed, 4 failed. The 4 failures are test_input_preparation.py::test_packed_metadata_supports_query_lengths_and_per_request_depths[*], which select FA4 (ModuleNotFoundError: No module named 'flash_attn'); FA4 is not installed on this host. Unrelated to this change.
• CPU: 800 passed / 34 skipped (macOS, no Triton). Ruff 0.16.8 check and format pass on both changed files.

[hsliuustc0106]

No actionable findings. Reviewed head 45565cc546b1df0b914d9f8d1fb50f5e3fe540bf, rechecked unchanged before posting.

The GC placement at cuda_graph.py:24–30 addresses the reported earlier-engine failure: warmup completes before collection, destructors run outside capture, and the active model, cache, stream, and inputs remain referenced. Both recurrent and coda captures benefit; replay is unchanged. PyTorch’s opt-in GC setting is a viable alternative, but the local call avoids depending on process configuration.

The regression test preserves an earlier engine’s cycles, forces collection during the next capture, requires actual captures, and compares exact tokens and exit depths.

• Checked: full diff, ownership and stream ordering, failure report, prior discussions, and CI. Local Ruff lint/format and diff checks passed. CPU CI passed: 800 passed, 357 skipped.
• Validation gaps: no PR runtime or GPU code executed locally. GH200 reproduction results remain author-reported; the exact offending destructor remains unidentified.
• Integration scope: merge base is 4a35034. At review time, PR metadata still reported that base while live main was ff05fe1. Synthetic merge 43de2ad contained the same two-file patch but had no CI run; existing CI tested older merge 85f1dbc.