Add cuda_device_id placement knob for GPU decoders#664
Draft
kvmto wants to merge 2 commits into
Draft
Conversation
Decoders can now be pinned to a specific CUDA device at construction via a cuda_device_id parameter, settable through C++/Python kwargs and the YAML realtime config (top-level decoder field, next to type/ transport). Model: one thread owns one decoder. decoder::get() validates the id (negative or >= device count throws), persistently pins the constructing thread with cudaSetDevice (no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id()). Plugin constructors therefore allocate on the right device with zero plugin changes -- this covers trt_decoder and the closed-source nv-qldpc-decoder transparently. decode_async() is the one exception: its fresh std::async worker pins itself for the call's duration via a lib-private RAII CudaDeviceGuard (libs/qec/lib/hardware_guards.h). The realtime host dispatcher (one thread serving all decoders) applies each decoder's device with set-if-different before enqueue and before DEVICE-mode graph capture; no-op for unpinned decoders. Tests: 7 C++ unit tests incl. 2-GPU placement and async-worker pinning, YAML round-trip + prepare_decoder_params coverage, Python kwargs tests. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up PR per review feedback on NVIDIA#634. Signed-off-by: kvmto <kmato@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First PR of the #634 split (per @bmhowe23's review):
cuda_device_idonly — GPU decoder pinning, settable at construction via kwargs and the YAML realtime config. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up draft PR (next release), Python interfaces for those to a third.What it does
decoder::get()reads and validatescuda_device_id(negative or >= device count →std::runtime_error), persistently pins the constructing thread (cudaSetDevice, no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id(), -1 = unpinned).trt_decoderand the closed-sourcenv-qldpc-decodertransparently. This addresses the lazily-allocating-decoder concern from the Add hardware pinning for QEC decoders #634 review without per-call guards or new decode entry points.decode_async()is the sole exception: its freshstd::asyncworker pins itself for the call's duration via a lib-private RAIICudaDeviceGuard(libs/qec/lib/hardware_guards.h, header-only; PR2 extends it).cuda_device_idis a top-leveldecoder_configfield (next totype/transport— placement knob common to any GPU decoder, not a per-decoder algorithm arg). Surfaced byprepare_decoder_params()for every decoder type; also exposed asdecoder_config.cuda_device_idin Python.qec.get_decoder("trt_decoder", H, cuda_device_id=1).To place N decoders on N GPUs, create each decoder on its own thread (
std::async/threading.Thread) — documented inrealtime_decoding.rst.Testing
DecoderCudaDeviceId.*): kwarg contract (absent/negative/out-of-range), key stripped before strict-validating plugin ctors, persistent pin observable after construction, two decoders on two GPUs from two threads, async worker pinning itself (verified RED→GREEN on a 2-GPU machine). GPU tests skip below the needed device count.prepare_decoder_paramssurfaces the knob for trt and non-trt types (ordering vs the trt-only early return is locked by test).cuda_device_idnamed in the error) and valid-id construct+decode. Fulltest_decoder.py61/61.test_decoders52/52,test_decoders_yaml20 pass / 2 skips (nv-qldpc-gated).Notes for reviewers
bad_castbecausehetMapFromKwargsstores Python ints assize_t(libs/corekwargs_utils.h) — the friendly negative-id message is only reachable from C++/YAML. Follow-up candidate in core.nv-qldpc-decoderinternal changes (mirroring the trt pattern) are a separate follow-up.Supersedes #663 (same content, reopened from the fork).
🤖 Generated with Claude Code