Skip to content

Add cuda_device_id placement knob for GPU decoders#664

Draft
kvmto wants to merge 2 commits into
NVIDIA:mainfrom
kvmto:decoder-gpu-pinning
Draft

Add cuda_device_id placement knob for GPU decoders#664
kvmto wants to merge 2 commits into
NVIDIA:mainfrom
kvmto:decoder-gpu-pinning

Conversation

@kvmto

@kvmto kvmto commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

First PR of the #634 split (per @bmhowe23's review): cuda_device_id only — GPU decoder pinning, settable at construction via kwargs and the YAML realtime config. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up draft PR (next release), Python interfaces for those to a third.

What it does

  • decoder::get() reads and validates cuda_device_id (negative or >= device count → std::runtime_error), persistently pins the constructing thread (cudaSetDevice, no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id(), -1 = unpinned).
  • Model: one thread owns one decoder. The thread that creates a decoder drives its decode calls; construction-time and lazy decode-time allocations all land on the pinned device with zero plugin changes — covers trt_decoder and the closed-source nv-qldpc-decoder transparently. This addresses the lazily-allocating-decoder concern from the Add hardware pinning for QEC decoders #634 review without per-call guards or new decode entry points.
  • decode_async() is the sole exception: its fresh std::async worker pins itself for the call's duration via a lib-private RAII CudaDeviceGuard (libs/qec/lib/hardware_guards.h, header-only; PR2 extends it).
  • YAML realtime config: cuda_device_id is a top-level decoder_config field (next to type/transport — placement knob common to any GPU decoder, not a per-decoder algorithm arg). Surfaced by prepare_decoder_params() for every decoder type; also exposed as decoder_config.cuda_device_id in Python.
  • Realtime session: the host dispatcher (one thread, all decoders) applies each decoder's device set-if-different (no restore) before enqueue and before DEVICE-mode graph capture; no-op for unpinned decoders.
  • Python kwargs work with no binding changes: qec.get_decoder("trt_decoder", H, cuda_device_id=1).

To place N decoders on N GPUs, create each decoder on its own thread (std::async / threading.Thread) — documented in realtime_decoding.rst.

Testing

  • 7 new C++ unit tests (DecoderCudaDeviceId.*): kwarg contract (absent/negative/out-of-range), key stripped before strict-validating plugin ctors, persistent pin observable after construction, two decoders on two GPUs from two threads, async worker pinning itself (verified RED→GREEN on a 2-GPU machine). GPU tests skip below the needed device count.
  • YAML: round-trip with the new field; prepare_decoder_params surfaces the knob for trt and non-trt types (ordering vs the trt-only early return is locked by test).
  • Python: kwargs error path (cuda_device_id named in the error) and valid-id construct+decode. Full test_decoder.py 61/61.
  • Full suites: test_decoders 52/52, test_decoders_yaml 20 pass / 2 skips (nv-qldpc-gated).

Notes for reviewers

  • Known pre-existing quirk (not introduced here): negative int kwargs from Python surface as an opaque bad_cast because hetMapFromKwargs stores Python ints as size_t (libs/core kwargs_utils.h) — the friendly negative-id message is only reachable from C++/YAML. Follow-up candidate in core.
  • nv-qldpc-decoder internal changes (mirroring the trt pattern) are a separate follow-up.

Supersedes #663 (same content, reopened from the fork).

🤖 Generated with Claude Code

kvmto and others added 2 commits July 9, 2026 17:31
Decoders can now be pinned to a specific CUDA device at construction
via a cuda_device_id parameter, settable through C++/Python kwargs
and the YAML realtime config (top-level decoder field, next to type/
transport).

Model: one thread owns one decoder. decoder::get() validates the id
(negative or >= device count throws), persistently pins the
constructing thread with cudaSetDevice (no restore), strips the key
before the plugin constructor, and stores it (get_cuda_device_id()).
Plugin constructors therefore allocate on the right device with zero
plugin changes -- this covers trt_decoder and the closed-source
nv-qldpc-decoder transparently.

decode_async() is the one exception: its fresh std::async worker pins
itself for the call's duration via a lib-private RAII CudaDeviceGuard
(libs/qec/lib/hardware_guards.h).

The realtime host dispatcher (one thread serving all decoders) applies
each decoder's device with set-if-different before enqueue and before
DEVICE-mode graph capture; no-op for unpinned decoders.

Tests: 7 C++ unit tests incl. 2-GPU placement and async-worker pinning,
YAML round-trip + prepare_decoder_params coverage, Python kwargs tests.

NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a
follow-up PR per review feedback on NVIDIA#634.

Signed-off-by: kvmto <kmato@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants