Decoding Server Updates#666
Open
bmhowe23 wants to merge 94 commits into
Open
Conversation
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Bring in PR #637's C++ surface-code orientation support (sc_orientation XV/XH/ZV/ZH; orientation-aware role and observable geometry) and its unit tests, so the realtime example can build the surface code at code_rotation XV to match the Ising-trained predecoder. Vendored verbatim; temporary until #637 merges upstream. Python bindings/exports omitted (this branch's consumer is the C++ example). Signed-off-by: Melody Ren <melodyr@nvidia.com>
Add surface_code-4-yaml, a YAML-switchable realtime surface-code decoder example (pymatching / trt+pymatching / nv-qldpc relay-bp over inproc_rpc), and build the code at code_rotation XV for both the stabilizer CNOT schedule and the logical observables. Over inproc_rpc the trt path loads an Ising-trained d7/T7 CNN predecoder ONNX and runs end-to-end; pymatching decodes at 0 LER and the example ctests pass at XV. Signed-off-by: Melody Ren <melodyr@nvidia.com>
Rework the surface_code-4-yaml DEM-generation kernel to emit the detector layout an Ising-trained predecoder expects (prep-basis single-term detectors + paired rounds + data-derived boundary detectors, X-then-Z), so the Ising CNN model decodes cudaqx's live syndromes. The trt+Ising config loads H/O/priors from the Ising bundle (Ising detector order) and a D_sparse that maps the cudaqx live measurement buffer onto Ising's detector rows (cudaqx's m2d with the Z-ancillas permuted to Ising order; X-ancillas and data identity); the final data measurements are enqueued so the boundary detectors close. End-to-end over inproc_rpc, trt + the Ising d7/T7 model decodes at 0/200 residual; pymatching and nv-qldpc keep cudaqx's own DEM (D = m2d). The example is single-volume (decoding one num_rounds-round volume; the sliding-window machinery is removed). --save_syndrome and --load_syndrome record the heterogeneous final data round so replay round-trips exactly. gen_dsparse.py prepares the bundle's D_sparse.txt from a local Ising checkout (--ising-repo); the trt+Ising path needs that external model and bundle and is not exercised in CI. The TensorRT predecoder runs with its internal CUDA graph enabled. Signed-off-by: Melody Ren <melodyr@nvidia.com>
…ance Bind a loaded YAML config to the experiment geometry: the load path validates the detector count and the measurement-buffer span (max(D_sparse)+1) against the run, which together pin distance and num_rounds; it also checks the observable count, and the trt+Ising path reads and enforces the bundle's metadata.txt (basis=Z, code_rotation=XV, distance, n_rounds). --load_syndrome now requires --yaml, and configure_decoders' return status is checked. Drop the num_rounds-multiple-of-distance constraint (single-volume decoding needs only num_rounds >= 1); num_rounds < distance warns rather than errors. Reject even or below-3 distance up front. Validation failures surface as a clean 'Error: ...' with a nonzero exit. Signed-off-by: Melody Ren <melodyr@nvidia.com>
Validate every decoder config in a loaded YAML (one observable each, consistent detector count and measurement-buffer span), supporting num_logical > 1 independent patches. Replay requires a structurally complete capture -- a present CORRECTIONS_START/END footer, one correction per shot, and per-shot round counts and widths matching the geometry -- and exits nonzero on a missing, truncated, or correction-mismatched file. Add native ctests covering the validation and boundary behavior: distance and round guards, geometry-bound YAML loading, X-basis bundle rejection, replay guards, and multi-logical generate+reload. Each negative case asserts both a nonzero exit and the expected diagnostic. Signed-off-by: Melody Ren <melodyr@nvidia.com>
- Neutralize internal-note and "for-Ising" comment framing; fix copyright year. - Rename gen_dsparse.py -> gen_dsparse_from_memory_circuit.py. - trt+Ising path: when the locally-generated predecoder bundle is absent, print the exact generation recipe and exit rather than failing deeper in. - Order the example after surface_code-3 in CMake; drop cross-example comments. Signed-off-by: Melody Ren <melodyr@nvidia.com>
…e-realtime-yaml-ising
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
…-with-pr624 Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Brings in #637 (surface-code orientation XV/XH/ZV/ZH). The stale vendored C++-only copy is replaced by upstream's version, which also adds the Python bindings the vendored copy omitted. Signed-off-by: Melody Ren <melodyr@nvidia.com>
Fold the d5t6 and multilogical generate/reload pairs into single driver runs (the driver already does generate + reload + decode and adds a residual-LER check), and drop the undersized-distance case (same guard and diagnostic as the even-distance case). 18 -> 15 tests, all passing; coverage unchanged. Signed-off-by: Melody Ren <melodyr@nvidia.com>
…port qec_decoding_daemon is a standalone decoding-server process: the service end of a CUDA-Q device_call transport, wiring the libcudaq-realtime HOST_CALL dispatcher onto a cpu_transport transceiver's rings and serving the decoding-server-cqr function table (enqueue_syndromes / get_corrections / reset_decoder). Both axes are configuration, not code: - decoders come from --config=<yaml> (multi_decoder_config::from_yaml_str) - transport comes from --transport=udp|cpu_roce (UDP loopback runs anywhere; cpu_roce is the real RDMA wire incl. the QP/rkey rendezvous, compiled in when libcudaq-realtime-cpu-transport + ibverbs are found) test_decoding_server runs the caller (simulated QPU, stim + -frealtime-lowering) side only -- no decoder in-process -- and verifies the round-trip plus the daemon's dispatch count. Two decoder configs are exercised: pymatching (plugin) and multi_error_lut (built-in). Transport selectable via QEC_DECODING_SERVER_TRANSPORT, with RDMA topology from the same CUDAQ_CPU_ROCE_TEST_* env vars as CUDA-Q's CpuRoceChannelTester. Requires the CUDA-Q `udp` device_call channel + UdpTransceiver (cuda-quantum branch decoding-server-udp-transport). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ransport # Conflicts: # libs/qec/lib/realtime/decoding-server-cqr/CMakeLists.txt # libs/qec/lib/realtime/decoding-server-cqr/decoding_server_cqr.cpp # libs/qec/lib/realtime/simulation-cqr/CMakeLists.txt # libs/qec/unittests/decoders/pymatching/CMakeLists.txt # libs/qec/unittests/decoders/pymatching/test_pymatching_device_call_realtime.cpp
…_runtime.md Adopt the spec's on-wire form for the default-route RPCs, with the packing done in the __qpu__ wrappers until the device_call lowering can emit it: - enqueue_syndromes now crosses the wire as (decoder_id, counter, syndrome_mapping_id, num_syndromes, syndrome_bits), with the bits packed LSB-first into a std::vector<uint8_t> placed last; the stdvec<u8> serialization's uint64 byte-count prefix is the spec's ARRAY_UINT8 prefix. - get_corrections sends (decoder_id, return_size, corrections_bytes, reset) and receives the correction bits packed; the kernel unpacks after the call. - The decoding server registers the matching 5-arg/4-arg schemas, validates byte-count == ceil(bit-count/8), unpacks on ingest, and packs responses. Capacity is now 512 bits (was 64 byte-per-bool). - The hidden host trampolines (direct/emulate channel) mirror the same unpack/pack around the byte-per-bool host:: API. Kernel bridge constraints shaped the packing loops: no vector<bool> proxy in an if-condition (branchless uint8 conversion instead) and identical integer widths on >>/& operands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tools/
qec_decoding_daemon was born as the server half of the two-process test but
is really the standalone decoding-server skeleton; give it a proper home at
libs/qec/tools/decoding-server (mirroring libs/solvers/tools), together with
its example decoder configs and the cudaq-realtime transport discovery. The
binary now lands in ${CMAKE_BINARY_DIR}/bin with the configs beside it, and
builds whenever its dependencies are found (no longer gated on tests).
test_decoding_server gates on the daemon target and receives the binary path
via -DQEC_DECODING_DAEMON_PATH (through cudaqx_add_device_code COMPILER_FLAGS,
since nvq++ compilation does not see target_compile_definitions), replacing
the same-directory assumption; configs resolve relative to the daemon.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cudaqx_add_device_code ran every nvq++ compile in the shared CMAKE_CURRENT_BINARY_DIR, but nvq++ drops intermediates named after the *source* (<src>.o, <src>.qke.o, <src>.classic.o, ...) into its working directory. Two targets compiling the same source file in the same directory (e.g. the surface_code-1 -local / -cqr / -quantinuum variants) therefore raced under parallel ninja and corrupted each other's objects -- reproduced deterministically as one variant's object containing another variant's preprocessor-conditional code. Give each object its own <target>_<source>.nvqpp/ working directory and pass -o as an absolute path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add surface_code-1-cqr: the same surface_code-1.cpp application (distance-3 memory circuit, per-round enqueue_syndromes(measure_result), windowed decode, transversal correction) compiled with -frealtime-lowering -DQEC_APP_CQR and linked against the simulation-cqr client + decoding-server-cqr service, so every cudaq::qec::decoding::* device_call crosses the cudaq-realtime host-dispatch ring as the spec's bit-packed frames instead of resolving to the in-process trampolines. A small QEC_APP_CQR block in the app brings the channel up (cudaq::realtime::initialize/finalize; QEC_DECODING_SERVER_PORT selects the udp channel for an external daemon) and prints the in-process service's dispatch count, which the test asserts is non-zero under CUDAQ_DEVICE_CALL_CHANNEL=host_dispatch -- 16 dispatches per shot (reset + 14 enqueues + get_corrections), 16000 across the 1000-shot run. This exercises what the synthetic 3-bit tests could not: the production measure_result path, multi-round accumulation, and real matching over the wire. The non-zero threshold is 60 (vs -local's 40): unseeded 1000-shot runs land ~35-45, so 40 flakes, while a broken decode path yields ~150+. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oding server
Run the surface_code-1-cqr application as a udp device_call client against
qec_decoding_daemon (tools/decoding-server) in a separate process:
1. the app's --save_dem pass produces the decoder YAML,
2. the daemon starts on an ephemeral udp port with that YAML (decoder setup
crosses the process boundary as configuration, not code),
3. the same app binary runs 1000 shots with QEC_DECODING_SERVER_PORT
routing every enqueue_syndromes / get_corrections / reset_decoder over
UDP loopback as the spec's bit-packed frames.
Self-verifying in both directions: the daemon's shutdown dispatch count must
cover every shot's device_calls (observed 16000 = 1000 shots x 16 calls) and
the app's in-process service count must be exactly 0, proving nothing decoded
locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Melody Ren <melodyr@nvidia.com>
…eplay test, flag/arg guards, dead debug) Signed-off-by: Melody Ren <melodyr@nvidia.com>
Signed-off-by: Melody Ren <melodyr@nvidia.com>
…_*_syndrome Signed-off-by: Melody Ren <melodyr@nvidia.com>
… qubits Each configured decoder now owns a worker thread and a bounded ring of request slots (always on, no config): the transport dispatcher thread VALIDATES and STAGES requests -- for the daemon path the syndrome bits are unpacked DIRECTLY into the decoder's ring slot (zero extra copy; the old path did two copies) -- and returns immediately for the fire-and-forget calls (enqueue_syndromes / reset_decoder). Decoder execution happens on the decoder's own thread, so N logical qubits' streams decode concurrently instead of serializing behind each other on the single cudaq_host_dispatcher_loop thread. Semantics: - Per-decoder FIFO order is preserved: single staging producer + per-decoder FIFO ring. Cross-decoder ordering is deliberately given up -- that is the parallelism. - get_corrections stages a rendezvous and blocks until that decoder's queue drains (end-of-shot only, bounded by one decode tail). - Validation errors still throw synchronously; a decoder exception during deferred execution becomes STICKY per-decoder state, rethrown at that decoder's next get_corrections (reset_decoder clears it). - Ring full -> staging blocks (natural backpressure onto the transport). New host APIs: stage_syndromes / commit_syndromes (zero-copy staging for transport handlers) and max_concurrent_decoder_workers() (busy high-water mark); the daemon prints QEC_DECODING_DAEMON_MAX_CONCURRENT_DECODERS at shutdown as concurrency evidence for tests. New test DecodingServerUdp.TwoProcessHostDispatchDualDecoders: one __qpu__ kernel drives TWO pymatching decoders (two logical qubits) in one daemon over UDP -- each decoder returns the correction for its own syndrome, no cross-talk, >= 6 dispatched calls, worker high-water mark >= 1. DaemonProcess test helper gains absolute --config paths, a configurable READY timeout, and MAX_CONCURRENT capture. Also documents (comment only) that cudaqx_add_device_code's nvq++ custom command emits no depfile, so header changes do not trigger recompilation of kernel objects; rm the stale <target>_<source>.o after header changes. Verified: full build with no extra cmake options, DecodingServerUdp 3/3, and the realtime/surface_code app_examples 18/18 (the py-* cases need PYTHONPATH=/usr/local/cudaq:<build>/python; see BRANCH_SUMMARY.txt). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
The dispatch_rpc catch block logs via the non-throwing cudaq::qec::error() free function rather than the CUDA_QEC_ERROR macro. The macro throws, and an exception escaping this handler into the transport dispatcher loop would terminate the process instead of returning the error response. Add a comment capturing that rationale (from PR #653) so the non-throwing call isn't "corrected" back into a throwing one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n bits Before cudaq PR 4816 the realtime device_call lowering could not bit-pack a std::vector<bool>, so the syndrome/correction bit vectors were hacked across the wire as a std::vector<uint8_t> (CUDAQ_TYPE_ARRAY_UINT8) with manual LSB-first packing. PR 4816 lowers std::vector<bool> to a native CUDAQ_TYPE_BIT_PACKED payload, so drop the hack and use it, keeping the change confined to the client/schema/translation boundary: - simulation_cqr_device.cpp: enqueue_syndromes device_call arg is now std::vector<bool>; the manual bit-packing loops are removed (the lowering packs it). simulation_cqr_host.cpp trampoline updated to the vector<bool> span semantics (length = # logical bits). - CqrTransceiver.h (parse_cqr_enqueue_frame): the stdvec array-length prefix (5th u64) is now the # of logical bits (= num_syndromes), not the byte count; the byte count is derived (ceil(bits/8)). This is the CUDAQ_TYPE_BIT_PACKED wire form. - decoding_server_cqr.cpp: declare the enqueue syndrome_bits arg and the get_corrections result as CUDAQ_TYPE_BIT_PACKED (matching the realtime unit tests), replacing the CUDAQ_TYPE_ARRAY_UINT8 stand-in. The internal EnqueuePayload / DecoderSession / rpc_producer path is unchanged; build_enqueue_frame still re-frames to the prefix-less internal layout. Verified: PyMatchingDeviceCallRealtime.HostDispatch, DecodingServerTwoProcess.*, PyMatchingRealtime.* and test_realtime_qldpc_graph_decoding all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The decode-server core (cudaq-qec-decoder-server) linked cudaq-qec PUBLIC, which propagated a NEEDED libcudaq-qec.so (and transitively libcustabilizer) into the cudaq-qec-realtime-decoding-server-cqr.so, violating the realtime-server dependency-closure contract (qec_realtime_server_dependency_closure) that forbids cudaq-qec but allows the lightweight cudaq-qec-decoders. The core only uses the decoder interface + logger (cudaq/qec/decoder.h, cudaq/qec/logger.h) and the realtime decoding config -- all provided by cudaq-qec-decoders + cudaq-qec-realtime-decoding -- so it never needed cudaq-qec. Looks like a target missed by the #645 "Remove CUDAQ Dependencies" work. After the change the .so needs only libcudaq-qec-decoders.so; #318 (qec_realtime_server_dependency_closure) and its selftest pass, and the full build + realtime functional tests still pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dding)
Drop the redundant explicit size arguments and all payload padding so the
cqr decode-server wire matches what the cc.device_call lowering emits. A
std::vector<bool> argument already carries its own element-count prefix, so
the hand-added size args (leftover from the ARRAY_UINT8 era) are redundant.
- enqueue_syndromes: drop num_syndromes from the device_call; the
syndrome_bits vector<bool> prefix carries N. parse_cqr_enqueue_frame reads
4 u64s (was 5); enqueue schema num_args 5 -> 4. Wire = 32 + B, no pad.
The lowered and daemon paths now emit the identical frame.
- get_corrections: drop return_size from the device_call (the OUT
vector<bool> length carries it); reset is a trailing bool. Request = 17
bytes, no pad; response result_len = ceil(R/8), no pad on either path
(removed align_to_8 in rpc_producer / qec_realtime_session).
Verified: 10/10 realtime + decoder_server_core, qldpc graph decode, and
55/55 app_examples pass. Enqueue arg_len probed at 32 + B (33 for N=3).
Wire format spec: NVIDIA/cudaq-spec decoder_server_runtime.md @ 725b33e
("decoder_server_runtime: make wire format lowering-accurate (no padding)").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-decoders main squash-merged PR #653; this branch already carried PR #653 plus the spec-compliant wire format and the centralized-decoder refactor, so the net delta is small. Conflict resolution (15 files): - Wire/schema (RpcWireFormat.h, CqrTransceiver.h, simulation_cqr_*, decoding_server_cqr.cpp schema): keep OURS — the lowering-accurate, no-padding format (32+B enqueue with BIT_PACKED element-count prefix, 17-byte get_corrections, result_len=ceil(R/8)). main was still on the old ARRAY_UINT8 / 5-arg / byte-count / padded format. - Implementation (DecoderSession.*, SessionRegistry.cpp, RoundAccumulator.*, DecoderServer.cpp, LoopbackTransceiver.h, RpcDispatcher.cpp): keep OURS — the ShotState/accepted_syndromes model is a superset of main's decoder_error bool, and centralized decoder construction supersedes main's inline H build. - Adopted from main: the g_transceiver post-shutdown null-check in decoding_server_cqr.cpp, and the _set/_get_syndrome_capture_callback rename (standardized underscore across realtime_decoding.*, decoding_server_cqr.cpp, surface_code-4-yaml.cpp). - Kept our non-throwing cudaq::qec::error() logger in the RPC handler (the CUDA_QEC_ERROR macro throws, which would kill the dispatcher thread). Updated BRANCH_SUMMARY.txt WIRE FORMAT section to the current spec. Tests: 10/10 realtime + decoder_server_core, qldpc graph decode, 55/55 app_examples, 17/17 dependency-closure/realtime — all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…der-server protocol fixes Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Collaborator
Author
|
/ok to test 89387c3 |
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
…e ring slots Two independent CI failures on the decoding-server branch: 1. CMake configure aborted with a FATAL_ERROR when CUDAQ_GPU_ROCE_AVAILABLE was set (hololink/DOCA present) but the proprietary cudevice archive was not provisioned. Hololink presence and the proprietary archive are provisioned independently, so gate the daemon's gpu_roce path on BOTH and fall back to udp/cpu_roce transports (with a WARNING) instead of failing configuration -- mirroring the existing if(TARGET ...) guard used by the app_examples tests. 2. Bus error (SIGBUS) in the realtime decode test at distance 5. HOST-mode ring slots skipped the 256-byte stride rounding that DEVICE mode applies, so slot_size_ computed to an unaligned 75 bytes at d5. write_response() performs a 4-byte __atomic RMW on the RPCHeader/RPCResponse `magic` field at slot offset i*slot_size_; an unaligned stride lands those atomics on cache-line-crossing addresses, which fault as split-lock SIGBUS on hosts with split-lock detection enabled (CI runners), silently tolerated elsewhere. Distance 3 floored at the aligned 64-byte minimum, so only d5 failed. Round the HOST-mode slot stride up to 16 bytes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inproc_rpc path unconditionally called cudaSetDeviceFlags(cudaDeviceMapHost) before decoder init, logging a spurious "CUDA driver version is insufficient" warning on CPU-only / GPU-less machines. That flag is only needed by the DEVICE-mode graph scheduler, which requires a usable GPU; CPU decoders run in HOST mode with plain host memory and never touch the device. Probe with cudaGetDeviceCount first and skip the flag entirely when no device is present. If a graph decoder is later selected without a usable GPU, qec_realtime_session::initialize() still fails with a clear DEVICE-mode error. Verified with CUDA_VISIBLE_DEVICES="": the warning no longer appears and the d5 pymatching realtime decode completes in HOST mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
/ok to test c252a9b |
## Description
Adds an end-to-end Holoscan-Sensor-Bridge (HSB) FPGA test for **CPU
decoders served by the
standalone decoding server** (`qec_decoding_daemon`), replacing the GPU
bridge used by the
qldpc flow. First supported decoder: **pymatching**; the orchestration
is generic so future
decoders plug in via data files alone.
```
hololink_fpga_syndrome_playback (UNCHANGED — qec_decoding_daemon --transport=cpu_roce --qp_config=hsb_fpga
sole FPGA control-plane writer)
1. cpu_roce ring up (one-shot start, peer QP from CLI) programs SIF with daemon QP/RKey/addr
2. prints "=== Bridge Ready ===" handshake writes per-round RPC frames to FPGA BRAM
3. HOST_CALL dispatch → pymatching (CPU) arms ILA, verifies captured RPCResponses
FPGA/emulator ──RDMA WRITE──► daemon rx ring (64 × 384B) ; daemon ──RDMA SEND──► FPGA SIF TX
```
## Changes
- **`qec_decoding_daemon`**: new `--qp_config=rendezvous|hsb_fpga` for
the `cpu_roce`
transport (plus `--peer-ip`, `--remote-qp` [default `0x2`, the FPGA
data-plane QP],
`--frame-size`). `hsb_fpga` is the HSB FPGA queue-pair-exchange method,
modeled on
cuda-quantum's `hsb_bridge_cpu.cpp`: peer QP from the CLI, one-shot
transceiver start (no
TCP rendezvous), and the daemon publishes its QP / RKey / Buffer Addr in
the canonical
`Bridge Ready` handshake (`hololink_bridge_common.h` format). The daemon
performs **no
Hololink control-plane traffic**; `num_slots` is clamped to the HSB WQE
depth (64). The
default rendezvous behavior and the udp transport are unchanged.
- **`hsb_fpga_decoder_server_test.sh`** (new; the qldpc script is
untouched): generic
orchestration with `--decoder NAME` resolving `config_NAME.yml` /
`syndromes_NAME.txt`.
Scrapes the daemon handshake and hands it to the playback tool, which
remains the sole
FPGA control-plane writer — the same division of labor as the qldpc
flow. Supports
`--emulate` (3-tool) and real-FPGA (2-tool) modes; `--device` accepts
one dual-port device
or two comma-separated single-port devices; `--build` verifies the
cuda-quantum checkout
against the `.cudaq_version` pin.
- **Data files**: `config_pymatching.yml` + `syndromes_pymatching.txt`,
generated with
`surface_code-4-yaml` (d=3 surface code, 4 rounds, `p_spam=0.01`, 100
shots; decomposed
graph-like DEM — every H column ≤ 2 ones as pymatching requires — 1
observable; generation
commands recorded in the file headers). Playback's per-round mode
replays 85 shots
(512-cycle playback BRAM ÷ 6 frames/shot).
No wire-format changes: playback and the server are byte-compatible per
the
"lowering-accurate (no padding)" `decoder_server_runtime.md`
convergence.
---------
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Collaborator
Author
|
/ok to test 9077024 |
bmhowe23
commented
Jul 9, 2026
Signed-off-by: Ben Howe <bhowe@nvidia.com>
…review Correctness fixes from a focused review of the decoder-server core (merged hastily in #653): - config.cpp: guard the process-wide multi_decoder_config with a mutex and hand out shared ownership, so a concurrent configure_decoders() cannot free the config while the CQR plugin's lazy init is copying it. - decoding_server_cqr.cpp: catch init_server failures in createDispatchSession (an escaping exception would terminate CUDAQ's channel setup) and publish g_transceiver only after the DecoderServer is fully constructed so a throwing constructor cannot leave it dangling. - ITransceiver.h: default-initialize RxFrame::vp_id/peer (Loopback recv left them indeterminate, nondeterministically poisoning shots); make release_fn a move-only fire-on-destruction holder so every drop path (dispatcher validation, queue-full, exceptions) returns the ring slot exactly once. - DecoderSession.cpp: add a last-resort catch around worker dispatch (an escaping exception would std::terminate the server); run the get_corrections reset before sending OK so a throw cannot emit a second response for an answered request; store the shutdown flag under queue_mutex and use an untimed cv wait (the previous store-outside-lock made a plain wait a lost-wakeup risk, papered over by 100 ms polling). - CqrTransceiver.h: fail oversized responses with INTERNAL_ERROR instead of silently truncating while result_len advertises bytes that were never written. Cleanups: deduplicate the RPC function-id constants (schema registration now uses the routing constants, pinned by static_asserts), derive kStatusHandlerException from RpcStatus instead of a literal 3, drop the unused CorrectionBits alias, and parse the YAML config once in DecoderServer(yaml) instead of twice with duplicated validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Collaborator
Author
|
/ok to test 7ad68ec |
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Collaborator
Author
|
/ok to test 034e055 |
The decoding-server daemon was build-tree-only: RUNTIME_OUTPUT_DIRECTORY
and a BUILD_RPATH, but no install() rule, while the libraries around it
(cudaq-qec-decoder-server, the decoding-server-cqr plugin) are installed.
Install the daemon and its two example YAML configs to
${CMAKE_INSTALL_BINDIR} under a new qec-tools component (mirroring
solvers-tools). The component is deliberately NOT added to the Python
wheel's install.components: the daemon depends on the CUDA-Q realtime
libs, which the cudaq wheel does not ship.
Give the binary an INSTALL_RPATH ($ORIGIN/../lib,
$ORIGIN/../lib/decoder-plugins, plus the absolute CUDA-Q realtime lib
dir) so the installed copy resolves libcudaq-qec and the decoder
plugins; previously only a build rpath existed. Verified with a
component install to a scratch DESTDIR: the installed daemon loads the
pymatching plugin from the install tree and serves a config end-to-end.
Drop "Test-only" from the daemon's @brief now that it is a shipped tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Collaborator
Author
|
/ok to test acfd5c1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR builds out the standalone realtime decoding server: a separate decoder process reached from the QPU-side program over a CUDA-Q
device_calltransport, with both the decoder selection and the transport being configuration, not code. It also adds the parallel/heterogeneous decoder support underneath it and the two-process and FPGA-in-the-loop tests that exercise the whole path.What's new
Decoding-server daemon (
libs/qec/tools/decoding-server/)qec_decoding_daemon: a standalone decoding-server process — the service end of a CUDA-Qdevice_calltransport, decoding on the CPU with whatever decoder(s) a YAML config selects (--config=<yaml>, parsed bymulti_decoder_config). The function table (enqueue_syndromes/get_corrections/reset_decoder) comes from thedecoding-server-cqrservice plugin regardless of transport or decoder.--transport=udp(loopback ring transceiver, runs anywhere) or--transport=cpu_roce(RDMA, requires a ConnectX NIC). Forcpu_roce,--qp_config=rendezvous|hsb_fpgaselects TCP QP/rkey rendezvous vs. the Holoscan-Sensor-Bridge FPGA handshake (daemon performs no FPGA control-plane traffic).QEC_DECODING_DAEMON_READYline for orchestration and aQEC_DECODING_DAEMON_DISPATCHED count=<N>line at shutdown as the cross-process dispatch proof.Transceiver layer (
decoding-server-cqr)ITransceiverabstraction with three implementations: the existing loopback, a newCpuRoceTransceiver(RDMA over the CPU HOST_CALL path), and a newGpuRoceTransceiver— a GPU RoCE path that wires the Hololink Sensor Bridge receiver into CUDA-Q's device-graph dispatcher (env-var configured, guarded byCUDAQ_GPU_ROCE_AVAILABLECMake detection with graceful fallback when no GPU/NIC is present).RpcWireFormat.h/decoder_rpc_ids.hto stay byte-exact with what CUDA-Q kernel lowering actually emits.Parallel / heterogeneous decoders
SessionRegistry+DecoderSessionrework so multiple decoders run concurrently on independent worker threads, including different decoder types served by one daemon; lifecycle races and hardening gaps found in the PR Add QEC decoder-server core and CQR adapter #653 review are fixed.create_realtime_decoder()centralizes realtime decoder construction (ID + O/D matrix wiring) so the in-process path, the daemon, and tests all build decoders the same way; decoder construction/initialization failures (e.g. TensorRT) now propagate instead of being swallowed.Tests
test_decoding_server: two-process test — this process is caller-only (no decoder configured at all), the daemon is spawned as a subprocess, and the daemon's dispatch count proves thedevice_calls crossed the process boundary. Parameterized overudpandcpu_roce/SoftRoCE transports.test_decoder_server_core/test_decoders_yaml: unit coverage for the server core and YAML decoder configuration.concurrency_test_decoder: an instrumented decoder plugin proving three decoder workers genuinely overlap in time.app_examples:surface_code-1(minimal realtime decode app + two-process CQR variant) andsurface_code-4-yaml(multi-patch surface code, one decoder type per patch, exercising heterogeneous decoders through the server), plusgen_dsparse_from_memory_circuit.pyfor generating the sparse detector matrices.hsb_fpga_decoder_server_test.sh: end-to-end orchestration over Holoscan-Sensor-Bridge RDMA — FPGA (or emulator) plays back syndrome frames into the daemon's RDMA ring and verifies the RPC responses; works with any CPU decoder the YAML selects.Size note
This is well over the ~1000-line guideline (~8,100 insertions), but the bulk is test infrastructure (
app_examples, the two-process tests, and the HSB orchestration script account for roughly 5,500 lines); the daemon, transceivers, and server-core changes are the reviewable core.Self-review checklist
Please confirm each item before requesting review. Check
[x]or strikethrough and explain.
Before requesting review
Scope and size
(if so, an issue has been raised).
Tests
just when it is missing.
EXPECT_*/assertchecks areinsufficient for algorithmic correctness.
Documentation
tracked.
Code style
snake_casevscamelCase) forthe area being modified.
Dependencies
OSRB tickets filed.