Skip to content

feat(cuda-oxide): shared-memory i32/addrspace(3) codegen and CTA-local TMA without SharedPtr - #1656

Open
nihalpasham wants to merge 3 commits into
NVIDIA:mainfrom
nihalpasham:feat/shared-addrspace-i32-codegen
Open

nihalpasham wants to merge 3 commits into
NVIDIA:mainfrom
nihalpasham:feat/shared-addrspace-i32-codegen

Conversation

@nihalpasham

Copy link
Copy Markdown
Collaborator

Summary

This PR closes shared-memory addressing and CTA-local TMA codegen gaps across both the cutlass-mlir (cute-rs) and native LLVM/PTX (ctm-rs / SIMT) backends:

  1. Propagate address_space::SHARED (addrspace(3)) across pointer offsets and casts (mir-importer):
    • SlotAddrSpaceMap (values.rs) and Rvalue::Cast (rvalue/expr.rs) now preserve non-generic address spaces across BinOp::Offset (ptr.add / ptr.offset) and CastKind::PtrToPtr / MutToConstPointer (ptr.cast), eliminating addrspace(3) -> addrspace(0) -> addrspace(3) round-trips on derived shared pointers.
  2. Use 32-bit i32 shared-memory offsets in cuda-oxide-mlir-export + #[grid_constant] in blockscale_gemm_cute:
    • Narrow MirPtrOffsetOp on !llvm.ptr<3> and SM120 mma / epilogue shared-memory index arithmetic from i64 to i32, allowing LLVM's LoadStoreVectorizer to form LDS.128 vector loads automatically and removing 64-bit SHF.R.U64 / IADD.64 instructions from the K-loop.
    • Pass the 5 TMA descriptors in blockscale_gemm_cute as #[grid_constant] &TensorMap.
  3. Use .shared::cta TMA loads with *mut u8 and remove SharedPtr<T>:
    • Change cuda_device::tma::cp_async_bulk_tensor_{1..5}d_g2s_cta to take dst: *mut u8 directly (matching cuda_intrinsics and barrier: *mut Barrier), since convert_g2s_cta already accepts both addrspace(3) and addrspace(0) pointers and SlotAddrSpaceMap keeps &raw mut TILE as *mut u8 in addrspace(3).
    • Remove SharedPtr<T>, SharedSlot<T>, and their special-case plumbing from cuda-device, cuda-intrinsics-gen, and mir-importer.
    • Route ctm_rs::tma::cp_async_bulk_tensor_shared_cta_global to cp_async_bulk_tensor_2d_g2s_cta (.shared::cta) and register CpAsyncBulkTensorG2sCtaTile2dOp in the SM120 MLIR export pack, eliminating the .shared::cluster runtime CTA-id check (CALL.ABS.NOINC __cuda_syscall_cp_async_bulk_tensor_2d_tile_unicast) and 8-byte stack frame on sm_120a.

Verification

  • scripts/smoketest.sh on RTX 5090 (sm_120a): 238 / 238 examples PASS (tma_copy/verify-code-shape.sh passes with .shared::cta, zero cvta.to.shared, and zero shared::cluster).
  • just check (fmt-check, clippy, test, test-cuda, check-guards, check-intrinsics main, doc-check): all PASS.
  • scripts/check-primitives-api-compile.py (--example api, --example fp16, --example blockscale): all PASS.
  • setmaxnreg + .shared::cta verification on sm_120a (ptxas -v -arch=sm_120a + nvdisasm): USETMAXREG.DEALLOC.CTAPOOL 0x28, USETMAXREG.TRY_ALLOC.CTAPOOL UP0, 0xe8, UTMALDG.2D, 168 registers, 0 stack frame.
  • blockscale_gemm_cute and blockscale_gemm_ctm on RTX 5090 (2048x2048x1024 MXFP4 GEMM, Nsight nsys profile kernel-active median of 31 launches after 100 warmups):
    • blockscale_gemm_cute (cutlass-mlir): 11.360 us -> 10.912 us (CudaGym gpu_active: 11.008 us vs Python CUTLASS baseline 10.928 us, 0.993x).
    • blockscale_gemm_ctm (native LLVM/PTX): 11.616 us -> 10.976 us (0 stack frame, 0 syscall fallback; vs Python CUTLASS CTM 10.880 us, within 0.9%).

…and grid_constant in blockscale_gemm_cute

Signed-off-by: nihalpasham <nihalp@nvidia.com>
…ts/casts and narrow MLIR shared GEP indices to i32

Signed-off-by: nihalpasham <nihalp@nvidia.com>
- Have `cuda_device::tma::cp_async_bulk_tensor_{1..5}d_g2s_cta` take `dst: *mut u8` directly, matching `cuda_intrinsics` and `barrier: *mut Barrier`.
- Remove `SharedPtr<T>`, `SharedSlot<T>`, and their special-case lowering hooks from `cuda-device`, `cuda-intrinsics-gen`, and `mir-importer`, along with the synthetic `SharedPtr` examples.
- Route `ctm_rs::tma::cp_async_bulk_tensor_shared_cta_global` to `cp_async_bulk_tensor_2d_g2s_cta` (`.shared::cta`) and register `CpAsyncBulkTensorG2sCtaTile2dOp` in the SM120 MLIR export pack, eliminating the `.shared::cluster` syscall fallback and 8-byte stack frame on `sm_120a`.

Signed-off-by: nihalpasham <nihalp@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant