Repository navigation
feat(cuda-oxide): shared-memory i32/addrspace(3) codegen and CTA-local TMA without SharedPtr - #1656
Open
nihalpasham wants to merge 3 commits into
Open
nihalpasham wants to merge 3 commits into
nihalpasham wants to merge 3 commits into
Conversation
…and grid_constant in blockscale_gemm_cute Signed-off-by: nihalpasham <nihalp@nvidia.com>
…ts/casts and narrow MLIR shared GEP indices to i32 Signed-off-by: nihalpasham <nihalp@nvidia.com>
- Have `cuda_device::tma::cp_async_bulk_tensor_{1..5}d_g2s_cta` take `dst: *mut u8` directly, matching `cuda_intrinsics` and `barrier: *mut Barrier`.
- Remove `SharedPtr<T>`, `SharedSlot<T>`, and their special-case lowering hooks from `cuda-device`, `cuda-intrinsics-gen`, and `mir-importer`, along with the synthetic `SharedPtr` examples.
- Route `ctm_rs::tma::cp_async_bulk_tensor_shared_cta_global` to `cp_async_bulk_tensor_2d_g2s_cta` (`.shared::cta`) and register `CpAsyncBulkTensorG2sCtaTile2dOp` in the SM120 MLIR export pack, eliminating the `.shared::cluster` syscall fallback and 8-byte stack frame on `sm_120a`.
Signed-off-by: nihalpasham <nihalp@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR closes shared-memory addressing and CTA-local TMA codegen gaps across both the
cutlass-mlir(cute-rs) and nativeLLVM/PTX(ctm-rs/ SIMT) backends:address_space::SHARED(addrspace(3)) across pointer offsets and casts (mir-importer):SlotAddrSpaceMap(values.rs) andRvalue::Cast(rvalue/expr.rs) now preserve non-generic address spaces acrossBinOp::Offset(ptr.add/ptr.offset) andCastKind::PtrToPtr/MutToConstPointer(ptr.cast), eliminatingaddrspace(3) -> addrspace(0) -> addrspace(3)round-trips on derived shared pointers.i32shared-memory offsets incuda-oxide-mlir-export+#[grid_constant]inblockscale_gemm_cute:MirPtrOffsetOpon!llvm.ptr<3>and SM120mma/epilogueshared-memory index arithmetic fromi64toi32, allowing LLVM'sLoadStoreVectorizerto formLDS.128vector loads automatically and removing 64-bitSHF.R.U64/IADD.64instructions from the K-loop.blockscale_gemm_cuteas#[grid_constant] &TensorMap..shared::ctaTMA loads with*mut u8and removeSharedPtr<T>:cuda_device::tma::cp_async_bulk_tensor_{1..5}d_g2s_ctato takedst: *mut u8directly (matchingcuda_intrinsicsandbarrier: *mut Barrier), sinceconvert_g2s_ctaalready accepts bothaddrspace(3)andaddrspace(0)pointers andSlotAddrSpaceMapkeeps&raw mut TILE as *mut u8inaddrspace(3).SharedPtr<T>,SharedSlot<T>, and their special-case plumbing fromcuda-device,cuda-intrinsics-gen, andmir-importer.ctm_rs::tma::cp_async_bulk_tensor_shared_cta_globaltocp_async_bulk_tensor_2d_g2s_cta(.shared::cta) and registerCpAsyncBulkTensorG2sCtaTile2dOpin the SM120 MLIR export pack, eliminating the.shared::clusterruntime CTA-id check (CALL.ABS.NOINC __cuda_syscall_cp_async_bulk_tensor_2d_tile_unicast) and 8-byte stack frame onsm_120a.Verification
scripts/smoketest.shon RTX 5090 (sm_120a): 238 / 238 examples PASS (tma_copy/verify-code-shape.shpasses with.shared::cta, zerocvta.to.shared, and zeroshared::cluster).just check(fmt-check,clippy,test,test-cuda,check-guards,check-intrinsics main,doc-check): all PASS.scripts/check-primitives-api-compile.py(--example api,--example fp16,--example blockscale): all PASS.setmaxnreg+.shared::ctaverification onsm_120a(ptxas -v -arch=sm_120a+nvdisasm):USETMAXREG.DEALLOC.CTAPOOL 0x28,USETMAXREG.TRY_ALLOC.CTAPOOL UP0, 0xe8,UTMALDG.2D,168registers,0stack frame.blockscale_gemm_cuteandblockscale_gemm_ctmon RTX 5090 (2048x2048x1024MXFP4 GEMM, Nsightnsys profilekernel-active median of 31 launches after 100 warmups):blockscale_gemm_cute(cutlass-mlir):11.360 us -> 10.912 us(CudaGymgpu_active:11.008 usvs Python CUTLASS baseline10.928 us,0.993x).blockscale_gemm_ctm(nativeLLVM/PTX):11.616 us -> 10.976 us(0stack frame,0syscall fallback; vs Python CUTLASS CTM10.880 us, within0.9%).