Skip to content

Data Races & Flawed Synchronization due to Invalid Atomic Memory Scope in SystemAtomicF32 and `SystemAtomicF64 #399

Description

@Snehal-Reddy

SystemAtomicF32 and SystemAtomicF64 types are mistakenly configured with device scope instead of system scope in their macro instantiations.

This presents the following issues:

  1. Flawed Synchronization: Code that uses SystemAtomicF32 or SystemAtomicF64 intends to synchronize memory accesses across the entire system (between the GPU and host CPU or across multiple GPUs over PCIe/NVLink).
  2. Data Races: Because the emitted PTX operations use the .gpu scope (atomic.global.gpu...) rather than the proper .sys scope (atomic.global.sys...), CPU host accesses and multi-GPU accesses to system memory will not observe proper cache coherency. This leads to data races and corrupted shared memory states without emitting any compilation errors.

Reproduction Case

Using SystemAtomicF32 or SystemAtomicF64 and calling atomic operations (like fetch_add, load, or store) will incorrectly generate device-scoped PTX:

use cuda_std::atomic::SystemAtomicF32;
use core::sync::atomic::Ordering;

#[cuda_std::kernel]
pub unsafe fn system_atomic_kernel(val: &SystemAtomicF32) {
    // PTX generates: atomic.global.gpu.add.f32 ...
    // Expected: atomic.global.sys.add.f32 ...
    val.fetch_add(1.0, Ordering::Relaxed);
}

Issue Details

In crates/cuda_std/src/atomic.rs, the type instantiations for system-level float atomics incorrectly pass the $scope argument as device instead of system to the macro:

// Current instantiations in atomic.rs
atomic_float!(f32, AtomicF32, 4, device, 32);
atomic_float!(f64, AtomicF64, 8, device, 64);
atomic_float!(f32, BlockAtomicF32, 4, block, 32, unsafe);
atomic_float!(f64, BlockAtomicF64, 8, block, 64, unsafe);
atomic_float!(f32, SystemAtomicF32, 4, device, 32); // <--- BUG: device instead of system
atomic_float!(f64, SystemAtomicF64, 8, device, 64); // <--- BUG: device instead of system

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions