Unix: sigaltstack is munmapped on thread exit without SS_DISABLE, leaving a dangling registration (memory corruption under ASan embedders)

[tiandiwonder]

Summary

On Unix, lazy_per_thread_init (crates/wasmtime/src/runtime/vm/sys/unix/signals.rs) registers a custom alternate signal stack per thread. Its TLS destructor deallocates it like this:

impl Drop for Stack {
    fn drop(&mut self) {
        unsafe {
            // Deallocate the stack memory.
            let r = rustix::mm::munmap(self.mmap_ptr, self.mmap_size);
            debug_assert!(r.is_ok(), "munmap failed during thread shutdown");
        }
    }
}

The stack is munmapped without first calling sigaltstack with SS_DISABLE (or restoring the previous stack), so the kernel keeps the freed range registered as the thread's alternate signal stack for the remainder of thread teardown.

Consequences

1. If any SA_ONSTACK signal is delivered to the thread after this destructor runs (other TLS destructors, sanitizer teardown, etc.), the kernel pushes the signal frame onto unmapped memory.

2. Worse, for AddressSanitizer embedders: ASan's thread-destruction hook calls UnsetAlternateSignalStack() (compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp), which queries the currently registered altstack and unconditionally UnmapOrDie(oldstack.ss_sp, oldstack.ss_size). With the dangling registration left by Stack::drop, ASan re-munmaps a 256 KiB range that was already freed — and possibly already reused by the allocator — silently destroying pages of live, unrelated heap allocations. The eventual crash appears at a random later point in whatever code owned the reused memory, which makes it extremely hard to attribute.

This looks like the actual mechanism behind the long-standing mystery documented in the comment at the top of lazy_per_thread_init (the ASan fuzzing crashes reproduced in https://gist.github.com/alexcrichton/6815a5d57a3c5ca94a8d816a9fcc91af): it is not TLS-destructor ordering corrupting ASan state — it is the dangling sigaltstack registration making ASan's UnsetAlternateSignalStack unmap a stale (reused) range. The existing cfg!(asan) guard only protects builds where the Rust code is compiled with -Zsanitizer=address; embedders that instrument only their C/C++ side with ASan and link wasmtime built without it (a common setup) still hit the bug.

Real-world impact

We (ClickHouse) chased sporadic ASan CI crashes for two months, where random large read buffers were found unmapped mid-use. Forensics of a core dump showed the faulted buffer was still registered as live in ASan's secondary allocator (never freed) and the destroyed page range measured exactly one wasmtime sigaltstack allocation (guard page + MIN_STACK_SIZE = 64×4096); a WASM module had executed on a pool thread minutes earlier, and the crash fired when that thread was reaped. Details: ClickHouse/ClickHouse#104692, fixed on our side by re-enabling cfg(asan) via ClickHouse/ClickHouse#109950.

Suggested fix

In Stack::drop, disable (or restore the previously registered) alternate signal stack before unmapping:

impl Drop for Stack {
    fn drop(&mut self) {
        unsafe {
            let disable = libc::stack_t {
                ss_sp: ptr::null_mut(),
                ss_flags: libc::SS_DISABLE,
                ss_size: MIN_STACK_SIZE,
            };
            let r = libc::sigaltstack(&disable, ptr::null_mut());
            debug_assert_eq!(r, 0, "sigaltstack(SS_DISABLE) failed during thread shutdown");
            let r = rustix::mm::munmap(self.mmap_ptr, self.mmap_size);
            debug_assert!(r.is_ok(), "munmap failed during thread shutdown");
        }
    }
}

With that in place the cfg!(asan) opt-out may even become unnecessary, since ASan's UnsetAlternateSignalStack would then see a disabled stack instead of a dangling one (worth verifying — UnmapOrDie(NULL, ...) on a disabled stack would fail loudly rather than corrupt, so ASan-instrumented-Rust builds may still want the guard).

[alexcrichton]

Thanks for the report! I'm not certain there's much we can do about this though other than recommending that asan builds indeed compile Rust with asan as well to disable the sigaltstack. This is a much better explanation to the current hand-wavy one of why we can't use a sigaltstack with asan, however.

What I'd be worried about is that I basically don't think we're equipped to reset the thread's sigaltstack back to the original on drop. While we could preserve whatever was leftover there we still don't have any guarantee about the order of TLS destructors so we don't know if we'd reset the stack from Wasmtime's stack to the prior or if the stack has already been reset by something else. Additionally I don't think we can just unset the sigaltstack because historical versions of LLVM seem to always call munmap on the return value of sigaltstack when unsetting.

Looking at LLVM tip-of-tree, however, it looks like this behavior has changed and might mean that this is now compatible? I think that the tip-of-tree behavior would be compatible with Wasmtime as-is, and it'd also be compatible with unsetting the sigaltstack as well in the handler.