ASan fails with "AddressSanitizer: nested bug in the same thread, aborting" because of allocation failures in __asan::AsanOnDeadlySignal

[evverx]

When ASan reports valid issues in fuzz targets using nallocinc built with the OSS-Fuzz tolchain

AddressSanitizer:DEADLYSIGNAL                                                                                                                                 
=================================================================                                                                                             
==8==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fd3cbccc7dd bp 0x7ffd79e37fd0 sp 0x7ffd79e37788 T0)                               
==8==The signal is caused by a READ memory access.                                                                                                            
==8==Hint: address points to the zero page.                                                                                                                   
SCARINESS: 10 (null-deref)                                                                                                                                    
    #0 0x7fd3cbccc7dd  (/lib/x86_64-linux-gnu/libc.so.6+0x18b7dd) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)                                         
    #1 0x557d2a5c9ef3 in strlen /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc                                  
    #2 0x557d2a69ace6 in avahi_alternative_host_name /src/avahi/avahi-common/alternative.c:123:33                                                             
    #3 0x557d2a699464 in LLVMFuzzerTestOneInput /src/avahi/fuzz/fuzz-domain.c:65:16 
...

it ends up failing itself along the way with

==8==ABORTING
MS: 1 InsertByte-libc++abi: bad_alloc was thrown in -fno-exceptions mode
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

With NALLOC_VERBOSE=1 the latest backtrace points to __asan::AsanOnDeadlySignal where an allocation failure is injected too

MS: 1 ShuffleBytes-failed malloc(48)

==8==ABORTING                                                                                                                                                 
MS: 1 ShuffleBytes-failed malloc(48)                                                                                                                          
    #0 0x55c932e5c131 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3                                             
    #1 0x55c932e94c0f in nalloc_backtrace_exclude /src/avahi/fuzz/nallocinc.c:196:5                                                                           
    #2 0x55c932e94c0f in nalloc_fail /src/avahi/fuzz/nallocinc.c:224:9
    #3 0x55c932e94f6a in malloc /src/avahi/fuzz/nallocinc.c:323:7
    #4 0x55c932e9c6f3 in operator new(unsigned long) cxa_noexception.cpp
    #5 0x55c932d71da9 in std::__Fuzzer::basic_stringbuf<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::overflow(int) cxa_noexception.cpp
    #6 0x55c932d6ad0d in std::__Fuzzer::basic_streambuf<char, std::__Fuzzer::char_traits<char>>::xsputn(char const*, long) cxa_noexception.cpp
    #7 0x55c932d190a5 in sputn /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/streambuf:233:12
    #8 0x55c932d190a5 in std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char>> std::__Fuzzer::__pad_and_output[abi:nn220000]<char, std::__Fuzzer::char_traits<char>>(std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char>>, char const*, char const*, char const*, std::__Fuzzer::ios_base&, char) /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__locale_dir/pad_and_output.h:75:22
    #9 0x55c932d7f434 in std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char>> std::__Fuzzer::num_put<char, std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char>>>::__do_put_integral[abi:nn220000]<unsigned long>(std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char>>, std::__Fuzzer::ios_base&, char, unsigned long) const cxa_noexception.cpp
    #10 0x55c932d6d0c8 in std::__Fuzzer::basic_ostream<char, std::__Fuzzer::char_traits<char>>& std::__Fuzzer::basic_ostream<char, std::__Fuzzer::char_traits<char>>::__put_num_integer_promote[abi:nn220000]<unsigned int>(unsigned int) cxa_noexception.cpp
    #11 0x55c932d47e68 in fuzzer::Sha1ToString(unsigned char const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerSHA1.cpp:214:57
    #12 0x55c932d2e9fa in fuzzer::Fuzzer::DumpCurrentUnit(char const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:179:31
    #13 0x55c932d2f26b in DeathCallback /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:191:3
    #14 0x55c932d2f26b in fuzzer::Fuzzer::StaticDeathCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:171:6
    #15 0x55c932e762ef in __sanitizer::Die() /src/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:52:5
    #16 0x55c932e5727a in __asan::ScopedInErrorReport::~ScopedInErrorReport() /src/llvm-project/compiler-rt/lib/asan/asan_report.cpp:221:7
    #17 0x55c932e56a43 in __asan::ReportDeadlySignal(__sanitizer::SignalContext const&) /src/llvm-project/compiler-rt/lib/asan/asan_report.cpp:249:1
    #18 0x55c932e560a7 in __asan::AsanOnDeadlySignal(int, void*, void*) /src/llvm-project/compiler-rt/lib/asan/asan_posix.cpp:40:3
    #19 0x7f75403ed32f  (/lib/x86_64-linux-gnu/libc.so.6+0x4532f) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)
    #20 0x7f75405337dc  (/lib/x86_64-linux-gnu/libc.so.6+0x18b7dc) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)
    #21 0x55c932dc5ef3 in strlen /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc
    #22 0x55c932e96ce6 in avahi_alternative_host_name /src/avahi/avahi-common/alternative.c:123:33
    #23 0x55c932e95464 in LLVMFuzzerTestOneInput /src/avahi/fuzz/fuzz-domain.c:65:16
    #24 0x55c932d30dfd in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:619:13
    #25 0x55c932d30435 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:516:7
    #26 0x55c932d32105 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:765:19
    #27 0x55c932d32d65 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:910:5
    #28 0x55c932d21915 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:917:6
    #29 0x55c932d4d572 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #30 0x7f75403d21c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)
    #31 0x7f75403d228a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)
    #32 0x55c932d14c64 in _start (/out/fuzz-domain-nalloc+0x42c64)

(on a somewhat related note prompted by commits like avahi/avahi@2a81a57 among other things I was planning to roll out fuzz targets like that on OSS-Fuzz but it seems OSS-Fuzz would have a hard time reproducing and reporting bugs reliably (unless I'm missing something))

[catenacyber]

So, I guess nalloc_backtrace_exclude should do more like it does in Suricata nallocinc version

Do you see another fix ?

[evverx]

I didn't know Suricata diverged.

I'll take a look at https://github.com/OISF/suricata/blob/a35760db9b703d58f88d080be67034263a9dabad/src/tests/fuzz/nallocinc.c (as far as I can see the difference is the presence of nalloc_prefix_{len|dir}) but as far as I understand even if it fixes the issue OSS-Fuzz couldn't reproduce bugs reliably. For example currently memory leaks are reported correctly with reproducers without crashes but when those reproducers are passed back to the fuzz targets they don't report those memory leaks unless NALLOC_FREQ is set or something like that.

[evverx]

I took nallocinc from Suricata and with NALLOC_RESTRICT_FILE_PREFIX the fuzz targets wrote the reproducers without crashing

...
artifact_prefix='./'; Test unit written to ./crash-58e6b3a414a1e090dfc6029add0f3555ccba127f
Base64: ZQ==

That being said I'm not sure how to replay the allocation failures for OSS-Fuzz to reproduce bugs reliably. I'll probably run them on CFLite or something like that where the backtraces should be enough.

[catenacyber]

NALLOC_RESTRICT_FILE_PREFIX will reduce the allocations that can crash to only the project.
Especially, 3rd party libraries will not have allocation failures, and they are not even accounted for when the next allocation should fail.

That is why the same reproducer does not crash with different values for NALLOC_RESTRICT_FILE_PREFIX

[catenacyber]

And for reproducer, you should typically have at least -runs=2 since the first run typically does some static allocations for the whole target lifetime

[evverx]

Unfortunately -runs isn't reliable in the sense that it can be different for different bugs (and it can't be controlled on OSS-Fuzz where the toolchain always passes 4). NALLOC_FREQ more or less works because it always resets nalloc_random_state in nalloc_start so the crashes depend on the input only but with it takes longer to arrive at those crashes based on my experiments.