[C++][CI] MinGW Gcc 15.2->16.1 toolchain regression - memory corruption in multi-threaded tests

[tadeja]

Describe the bug, including details regarding any error messages, version, and platform.

Update: not an Arrow bug - gcc 15.2->16.1 toolchain regression. Diagnosis in #49961 + comment below.
Keeping workaround of #49931's MSYS2 pin step. #49945 stays waiting, parent issue: #49948.

On AMD64 Windows MinGW MINGW64 C++ with MSYS gcc 15.2 -> 16.1 update (post 2026 April 30: mingw-w64-x86_64-gcc-16.1.0-3 and mingw-w64-x86_64-gcc-libs) two tests started failing every run with std::bad_weak_ptr: arrow-dataset-dataset-writer-test and arrow-dataset-file-test. Also affected, with no dataset code involved: arrow-async-utility-test (heap corruption in PushGenerator.Stress), arrow-threading-utility-test (Timeout), ...
CLANG64 C++ (libc++) and gcc 15.2 are unaffected!

Example CI log

[ RUN      ] TestFileSystemDataset.MultiThreadedWritePersistsOrder
terminate called after throwing an instance of 'std::bad_weak_ptr'
  what():  bad_weak_ptr
...
[ RUN      ] DatasetWriterTestFixture.BatchWriteConcurrent
terminate called after throwing an instance of 'std::bad_weak_ptr'
  what():  bad_weak_ptr

Component(s)

C++, Continuous Integration

[tadeja]

Cross-posting from pr for visibility:

Diagnosed further - this isn't a dataset_writer bug. It looks like lower-level memory corruption, likely a gcc-16/MinGW regression. Keeping workaround of #49931's MSYS2 pin for now, #49945 stays waiting, tracked under #49948.

• The bad_weak_ptr isn't an Arrow lifetime bug: in BatchWriteConcurrent the queue's only shared_ptr is not reset until Finish(), so weak_from_this().expired() there means a corrupted control block. reset -> make_shared did not fix this.
• adding fprintf to the constructor/destructor of DatasetWriterFileQueue made it a Heisenbug - failed once, retry passed OK:
  https://github.com/apache/arrow/actions/runs/25737251706/job/75577465287?pr=49961#step:12:604

    Test #62: arrow-dataset-dataset-writer-test ............Exit code 0xc0000374***Exception:   0.30 sec
...
    Test #62: arrow-dataset-dataset-writer-test ............   Passed    4.85 sec
...
The following tests FAILED:
	 41 - arrow-async-utility-test (Exit code 0xc0000374)   arrow-tests unittest
	 44 - arrow-threading-utility-test (Timeout)            arrow-tests unittest
	 65 - arrow-dataset-file-test (Failed)                  arrow_dataset unittest
	 78 - arrow-flight-test (Failed)                        arrow_flight unittest
Error: Process completed with exit code 8.

• Independent reproducers without dataset_writer are: arrow-async-utility-test always dies at PushGenerator.Stress and arrow-threading-utility-test always times out. Both fine on CLANG64/libc++ and on gcc‑15.2. Only gcc 15.2->gcc 16.1 is affected.

[kou]

Thanks for debugging this. If this is a MinGW GCC 16.1 problem, it's better that we report it to the upstream. Can we create a (small) program code that reproduces this?