Multithreaded ne.evaluate() intermittently returns all-zero output on Windows / I7 14700K CPU (reproduces 2016 SO #36484326)

[Undi123]

numexpr.evaluate() with default multithreading occasionally writes an all-zero array into the output buffer, even when inputs are constant non-zero arrays. The same issue was reported on Stack Overflow in 2016 (https://stackoverflow.com/questions/36484326) and never resolved. Confirming it still reproduces on current numexpr.

Environment
Python 3.13.9 (Anaconda) on Windows 11
pandas 2.3.3 — pd.options.mode.copy_on_write = False, compute.use_numexpr = True, expressions._MIN_ELEMENTS = 1_000_000
numpy 2.3.5
numexpr 2.14.1 — detect_number_of_threads()=28, nthreads=16
pyarrow 21.0.0, bottleneck 1.4.2, numba 0.62.1

Minimal reproducer

import numpy as np
import numexpr as ne

x = np.ones(1000000)
y = np.ones(1000000)

for ii in range(1000):

    rr = ne.evaluate('x * y')
    test = np.all(rr == 0)
    if test:
        print('Gotcha! %d' % ii)

print('Complete!')

Observed

%runcell -i 0 F:/untitled0.py
Gotcha! 30
Gotcha! 38
Gotcha! 46
Gotcha! 52
Gotcha! 135
Gotcha! 197
Gotcha! 245
Gotcha! 285
Gotcha! 353
Gotcha! 505
Gotcha! 542
Gotcha! 543
Gotcha! 559
Gotcha! 612
Gotcha! 629
Gotcha! 678
Gotcha! 877
Gotcha! 899
Gotcha! 912
Gotcha! 938
Gotcha! 992
Complete!

Workaround confirmed
ne.set_num_threads(1)
0 all-zero outputs across 1000 iterations

[27rabbitlt]

looking into this

[27rabbitlt]

I cannot reproduce this bug on my Win11 PC with numpy and numexpr version the same. I have 8 cores 16 threads without hybrid cores so maybe hybrid cores and scheduling is the reason why I cannot reproduce.

I tried to fix this without any testing, so finger crossed, would you like to test on my fix branch and see if the problem is still there?

https://github.com/27rabbitlt/numexpr/tree/fix/race-condition

[Undi123]

I think it is fixed. Thanks for the quick turnaround! Here's what I ran:

Environment

Windows 11, I7 14700K CPU
Fresh conda env, Python 3.11
Installed from your branch: pip install git+https://github.com/27rabbitlt/numexpr.git@fix/race-condition
Version confirmed: 2.14.2.dev0

Tests

1. Original minimal reproducer

import numpy as np
import numexpr as ne

x = np.ones(1000000)
y = np.ones(1000000)

for ii in range(1000):

    rr = ne.evaluate('x * y')
    test = np.all(rr == 0)
    if test:
        print('Gotcha! %d' % ii)

print('Complete!')

completed cleanly.

2. A heavier variant

import numexpr as ne
import numpy as np

print("numexpr version:", ne.__version__)
print("numexpr path:", ne.__file__)

ne.set_num_threads(16) 
a = np.random.randn(9202593).astype(np.float64)
b = np.random.randn(9202593).astype(np.float64)

fail = 0
for i in range(200):
    out = ne.evaluate("a * b + 1.0")
    nonzero = np.count_nonzero(out)
    if nonzero == 0:
        fail += 1
        print(f"iter {i}: all-zero!")
print(f"total fail: {fail}/200")

0 failures.

3. python -c "import numexpr as ne; ne.test()" — all passed.

[27rabbitlt]

I actually don't know why this would solve your problem. After re-thinking the code change, I think it should make no difference at all. I thought there was a bug in module.cpp where multithreading coordination has a minor issue, but now I don't see any problem there.

Would you try: pip install git+https://github.com/pydata/numexpr.git@v2.14.1
and see if you compile and build directly from the 2.14.1 source code what would happen?

[Undi123]

Wait. You mean you didn't do anything? That's weird.

I tried to locate the bug according to suggetion from Claude, maybe it is related to the PyPI Windows wheel .

I ran a controlled comparison to isolate the bug.

Same machine, same Python, same MKL-based numpy — only the numexpr build source differs:

#	numpy BLAS	numexpr	Build source	fail/200
1	MKL	2.14.1	PyPI Windows wheel	1/200 ❌
2	MKL	2.14.1	Source build (same v2.14.1 tag)	0/200 ✅
3	MKL	2.14.2.dev0	Source build (master)	0/200 ✅
4	OpenBLAS	2.14.1	Source build	0/200 ✅

So I guess the only difference is PyPI wheel vs local source build. The intermittent all-zero result only reproduces with the PyPI wheel.

[27rabbitlt]

So I guess the only difference is PyPI wheel vs local source build. The intermittent all-zero result only reproduces with the PyPI wheel.

Yes, exactly.

Will spend some time on this. Also cc @FrancescAlted

[FrancescAlted]

Thanks for your investigation @27rabbitlt. That's interesting, and quite puzzling actually. @Undi123, it would be nice to get more feedback from e.g. conda-forge builds and see how that goes.

[Undi123]

I tried to locate the bug, and I think the root cause is related to tbb 2022.0.0

Diff between the failing base env and a clean working env showed only a handful of relevant packages differed. The key ones:

Package Buggy (base) Fixed
tbb 2022.0.0 2022.3.0
tbb-devel 2022.0.0 2022.3.0
numexpr 2.14.1 build _0 2.14.1 build _1

This is becuase anaconda=2025.12 meta-package pins tbb==2022.0.0 and numexpr build _0 exactly. This means any user installing the newest full Anaconda distribution on Python 3.13 + Windows will land in the buggy combination by default.