Skip to content

fix: validate unsigned AVX2 take indices - #9036

Open
connortsui20 wants to merge 3 commits into
developfrom
ct/avx-signed-take-bug
Open

fix: validate unsigned AVX2 take indices#9036
connortsui20 wants to merge 3 commits into
developfrom
ct/avx-signed-take-bug

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

Fix AVX2 primitive take bounds checks for unsigned indices. AVX2 exposes signed integer comparisons (_mm256_cmpgt_epi32 / _mm256_cmpgt_epi64), not unsigned equivalents, and its gather instructions interpret offsets as signed integers. XOR-bias before comparing, pass the resulting valid mask to the gather, and panic before gathering if a lane is invalid. Without the mask, an out-of-bounds gather is undefined behavior. A length that does not fit in its index type means every representable index is valid.

Why this matters

Example: values has 8 u32 elements

  values
  index:   0      1      2      3      4      5      6      7
         +------+------+------+------+------+------+------+------+
  memory |  v0  |  v1  |  v2  |  v3  |  v4  |  v5  |  v6  |  v7  |
         +------+------+------+------+------+------+------+------+
         ^ base

  Bad input lane: u32::MAX

  bits as unsigned:  0xffff_ffff  = 4,294,967,295
  bits as signed:    0xffff_ffff  = -1

  OLD CODE

      -1 > 8     = false
      !(false)   = true       <-- gather mask enables this lane

      address = base + (signed_index * element_size)
              = base + (-1 * 4)
              = base - 4              <-- outside the slice: undefined behavior

  FIXED CODE

      XOR the sign bit before comparing:

      index:       0xffff_ffff        length:      0x0000_0008
      XOR sign:    0x7fff_ffff        XOR sign:    0x8000_0008

      biased_length > biased_index  = false
      valid gather mask             = 0
      panic before the gather instruction executes

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 added the changelog/fix A bug fix label Jul 28, 2026
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 requested review from a10y and robert3005 July 28, 2026 16:21
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 marked this pull request as ready for review July 28, 2026 16:35
@connortsui20

Copy link
Copy Markdown
Member Author

I wonder if we its better to just always mask it out instead of checking for better perf? do we even allow take indices larger than i32::MAX?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant