Repository navigation
Conversation
Random data mispredicts the range checks in the hot loop. Compute the comparisons first and select the result, with the value clamped so the conversion to int is in range for any input, including NaN. The result is identical to the previous code for all inputs. 1D regular axis, 1M normal doubles, ns/element: clang, variant argument 3.4 -> 1.1; gcc 14 default flags 3.6 -> 2.2. Assisted-by: ClaudeCode:claude-fable-5-1
henryiii
changed the title
Replaces the nested ternary clamp with fmax(fmin(z, 1), 0). This maps to min/max instructions on clang and on gcc with -fno-trapping-math, and halves most of the remaining regressions vs develop. Assisted-by: ClaudeCode:claude-fable-5-1
The 128 kB buffer spills out of L1 on recent CPUs. With 32 kB, bulk fills where every value lands in the same bin run 2x faster on Apple M5 with clang and gcc; other cases are unchanged. Stack usage drops as well. Assisted-by: ClaudeCode:claude-fable-5-1
Axes without underflow or overflow bins use optional_index. The validity checks in linearize and in the storage loop mispredict on data that is partly out of range. Replace the checks with mask arithmetic, and move the valid indices to the front before the storage loop when no weight or sample arrays are passed. 1D regular axis without flow bins, 1M normal doubles, ns/element: clang 4.8 -> 1.2; gcc 14 with -fno-trapping-math 4.9 -> 1.3. Assisted-by: ClaudeCode:claude-fable-5-1
The single-value fill path keeps the branching version. Its checks are predictable there, and the mask version cost 10 to 30 percent on gcc for axes without flow bins. Assisted-by: ClaudeCode:claude-fable-5-1
henryiii
force-pushed
the
perf/branchless-optional-index
branch
from
November 3, 2026 16:26
be4b4f2 to
3d36eec
Compare
henryiii
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build on #452; the first commit is that PR. This fixes the case when you disable underflow or overflow bins; before this PR they are much slower (and is a pretty common case in boost-histogram, as this includes the the "numpy" axis type).
🤖 AI text below 🤖
Axes without underflow or overflow bins use
optional_index, and on data that is partly out of range the validity checks mispredict. Making onlylinearizeselect-based did not help, because clang turns the selects back into branches; the disassembly of the fill loop showed three mispredicting branches per element. This PR:detail/linearize.hpp: theoptional_indexoverload computes validity with non-short-circuit boolean ops and applies an all-ones mask instead of branching. All-ones is alsoinvalid_index, so an invalid input stays invalid.detail/fill_n.hpp: after the index pass, a branch-free compaction moves the valid indices to the front, so the storage loop has no data-dependent branch. This runs only when no weight or sample arrays are passed and the index type isoptional_index.axis/regular.hpp: the "upper edge of the last bin is inclusive" select becomes a subtraction, since clang also turned that select into a branch.Measured with a 1D
regular<..., option::none_t>axis of 100 bins,dense_storage<int64_t>, 1M normally distributed doubles on[-1, 1](32% out of range), ns/element (Apple M5 Pro, aarch64):spanargumentvariant<double, span>argument-fno-trapping-mathThe all-in-range case pays about 0.2 ns/element for the compaction pass. The gcc default-flags row is limited by the trapping-math issue described in #452. Fills with weights or samples keep the branch in the storage loop and only get the
linearizeimprovement; carrying positions through the compaction would extend it to them.