Fix remaining flaky CodSpeed microbenchmarks#8861
Conversation
`true_count_vortex_buffer[128]` flipped between the same two values (580.6ns <-> 522.2ns, +/-11.17%) on at least nine unrelated PRs since #8742, including java-only (#8811, #8812), Cargo.toml-only (#8820), website-removal-only (#8805), zstd-only (#8843), and geo-only (#8803) changes. A 128-bit true_count is a popcount over two u64 words, so the measurement is fixed dispatch/harness overhead and binary code layout rather than the count itself. Keep only the sizes where the popcount loop dominates, mirroring the bitwise_not_vortex_buffer_mut fix from #8742. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
`copy_nullable[65536]` and `copy_non_nullable[65536]` were on the #8742 watch list and have now been flagged with byte-identical values on two unrelated open PRs (#8838 union scalars, #8724 decimal add/sub kernels; -24.3% / -16.59% each). The copy casts allocate their output buffer in the timed region, so glibc malloc differences across CodSpeed runner images show up as benchmark changes. Install vendored mimalloc as the global allocator, as #8742 did for the other alloc-heavy vortex-array bench files. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
`compress[(100000, 4)]` flipped between the same two values (1.8ms <-> 1.6ms, ~+/-11.9%) on three unrelated PRs since #8742: #8805 (deletes a JS website), #8750, and #8856 (list min/max expressions). runend_encode allocates its output buffers inside the timed region and the runend bench binaries still use glibc malloc, whose code differs across CodSpeed runner images. Install vendored mimalloc as the global allocator, matching the #8742 fix for the flaky vortex-array benches. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: Vortex queries 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.987x ➖, 0↑ 0↓)
datafusion / parquet (0.937x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.980x ➖, 0↑ 0↓)
duckdb / parquet (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Merging this PR will degrade performance by 16.43%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
|
On the 6 "regressions" in the CodSpeed report: they are the expected one-time step change from the allocator swap, not real regressions. All 6 are in #8742 had the same ±10–20% level shifts on benchmarks sharing files with the fixed ones and was resolved by a one-time acknowledgment on the dashboard. After that, every PR compares mimalloc-vs-mimalloc, and these benchmarks measure only Vortex changes — that stability is the point of the swap. Generated by Claude Code |
Rationale for this change
Three benchmarks stayed flaky after #8742, flipping between the same two values on PRs that can't affect them. Same root causes and fixes as #8742:
true_count_vortex_buffer[128]compress[(100000, 4)]cast_decimalcopy_*[65536]Left alone:
compact_sliced[(4096, 90)](single sighting) and the CUDA walltime benches (hosted-runner walltime noise, a runner config issue).The allocator swap shifts every benchmark in the two touched binaries once — see the comment below. Needs a one-time CodSpeed acknowledgment, like #8742.
What changes are included in this PR?
One commit per benchmark; bench files only. Ran
cargo check+clippyon the three bench targets, smoke-ran the binaries,cargo +nightly fmt.