Fix the two root causes of CodSpeed flakes: substituted baselines and harness-dominated benchmarks - #9011
Fix the two root causes of CodSpeed flakes: substituted baselines and harness-dominated benchmarks#9011connortsui20 wants to merge 2 commits into
Performance Regression: -91.08%
⚠️ Unknown Walltime execution environment detected
Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.
For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.
⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 1880 untouched benchmarks
🆕 1 new benchmark
⏩ 12 skipped benchmarks1
Warning
Please fix the performance issues or acknowledge them on CodSpeed.
Performance Changes
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | binary_search_vortex |
486.7 ns | 25,683.6 ns | -98.11% |
| ❌ | Simulation | binary_search_std |
486.1 ns | 15,222.2 ns | -96.81% |
| ⚡ | WallTime | cuda/bitpacked_u8/unpack/3bw[100M] |
351.5 µs | 299.3 µs | +17.42% |
| 🆕 | Simulation | slice_vortex_buffer |
N/A | 5.6 µs | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/codspeed-flake-rootcause (aa146bf) with develop (b15394a)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩