feat: add --omit-failed-runs to exclude failures from statistics (fixes #827)#891
feat: add --omit-failed-runs to exclude failures from statistics (fixes #827)#891leno23 wants to merge 1 commit into
Conversation
When used together with --ignore-failure, non-zero benchmark runs are removed from mean/median/stddev and exported timings while reporting how many runs failed. Fixes sharkdp#827 Co-authored-by: Cursor <cursoragent@cursor.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 426887923b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| exit_codes = keep_indices | ||
| .iter() | ||
| .map(|&index| exit_codes[index]) | ||
| .collect(); |
There was a problem hiding this comment.
When --omit-failed-runs is enabled, this block rewrites exit_codes to only successful runs, which drops the non-zero/None statuses from the exported result even though those runs happened. That makes downstream JSON consumers unable to detect failure rate and conflicts with BenchmarkResult's documented meaning of exit_codes as all command invocations; only timing/statistical inputs should be filtered, not the exit-code history.
Useful? React with 👍 / 👎.
1 participant
Summary
--omit-failed-runsto exclude benchmark runs with non-zero exit codes from summary statistics and exported timing data.--ignore-failureso failed runs can complete instead of aborting the benchmark.Motivation
Fixes #827. Useful for flaky commands where failed runs should not skew mean/median/stddev, while still recording that failures occurred.
Example
hyperfine --ignore-failure --omit-failed-runs --runs 20 './flaky-test.sh'If 3 of 20 runs exit non-zero, summary statistics are computed from the remaining 17 runs.
Test plan
cargo test--omit-failed-runsrequires--ignore-failure