Skip to content

perf: add direct binary A/B bench runner#129

Merged
ianm199 merged 1 commit into
mainfrom
perf/rust-ab-runner
Jun 2, 2026
Merged

perf: add direct binary A/B bench runner#129
ianm199 merged 1 commit into
mainfrom
perf/rust-ab-runner

Conversation

@ianm199
Copy link
Copy Markdown
Owner

@ianm199 ianm199 commented Jun 2, 2026

Summary

  • add harness/bench/compare_bins.sh, a direct A/B runner for two arbitrary Lua interpreter binaries
  • document when to use it: local Rust-vs-Rust packet validation without appending dashboard ledger rows

Why

The intern-retain packet needed a direct Rust A/B check between two built artifacts (/tmp/lua-rs-intern-baseline and /tmp/lua-rs-intern-skip). That was done with an ad hoc shell loop. This runner makes that workflow reusable for future small packets where reference-C timing noise is the wrong comparison.

The runner:

  • accepts --a and --b binaries plus labels
  • supports --runs N and comma-separated --workloads
  • checks byte-identical workload output between binaries
  • reports best-of-N wall time and max RSS
  • writes ignored TSV/JSON artifacts under harness/bench/results/
  • intentionally does not append harness/evidence/ledger.jsonl

Validation

  • bash -n harness/bench/compare_bins.sh
  • git diff --check
  • self-compare smoke:
    • bash harness/bench/compare_bins.sh --a target/release/lua-rs --b target/release/lua-rs --label-a same_a --label-b same_b --runs 2 --workloads gc_pressure,binarytrees
    • output artifact: harness/bench/results/20260602T200046Z-bd1b0a3-bin-ab.tsv
    • JSON artifact parsed with python3 -m json.tool harness/bench/results/20260602T200046Z-bd1b0a3-bin-ab.json

Stacked on #128.

@ianm199 ianm199 changed the base branch from perf/intern-retain-fastpath to main June 2, 2026 20:54
@ianm199 ianm199 merged commit 70bff4e into main Jun 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant