test(integration): add GetChangedTargets benchmarks#204
Conversation
Add a `make benchmark` target that measures GetChangedTargets latency against three fixed, checked-in commit pairs of increasing diff size (small/medium/large). This is measurement-only: it has no pass/fail thresholds, is not part of `make test` / `make test-integration`, and is never invoked from CI, so a slow run can never fail the build. - Widen repoRemote/startServer/newClient/getChangedTargets in integration_test.go from *testing.T to testing.TB so the existing integration test harness can be reused from Benchmark* functions. - Add integration/benchmark_test.go with three benchmarks pinned to fixed commit SHAs in this repo's own history. - Add benchmark_test.go to the integration_test BUILD.bazel target via `make gazelle`. - Add the `make benchmark` target, which runs the same bazel go_test binary with -test.run=^$ -test.bench=. so only benchmarks execute. Test Plan: - make gazelle (confirmed BUILD.bazel picked up the new file) - make test-integration (existing tests still pass unchanged) - make benchmark (all three benchmarks run and report ns/op, exits 0)
ns/op is hard to eyeball at these durations (calls take seconds, not nanoseconds). Add a sec/op custom metric via b.ReportMetric alongside the built-in ns/op, computed from b.Elapsed() so it reflects the same timer window (post-ResetTimer).
sywhang
left a comment
There was a problem hiding this comment.
What's the current measured benchmark on this on your local box?
Address review comment: shorter target name consistent with Go convention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
uncached run is the initial request, cached run is the subsequent calls with the same request. |
| @echo "Running benchmarks..." | ||
| @$(BAZEL) test //integration:integration_test \ | ||
| --test_output=all \ | ||
| --test_arg=-test.run=^$$ \ |
There was a problem hiding this comment.
let's run it with higher iteration by default. like 15 or 20.
There was a problem hiding this comment.
I decided changed the test to run getChangedTargets on different commits to test cold cache calls. Since each iteration needs a unique commit pair for a cache miss, b.N is capped at the number of pairs we have. BenchmarkGetChangedTargets_Cold tests for the requests with cache miss (full bazel query) with 15 distinctive commit pairs, while the other BenchmarkGetChangedTargets_Cached tests for the duration of cache hits.
This lets us measure the endpoint latency without the server startup.
There was a problem hiding this comment.
Updated benchtime to run 15x
Address review comment: run more iterations for stable measurements. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Address review comment: run 15 cached iterations for stable measurements. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Run the full benchmark function 15 times (including cold call) instead of repeating only the cached b.N loop. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Each iteration uses a distinct pair of consecutive commits, guaranteeing a cache miss per call. This measures true cold-path performance (git checkout + bazel query + compare) without restarting the server 15 times. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace early-history commits (broken Bazel configs, README-only diffs) with pairs from the stable post-5716262 range that all touch Go source. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Keep both `cover` (from main) and `bench` (from this branch) in .PHONY. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
make benchtarget that measuresGetChangedTargetslatency with two benchmarks:startServerWithLoggerto allow benchmarks to inject a nop logger, suppressing service log noisemake test/make test-integration, and never invoked from CIrepoRemote/startServer/newClient/getChangedTargetsinintegration_test.gofrom*testing.Ttotesting.TBso the existing integration harness can be reused fromBenchmark*functionsTest plan
make gazelle—integration/BUILD.bazelpicked up the newbenchmark_test.gomake test-integration— existing integration tests still pass unchangedmake bench— both benchmarks run 15 iterations and reportns/op, exits 0