PR-A5: accuracy benchmark loop — synthetic corpus + CI threshold gate#22
Merged
Conversation
analysis._bpm_from_beats returned 120.0 on every low-evidence path (<3 beats, non-positive regression slope, non-positive median interval), so an un-estimable tempo was silently reported as 120 instead of unknown. This regression asserts those paths yield 0.0 (unknown); it fails on the current tree (120.0 == 0.0), pinning the silent fallback ahead of the fix. Receipt: analysis.py:96,113,119,123 (verified at main). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
…chain
Stop passing off assumed tempo as measured, and record why each engine fallback
happened instead of swallowing it.
analysis:
- _bpm_from_beats returns 0.0 (UNKNOWN) — never a silent 120 — for <3 beats, a
non-positive slope, or a non-positive median interval.
- BeatGrid gains bpm_confidence, bpm_candidates (always incl. half/double),
meter_confidence, tempo_assumed, and fallback_chain; source_bpm==0.0 means
unknown (no Optional spread). New helpers _bpm_confidence / _bpm_candidates /
_meter_confidence back them.
- The librosa downbeats are a 4/4 assumption, so that path is flagged
tempo_assumed with low meter_confidence.
- analyze() records the beat_this->librosa hop as EngineAttempt evidence.
io_utils: shared EngineAttempt {engine,status,reason} schema + sanitize_reason
(drops absolute paths, collapses subprocess dumps, deterministic).
drum_midi: when the grid tempo is unknown the .mid still needs a tempo to
serialize, so 120 is written but flagged tempo_source=midi_serialization_default
with tempo_assumed=true; a measured tempo is tempo_source=beat_grid.
stretch: the rubberband->signalsmith->librosa chain records a sanitized
EngineAttempt per hop (the reasons the bare excepts dropped); match_bpm_file and
stretch_stems manifests carry attempts[].
Turns the A2 regression green.
Receipt: analysis.py:96,98,113,119,123; drum_midi.py:56; stretch.py hops.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
…m wiring Round out PR-A2 acceptance beyond the silent-120 regression: - known beat sequence still yields ~120 with high confidence; candidates always include half/double; unknown BPM has zero confidence and no candidates. - meter_confidence is high for a clean 4/4 grid, lower for ragged bars. - BeatGrid.to_dict carries the evidence fields. - analyze records the beat_this->librosa fallback chain with a sanitized reason; an un-estimable tempo is source_bpm 0.0 + tempo_assumed; the librosa meter is flagged assumed with low meter_confidence. - EngineAttempt.to_dict + sanitize_reason (path scrub, newline collapse, deterministic). - match_bpm_file manifest carries attempts[]; stretch_with_engine records fell_through hops with sanitized reasons. - drum_midi flags the 120 serialization default vs a measured beat_grid tempo. - source guard: analysis has no silent-120 fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
drum_midi had no cross-part de-duplication, so separation bleed — a weak phantom onset in one part coincident with a loud hit in another — survived as a duplicate coincident note. This regression drops a weak phantom into the toms part on top of each loud kick and asserts no coincident cross-pitch notes survive within 18 ms. It fails on the current tree (3 bleed duplicates survive), pinning the loss. Receipt: drum_midi.py parts-based path (no cross-part de-dup) at main. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
…t, config Rework the parts-based drum transcription around calibrated, configurable stages, and record the per-event evidence. - Per-part velocity normalization against each part's ~98th-percentile strength (was a single global peak), then a kit-level balance gain — a quiet hi-hat and a loud kick both use the full velocity range and ghost notes stay audible. - Cross-part de-dup: cluster onsets within dedupe_window_ms and drop a coincident cross-part hit clearly weaker than the loudest (separation bleed), keeping the most probable. Comparable simultaneous hits (real kick+snare) and same-part ghosts survive. Manifest reports duplicates_removed. - Toms split low/mid/high by dominant pitch (GM 45/47/50); off -> single mid tom. - Hi-hat open/closed decay windows + ratio, and the PR-A1 cymbal thresholds, are moved into DrumMidiCfg (config-ified) with a validate() that fails fast on an out-of-range value; load_config runs it. - Manifest gains a per-event record (time, part, note, raw_strength, normalized_strength, velocity). Turns the bleed regression green. Inherited tests adapted to the new velocity function and the now-config-ified cymbal knobs. Receipt: drum_midi.py velocity/global-peak, no cross-part de-dup, single tom note, hard-coded hat test (verified at main). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
…validation Round out PR-A4 acceptance beyond the bleed regression: - per-part velocity normalization: a quiet part still reaches a strong velocity for its loudest hit, and every hit (incl. a ghost) stays above the audible floor — ghost notes survive. - toms split low/mid/high by dominant pitch; tom_split off collapses to one note. - a real, comparable kick+snare on the same beat is NOT treated as bleed (both survive; duplicates_removed == 0). - manifest carries per-event raw/normalized strength + velocity. - config validation rejects out-of-range values (percentile, weak-ratio, cymbal_ambiguous, tom bounds) directly and via load_config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
…d gate A render-roundtrip benchmark that measures the A1..A4 accuracy work and fails the build on a regression. - benchmarks/corpus.py: deterministic synthetic fixtures with a known ground truth — constant 90/120/140 BPM tempo trains, and a backbeat drum groove with per-part hits, ghost notes, a low-tom fill, and a cross-part bleed variant. - benchmarks/metrics.py: pure metrics — tempo abs error, half/double (octave) correctness, drum onset F1, cross-part duplicate rate, velocity rank correlation. - benchmarks/run_benchmarks.py: runs analysis + drum_midi over the corpus, computes metrics, compares to thresholds.yaml, writes a JSON report, and exits non-zero on any violation. Metrics needing an absent model (melodic F1, etc.) are reported as skipped with a reason — never silently dropped. - Every report embeds provenance: fixture SHA-256, drum-midi config hash, and library/model versions captured at run time. - thresholds.yaml: bounds with margin over a clean baseline (tempo err 2.3, drum F1 1.0, duplicate rate 0.0, velocity corr 0.64). - .github/workflows/benchmarks.yml: unit tests + the benchmark threshold gate on every PR/push, uploading the report artifact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
- pure metric functions (match/F1, cross-pitch duplicate rate, half/double, rank correlation, unknown-tempo full miss). - corpus determinism (stable fixture SHA-256, expected fixture set). - end-to-end harness: all gated metrics present, the shipped thresholds pass on the current tree, provenance (fixture sha256 + config hash + versions) present, and skipped metrics reported with a reason. - threshold enforcement actually gates (an unreachable bound trips a violation). - the CLI returns 0 on the passing baseline and writes the JSON report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
MacFall7
marked this pull request as ready for review
July 10, 2026 03:23
The benchmarks workflow installed only a handful of runtime deps and then ran `pytest -q`, which failed with "pytest: command not found" (exit 127) — pytest (and the webapp test deps fastapi/httpx/uvicorn/python-multipart) were never installed. Install the project with its dev+ui extras instead; torch stays out (heavy libs load lazily and the GPU paths are mocked, so the suite is CPU-only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
…orts Under a bare `pytest` invocation (CI) the repo root is not on sys.path, so `tests/test_benchmarks.py`'s `import benchmarks` failed with ModuleNotFoundError (it worked locally only because `python -m pytest` adds the CWD). Set `pythonpath = ["."]` in the pytest config so the top-level benchmarks package resolves under both invocations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
GitHub Actions sets FORCE_COLOR, so Typer/Rich colorizes `--help` and splits the `--preset` token with escape codes (`-\x1b[0m\x1b[1;36m-preset`), breaking the raw substring check — the test passed locally (no color) but failed in CI. Compare against the ANSI-stripped output instead. Full suite is green under FORCE_COLOR=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fifth bounded PR of the StemForge Accuracy Series. A render-roundtrip benchmark that measures the A1–A4 accuracy work against a known ground truth and fails the build on a regression.
What changed (all new files)
benchmarks/corpus.py— deterministic synthetic fixtures (seeded RNG → stable SHA-256): constant 90/120/140 BPM tempo trains; a backbeat drum groove with per-part ground truth, ghost notes, a low-tom fill, and a cross-part bleed variant.benchmarks/metrics.py— pure metrics: tempo abs error, half/double (octave) correctness, drum onset F1, cross-part duplicate rate, velocity rank correlation.benchmarks/run_benchmarks.py— runsanalysis+drum_midiover the corpus, computes metrics, compares tothresholds.yaml, writes a JSON report, exits non-zero on any violation.benchmarks/thresholds.yaml— bounds with margin over a clean baseline..github/workflows/benchmarks.yml— unit tests + the benchmark threshold gate on every PR/push, uploading the report artifact.Metrics gated
Provenance on every report: fixture SHA-256, drum-midi config hash, library/model versions (captured at run one).
No silent caps — metrics needing an absent model (melodic onset/pitch F1, fragmentation; beat alignment; quantization displacement — covered by A3's ledger) are reported as
skippedwith a reason, never dropped.Verification
pytest -q→ 188 passed, exit 0 (GPU-free; +7 new).python -m benchmarks.run_benchmarks→ BENCHMARK PASSED, exit 0.Stacking
Branched on PR-A4 (
claude/a4-drum-calibration), which is on A2→A1 — the benchmark must run against the improved tempo (A2) and calibrated-drum (A1/A4) code to be meaningful (e.g. the duplicate-rate gate only holds because A4's de-dup is present). It carries no source edits of its own, only newbenchmarks/files + the CI workflow. Until #18/#19/#21 merge, this PR's diff transiently shows A1+A2+A4+A5 combined; it collapses to the A5-only (benchmarks/ + workflow) delta once they land. Merge order: A1 → A2 → A3 → A4 → A5.Draft — not for merge without Mac's gate.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Dn8XfTBWPfKA9poDbkYQU6
Generated by Claude Code