Repo cleanup + documentation refresh for the accuracy series

README.md

@@ -153,27 +153,62 @@ Every run writes a per-song bundle:
 
 ```
 out/<song>/
-├─ stems/       vocals.wav drums.wav bass.wav other.wav [guitar.wav piano.wav]
-├─ midi/        bass.mid other.mid ...
-├─ drums/       kick.wav snare.wav toms.wav other.wav  +  drums.mid
-├─ stretched/   <stem>_<bpm>bpm.wav
-└─ manifest.json   source/target BPM, beat grid, model versions, per-stem ratios
+├─ stems/           vocals.wav drums.wav bass.wav other.wav [guitar.wav piano.wav]
+├─ midi/<stem>/     raw.mid  cleaned.mid  quantized.mid  events.json   # transform ledger
+├─ drums/           kick.wav snare.wav toms.wav other.wav  +  drums.mid
+├─ stretched/       <stem>_<bpm>bpm.wav
+└─ manifest.json    run outcome + file receipt, tempo evidence, per-stem ratios, versions
 ```
 
 ---
 
+## Accuracy & evidence
+
+Every stage records *why* it produced what it did, so a result is auditable — not
+a black box. The manifest never lies about what happened:
+
+- **Honest run outcome** — `manifest.outcome ∈ {success, partial, failed}` from a
+  required-stage set; the CLI exits non-zero when a required stage errored. Each
+  run stages into a unique dir and is atomically finalized, and the manifest
+  carries a **sha256 receipt** of every produced file (collision-free paths).
+- **Tempo evidence** — the beat grid reports `source_bpm` (`0.0` == *unknown*,
+  never a silent 120), `bpm_confidence`, `bpm_candidates` (always incl. half/double),
+  `meter_confidence`, `tempo_assumed`, and a `fallback_chain` of `EngineAttempt`
+  records. The stretch chain (rubberband → signalsmith → librosa) records the
+  same sanitized `attempts[]` — no swallowed fallback reasons.
+- **Cymbal MIDI on the default path** — the drum teardown's `other` stem (which
+  bundles all cymbals) is classified per-onset into closed/open hi-hat, ride, or
+  crash; the manifest reports `cymbal_classes` + `cymbal_rejected`.
+- **Calibrated drum MIDI** — per-part velocity normalization, cross-part bleed
+  de-dup (`duplicates_removed`), toms split low/mid/high by pitch, and a per-event
+  record `{time, part, note, raw_strength, normalized_strength, velocity}`. Every
+  threshold is a validated config knob.
+- **Melodic MIDI ledger** — the raw Basic Pitch prediction is preserved
+  (`raw.mid`, never overwritten); `events.json` records each note's confidence,
+  drop reason (monophonic cleanup), and quantize delta, and **reconstructs**
+  `cleaned.mid`/`quantized.mid` from `raw` exactly.
+
+A synthetic **benchmark corpus** (`benchmarks/`) with a known ground truth gates
+these on every push: `python -m benchmarks.run_benchmarks` measures tempo error,
+half/double correctness, drum onset F1, cross-part duplicate rate, and velocity
+rank correlation against `benchmarks/thresholds.yaml`, and CI fails the build on a
+regression. Every report embeds provenance (fixture SHA-256, config hash,
+library/model versions).
+
+---
+
 ## Build status (roadmap phases)
 
 | Phase | Module | Status |
 |-------|--------|--------|
 | 0 | env / GPU sanity | ✅ `stemforge doctor` |
 | 1 | Demucs separation | ✅ working |
-| 2 | analysis (beat_this/librosa) + Rubber Band stretch | ✅ working |
-| 3 | melodic MIDI (Basic Pitch ONNX) | 🟡 interface complete; drop in `basic_pitch.onnx` |
+| 2 | analysis (beat_this/librosa) + Rubber Band stretch | ✅ working — with tempo evidence |
+| 3 | melodic MIDI (Basic Pitch ONNX) | 🟡 interface + transform ledger complete; drop in `basic_pitch.onnx` |
 | 4 | drum teardown (inagoy/drumsep Demucs, main env) | ✅ working — loop/stem → hit stems |
-| 5 | drum MIDI (parts-based, GM + velocity) | ✅ working — loop → GM `.mid` (ADTOF path optional) |
+| 5 | drum MIDI (parts-based, GM + velocity) | ✅ working — cymbals + calibrated velocity/de-dup |
 | 6 | orchestrator + StemForge workstation UI | ✅ working |
-| 7 | hardening / golden-file tests | 🟡 in progress |
+| 7 | hardening / accuracy benchmarks | ✅ server hardening + CI-gated benchmark corpus |
 
 ✅ = runs today · 🟡 = interface + logic in place, needs the external model weight/repo wired
 
@@ -188,6 +223,8 @@ stemforge/
 ├─ pyproject.toml
 ├─ configs/default.yaml       # every knob, maps to RunConfig
 ├─ models/                    # downloaded weights (git-ignored)
+├─ benchmarks/                # synthetic accuracy corpus + metrics + threshold gate
+├─ .github/workflows/         # CI: unit tests + the benchmark gate
 ├─ src/stemforge/
 │  ├─ ingest.py  analysis.py  separate.py  separate_uvr.py  midi_melodic.py
 │  ├─ drum_split.py  drum_midi.py  stretch.py  desktop.py

models/README.md

@@ -5,13 +5,19 @@ Weights are **not** committed (git-ignored). Drop them here.
 | File / dir | Used by | Source |
 |------------|---------|--------|
 | `basic_pitch.onnx` | `midi_melodic.py` | Basic Pitch ONNX export (~230 KB) — HuggingFace `AEmotionStudio/basic-pitch-onnx-models` |
-| `drumsep_roformer/` | `drum_split.py` | Jarredou/aufr33 MelBand Roformer 6-stem (config + checkpoint) |
 | `larsnet/` | `drum_split.py` (fallback) | LarsNet 5-stem U-Nets — `polimi-ispl/larsnet` |
-| `adtof/` | `drum_midi.py` | ADTOF CRNN checkpoint — from the ADTOF repo |
+| `adtof/` | `drum_midi.py` (optional SOTA drum MIDI) | ADTOF CRNN checkpoint — from the ADTOF repo |
 
 Auto-downloaded (no action needed):
 - **Demucs** `htdemucs_ft` / `htdemucs_6s` — cached by the `demucs` package on first run.
+- **inagoy/drumsep** (`drums.split` default) — a Demucs checkpoint fetched to
+  `models/drumsep/modelo_final.th` on first use. It is **sha256-pinned**
+  (`drums.split.inagoy_sha256`) and refused unless the hash matches — no verified
+  hash, no unsafe `torch.load`. See the security note in the top-level README.
+- **MDX23C DrumSep** (`backend: uvr_drumsep`, optional) — per-hit model run in the
+  isolated `.venv-uvr` (`stemforge setup-sota`), cached under `models/uvr/`.
 - **beat_this** — weights fetched on first inference.
 
 Point non-default locations at these via `configs/default.yaml`
-(`midi.onnx_model_path`, `drums.split.external_cmd`, `drums.midi.external_cmd`).
+(`midi.onnx_model_path`, `drums.split.inagoy_url`/`inagoy_sha256`,
+`drums.split.external_cmd`, `drums.midi.external_cmd`).

tests/README.md

@@ -1,19 +1,39 @@
 # Tests
 
 Runnable **without a GPU or ML frameworks** — heavy libs are lazily imported and
-skipped via `pytest.importorskip`.
+skipped via `pytest.importorskip`; the GPU/subprocess paths are mocked. The full
+suite is CPU-only and deterministic (seeded RNG, synthetic audio).
+
+```bash
+pip install -e ".[dev,ui]"
+pytest                       # or: python -m pytest
+```
+
+## Core
 
 - `test_config.py` — config load, dotted overrides, unknown-key rejection
 - `test_io.py` — AudioTensor, WAV round-trip, sha256 receipts, slugify
 - `test_dsp.py` — GM map, velocity scaling, RMS peak window, open/closed hi-hat
 - `test_analysis.py` — regression BPM (clean + jittered), BeatGrid
-- `test_stretch.py` — identity no-op; librosa length check (skipped if absent)
-- `test_pipeline.py` — end-to-end ingest→manifest with stages disabled (no ML deps)
+- `test_stretch.py` — identity no-op; librosa length check; match-BPM
+- `test_pipeline.py` — end-to-end ingest→manifest with stages disabled
+- `test_separation_backends.py` — demucs / uvr / hybrid backend selection
+- `test_drum_split.py` — drum teardown (inagoy + uvr), C1 sha256 hash-pin gate
+- `test_desktop.py` / `test_cli.py` — launcher + CLI surface
 
-Golden-file tests (Phase 7): drop reference stems/MIDI under `tests/golden/` and
-compare module outputs against them once the model weights are wired locally.
+## Hardening
 
-```bash
-pip install -e ".[dev]"
-pytest
-```
+- `test_rcpt.py` — trustworthy manifest: receipt collisions, honest run outcome
+- `test_h2v2.py` — concurrency: single-worker queue, per-call scratch, locked loader
+- `test_h1.py` / `test_webapp.py` — local server hardening + web API
+
+## Accuracy series
+
+- `test_drum_midi_cymbals.py` — cymbal MIDI on the default path (A1)
+- `test_tempo_evidence.py` — honest tempo unknowns + engine fallback chains (A2)
+- `test_midi_evidence.py` — Basic Pitch transform ledger, exact reconstruction (A3)
+- `test_drum_calibration.py` — per-part velocity, bleed de-dup, tom split, config (A4)
+- `test_benchmarks.py` — the `benchmarks/` corpus, metrics, and threshold gate (A5)
+
+The synthetic accuracy corpus and its CI threshold gate live in `benchmarks/`
+(see `benchmarks/README.md`).