diff --git a/README.md b/README.md index 8f188d7..319847f 100644 --- a/README.md +++ b/README.md @@ -153,27 +153,62 @@ Every run writes a per-song bundle: ``` out// -├─ stems/ vocals.wav drums.wav bass.wav other.wav [guitar.wav piano.wav] -├─ midi/ bass.mid other.mid ... -├─ drums/ kick.wav snare.wav toms.wav other.wav + drums.mid -├─ stretched/ _bpm.wav -└─ manifest.json source/target BPM, beat grid, model versions, per-stem ratios +├─ stems/ vocals.wav drums.wav bass.wav other.wav [guitar.wav piano.wav] +├─ midi// raw.mid cleaned.mid quantized.mid events.json # transform ledger +├─ drums/ kick.wav snare.wav toms.wav other.wav + drums.mid +├─ stretched/ _bpm.wav +└─ manifest.json run outcome + file receipt, tempo evidence, per-stem ratios, versions ``` --- +## Accuracy & evidence + +Every stage records *why* it produced what it did, so a result is auditable — not +a black box. The manifest never lies about what happened: + +- **Honest run outcome** — `manifest.outcome ∈ {success, partial, failed}` from a + required-stage set; the CLI exits non-zero when a required stage errored. Each + run stages into a unique dir and is atomically finalized, and the manifest + carries a **sha256 receipt** of every produced file (collision-free paths). +- **Tempo evidence** — the beat grid reports `source_bpm` (`0.0` == *unknown*, + never a silent 120), `bpm_confidence`, `bpm_candidates` (always incl. half/double), + `meter_confidence`, `tempo_assumed`, and a `fallback_chain` of `EngineAttempt` + records. The stretch chain (rubberband → signalsmith → librosa) records the + same sanitized `attempts[]` — no swallowed fallback reasons. +- **Cymbal MIDI on the default path** — the drum teardown's `other` stem (which + bundles all cymbals) is classified per-onset into closed/open hi-hat, ride, or + crash; the manifest reports `cymbal_classes` + `cymbal_rejected`. +- **Calibrated drum MIDI** — per-part velocity normalization, cross-part bleed + de-dup (`duplicates_removed`), toms split low/mid/high by pitch, and a per-event + record `{time, part, note, raw_strength, normalized_strength, velocity}`. Every + threshold is a validated config knob. +- **Melodic MIDI ledger** — the raw Basic Pitch prediction is preserved + (`raw.mid`, never overwritten); `events.json` records each note's confidence, + drop reason (monophonic cleanup), and quantize delta, and **reconstructs** + `cleaned.mid`/`quantized.mid` from `raw` exactly. + +A synthetic **benchmark corpus** (`benchmarks/`) with a known ground truth gates +these on every push: `python -m benchmarks.run_benchmarks` measures tempo error, +half/double correctness, drum onset F1, cross-part duplicate rate, and velocity +rank correlation against `benchmarks/thresholds.yaml`, and CI fails the build on a +regression. Every report embeds provenance (fixture SHA-256, config hash, +library/model versions). + +--- + ## Build status (roadmap phases) | Phase | Module | Status | |-------|--------|--------| | 0 | env / GPU sanity | ✅ `stemforge doctor` | | 1 | Demucs separation | ✅ working | -| 2 | analysis (beat_this/librosa) + Rubber Band stretch | ✅ working | -| 3 | melodic MIDI (Basic Pitch ONNX) | 🟡 interface complete; drop in `basic_pitch.onnx` | +| 2 | analysis (beat_this/librosa) + Rubber Band stretch | ✅ working — with tempo evidence | +| 3 | melodic MIDI (Basic Pitch ONNX) | 🟡 interface + transform ledger complete; drop in `basic_pitch.onnx` | | 4 | drum teardown (inagoy/drumsep Demucs, main env) | ✅ working — loop/stem → hit stems | -| 5 | drum MIDI (parts-based, GM + velocity) | ✅ working — loop → GM `.mid` (ADTOF path optional) | +| 5 | drum MIDI (parts-based, GM + velocity) | ✅ working — cymbals + calibrated velocity/de-dup | | 6 | orchestrator + StemForge workstation UI | ✅ working | -| 7 | hardening / golden-file tests | 🟡 in progress | +| 7 | hardening / accuracy benchmarks | ✅ server hardening + CI-gated benchmark corpus | ✅ = runs today · 🟡 = interface + logic in place, needs the external model weight/repo wired @@ -188,6 +223,8 @@ stemforge/ ├─ pyproject.toml ├─ configs/default.yaml # every knob, maps to RunConfig ├─ models/ # downloaded weights (git-ignored) +├─ benchmarks/ # synthetic accuracy corpus + metrics + threshold gate +├─ .github/workflows/ # CI: unit tests + the benchmark gate ├─ src/stemforge/ │ ├─ ingest.py analysis.py separate.py separate_uvr.py midi_melodic.py │ ├─ drum_split.py drum_midi.py stretch.py desktop.py diff --git a/models/README.md b/models/README.md index 7eda6f0..39b80f8 100644 --- a/models/README.md +++ b/models/README.md @@ -5,13 +5,19 @@ Weights are **not** committed (git-ignored). Drop them here. | File / dir | Used by | Source | |------------|---------|--------| | `basic_pitch.onnx` | `midi_melodic.py` | Basic Pitch ONNX export (~230 KB) — HuggingFace `AEmotionStudio/basic-pitch-onnx-models` | -| `drumsep_roformer/` | `drum_split.py` | Jarredou/aufr33 MelBand Roformer 6-stem (config + checkpoint) | | `larsnet/` | `drum_split.py` (fallback) | LarsNet 5-stem U-Nets — `polimi-ispl/larsnet` | -| `adtof/` | `drum_midi.py` | ADTOF CRNN checkpoint — from the ADTOF repo | +| `adtof/` | `drum_midi.py` (optional SOTA drum MIDI) | ADTOF CRNN checkpoint — from the ADTOF repo | Auto-downloaded (no action needed): - **Demucs** `htdemucs_ft` / `htdemucs_6s` — cached by the `demucs` package on first run. +- **inagoy/drumsep** (`drums.split` default) — a Demucs checkpoint fetched to + `models/drumsep/modelo_final.th` on first use. It is **sha256-pinned** + (`drums.split.inagoy_sha256`) and refused unless the hash matches — no verified + hash, no unsafe `torch.load`. See the security note in the top-level README. +- **MDX23C DrumSep** (`backend: uvr_drumsep`, optional) — per-hit model run in the + isolated `.venv-uvr` (`stemforge setup-sota`), cached under `models/uvr/`. - **beat_this** — weights fetched on first inference. Point non-default locations at these via `configs/default.yaml` -(`midi.onnx_model_path`, `drums.split.external_cmd`, `drums.midi.external_cmd`). +(`midi.onnx_model_path`, `drums.split.inagoy_url`/`inagoy_sha256`, +`drums.split.external_cmd`, `drums.midi.external_cmd`). diff --git a/tests/README.md b/tests/README.md index 4cc4bcf..58eb581 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,19 +1,39 @@ # Tests Runnable **without a GPU or ML frameworks** — heavy libs are lazily imported and -skipped via `pytest.importorskip`. +skipped via `pytest.importorskip`; the GPU/subprocess paths are mocked. The full +suite is CPU-only and deterministic (seeded RNG, synthetic audio). + +```bash +pip install -e ".[dev,ui]" +pytest # or: python -m pytest +``` + +## Core - `test_config.py` — config load, dotted overrides, unknown-key rejection - `test_io.py` — AudioTensor, WAV round-trip, sha256 receipts, slugify - `test_dsp.py` — GM map, velocity scaling, RMS peak window, open/closed hi-hat - `test_analysis.py` — regression BPM (clean + jittered), BeatGrid -- `test_stretch.py` — identity no-op; librosa length check (skipped if absent) -- `test_pipeline.py` — end-to-end ingest→manifest with stages disabled (no ML deps) +- `test_stretch.py` — identity no-op; librosa length check; match-BPM +- `test_pipeline.py` — end-to-end ingest→manifest with stages disabled +- `test_separation_backends.py` — demucs / uvr / hybrid backend selection +- `test_drum_split.py` — drum teardown (inagoy + uvr), C1 sha256 hash-pin gate +- `test_desktop.py` / `test_cli.py` — launcher + CLI surface -Golden-file tests (Phase 7): drop reference stems/MIDI under `tests/golden/` and -compare module outputs against them once the model weights are wired locally. +## Hardening -```bash -pip install -e ".[dev]" -pytest -``` +- `test_rcpt.py` — trustworthy manifest: receipt collisions, honest run outcome +- `test_h2v2.py` — concurrency: single-worker queue, per-call scratch, locked loader +- `test_h1.py` / `test_webapp.py` — local server hardening + web API + +## Accuracy series + +- `test_drum_midi_cymbals.py` — cymbal MIDI on the default path (A1) +- `test_tempo_evidence.py` — honest tempo unknowns + engine fallback chains (A2) +- `test_midi_evidence.py` — Basic Pitch transform ledger, exact reconstruction (A3) +- `test_drum_calibration.py` — per-part velocity, bleed de-dup, tom split, config (A4) +- `test_benchmarks.py` — the `benchmarks/` corpus, metrics, and threshold gate (A5) + +The synthetic accuracy corpus and its CI threshold gate live in `benchmarks/` +(see `benchmarks/README.md`).