Skip to content

Export: the audio analysis as JSON, CSV and MIDI - #857

Merged
CaYatur merged 2 commits into
mainfrom
feat/analysis-export
Oct 27, 2026
Merged

CaYatur merged 2 commits into
mainfrom
feat/analysis-export

Conversation

@CaYatur

@CaYatur CaYatur commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Writes what the app measures in a song to files, for video editors, DAWs and spreadsheets. It ships with 3.1.6, as the owner asked.

What it does

  • A new export option, "Write the audio analysis next to it (JSON, CSV, MIDI)", is off by default. Frame sequences and chapter exports write analysis.* inside their folder. It is not available with a seamless loop, because that output starts mid-render.
  • "Audio Analysis Only" writes the same files from an audio file without rendering video. A 3-minute song at 60 fps takes about 9 s.
  • <name> - analysis.json and .csv: one row per video frame (row N = frame N).
    • Every modulation source measured from the audio: bands, custom bands, loudness, timbre, pitch classes, drum detectors.
    • Plus beat, downbeat, pitchHz, chord and key.
    • The JSON also has a summary (BPM, key, beatGridFit), events (beats, onsets, kick, snare and hi-hat with strength), chord and key sections, and a description of every field.
  • <name> - analysis.mid: an SMF type 1 file with four tracks, at the detected tempo.
    • Beats.
    • Kick, snare and hi-hat as GM 36/38/42 on channel 10.
    • Chords as notes on channel 1.
    • BPM and key markers.
  • MCP: analysis: true on sv_start_export, and a new tool, sv_export_analysis (97 tools now).

Why the summary uses the whole file
On a 20 s excerpt, the live tempo engine wandered from 172 to 80 BPM and the live chord changed 136 times. The files use the whole song instead:

  • Tempo comes from the autocorrelation of the loudness rise, with a log-Gaussian prior around 120 BPM.
    • On four released tracks it matched the kick spacing: 126.3, 126.3, 128.55 and 104.85 BPM.
    • An envelope from the eight spectrum bands gave 96 BPM on one of them, because those bands saturate on loud masters.
  • Beat grid: constant tempo, phase fitted to the onsets, downbeats where the kick is strongest. beatGridFit reports how well the grid fits.
  • Chords: average pitch classes per beat, matched to the templates. A one-beat deviation joins its neighbours, and a chord lasts at least two beats. The median chord section on the four tracks went from 0.48 s to about 2 s.
  • Key: from the pitch classes of the whole file.
  • Event times move back by half the analysis window (21 ms), so they line up with the audio. Frame columns stay as frame N saw them.

The video does not change
The recorder has its own SVAudio and Modulator, fed the same frame bytes. getBars advances state on every call, so sharing the render engine would have changed the video. Measured with --bench-export:

  • same frame sha1 with the option off and on;
  • byte-identical JSON, CSV and MIDI across two runs and an analysis-only run of the same file;
  • a cancelled analysis leaves no files.

Tests: npm test 3573/3573. tests/analysis-export.test.js checks known answers on synthetic audio run through the exporter's own FFT and byte scale:

  • a 120 BPM kick train gives 120 ±0.5 BPM, with the grid within one frame of the kicks;
  • a C major triad gives a C chord and the key of C;
  • an eighth-note pulse picks 128.6 BPM over its half and third;
  • the MIDI file parses back: chunk lengths match, times only increase, every note closes, and kick times match the JSON;
  • CSV shape, determinism, the per-beat chord smoothing, MCP validation, and both request paths (direct and queue).

Checked in the running app, isolated profile, Turkish and English:

  • the option and the button render;
  • an analysis-only run shows "Analysing the audio… N%" and "Audio analysis written (JSON, CSV, MIDI) → path";
  • the files are written.

Also in this PR

  • The export window is opened by one shared function, so the helper-window shutdown check still counts one site.
  • README: the feature bullet in EN and TR, with the Excel note that the CSV uses commas and a decimal point.

CaYatur and others added 2 commits October 27, 2026 16:51
A new export option, off by default, writes the audio analysis next to
the video. "Audio Analysis Only" writes it from an audio file without
rendering video. The files:

- JSON and CSV: one row per video frame. Each row has every modulation
  source measured from the audio, the beat grid, the chord of the
  current beat and the key. The JSON also has a summary, events, and
  chord and key sections.
- MIDI: beats, kick, snare and hi-hat on channel 10, and the chords as
  notes, at the detected tempo.

The recorder runs on its own audio engine, so the video does not
change: the frame digests match with the option on and off. Tempo,
beat grid, chords and key come from the whole file:

- tempo from the autocorrelation of the loudness rise, weighted around
  120 BPM;
- a steady grid with its phase fitted to the onsets;
- chords from the average pitch classes of each beat, at least two
  beats long;
- the key from the pitch classes of the whole file.

Event times are corrected for the analysis window.

Also over MCP: `analysis` on sv_start_export, and a new tool,
sv_export_analysis.

Co-authored-by: caganturgut <271973834+caganturgut@users.noreply.github.com>
Co-authored-by: caganturgut <271973834+caganturgut@users.noreply.github.com>
@CaYatur
CaYatur merged commit 6578db0 into main Oct 27, 2026
5 checks passed
@CaYatur
CaYatur deleted the feat/analysis-export branch October 27, 2026 13:53
@cayadraft

cayadraft Bot commented Oct 27, 2026

Copy link
Copy Markdown

@CaYatur

CaYaDraft — pull request

Summary
Export: the audio analysis as JSON, CSV and MIDI

Files
CHANGELOG.md (modified, +23/-0)
@@ -35,6 +35,29 @@ CaYaDev presets and the audio bridge came after the 3.1.6-beta build; the ones a

  • YouTube chapters from the timeline markers [837]. "Copy YouTube Chapters" in the timeline's marker list puts them on the clipboard as "0:00 Intro" lines for a video description. A new export option (off by default) writes the same list to "

README.md (modified, +18/-7)
@@ -17,7 +17,7 @@ layers and 40 GPU effects, an OBS overlay, Spout / Syphon, **project
Downloads
License: MIT
[![Platform](https://img.shields.io/badge/platform-Windows%20%…

README.tr.md (modified, +18/-7)
@@ -18,7 +18,7 @@ destesi**, ve MCP üzerinden yapay zekâ kontrolü.
İndirme
Lisans: MIT
[![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20macOS%20%7C%20Linux-111997.sv…

ROADMAP.md (modified, +2/-1)
@@ -138,7 +138,7 @@ npm test
npm start -- --smoke


-- **3559 unit tests, all passing** on `main`. 703 of those shipped in v3.1.0;
+- **3573 unit tests, all passing** on `main`. 703 of those shipped in v3.1.0;
  105 came with v3.1.1; 163 came with v3.1.2; 157 came with v3.1.3 — 1128 at
  that tag — 469 more with v3.1.4, most of them from the MilkDrop work — 1597 at
  that tag — 1402 more…

src/admin/admin.js (modified, +52/-4)
@@ -49,6 +49,7 @@

  // Video dışa aktarma durumu
  let exportAudioPath = null;
+  let exportKind = 'video'; // süren iş: 'video' ya da yalnız 'analysis'
  let exportAudioName = '';
  let exporting = false;
  let exportPct = 0;
@@ -1764,6 +1765,7 @@
      chapters: !!e.chapters && (!fmt || fmt.kind === 'video'),
      chapterList: !!e.chapterList && (!fmt || fmt.kind === 'video'),…

src/exporter/exporter.js (modified, +56/-0)
@@ -402,6 +402,28 @@
  }

  // Tek kareyi çiz ve birleştirme yüzeyinin RGBA baytlarını döndür.
+  /* Ses analizi kaydı. Kendi ses motoru ve modülatörü var: çizimin
+     kullandığı nesnelere yalnız bakmak bile yetmezdi, çünkü getBars her
+     çağrıda durum ilerletiyor. Ayrı nesneler aynı…

**Note**
The local model is unavailable, so this is only the title and file list.

---
Stopping here. I will not merge, push, or change this pull request until you ask.

`/caya explain` describes the diff. `/caya plan` previews the next edit. `/caya revise` adds a commit.

<!-- cayadraft -->

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant