berd-call: replay VAD gates and compare offline profiles

src-tauri/crates/berd-call/VAD_REPLAY.md

@@ -0,0 +1,21 @@
+# Offline VAD replay
+
+This command evaluates continuous-recognition activity gates from normalized JSON fixtures and measures observed waiting from call capture event logs.
+
+Run `berd-call vad-replay INPUT.json --candidates CANDIDATES.json`. It emits JSONL, one report for the recorded settings followed by each named candidate, without opening a microphone or reading/writing live profiles. Candidate settings must be static. Recorded settings may include dynamic values because each frame records the values actually applied; replay does not simulate recognizer-driven learning.
+
+Input has `schema_version: 1`, `session_id`, `held_out`, `frames`, `labels` and `reply_waiting`. Frames contain `end_ms`, `epoch`, `confidence`, `mode` (listening/interruption), and `settings` using the existing DetectionSettings schema. Each frame represents 16 ms. Epoch changes are explicit live gate resets; missing frame intervals leave state unknown until a full onset/release run determines it again.
+
+Labels contain `start_ms`, `end_ms`, `speech` and `provenance` (human_checked/inferred). Label intervals must be ordered and nonoverlapping. Silence must be labeled explicitly; unlabelled audio is not assumed silent. Reply-waiting intervals contain start_ms/end_ms, also ordered and nonoverlapping. All times share the fixture's audio timeline. Candidate files contain an array of `{name, settings}`; names must be unique.
+
+Scores report human-checked speech missed and false activity durations at 16 ms resolution, silent activity overlapping a waiting reply, onset-delay samples, missed/excluded utterances and fragmentation. Inferred labels are counted but excluded from correctness scores. Full onset samples require uninterrupted known detector coverage over the annotation. Interruption detection samples use the recorded playback mode at speech onset. They measure gate detection, not audible agent stopping. Raw samples allow tail analysis without hiding misses in an average.
+
+These reports do not estimate a changed conversation's playback schedule. In particular, original reply-ready intervals and modes are fixed evidence. Replaying a stronger gate might change later playback in reality. Compare candidates on held-out sessions, inspect missed speech alongside latency, and trial improvements live before promoting them. No candidate is promoted by this command.
+
+Each input file is capped at 128 MiB and a comparison at 64 candidates. Split long recordings into fixtures with preserved context and epoch boundaries. Raw recordings and fixtures stay local unless separately shared.
+
+Run `berd-call vad-replay events CAPTURE_DIR` to analyze a `berd-call-capture/1` directory's `events.jsonl`. It measures the union of outstanding prepare-request intervals, their overlaps with VAD and pending recognition (reported separately and together), and admitted-request delay samples. Pending/not-admitted outcomes end a request interval without a successful admission sample. Gaps and capture limits censor outstanding requests; the interval immediately before a gap is excluded because its events may be missing. Requests left open at file end are reported unresolved. Logs must be ordered by monotonic microseconds; malformed/unknown-version input fails instead of producing a success score.
+
+Request-wait overlap is observational. It includes preparation and other scheduling, so it is not proof of delay caused solely by VAD or recognition. The event analyzer uses the capture clock; normalized frame fixtures use the audio timeline. Converting the per-frame capture trace into candidate fixtures requires explicit sample/clock and epoch mapping. It must never infer timestamps by concatenating across muted/dropped input.
+
+Pending recognition can hold replies after VAD clears. The Apple worker currently uses VAD end to start a five-second no-result deadline, then cancels and recreates recognition. Gate replay cannot prove transcript preservation under changed reset policies. Those trials require original audio, repeated recognition runs against the current-policy baseline, and omission/duplicate/late-result evaluation.

src-tauri/crates/berd-call/src/capture_metrics.rs

@@ -0,0 +1,141 @@
+//! Observed request/recognition timing from capture events, not causal attribution.
+use serde::Serialize;
+use serde_json::Value;
+use std::collections::HashMap;
+
+#[derive(Default, Serialize)]
+pub struct CaptureMetrics {
+    pub request_wait_us: u64,
+    pub request_wait_with_vad_us: u64,
+    pub request_wait_with_recognition_pending_us: u64,
+    pub request_wait_with_both_us: u64,
+    pub request_wait_with_unknown_state_us: u64,
+    pub admitted_request_delays_us: Vec<u64>,
+    pub unresolved_requests: u64,
+    pub requests_censored_by_gap: u64,
+    pub capture_gaps: u64,
+    pub recognizer_events: u64,
+    pub excluded_before_gap_us: u64,
+    pub capture_limited: bool,
+}
+
+pub fn measure(events: &[Value]) -> Result<CaptureMetrics, String> {
+    let mut result = CaptureMetrics::default();
+    let mut pending = HashMap::<u64, u64>::new();
+    let mut speaking = Some(false);
+    let mut recognition = Some(false);
+    let mut previous = None;
+    for event in events {
+        let us = event["us"]
+            .as_u64()
+            .ok_or("capture event needs unsigned us")?;
+        let kind = event["type"].as_str().ok_or("capture event needs type")?;
+        if previous.is_some_and(|previous| us < previous) {
+            return Err("capture events must be ordered by monotonic us".into());
+        }
+        let duration = us - previous.unwrap_or(us);
+        if matches!(kind, "gap" | "limit") {
+            result.excluded_before_gap_us += duration;
+        } else if !pending.is_empty() {
+            result.request_wait_us += duration;
+            match (speaking, recognition) {
+                (Some(true), Some(true)) => result.request_wait_with_both_us += duration,
+                (Some(true), Some(false)) => result.request_wait_with_vad_us += duration,
+                (Some(false), Some(true)) => {
+                    result.request_wait_with_recognition_pending_us += duration
+                }
+                (None, _) | (_, None) => result.request_wait_with_unknown_state_us += duration,
+                _ => {}
+            }
+        }
+        previous = Some(us);
+        match kind {
+            "gap" | "limit" => {
+                result.capture_gaps += 1;
+                result.capture_limited |= kind == "limit";
+                result.requests_censored_by_gap += pending.len() as u64;
+                pending.clear();
+                speaking = None;
+                recognition = None;
+            }
+            "recognizer" => {
+                result.recognizer_events += 1;
+            }
+            "in" | "out" => {
+                let message = &event["message"];
+                let message_type = message["type"]
+                    .as_str()
+                    .ok_or("protocol event needs message.type")?;
+                match (kind, message_type) {
+                    ("in", "prepare_speak") => {
+                        let id = message["id"].as_u64().ok_or("prepare_speak needs id")?;
+                        if pending.insert(id, us).is_some() {
+                            return Err("duplicate outstanding prepare_speak id".into());
+                        }
+                    }
+                    ("out", "admitted" | "not_admitted" | "pending") => {
+                        let id = message["id"].as_u64().ok_or("prepare outcome needs id")?;
+                        if let Some(start) = pending.remove(&id) {
+                            if message_type == "admitted" {
+                                result.admitted_request_delays_us.push(us - start);
+                            }
+                        }
+                    }
+                    ("out", "input_speaking") => {
+                        speaking = Some(
+                            message["active"]
+                                .as_bool()
+                                .ok_or("input_speaking needs active")?,
+                        );
+                    }
+                    ("out", "recognition_pending") => {
+                        recognition = Some(
+                            message["active"]
+                                .as_bool()
+                                .ok_or("recognition_pending needs active")?,
+                        );
+                    }
+                    _ => {}
+                }
+            }
+            _ => {}
+        }
+    }
+    result.unresolved_requests = pending.len() as u64;
+    Ok(result)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use serde_json::json;
+    #[test]
+    fn separates_vad_from_recognition_only_waiting() {
+        let events = vec![
+            json!({"us":0,"type":"out","message":{"type":"input_speaking","active":true}}),
+            json!({"us":0,"type":"out","message":{"type":"recognition_pending","active":true}}),
+            json!({"us":10,"type":"in","message":{"type":"prepare_speak","id":1}}),
+            json!({"us":20,"type":"out","message":{"type":"input_speaking","active":false}}),
+            json!({"us":50,"type":"out","message":{"type":"recognition_pending","active":false}}),
+            json!({"us":60,"type":"out","message":{"type":"admitted","id":1}}),
+        ];
+        let result = measure(&events).unwrap();
+        assert_eq!(result.request_wait_us, 50);
+        assert_eq!(result.request_wait_with_both_us, 10);
+        assert_eq!(result.request_wait_with_recognition_pending_us, 30);
+        assert_eq!(result.admitted_request_delays_us, vec![50]);
+    }
+    #[test]
+    fn gaps_censor_requests_instead_of_scoring_success() {
+        let events = vec![
+            json!({"us":0,"type":"in","message":{"type":"prepare_speak","id":1}}),
+            json!({"us":20,"type":"gap","reason":"capture_dropped"}),
+            json!({"us":30,"type":"out","message":{"type":"admitted","id":1}}),
+        ];
+        let result = measure(&events).unwrap();
+        assert_eq!(result.requests_censored_by_gap, 1);
+        assert_eq!(result.request_wait_us, 0);
+        assert_eq!(result.excluded_before_gap_us, 20);
+        assert!(result.admitted_request_delays_us.is_empty());
+    }
+}

src-tauri/crates/berd-call/src/cli_help.rs

@@ -22,6 +22,7 @@ Speech and models:
   models                  Inspect or install speech models
 
 Diagnostics:
+  vad-replay              Compare offline continuous-recognition VAD fixtures
   benchmark tts           Benchmark a TTS backend
   benchmark stt           Benchmark an STT backend
 
@@ -275,6 +276,7 @@ fn help_for(topic: &[&str]) -> Option<&'static str> {
         | ["models", "openai", "voices"]
         | ["models", "pocket", "status" | "install" | "voices"]
         | ["models", "parakeet", "status" | "install"] => Some(MODELS_HELP),
+        ["vad-replay"] => Some("Usage: berd-call vad-replay INPUT.json [--candidates CANDIDATES.json]\n       berd-call vad-replay events CAPTURE_DIR\n\nReplays continuous-recognition VAD fixtures without changing live profiles.\nWrites one JSON report per recorded/candidate profile. See VAD_REPLAY.md for the fixture schema and scoring limits."),
         ["benchmark"] => Some(BENCHMARK_HELP),
         ["benchmark", "tts"] => Some(BENCHMARK_TTS_HELP),
         ["benchmark", "stt"] => Some(BENCHMARK_STT_HELP),

src-tauri/crates/berd-call/src/lib.rs

@@ -6,6 +6,7 @@ pub const MAC_SPEECH_RECOGNITION_FINISH_TIMEOUT_SECONDS: u64 = 5;
 mod asset_verification;
 mod audio_output;
 pub mod benchmark;
+pub mod capture_metrics;
 pub mod causal_inbox;
 mod configured_tts;
 pub mod detection;
@@ -42,6 +43,7 @@ mod synthesis;
 mod tts;
 #[cfg(target_os = "macos")]
 pub mod update_guard;
+pub mod vad_replay;
 
 pub use audio_output::{wait_until_drained, PcmAudioOutput};
 pub use configured_tts::{

src-tauri/crates/berd-call/src/main.rs

@@ -82,6 +82,7 @@ mod session_audio;
 mod session_framing;
 #[cfg(target_os = "macos")]
 mod system_input_mute;
+mod vad_replay_cli;
 
 use session_audio::{
     AudioHostAck, AudioOutputControlRequest, AudioPipeTransport, RemotePcmAudioOutput,
@@ -550,6 +551,13 @@ fn main() {
         None => {}
     }
     match args.get(1).map(String::as_str) {
+        Some("vad-replay") => {
+            if let Err(error) = vad_replay_cli::run(&args[2..]) {
+                eprintln!("berd-call vad-replay: {error}");
+                std::process::exit(2);
+            }
+        }
+
         #[cfg(target_os = "macos")]
         Some("detection-profile") => {
             let result = (|| -> Result<serde_json::Value, String> {