Berd Call: clear stuck speech holds, show interim words, playback cancel fixes

src-tauri/crates/berd-call/PROTOCOL.md

@@ -301,13 +301,41 @@ The child emits:
 {"type":"live_event","token":u64,"text":string,"origin"?:"user"|"spokesperson"|"handoff"}
 ```
 
+On macOS, `recognition_pending.active` reports the agent speech admission hold, not
+the count of unresolved native recognitions. After local speech ends the hold
+expires after five seconds without a final, or, while Apple's latest result is
+interim words that no final has followed, after up to ten more seconds. Noise with no
+interim words keeps the five seconds. Empty finals settle their audio and resolve
+their turns without emitting a `live_event`. Expiry emits
+`recognition_pending.active:false` without cancelling recognition, clearing audio,
+or changing the recognizer generation, so a late final is still forwarded as a
+`live_event`. New speech reasserts the hold. Native finals carry no utterance
+identity, so each final carries the end of its audio and resolves the ended speech
+turns that audio covers, oldest first, which may include expired ones. A final for
+an earlier phrase of a turn still being spoken resolves nothing, and a final with no
+end resolves the oldest turn. Credit accounting is capped at 1024; at the cap attribution stays ambiguous
+until the recognizer resets, and finals still forward but only the timeout
+releases the hold.
+
 For every final live-side event, the child allocates a strictly increasing token,
 stores it in `SessionCore`, acknowledges the runtime storage receipt, and emits
 `live_event`. Conventional sessions omit `origin`, which means `user`.
 Expert-Spokesperson sessions use `origin` to distinguish user transcripts,
 Spokesperson transcripts, and handoffs. User finals interrupt reserved or playing
 assistant output only after storage and emission. Final text is at most 64 KiB.
 
+Conventional sessions on macOS also emit the user's words while they are still
+being transcribed:
+
+```text
+{"type":"user_interim","token":u64,"text":string}
+```
+
+Each `user_interim` replaces the previous one, and empty text clears it.
+`token` is the token the next user final will take, so a reply composed against
+interim words is still current once they finalize. The message is never stored,
+never allocates a token, and never interrupts output.
+
 Expert-Spokesperson sessions also emit two previews for hosts that show the
 conversation as it happens:
 
@@ -355,7 +383,7 @@ ID and speech ID when it is ready to accept output:
 Before emitting `accepted`, the child installs the admitted speech's leased
 assistant-activity guard. `suppress_input` stops PCM admission so user input
 cannot interrupt the speech; `allow_barge_in` continues PCM admission with the
-assistant-sensitive VAD threshold. `accepted` transfers output authority. The
+same VAD threshold. `accepted` transfers output authority. The
 child then writes Begin and waits for `audio_begin_accepted` before delivering
 PCM. Readiness is bounded to two seconds; expiry emits `speech_failed` with zero output. With `allow_barge_in`, speaking or recognition pending provisionally suspends started output. If both clear without a final, the same speech and player resume. A final, targeted cancellation, host mute/reset while provisionally suspended, pause, or shutdown discards the hold and interrupts exactly once.
 Accepted output owns exactly one assistant-activity guard. The child removes

src-tauri/crates/berd-call/README.md

@@ -334,7 +334,7 @@ updates keep the prior configuration, and private credentials, endpoints, and
 bundle paths never enter the snapshot.
 
 The session also projects a separately revisioned `input_during_tts` snapshot.
-`allow_barge_in` keeps PCM flowing through assistant-sensitive VAD, while
+`allow_barge_in` keeps PCM flowing through the same VAD threshold, while
 `suppress_input` drops PCM at the shared runtime for the admitted utterance.
 Live policy updates apply to the next admission; host mute remains an
 independent reason, so neither state can accidentally clear the other.
@@ -432,7 +432,7 @@ CC BY 4.0, as reflected by the crate's aggregate package-license metadata.
 Input is paced in real time as exact 960-sample frames every 20 ms. Each clip
 has one second of leading silence and 6.5 seconds of trailing silence. The long
 tail deliberately keeps continuous recognizers supplied with capture-like PCM
-through VAD settlement and the runtime's five-second live no-result bound; it
+through VAD settlement and the runtime's five-second pending-utterance timeout; it
 is included in the reported workload. A final transcript is validated and
 stored in its per-utterance result before its storage receipt is acknowledged,
 and the next clip does not begin until authoritative speaking and
@@ -462,3 +462,5 @@ Auto is the default input-during-speech preference. It mutes microphone input du
 ### Input activity in stdout streams
 
 `start --stream` (including `--stream live`) emits ephemeral `0\tinput_activity\t{"speaking":true,"recognizing":false}` rows when input speech or recognition state changes. Both booleans are snapshots; show activity while either is true. These rows have cursor zero, are not utterances, and must never advance a speech cursor, trigger an agent turn, or enter transcript history. They are not delivered to Codex tasks. Clear client activity on session restart, stream end, or failure. Older CLIs omit these rows.
+
+On macOS, conventional sessions with `--stream` also emit ephemeral `0\tuser_interim\t{"cursor":12,"text":"you should"}` rows while your words are still being transcribed. Each row replaces the previous one; empty text clears it, as do mute, input reset and session restart. `cursor` is the cursor the next `user` row will carry, so a client can show the words as a caption or let an agent start on them and keep its reply current when the final arrives. The final `user` row stays authoritative. Like `input_activity`, these rows never advance a speech cursor or enter transcript history, and older CLIs omit them.

src-tauri/crates/berd-call/src/benchmark/stt.rs

@@ -563,6 +563,7 @@ fn observe_event(
                 tracker.speaking_ended = Some(now);
             }
         }
+        Some(VoiceInputEvent::InterimTranscript(_)) => {}
         Some(VoiceInputEvent::RecognitionPendingChanged(active)) => {
             tracker.pending = active;
             if active {

src-tauri/crates/berd-call/src/host_session.rs

@@ -197,6 +197,13 @@ impl Transcript {
     }
 }
 
+fn user_interim_line(cursor: u64, text: &str) -> String {
+    format!(
+        "0\tuser_interim\t{}",
+        json!({"cursor": cursor, "text": text})
+    )
+}
+
 fn input_activity_line(speaking: bool, recognizing: bool) -> String {
     format!(
         "0\tinput_activity\t{}",
@@ -978,6 +985,7 @@ struct SessionActor {
     saved: Option<(std::path::PathBuf, crate::saved_settings::SavedSettings)>,
     input_speaking: bool,
     recognition_pending: bool,
+    interim_shown: bool,
     pending_polls: Vec<PendingInputPoll>,
     writer: Arc<Mutex<ChildStdin>>,
     events: Receiver<SessionMessage>,
@@ -1082,6 +1090,7 @@ impl SessionActor {
             saved: None,
             input_speaking: false,
             recognition_pending: false,
+            interim_shown: false,
             pending_polls: Vec::new(),
             writer,
             events,
@@ -1586,11 +1595,17 @@ impl SessionActor {
             SessionMessage::InputMuteApplied { id, active } => {
                 // Commit before acknowledging so a restart cannot read a stale value.
                 self.muted.store(active, Ordering::SeqCst);
+                if active {
+                    self.stream_user_interim(0, "")?;
+                }
                 self.stream_input_activity()?;
                 if let Some(response) = self.pending_settings.remove(&id) {
                     let _ = response.send(Ok(json!({"muted":active})));
                 }
             }
+            SessionMessage::UserInterim { token, text } if !self.expert_spokesperson => {
+                self.stream_user_interim(token, &text)?
+            }
             SessionMessage::LiveEvent {
                 token,
                 text,
@@ -1730,9 +1745,26 @@ impl SessionActor {
     fn reset_input_activity(&mut self) -> Result<(), String> {
         self.input_speaking = false;
         self.recognition_pending = false;
+        self.stream_user_interim(0, "")?;
         self.stream_input_activity()
     }
 
+    /// Ephemeral like input activity. Muted words are never shown.
+    fn stream_user_interim(&mut self, cursor: u64, text: &str) -> Result<(), String> {
+        let text = if self.muted.load(Ordering::SeqCst) {
+            ""
+        } else {
+            text
+        };
+        if matches!(&*self.transcript, Transcript::Stdout { .. })
+            && (self.interim_shown || !text.is_empty())
+        {
+            self.interim_shown = !text.is_empty();
+            print_flushed(&user_interim_line(cursor, text))?;
+        }
+        Ok(())
+    }
+
     fn stream_live_event(
         &self,
         token: u64,
@@ -4419,6 +4451,87 @@ mod tests {
         )
     }
 
+    #[test]
+    fn user_interim_stream_shows_words_and_clears_on_mute_and_reset() {
+        const CHILD: &str = "BERD_CALL_TEST_INTERIM_CHILD";
+        if std::env::var_os(CHILD).is_none() {
+            let output = Command::new(std::env::current_exe().unwrap())
+                .args(["--exact", "host_session::tests::user_interim_stream_shows_words_and_clears_on_mute_and_reset", "--nocapture"])
+                .env(CHILD, "1")
+                .output()
+                .unwrap();
+            assert!(
+                output.status.success(),
+                "{}",
+                String::from_utf8_lossy(&output.stderr)
+            );
+            let stdout = String::from_utf8(output.stdout).unwrap();
+            let rows: Vec<Value> = stdout
+                .lines()
+                .filter_map(|line| {
+                    line.strip_prefix("0\tuser_interim\t")
+                        .map(|text| serde_json::from_str(text).unwrap())
+                })
+                .collect();
+            let expected: Vec<_> = [
+                (3, "you sh"),
+                (3, "you should\tgo"),
+                (3, ""),
+                (4, "next"),
+                (0, ""),
+                (5, "back"),
+                (0, ""),
+            ]
+            .into_iter()
+            .map(|(cursor, text)| json!({"cursor": cursor, "text": text}))
+            .collect();
+            assert_eq!(rows, expected, "{stdout}");
+            return;
+        }
+        for (transcript, expert) in [
+            (Transcript::Stdout { live: false }, false),
+            (Transcript::Silent, false),
+            (Transcript::Stdout { live: true }, true),
+        ] {
+            let mut child = Command::new("/bin/cat")
+                .stdin(Stdio::piped())
+                .stdout(Stdio::null())
+                .spawn()
+                .unwrap();
+            let writer = Arc::new(Mutex::new(child.stdin.take().unwrap()));
+            let (_event_tx, events) = mpsc::sync_channel(1);
+            let (_command_tx, commands) = mpsc::sync_channel(1);
+            let (audio, _audio_rx) = mpsc::sync_channel(1);
+            let mut actor = test_actor(writer, events, commands, audio);
+            actor.transcript = Arc::new(transcript);
+            actor.expert_spokesperson = expert;
+            actor.muted.store(false, Ordering::SeqCst);
+            let interim = |token, text: &str| SessionMessage::UserInterim {
+                token,
+                text: text.into(),
+            };
+            let mute = |active| SessionMessage::InputMuteApplied { id: 0, active };
+            for event in [
+                interim(3, "you sh"),
+                interim(3, "you should\tgo"),
+                interim(3, ""),
+                interim(4, ""),
+                interim(4, "next"),
+                mute(true),
+                interim(4, "muted words"),
+                mute(false),
+                interim(5, "back"),
+            ] {
+                actor.handle_event(event).unwrap();
+            }
+            actor.reset_input_activity().unwrap();
+            actor.reset_input_activity().unwrap();
+            assert!(actor.transcribed.is_empty());
+            child.kill().unwrap();
+            child.wait().unwrap();
+        }
+    }
+
     #[test]
     fn input_activity_is_cursorless_and_preserves_both_states() {
         for speaking in [false, true] {