Skip to content

Repository navigation

berd-call: stream the user's words while they are transcribed - #428

Closed
johnmatthewtennant wants to merge 2 commits into
jtennant/quinn/call-learning-loopfrom
jtennant/henry/call-interim-caption
Closed

berd-call: stream the user's words while they are transcribed#428
johnmatthewtennant wants to merge 2 commits into
jtennant/quinn/call-learning-loopfrom
jtennant/henry/call-interim-caption

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Summary

During a Berd Call, what you say shows up only once Apple finalizes it, often a few seconds after you stop talking. This streams Apple's interim words to hosts so they can show them as you speak, or let agents start on them early.

Conventional macOS sessions now emit a user_interim session message with Apple's interim text. Each update replaces the previous one, and empty text clears it. Its token is the token the next final will take, so a reply composed against the interim words is still current once they finalize. It's never stored, never allocates a token, and never interrupts output.

start --stream carries it as an ephemeral row like input_activity: 0\tuser_interim\t{"cursor":12,"text":"you should"}. The row never advances a speech cursor or enters transcript history, and Codex tasks never see it. It clears on the final, mute, input reset and session restart. Berd's own native voice conversations ignore the event for now.

The Buzz plugin already handles these rows (squareup/jtennant-agent-config 6885f90, 6c94934). It shows a caption above the composer and has an experimental setting that wakes agents with the interim words when you pause. A reply to that wake is spoken with the interim cursor, so it plays once the words are final instead of being refused as stale.

Stacked on #426.

Testing

  • cargo test -p berd-call passes on macOS, including new tests for the interim token matching its final, and for interim rows that show, clear, stay hidden while muted, and stay off for silent and Expert–Spokesperson transcripts.
  • just clippy passes.
  • The plugin's 238 tests pass, including the caption and the heads-up wake with its reply cursor.

🤖 Generated with Claude Code

@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/henry/call-interim-caption branch from d4d7e28 to e347e12 Compare October 23, 2026 12:37
Henry and others added 2 commits October 23, 2026 09:38
Henry @claude Zack
feat(berd-call): stream the user's words while they are transcribed
37f5054 
Hosts could only show what the user said once Apple finalized it, often
seconds after they stopped talking. Conventional sessions on macOS now
emit `user_interim` with Apple's interim text, replaced by each update
and cleared when its final arrives, on mute, input reset or restart.

`--stream` carries it as an ephemeral `0\tuser_interim` row whose JSON
gives the text and the cursor the next `user` row will take. A client
can show the words as a caption, or let an agent start on them and keep
its reply current when the final lands. The row never advances a cursor
or enters transcript history, and Codex tasks never see it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Henry <a0d9f90f86da5f023099d22b8e3599af162d9423a422d8f385a12815de879cf2@buzz.block.builderlab.xyz>
Henry @claude Zack
fix(berd): ignore interim words in native voice conversations
6174276 
Berd's native voice handler matches every voice input event, so the new
interim transcript event broke its build. Berd doesn't show interim
words yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Henry <a0d9f90f86da5f023099d22b8e3599af162d9423a422d8f385a12815de879cf2@buzz.block.builderlab.xyz>
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/henry/call-interim-caption branch from e347e12 to 6174276 Compare October 23, 2026 13:39
@johnmatthewtennant johnmatthewtennant mentioned this pull request Oct 23, 2026
Merged
johnmatthewtennant added a commit that referenced this pull request Oct 23, 2026
@johnmatthewtennant Zack
Henry @claude Aaron
Berd Call: clear stuck speech holds, show interim words, playback can…
3217cdc 
…cel fixes (#429)

## Summary

Berd Call kept holding the agent's reply after the user had finished,
and showed the user's words only once Apple finalized them. This
collects the release-safe fixes from the call learning work, without its
capture, replay or tuning infrastructure.

**Stuck speech holds.** The hold that keeps an agent from speaking over
the user is released by Apple's finals, but Apple emits one final for a
sentence that pauses split into several VAD turns. Resolving one turn
per final left the rest outstanding, so in recent calls 55 of 57 holds
ran the full 5 s timeout even though Apple had finished 0 to 4 s after
the user stopped. Expiry also cancelled the recognizer, throwing away a
transcript still on its way, and the hold lifted at the timeout while
Apple was still transcribing, so a reply started and was then cut off by
the late transcript.

- Expiry releases only the hold. Recognition, its queued audio and the
recognizer generation continue, so a late final still reaches the
conversation.
- Each final carries the end of its audio through the Swift bridge and
resolves exactly the ended turns it covers. A final that arrives during
the silence window, before its turn ends, resolves that turn when it
ends. A final for an earlier phrase of a turn still being spoken
resolves nothing.
- The bridge reports Apple's interim results. While Apple's latest
result is interim words, the hold lasts up to 10 s beyond the 5 s
timeout, until a final follows. Noise with no interim words keeps 5 s.

**Interim words.** Conventional macOS sessions emit `user_interim`
(`--stream` carries it as an ephemeral `0\tuser_interim` row) with
Apple's interim text, replaced by each update and cleared on its final,
mute, input reset or restart. It is never stored, never takes a token
and never interrupts output.

**One VAD threshold.** Listening and interruption during agent playback
use the same speech probability, 0.9, instead of separate thresholds.

**Playback and cancel fixes.** A cancel wake could be lost between a
playback worker's flag check and its wait, idle live playback did not
wake on cancellation, and handoff terminal failures lost their order.

### What came from where

| Commit | Source |
| --- | --- |
| wake idle live playback on cancellation | `80b97e4c2` in #426
(cherry-pick) |
| cover a cancel wake racing the flag check (test) | `f6097e517` in #426
(cherry-pick) |
| order cancel wakes after the flag check | `14943c91e` in #426
(cherry-pick) |
| preserve ordered handoff terminal failures | `6b5afdf00` in #426
(cherry-pick) |
| tolerate audio replies after child shutdown (test) | `46fdf603b` in
#426 (cherry-pick) |
| validate partial audio EOF after intentional disconnect (test) |
`71214694e` in #426 (cherry-pick, conflict resolved by dropping the
capture-only replay test) |
| expire speech holds without cancelling recognition | `2eda9918c` in
#426 (ported) |
| release the speech hold when a final covers its turns | `43efb0e00` in
#426 (ported) |
| keep the speech hold while Apple still has words to finalize |
`09ee43fd9`, `e173655ac`, `4d2f03488` in #426 (ported, final combined
behavior) |
| stream the user's words while they are transcribed | `7b28c3ecd` in
#428 (ported) |
| ignore interim words in native voice conversations | `e347e12a6` in
#428 (cherry-pick) |

The ports differ from #426 only where #426 depended on infrastructure
this branch leaves out:

- The pending-utterance timeout is a fixed 5 s constant and the interim
grace a fixed 10 s, not `DetectionSettings` fields, CLI flags or a
protocol bump.
- Capture hooks and the capture-only `heard` and `settled` records are
gone, along with the detection-evidence handling in finals
(`result_is_current` still gates hold accounting).
- A turn's end is the pushed audio minus main's 1.2 s silence flush
window, where #426 subtracted its configurable `release_ms`.

### Left in #426

Capture and replay, VAD debug cues, failure fixtures, speech-profile
evaluation, and the learning and tuning work (dynamic timeouts,
thresholds and release).

### Related issue

Follows #426 and #428. None found otherwise.

### Testing

- Real Apple recognition: the ignored on-device test
`mac_speech::tests::final_end_is_measured_on_the_pushed_audio_timeline`
passes, so a final's end is on the same pushed-audio timeline the VAD
turns use. That is what makes the coverage rule sound.
- Hold release, red/green:
`mac_final_before_its_turn_ends_releases_the_hold_when_it_does` fails
before the coverage fix (the turn stays pending until the 5 s timeout)
and passes after it. Companion tests cover a final resolving every turn
its audio covers, an earlier-phrase final keeping the hold, the hold
waiting for interim words and returning to 5 s once a final follows, and
late finals surviving hold expiry.
- Interim words: tests cover the stream through mute, reset and restart,
and the interim token matching the next final.
- Playback cancellation: the cherry-picked `session_protocol`
regressions run the real `berd-call` child against a scripted audio
host.

Not yet exercised on a live call: the end-to-end hold timing and interim
captions with a real microphone.

---------

Signed-off-by: Zack <04f47b290b155b3b8b782b11501cdb15f1e21e82dfceefd01ff0ed405459d927@buzz.block.builderlab.xyz>
Signed-off-by: Henry <a0d9f90f86da5f023099d22b8e3599af162d9423a422d8f385a12815de879cf2@buzz.block.builderlab.xyz>
Signed-off-by: Aaron <f425b86c3b780d43f8d9cf1802429d04dd985bf92eac44fca9468b5ad3887c5e@buzz.block.builderlab.xyz>
Co-authored-by: Zack <04f47b290b155b3b8b782b11501cdb15f1e21e82dfceefd01ff0ed405459d927@buzz.block.builderlab.xyz>
Co-authored-by: Henry <a0d9f90f86da5f023099d22b8e3599af162d9423a422d8f385a12815de879cf2@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Aaron <f425b86c3b780d43f8d9cf1802429d04dd985bf92eac44fca9468b5ad3887c5e@buzz.block.builderlab.xyz>
johnmatthewtennant pushed a commit that referenced this pull request Oct 23, 2026
Zack @claude
Merge main into the learning loop after 429 and 430
d5af7d3 
Main now carries the speech-hold, interim-word and playback-cancel
fixes from #429, which this branch and #428 already had in their own
form, plus the Swift link-path fix from #430.

Conflicts take this branch's versions, with #428's interim words and
the final_end coverage fix layered on, so the configurable and learned
thresholds stay in place of #429's single fixed VAD threshold. The
swiftbuild link-path change from #430 is kept as is. The merged tree
matches the combined build that passed the full berd-call suite, plus
the two build.rs changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
johnmatthewtennant pushed a commit that referenced this pull request Oct 23, 2026
Henry @claude
Merge main into the learning loop after 429 and 430
47000de 
Main now carries the speech-hold, interim-word and playback-cancel
fixes from #429, which this branch and #428 already had in their own
form, plus the Swift link-path fix from #430.

Conflicts take this branch's versions, with #428's interim words and
the final_end coverage fix layered on, so the configurable and learned
thresholds stay in place of #429's single fixed VAD threshold. The
swiftbuild link-path change from #430 is kept as is. The merged tree
matches the combined build that passed the full berd-call suite, plus
the two build.rs changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Henry <a0d9f90f86da5f023099d22b8e3599af162d9423a422d8f385a12815de879cf2@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@johnmatthewtennant