SMOODEV-2534: voice input/output — mic capture + TTS playback over the browser-voice WS (flag OFF)#31
Merged
Merged
Conversation
…e browser-voice WS (flag OFF) Adds a framework-free VoiceSession speaking the frozen browser-voice protocol (start/interrupt/stop JSON + 16 kHz linear16 binary both ways), a composer mic toggle gated by the new voice.enabled config (OFF by default, zero UI when off), RMS barge-in that interrupts + flushes playback, and voice turns rendered through the existing message path so the transcript stays coherent with text chat. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 5a449f8 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: SMOODEV-2534
Adds premium browser voice to the widget (ADR-084), shipped dark behind
voice: { enabled: false, url? }— OFF by default, zero voice UI when off. Live integration is blocked on the server-side browser-voice WS endpoint PR in the monorepo; everything here is built and tested against the frozen protocol with a mock WS.Protocol (frozen)
Endpoint
wss://twilio-voice.smoo.ai/browser-voice/ws(overridable viavoice.url).{"type":"start","agent_id","conversation_id?","token?"}(public agents auth by the browser-sentOriginheader), then binary PCM linear16 mono @ 16 kHz mic chunks,{"type":"interrupt"},{"type":"stop"}.transcript_partial/transcript_final/reply_text/speaking_started/ binary 16 kHz TTS PCM /speaking_done/handoff/error {code}.What's here
src/voice-session.ts— framework-freeVoiceSession: WS lifecycle, mic capture (getUserMedia → AudioWorklet with ScriptProcessor fallback → bucket-average downsample to 16 kHz Int16), gapless playback (PcmPlayerschedules chunks back-to-back through a 16 kHz AudioContext), and events out. Browser audio + WS are thin injectable seams so tests run in jsdom. No new runtime deps.transcript_final→ user bubble,reply_text→ assistant bubble) so history stays coherent with text chat.conversationId(now captured from create/resume) rides the start frame asconversation_id, so voice resumes the same thread.Barge-in
While the agent's TTS is playing, any mic frame whose RMS exceeds a small threshold (default 0.02, post-AGC) sends
{"type":"interrupt"}and flushes the playback queue immediately; the barge-in speech itself keeps streaming so the server transcribes it. Hitting the mic button mid-playback interrupts + flushes the same way before ending the session.Follow-up
The frozen protocol never returns the voice-created conversation id to the client, so a fresh voice-first session can't be adopted for subsequent text turns — needs a server-side protocol addition if we want it.
Verification
pnpm test— 167 passed (12 files), incl. new coverage: start-frame framing, binary passthrough + drop-until-open, barge-in (loud/quiet/manual), downsampler correctness on a synthetic 48 kHz sine (+44.1 kHz ratio, clamping), playback ordering/re-anchor/flush, server-event fan-out, teardown idempotence, and config gating (no mic button unless enabled).pnpm typecheck,pnpm typecheck:e2e,pnpm build— green.🤖 Generated with Claude Code