docs: speaker focus and speaker memory for voice agents#287
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
petemomo
left a comment
petemomo
left a comment
There was a problem hiding this comment.
Thanks for these — the writing is strong and the terminology framing is careful (clear that Speechmatics provides the STT layer, not the full pipeline). A few things to work through before merge. Line-specific notes are inline; the cross-cutting points are here.
Placement — which product surface do these belong to?
As filed, I'm not sure these belong under Speech to Text → Features. Everything else in speech-to-text/features/ is a mode-agnostic API capability you switch on via the Realtime or Batch API (diarization, translation, speaker identification, audio events). These two read as something different — voice-agent patterns built on the Python Voice SDK, with agent tool sets, system-prompt fragments, and Pipecat/LiveKit recipes. The pages say as much themselves:
- Speaker focus: "a helper provided by the Speechmatics Python voice SDK… The Realtime API does not expose speaker focus directly today, and the voice SDK is Python only."
- Speaker memory: "an application of speaker identification… This page covers the voice-agent pattern."
Two questions to make sure the placement is deliberate:
- Which product surface do these belong to? They're built on the current Realtime streaming pattern + the Python Voice SDK, and they cross-link heavily to the Voice agents section — but they sit under general STT features. They're also distinct from the coming-soon agent STT (Linden 1) interaction pattern, so I'd like to avoid readers conflating the two. What was the thinking behind STT → Features over the Voice agents section?
- Speaker memory vs speaker identification — this overlaps a lot with the existing speaker identification pages (the Features page plus the realtime and batch ones). Should it be a section on/next to that content rather than a fourth standalone speaker-ID page, so readers know which one to use?
Happy to help reshuffle once we've agreed the home — if the pages move, we'll need sidebar entries and redirects.
Content type & duplication
Both pages blend a lot of Explanation (why identifiers work, focus vs focus mode) with the How-to steps. Per the style guide, where Explanation outweighs How-to it should split into its own page. Combined with the speaker-identification overlap above, worth deciding what's concept vs task and where each lives.
Nav-item length
As autogenerated, the sidebar labels would be "Speaker focus for voice agents" / "Speaker memory for voice agents" (5 words) against the 1–3 word nav guideline. If the pages stay, set a short sidebar_label ("Speaker focus" / "Speaker memory"). This ties into the placement decision.
Register / conciseness
The house voice is concise and instructional; these lean conversational in places. A tightening pass would bring them in line. A few examples:
- "Put a voice agent in a room with more than one person and you hit a problem straight away…"
- "Without this rule the model guesses, and it guesses badly."
- Imprecise qualifiers the guide flags: "a good rule of thumb", "matters more than they look", "worse than useless".
| The following pseudocode defines a sensible tool set: | ||
|
|
||
| ```python | ||
| focus_on_speaker(speaker_ids: string[]) |
There was a problem hiding this comment.
Question (may be intentional): this block — and the configure_stt/tool-set blocks around it — is fenced as ```python but is pseudocode (string[], true, none aren't Python). The Voice SDK, Pipecat, and LiveKit blocks lower down are real, runnable Python. Was mixing illustrative and runnable code under the same python tag deliberate? If not, fencing the illustrative ones as text (or making them valid Python) would tell readers what they can paste and run.
|
|
||
| `RETAIN` is the safer default for a chatty agent, because a background speaker can be brought in without a config change. `IGNORE` is blunter. Reach for it when you truly want the engine to stop listening, not just stop responding. | ||
|
|
||
| Keep the agent's own voice out of the loop, otherwise its audio output is transcribed straight back in and it starts talking to itself. Any speaker enrolled with a label wrapped in double underscores (such as `__ASSISTANT__`) is excluded automatically, so you never have to ignore it explicitly. To use this, pass the agent's voice as a [known speaker](/speech-to-text/features/speaker-memory-for-voice-agents) at session start, labeled `__ASSISTANT__`, with a speaker identifier captured from an earlier session or from enrollment. The engine recognizes the agent's own audio and drops it before it reaches the transcript. This is the clean option for complex speaker or microphone setups, where the agent's voice can leak back in through the mic. You can pass several identifiers under the one label to cover that voice across different output paths or capture devices. Adding the agent's speaker to the ignore list at runtime also works. |
There was a problem hiding this comment.
This paragraph runs ~7 sentences; the style guide asks for 1–3 (one idea each) for scannability and retrieval. Consider splitting — e.g. the __ASSISTANT__ auto-exclude mechanic, then the multi-identifier tip, then the runtime ignore-list alternative.
There was a problem hiding this comment.
Done split into three paragraphs along the lines you suggested: the auto-exclude mechanic, enrollment with multi-device identifiers, then the runtime ignore-list alternative.
Introduces a new page, "How speaker identifiers work," to centralize explanations of speaker identifiers, their scope, and persistence. This allows the "Speaker memory for voice agents" page to focus on the voice agent application pattern, reducing redundancy. Also includes minor improvements to the "Speaker focus for voice agents" page, like adding a title and code block line numbers.
Enhances the presentation and clarity of speaker-related documentation pages. Updates `sidebar_label` and `sidebar_position` for improved navigation within the docs. Standardizes "Voice SDK" capitalization and adds line numbers to code examples for better readability. Includes minor wording improvements and adds "awaitable" to the custom words list.
There was a problem hiding this comment.
Thanks for the quick turnaround — the casing fix, the paragraph splits, the tightening pass, the sidebar labels, and extracting the foundational page all land well. On the pseudocode fencing: agreed, keep python — replied in the thread.
Placement — sharper after the refactor
The refactor actually splits the placement question:
- How speaker identifiers work is mode-agnostic (Realtime + Batch), so Features is the right home for it.
- Speaker focus for voice agents and Speaker memory for voice agents still don't fit. The Features group hosts capabilities of the streaming and pre-recorded interaction patterns; these two pages are voice-agent material, specific to the Voice SDK. The Voice SDK is the SDK closest to agent STT, and agent STT doesn't yet exist as an entity in the docs — so this content has no natural home today. The Voice SDK itself sits in the Voice agents section, which isn't ideal either, but it's arguably the closest existing location.
This is an information architecture gap rather than a fault of the pages themselves. Proposal: until agent STT content has a proper home, move the two "…for voice agents" pages into the Voice agents section next to the Voice SDK page they build on, keep How speaker identifiers work in Features, and add the sidebar entries and redirects. Happy to pair on the reshuffle — IA calls sit with the docs platform owners, so this one is ours to sort out with you.
Sidebar grouping (for whatever stays in Features)
The Features sidebar currently renders as … Diarization → Speaker identification → Custom dictionary → Speaker identifiers → Speaker focus → Speaker memory. Two nudges: group the speaker pages (adjust sidebar_position so Custom dictionary isn't in the middle of the cluster), and "Speaker identifiers" sitting near "Speaker identification" is easy to misread — if both remain siblings, consider a more distinct label.
New issues from the latest commits
Flagged inline: the description now renders twice at the top of all three pages, the foundational page's intro paragraph length and a dangling modifier, the "Best practices" heading, and a question about where the identifier-storage/GDPR warning should live.
|
|
||
| # How speaker identifiers work | ||
|
|
||
| Understand how the Speechmatics engine produces speaker identifiers and how they differ from diarization labels. |
There was a problem hiding this comment.
The site theme renders the frontmatter description as a subtitle directly under the H1 — so this line displays twice in a row on the rendered page. The same doubling appears on all three pages: verbatim here, near-verbatim on speaker memory, paraphrased on speaker focus. Cut the body line, or open with a sentence that adds something beyond the description.
|
|
||
| Understand how the Speechmatics engine produces speaker identifiers and how they differ from diarization labels. | ||
|
|
||
| Diarization assigns per-session labels: `S1`, `S2`, `S3`. These labels last only for one session. The person who is `S1` in one session may be `S2` in the next. Speaker identifiers give those speakers durable memory. During a session the engine builds a voice fingerprint for each speaker, and you can save those fingerprints as identifiers. Supplied to a later session, the engine attributes matching voices to a name you chose, such as `Sam` instead of `S1`. |
There was a problem hiding this comment.
Two things here. The paragraph runs 6 sentences against the style guide's 1–3 per paragraph. And "Supplied to a later session, the engine attributes…" is a dangling modifier — the engine isn't what's supplied. Suggest splitting after "…durable memory." and rephrasing the last sentence: "Supply them to a later session and the engine attributes matching voices to a name you chose, such as Sam instead of S1."
|
|
||
| Diarization assigns per-session labels: `S1`, `S2`, `S3`. These labels last only for one session. The person who is `S1` in one session may be `S2` in the next. Speaker identifiers give those speakers durable memory. During a session the engine builds a voice fingerprint for each speaker, and you can save those fingerprints as identifiers. Supplied to a later session, the engine attributes matching voices to a name you chose, such as `Sam` instead of `S1`. | ||
|
|
||
| ## What a speaker identifier is |
There was a problem hiding this comment.
Question on where the storage/biometric warning lives. It stayed on Speaker memory, but this page now owns what an identifier is — including account scoping — and readers can land here from either page. Should the "Speechmatics does not store identifiers / may qualify as biometric data" warning move here, or appear on both?
|
|
||
| # Speaker memory for voice agents | ||
|
|
||
| Persist speaker identifiers so a voice agent recognizes a returning speaker by name across sessions. |
There was a problem hiding this comment.
Near-verbatim repeat of the description rendered just above it ("returning speakers" vs "a returning speaker"). Same fix as on How speaker identifiers work — cut the body line or differentiate it.
| ) | ||
| ``` | ||
|
|
||
| ## Best practices |
There was a problem hiding this comment.
This heading lost its subject in the rename ("Best practices for speaker memory" → "Best practices"). Headings need to make sense without the page context, and the focus page kept "Best practices for speaker focus" — suggest restoring "Best practices for speaker memory".
|
|
||
| # Speaker focus for voice agents | ||
|
|
||
| Speaker focus lets a voice agent decide which speakers to respond to and how to treat everyone else, and update that choice live during a session. |
There was a problem hiding this comment.
Softer version of the doubling flagged on the other two pages: this paraphrases the description shown directly above it, so the rendered page opens by saying the same thing twice in different words. Consider cutting it and letting "A voice agent in a room…" open the body.
3 participants
Adds two feature pages under Speech to Text → Features, covering how to use
speaker focus and speaker identifiers when building a voice agent.
as background, driven live by LLM tool calls. Python voice SDK only for now.
across sessions so the agent recognizes returning speakers by name. This is
the voice-agent application of speaker identification, which is a general
Realtime and Batch capability, not voice-specific.
Both include the agent tool patterns, system prompt guidance, and short
Pipecat and LiveKit examples, and cross-link to the Voice SDK and speaker
identification pages. Also adds back-links from the Voice SDK page and
Pipecatto the cspell dictionary.