th-9a5794: stream_preamble — fast-model preamble to cover reasoning TTFT#234
Merged
Conversation
When SMOOTH_AGENT_PREAMBLE_MODEL is set (e.g. groq-gpt-oss-20b), the streaming runner fires a small fast model IN PARALLEL with each turn and streams ONE short present-tense "what I'm about to do" sentence over a new stream_preamble wire event — an ephemeral status line the real answer replaces. Covers the reasoning model's (gpt-oss-120b) time-to-first-token dead air. Validated live against the llm.smoo.ai gateway: 20b returns a clean sentence in ~466ms, ahead of the 120b's first answer token; on tool-using turns (2-4s) it fills the whole gap. - runner.rs: parallel preamble task, best-effort/fail-silent, guarded by an answer-started flag so a slow preamble never pops in after the reply. - protocol.rs: stream_preamble builder (mirrors stream_reasoning shape). - config.rs: document SMOOTH_AGENT_PREAMBLE_MODEL (unset = off). - spec: stream-preamble.schema.json; TS SDK: StreamPreamble in the union + validate map. - Off unless configured — every existing consumer unchanged. 101 server tests + protocol test pass; SDK typechecks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WEUZgsyfGVab8uqSFPbr23
🦋 Changeset detectedLatest commit: b4af670 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an optional fast-model preamble to streaming turns. When the server is configured with
SMOOTH_AGENT_PREAMBLE_MODEL(e.g.groq-gpt-oss-20b), a small fast model runs in parallel with each turn and streams ONE short present-tense "what I'm about to do" sentence over a newstream_preamblewire event — an ephemeral status line the real answer replaces. It covers the reasoning model's (gpt-oss-120b) time-to-first-token dead air.Why
Perceived operator slowness is latency-to-first-token, not throughput — a reasoning model behind the gateway sits on dead air while it reasons + resolves its first tool call. The gateway exposes no clean harmony preamble channel, so we translate intent via a fast model.
Validated live against
llm.smoo.ai: the 20b returns a clean sentence ("Let me retrieve your recent conversation statistics.") in ~466 ms — ahead of the 120b's first answer token, and on tool-using turns (2–4 s) it fills the whole gap.Changes
runner.rs: parallel preamble task, best-effort/fail-silent, guarded by an answer-started flag so a slow preamble never pops in after the reply began.protocol.rs:stream_preamblebuilder (mirrorsstream_reasoning).config.rs: documentsSMOOTH_AGENT_PREAMBLE_MODEL(unset = off).spec/events/stream-preamble.schema.json; TS SDKStreamPreamblein the union + validate map (regenerated).Safety
Off unless configured — every existing consumer is byte-for-byte unchanged. 101 server tests + a new protocol test pass; SDK typechecks. Uses the same gateway/key as the main model. Skipped when a test LLM provider is injected.
Rollout
Merge → the 🦋 version PR publishes server crate + SDK at 1.22.15 → chat-ws bumps the pin + sets
SMOOTH_AGENT_PREAMBLE_MODEL=groq-gpt-oss-20b→@smooai/chat-widgetrenders thestream_preambleframe (separate PR).🤖 Generated with Claude Code
https://claude.ai/code/session_01WEUZgsyfGVab8uqSFPbr23