th-9a5794: fast-model preamble to cover reasoning-model time-to-first-token

[brentrager]

Problem

The gpt-oss-120b operator brain feels slow — but not from throughput (Groq is sub-second p95). It's latency-to-first-token: a reasoning model behind the llm.smoo.ai gateway sits on dead air while it reasons and resolves its first tool call. The gateway only surfaces reasoning_content + content/tool_calls — there's no clean harmony preamble channel — so the raw thought is unshowable.

Solution

An optional, off-by-default parallel preamble. When AgentConfig.preamble is set, run_with_channel fires a small fast model (e.g. groq-gpt-oss-20b) concurrently with the main model's first turn and streams ONE short present-tense "what I'm about to do" sentence as a new additive AgentEvent::PreambleDelta — an ephemeral status line the real answer replaces.

• Best-effort: any error/slowness is swallowed → it can never block or break the real turn.
• In the core engine so every SmooAI brain (chat-ws, voice, HeyPage, general-agent→Rust) inherits it — no per-service copy, no new TypeScript.
• Additive AgentEvent → older consumers skip the unknown variant (no preamble shown), everything else unchanged.
• Model routing: prod builds a dedicated fast client on preamble.model sharing the main gateway/key; tests route through the injected mock (hermetic).

Scope / what's NOT here yet

This is the reusable core slice only. To actually turn it on end-to-end (follow-ups):

1. smooth-operator protocol/spec — a preamble/status wire event so the widget renders it distinctly (ephemeral, replaced) vs. permanent chat content.
2. @smooai/chat-widget — render that affordance.
3. chat-ws / voice wiring — set AgentConfig.preamble (gated by a flag/config key) + crate version bump + core-v* publish.

Known limitation for the wiring phase: the preamble fires on every turn, so a rare instant (no-reasoning) follow-up could show it briefly before the answer replaces it. A "only if first token hasn't arrived within Nms" gate is a cheap future refinement — deliberately not built yet.

// ponytail: Rust reference impl only; the TS/Go/.NET/Python core ports skip this until a non-Rust consumer needs it.

Tests

• preamble_emits_delta_and_does_not_break_the_turn — asserts the PreambleDelta fires AND the real answer still lands.
• preamble_off_by_default_emits_nothing — no preamble config → zero extra LLM calls, no event.
• Full suite: 546 lib tests pass, clippy clean (2 pre-existing warnings in checkpoint.rs).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WEUZgsyfGVab8uqSFPbr23

[changeset-bot]

⚠️ No Changeset found

Latest commit: 91c7e4a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR