Skip to content

perf(flow-chat): cut per-flush cost on the streaming render path - #3545

Open
Raing5Days wants to merge 1 commit into
GCWing:mainfrom
Raing5Days:perf/flow-chat-streaming-hot-path
Open

Raing5Days wants to merge 1 commit into
GCWing:mainfrom
Raing5Days:perf/flow-chat-streaming-hot-path

Conversation

@Raing5Days

Copy link
Copy Markdown
Contributor

Summary

Profiling the Web UI streaming render path turned up four places where work ran
unconditionally on every batched flush (every 32ms while text streams). All four are
fixed here as one focused change set on the same hot path.

  • TextChunkModule — newline coalescing re-scanned the whole accumulated response on
    every chunk: (currentContent + text).replace(/\n{3,}/g, '\n\n'). That is O(N) per
    chunk and O(N^2) over a long answer, on the main thread. Both sides of an append are
    already clean, so only the two-character join boundary plus the incoming chunk need
    cleaning. Recovered item content is cleaned once when reused, so the buffer invariant
    still holds.
  • EventBatcher — getBatchedEventsLogPayload() ran before log.trace(), and the
    logger drops the entry after its level check. With the default level (WARN in
    production, DEBUG in dev) every flush allocated a full per-event map for a log line
    that was then thrown away. The payload is now built only when trace output can reach a
    sink; the slow-flush warning keeps its counts.
  • modernFlowChatStore — dropped the immer middleware. The three setters only assign
    whole fields, never a draft, so produce() plus immer's deep freeze walked the entire
    session and virtual-item tree on every flush. Plain set() keeps the same reference
    semantics. clear() now also resets the needsUserAttention projection cache guard
    alongside the other cached fields.
  • useConversationTimeline — the prewarm effect depends on [sources], which changes on
    every flush, so it re-walked the whole timeline including items already queued or
    parsed. A WeakSet of seen items skips them.

Fixes #

Type and Areas

Type: performance / refactor

Areas: web UI (src/web-ui/src/flow_chat)

Motivation / Impact

Streaming a long answer got progressively more expensive per flush, and part of that
cost bought nothing: a redundant full-text rescan, a log payload that was discarded, a
state-tree deep freeze nothing depended on, and a repeated timeline sweep.

User-facing effect: long streaming responses keep a steadier frame budget and the cost
per flush stops growing with the length of the conversation. No behavior, API, or
persisted-shape change.

Verification

Run on this branch, based on origin/main at e17e1803e:

  • pnpm --dir src/web-ui run test:run src/flow_chat/services/flow-chat-manager/TextChunkModule.test.ts src/flow_chat/services/EventBatcher.test.ts src/flow_chat/store/modernFlowChatStore.test.ts → 109 passed
  • pnpm --dir src/web-ui run test:run src/flow_chat/timeline/ → 24 passed across 8 files
  • pnpm --dir src/web-ui exec tsc --noEmit -p tsconfig.json → clean
  • pnpm --dir src/web-ui run lint → 0 errors (14 pre-existing warnings, none in the changed files)

TextChunkModule.test.ts did not assert newline coalescing directly before this change;
behavior equivalence there is covered by the existing streaming tests. A dedicated
"chunk boundary crosses 3+ newlines" case would be a good follow-up.

Reviewer Notes

Two points worth a careful look:

  1. modernFlowChatStore no longer freezes its state. Nothing reads frozen state or hands
    an updater to setState on this store, and useFlowChatActions selects actions through
    useShallow, so it is unaffected. canvasStore keeps its immer middleware.
  2. The newline fix relies on the invariant "the content buffer never holds a 3+ newline
    run". Both buffer writers go through the new helper, and recovered item content is
    cleaned once on reuse.

Remote scenarios: this touches the local Web UI render path only. No changes to remote
workspace transport, remote control, Peer Device Mode command handling, or the detached
dispatch protocol, and none of those paths consume these modules.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, or skipped checks are explained.
  • User-facing strings, docs, and locales are updated where applicable. (No user-facing copy changed, so nothing to update.)

Four fixes found while profiling the Web UI streaming hot path. Each removes
work that ran unconditionally on every batched flush (32ms while text streams).

- TextChunkModule: newline coalescing re-scanned the whole accumulated response
  on every chunk, which is O(N^2) over a long answer. Both sides of an append
  are already clean, so only the two-character join boundary plus the incoming
  chunk need cleaning. Reused item content is cleaned once when recovered so the
  buffer invariant still holds.
- EventBatcher: build the trace payload only when trace output can reach a sink.
  The logger drops the entry after its level check, so every flush previously
  allocated a full event map for a discarded log line. The slow-flush warning
  keeps its counts.
- modernFlowChatStore: drop the immer middleware. The setters only assign whole
  fields, so produce() and its deep freeze walked the entire session and virtual
  item tree on every flush for no benefit. Also reset the needsUserAttention
  projection cache guard in clear().
- useConversationTimeline: remember prewarmed text items in a WeakSet so a
  streaming flush no longer re-walks the whole timeline.

Verification: 133 vitest cases across the touched modules pass, tsc --noEmit is
clean, eslint reports 0 errors. Web UI only; no remote-workspace, remote-control,
peer-device, or dispatch protocol behavior changes.
@Raing5Days
Raing5Days force-pushed the perf/flow-chat-streaming-hot-path branch from b53bbd7 to b4fc0d1 Compare October 24, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant