feat: add Force Stream channel setting for stream-only upstreams#6102
Open
xcrong wants to merge 5 commits into
Open
feat: add Force Stream channel setting for stream-only upstreams#6102xcrong wants to merge 5 commits into
xcrong wants to merge 5 commits into
Conversation
Add a per-channel 'Force Stream' toggle that forces stream:true upstream for non-streaming client requests, buffers the SSE response, and returns it as a single non-streaming JSON response. This enables channels backed by upstream providers that only support streaming to serve non-streaming clients transparently. Backend: - New ChannelSettings.ForceStream field (dto/channel_settings.go) - New RelayInfo.ForceStreamBuffer flag (relay/common/relay_info.go) - Force stream upstream in TextHelper when setting is active and pass-through is disabled (relay/compatible_handler.go) - New OaiStreamBufferHandler that accumulates SSE deltas (content, reasoning_content, tool_calls) into an OpenAITextResponse (relay/channel/openai/relay-stream-buffer.go) - Response dispatch routes to buffer handler when ForceStreamBuffer is set (relay/channel/openai/adaptor.go) - Extracted shared helpers markContentFilterReject and marshalTextResponse used by both OpenaiHandler and OaiStreamBufferHandler (relay/channel/openai/helper.go) Frontend: - Channel form schema, defaults, transform, and build logic - Switch toggle in channel advanced settings drawer - i18n translations for all 7 locales (en, zh, zh-TW, fr, ja, ru, vi) Tests: - 14 deterministic table tests covering content/reasoning accumulation, tool-call delta merging, multi-choice responses, usage extraction with cached-token data, error detection, and SSE parsing This code was AI-assisted.
The ForceStream 'stream-to-nonstream' logic only existed in TextHelper (relay path). The channel test path (testChannel) builds and runs the adaptor flow independently, so ForceStream channels were never exercised correctly during manual/automatic tests: a non-streaming probe hit an upstream that only supports streaming and failed, causing auto-ban. Extract the shared logic into ApplyForceStream and invoke it from both TextHelper and testChannel. Only GeneralOpenAIRequest is handled, since OaiStreamBufferHandler only parses OpenAI chat-completion SSE deltas. The buffered handler returns a single JSON with non-nil usage, so the existing downstream helpers (readTestResponseBody, validateTestResponseBody, coerceTestUsage) work unchanged with the original isStream=false. Added table-driven tests for ApplyForceStream covering the active branch and all four suppression conditions. This code was AI-assisted.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdds a configurable ChangesForce-stream configuration and UI
Request-path integration
Buffered OpenAI responses
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant TextHelper
participant ApplyForceStream
participant Upstream
participant OaiStreamBufferHandler
Client->>TextHelper: Send non-streaming GeneralOpenAIRequest
TextHelper->>ApplyForceStream: Evaluate ForceStream and pass-through settings
ApplyForceStream->>TextHelper: Enable streaming and ForceStreamBuffer
TextHelper->>Upstream: Forward streaming request
Upstream->>OaiStreamBufferHandler: Return SSE chunks
OaiStreamBufferHandler->>OaiStreamBufferHandler: Merge chunks and assemble JSON
OaiStreamBufferHandler->>Client: Return non-streaming JSON
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
web/default/src/features/channels/lib/channel-form.ts (1)
214-294: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider validating the
force_stream+pass_through_body_enabledconflict.The description text states Force Stream is "Incompatible with Pass Through Body," but the schema's
superRefinehas no check for this. Enabling both silently no-ops Force Stream on the backend (perApplyForceStream), which could confuse admins.♻️ Suggested validation
.superRefine((data, ctx) => { + if (data.force_stream && data.pass_through_body_enabled) { + addRequiredIssue( + ctx, + 'force_stream', + 'Force Stream is incompatible with Pass Through Body' + ) + } + if ([3, 8, 36, 45].includes(data.type) && !data.base_url?.trim()) {Also applies to: 188-194
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/default/src/features/channels/lib/channel-form.ts` around lines 214 - 294, Add validation in the schema’s superRefine callback to reject configurations where both force_stream and pass_through_body_enabled are enabled, using addRequiredIssue on the relevant field with a clear incompatibility message. Locate the existing channel-specific checks within superRefine and ensure the rule applies consistently to the corresponding fields noted in the comment.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@relay/compatible_handler.go`:
- Around line 25-38: ApplyForceStream must recompute info.ForceStreamBuffer on
every invocation rather than only setting it true: assign it directly from the
full force-stream condition, then set request.Stream to true only when that
condition is satisfied. This clears stale state when RelayInfo is reused for
retries on channels that do not force streaming.
In
`@web/default/src/features/channels/components/drawers/channel-mutate-drawer.tsx`:
- Around line 4095-4119: Only render the force_stream FormField when the current
channel type is OpenAI-compatible, matching the channels handled by
ForceStreamBuffer in the OpenAI adaptor; use the existing channel-type state or
helper near the drawer component to gate the Switch and its label, while
preserving form behavior for supported channels.
In `@web/default/src/i18n/locales/zh-TW.json`:
- Line 1505: Restore the Traditional Chinese translation for the specified
referral-rewards localization key in zh-TW.json, matching the correct existing
translation in zh.json rather than leaving the value identical to the English
key.
---
Nitpick comments:
In `@web/default/src/features/channels/lib/channel-form.ts`:
- Around line 214-294: Add validation in the schema’s superRefine callback to
reject configurations where both force_stream and pass_through_body_enabled are
enabled, using addRequiredIssue on the relevant field with a clear
incompatibility message. Locate the existing channel-specific checks within
superRefine and ensure the rule applies consistently to the corresponding fields
noted in the comment.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7138f991-12e9-4f74-bc6b-f07a96c2e633
📒 Files selected for processing (21)
controller/channel-test.godto/channel_settings.gorelay/channel/openai/adaptor.gorelay/channel/openai/helper.gorelay/channel/openai/relay-openai.gorelay/channel/openai/relay-stream-buffer.gorelay/channel/openai/relay-stream-buffer_test.gorelay/common/relay_info.gorelay/compatible_handler.gorelay/compatible_handler_test.goweb/default/src/features/channels/components/drawers/channel-mutate-drawer.tsxweb/default/src/features/channels/lib/channel-form-errors.tsweb/default/src/features/channels/lib/channel-form.tsweb/default/src/features/channels/types.tsweb/default/src/i18n/locales/en.jsonweb/default/src/i18n/locales/fr.jsonweb/default/src/i18n/locales/ja.jsonweb/default/src/i18n/locales/ru.jsonweb/default/src/i18n/locales/vi.jsonweb/default/src/i18n/locales/zh-TW.jsonweb/default/src/i18n/locales/zh.json
- Clear stale ForceStreamBuffer on retry: ApplyForceStream now reassigns the flag unconditionally, so a RelayInfo reused across a retry onto a non-ForceStream channel no longer keeps buffering on (which would route the upstream's plain JSON into OaiStreamBufferHandler and misparse it as SSE). - Restore the zh-TW translation for the referral-rewards key that was overwritten with the untranslated English source. - Gate the Force Stream toggle to OpenAI/Azure/Custom channel types, matching the adaptor that actually consumes ForceStreamBuffer; avoids a misleading switch on Claude/Gemini channels where it would break the non-streaming response. Adds a regression test for the retry flag-clearing invariant. This code was AI-assisted.
The Force Stream toggle was gated to channel types [1, 3, 8] (OpenAI, Azure, Custom), hiding it for advanced custom channels. The buffering logic runs on the advanced custom native OpenAI-format path as well (ApplyForceStream in TextHelper + ForceStreamBuffer consumed in the OpenAI adaptor default branch), so include CHANNEL_TYPE_ADVANCED_CUSTOM in the visibility gate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 变更描述 / Description
背景：部分上游 provider（以及某些自建网关）只支持流式响应，不接受
stream:false。现有渠道无法把这类上游透明地暴露给发非流式请求的客户端。方案：新增渠道级 Force Stream 开关。开启后，当客户端请求为非流式时，系统会：
stream:true；OaiStreamBufferHandler把上游 SSE 增量（content/reasoning_content/tool_calls）累积成一条完整的OpenAITextResponse；application/json单体响应返回给客户端，客户端无感知。后端：
dto/channel_settings.go：新增ChannelSettings.ForceStreamrelay/common/relay_info.go：新增RelayInfo.ForceStreamBuffer标志relay/compatible_handler.go：抽出公共函数ApplyForceStream，在TextHelper（正常请求路径）中调用relay/channel/openai/relay-stream-buffer.go：新增OaiStreamBufferHandler，累积 SSE delta 为单体响应relay/channel/openai/adaptor.go：DoResponse在ForceStreamBuffer时路由到 buffer handlerrelay/channel/openai/helper.go：抽出markContentFilterReject/marshalTextResponse供两个 handler 复用controller/channel-test.go：渠道测试路径同样调用ApplyForceStream——之前测试路径独立于TextHelper，ForceStream 渠道在手动/自动健康检查时会用非流式探测只支持流式的上游而失败，甚至被自动禁用前端（
web/default）：为什么渠道测试也要改：
testChannel自己跑buildTestRequest → ConvertOpenAIRequest → DoRequest → DoResponse，不经过TextHelper，所以最初遗漏了 ForceStream 逻辑。由于OaiStreamBufferHandler返回的是带非 nil usage 的单体 JSON，渠道测试下游的readTestResponseBody/validateTestResponseBody/coerceTestUsage在原始isStream=false下行为完全正确，无需额外改动。🚀 变更类型 / Type of change
🔗 关联任务 / Related Issue
✅ 提交前检查项 / Checklist
OaiStreamBufferHandler只解析该格式的 SSE delta）；与 pass-through 模式互斥。common.*）、跨数据库兼容、计费安全等规范。📸 运行证明 / Proof of Work
单元测试（
OaiStreamBufferHandler累积逻辑 +ApplyForceStream）：TestApplyForceStream覆盖激活分支（非流式 / stream=nil）与全部四个抑制条件（客户端已流式 / 开关关闭 / 渠道 pass-through / 全局 pass-through），断言ForceStreamBuffer标志与request.Stream的精确状态。OaiStreamBufferHandler的 14 条表驱动测试覆盖 content / reasoning 累积、tool_calls delta 合并、多 choice、usage 提取（含 cached token）、错误检测、SSE 解析。编译与回归：
go build ./...通过；go test ./relay/... ./controller/...全绿。Summary by CodeRabbit
force_stream) channel setting, with UI + i18n support.