Skip to content

knowledge(vllm-omni): 1 change(s) from intake #397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tzhouam merged 1 commit into from
Oct 23, 2026
Merged

knowledge(vllm-omni): 1 change(s) from intake #397

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions knowledge/repos/vllm-omni/components/serving/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,4 @@ sources: []
- [多模态 processor cache 的 replica identity 合同](rules-multimodal-cache.md)。
- [视频 timeline guides 规则](rules-timeline-guides.md)
- [vLLM-Omni 服务：Gradio 示例 PCM 音频转换规则](rules-gradio-pcm.md)
- [vLLM-Omni Serving：Seed-TTS 基准与准确率规则](rules-seed-tts.md)
19 changes: 19 additions & 0 deletions knowledge/repos/vllm-omni/components/serving/rules-seed-tts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
title: "vLLM-Omni Serving：Seed-TTS 基准与准确率规则"
created: 2026-10-23
updated: 2026-10-23
type: rule
tags: [vllm-omni, components, serving]
sources: ["PR #9284"]
---

# vLLM-Omni Serving：Seed-TTS 基准与准确率规则

## VLLM-OMNI-PR9284-SEEDTTS-VERBATIM — Seed-TTS 默认系统提示须令模型逐字朗读：问题/指令是待读文本，WER 门槛收紧至 0.03 并配文本级回归

- 触发：改动 `vllm_omni/benchmarks/data_modules/seed_tts_dataset.py` 中的 `SEED_TTS_DEFAULT_OMNI_SYSTEM_PROMPT`，或 `tests/e2e/accuracy/qwen3_omni/test_qwen3_omni.py` 中 Seed-TTS 门槛常量（`_MAX_SEED_TTS_MEAN_WER` 等）与逐字朗读回归测试，或 `docs/cli/bench/serve.md` 的 "Seed-TTS reference speech as real audio input" 小节。

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Split the rule across its actual owners

For future changes limited to these paths, the owner map assigns vllm_omni/benchmarks/ to the benchmark owner (adapters/vllm_omni/release_baseline.yaml:28-30), while the WER threshold and regression are explicitly Qwen3-Omni-specific; indexing the combined rule only under serving means the relevant benchmark/model review routes will not load this guard. Move the shared dataset/CLI contract to the benchmark rules and the model-specific threshold/test contract to the Qwen-Omni rules, updating both nearest indexes.

AGENTS.md reference: AGENTS.md:L12-L17

Useful? React with 👍 / 👎.

- 强制：默认系统提示保留 Qwen 身份前缀，并明确要求把整条 user 消息当作待朗读文本逐字读出：问题与指令是要说的内容而非要回答或执行的任务，文本响应必须与 user 消息完全一致，不加引言、解释、回答或其他词语，不改写、翻译或续写；保留 `--seed-tts-system-prompt` 覆盖入口。依据 PR #9284：旧提示未区分"读问题"与"答问题"，nightly build #17816 中 1088 条英文 mean WER 0.224804（中位数 0）仍通过旧 0.35 门槛，`I feel sick, what should I do?` 被答成 67 秒建议音频；同 1088 条目标上旧提示 mean WER 0.267329（74 条 WER>1），逐字提示 0.028078（0 条 WER>1），据此 `_MAX_SEED_TTS_MEAN_WER` 收紧为 0.03，数据集目标与全音频 WER 计算保持不变。新增 `test_qwen3_omni_seed_tts_reads_questions_verbatim`：以默认提示直连 chat completions（temperature 0、max_tokens 128、仅 text 模态），断言 `finish_reason == "stop"` 且 `message.content == target`，参数化覆盖 nightly #17816 的两条已知失败目标。文档须说明：发送 `ref_audio`/`ref_text` 字段不等于声音克隆——克隆需要模型与端点支持这些字段，对 Qwen3-Omni chat 不生效；`--seed-tts-wer-save-items` 用于排查内容错误时在结果 JSON 保留每条目标、ASR 转写与 WER（准确率 CI 不启用逐项保存）；WER 的对比对象是数据集目标而非模型生成的回复。

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove one-off incident evidence from the rule

When this page is injected into later reviews, the nightly build number, exact sample counts and measured values, 67-second anecdote, and two historical failures become permanent rule text even though the intake contract classifies one-time counts and “why it happened then” as non-rule material (doc/knowledge/contributing/incidents.md:46). Keep the executable prompt, threshold, and regression requirements, but leave this historical evidence in the source PR so the rule remains concise and reusable.

AGENTS.md reference: AGENTS.md:L12-L17

Useful? React with 👍 / 👎.

- 禁止：把默认提示改回仅描述 TTS/voice-clone 角色而不要求逐字输出、使模型重新回答或执行目标文本的措辞；删除 Qwen 身份前缀或 `--seed-tts-system-prompt` 覆盖；在未换提示、未在同数据集复测的情况下放宽 `_MAX_SEED_TTS_MEAN_WER`；删除或弱化逐字朗读回归（例如只断言 finish_reason 而不比对内容）；在文档中声称向 Qwen3-Omni chat 发送参考字段即可获得声音克隆，或把 WER 描述为与模型回复对比。
- 验收：`SEED_TTS_DEFAULT_OMNI_SYSTEM_PROMPT` 含逐字朗读与不回答/不执行的语义；`test_qwen3_omni_seed_tts_reads_questions_verbatim` 及 `tests/e2e/accuracy/qwen3_omni/` 目标测试通过；`_MAX_SEED_TTS_MEAN_WER == 0.03`；`docs/cli/bench/serve.md` 记录默认提示、覆盖旗标、`--seed-tts-wer-save-items` 与 WER 对比语义。^[PR #9284]

<!-- kb:rule status=active since=v0.30.0 -->
Loading