Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion knowledge/repos/vllm-omni/benchmark/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ sources: [benchmarks/tts/bench_tts.py, benchmarks/tts/model_configs.yaml, "PR #6

| 遇到什么 | 查看哪里 |
|---|---|
| 新增/审查 perf JSON、warmup 覆盖、realtime artifact 或 Hub dataset fallback | [benchmark rules](rules.md) |
| 新增/审查 perf JSON、warmup 覆盖、realtime artifact、Hub dataset fallback、per-request timeout 或 `/v1/videos` 轮询预算 | [benchmark rules](rules.md) |
| 查看旧 benchmark 总览 | [overview](overview.md) |
| 运行 AR graph、Hunyuan 或 MiniCPM-o Omni-DuplexEval benchmark | [benchmark guides](guides/_index.md) |
| 调查 profiling、模型加载和性能验证错误 | [benchmark incidents](incidents/_index.md) |
Expand Down
18 changes: 16 additions & 2 deletions knowledge/repos/vllm-omni/benchmark/rules.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "vLLM-Omni Benchmark 规则"
created: 2026-09-05
updated: 2026-09-05
updated: 2026-09-10
type: rule
tags: [vllm-omni, benchmark]
sources: ["PR #6817", tests/dfx/perf/scripts/run_benchmark.py, tests/benchmarks/test_omniinteract.py]
sources: ["PR #6817", tests/dfx/perf/scripts/run_benchmark.py, tests/benchmarks/test_omniinteract.py, "PR #7130", "PR #7259"]
confidence: high
---

Expand Down Expand Up @@ -46,3 +46,17 @@ confidence: high
- 禁止:让已提交的性能 workload 跟随 Hub 默认分支,或只测试不带 revision 的 fallback。
- 验收:配置中的 dataset 标识含 revision;fallback unit coverage 覆盖 plain 与 revision
两种输入;每个固定 subset 的 case 数、并发与 warmup 设置可由 JSON 直接审计。 ^[PR #6817]

## BENCH-1d — omni bench 共享 session 必须有界 per-request total timeout

- 触发:修改 `vllm bench serve` / `benchmarks/patch` 的 `aiohttp.ClientSession`、`--omni-request-timeout-s` 或请求失败记账。
- 强制:默认 per-request `ClientTimeout.total` 为 900s;显式正值覆盖默认;`<= 0` 才恢复 legacy 6h。超时必须经既有 request-func 记为 `failed`,使挂起的 server 结束 run 并写出结果 JSON,而不是占满 concurrency slot。
- 禁止:把 6h 当默认“无超时”;只改打印文案不改 session timeout;用 warmup 成功推断测量请求不会 hang。
- 验收:默认/显式/`<=0` 三态与 session `timeout.total`;对 accept-then-silence 的假 server 断言 `success is False` 且 run 在 deadline 内返回。^[PR #7130]

## BENCH-1e — `/v1/videos` 轮询预算可配置,失败请求必须计入进度与报告

- 触发:修改 diffusion serving benchmark 的 video job POST/poll/cleanup、`--video-job-timeout` 或失败汇总。
- 强制:轮询 deadline 来自 per-request `video_job_timeout`(默认 900s,可 CLI 加大),从 job 创建后起算含排队;仅当 status 仍非 `completed`/`failed` 且超时才判失败。progress/latency 在 cleanup 路径更新,使失败请求也推进进度条;报告打印失败数与样例错误,JSON 写入全部 `request_errors`。
- 禁止:硬编码 600s 后删除仍可能完成的 job;成功路径才 `pbar.update`;只报告成功计数而隐藏失败原因。
- 验收:覆盖默认/加大 timeout、deadline 时已 `completed` 仍取回、混合成功/失败时进度与 `failed_requests`/`request_errors` 一致。^[PR #7259]
4 changes: 2 additions & 2 deletions knowledge/repos/vllm-omni/ci/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,10 +210,10 @@ sources: [.buildkite/cuda/pipeline.yml, docs/contributing/ci/test_system_overvie
| 遇到什么 | 查看哪里 |
|---|---|
| 审查硬件 lane(含 MiniMax-H3 DLO DP2 ready smoke 的证据边界)、回归 fence、CI 工具供应链、ASR 文本比较或 xdist/shared fixture | [CI rules](rules.md) |
| AMD/ROCm timeout、quarantine、memory signal、Qwen3-TTS argv 或 Qwen3-Omni control-plane fixture | [AMD/ROCm CI rules](rules-amd.md) |
| AMD/ROCm timeout、quarantine、memory signal、single-card diffusion job 的 multi-card marker 排除、Qwen3-TTS argv 或 Qwen3-Omni control-plane fixture | [AMD/ROCm CI rules](rules-amd.md) |
| CUDA L4 Kubernetes preset、GPU-count shard、resource/retry policy | [L4 Kubernetes CI rules](rules-l4-k8s.md) |
| Whisper 转写 helper 的 GPU 首选、16 GiB 门槛、CPU fallback 或 CUDA ready/merge source dependency | [Whisper 转写 CI 规则](rules-whisper-transcription.md) |
| 查看仓库特有 CI 陷阱 | [CI guides](guides/_index.md) |
| 调查历史 CI 失败 | [CI incidents](incidents/_index.md) |

- [CI 并行测试与 engine fixture 合同](rules-test-fixtures.md):`OMNI-CI-2b`, `OMNI-CI-2c`, `OMNI-CI-2d`。
- [CI 并行测试与 engine fixture 合同](rules-test-fixtures.md):`OMNI-CI-2b`, `OMNI-CI-2c`, `OMNI-CI-2d`, `OMNI-CI-2h`。
11 changes: 9 additions & 2 deletions knowledge/repos/vllm-omni/ci/rules-amd.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "AMD/ROCm CI 规则"
created: 2026-09-05
updated: 2026-09-05
updated: 2026-09-10
type: rule
tags: [vllm-omni, ci]
sources: ["PR #6704", "PR #6830", "PR #6884", .buildkite/amd/, tests/helpers/clean.py, tests/helpers/stage_config.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py]
sources: ["PR #6704", "PR #6830", "PR #6884", .buildkite/amd/, tests/helpers/clean.py, tests/helpers/stage_config.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #7234"]
confidence: high
---

Expand Down Expand Up @@ -35,3 +35,10 @@ confidence: high
sampling/cleanup 与 model assertion。PR #6884 必须 parse argv 与 platform overlay,断言 marker/run-level、
CUDA preserve、ROCm-only clear/pin;最终需 exact final-head real MI300 L3 跑完两个 target jobs。该 PR
未提供该 run,现有证据仅 author-reported local/static validation。^[PR #6704] ^[PR #6830] ^[PR #6884]

## OMNI-CI-2g — 单卡 AMD diffusion model job 必须排除全部 multi-card marker

- 触发:修改 `.buildkite/amd` 的 Diffusion Model Test、sequence-parallel/`mi300_2` 任务,或给 diffusion 模型测试加 `cards_N` / ROCm hardware marker。
- 强制:`mi300_1` 一类单卡 model job 的 pytest marker 必须 `not (cards_2 or … or cards_8)`,同时保留无 `cards_1` 的 legacy 单卡用例;真正需要双卡的用例(如 LTX2 Ulysses parity)改到已有双卡 lane,并声明 `rocm` 资源与 `device_count >= world_size` 早失败。
- 禁止:让 `cards_2+` 测试在单卡 worker 上 spawn rank1→GPU1 导致 `invalid device ordinal`;用邻近 green shard 宣称 multi-GPU routing 已修好。
- 验收:pipeline argv/collection 断言单卡 job 排除 multi-card markers、双卡 job 收集目标文件;硬件 marker helper 覆盖 ROCm 声明。^[PR #7234]
11 changes: 9 additions & 2 deletions knowledge/repos/vllm-omni/ci/rules-test-fixtures.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "CI 并行测试与 engine fixture 合同"
created: 2026-08-23
updated: 2026-09-07
updated: 2026-09-10
type: rule
tags: [vllm-omni, ci]
sources: ["PR #3422", "PR #5074", "PR #5255", "PR #5310", "PR #5402", "PR #5524", "PR #5543", "PR #5670", "PR #5713", "PR #5780", "PR #5823", "PR #5836", "PR #5957", "PR #5976", docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, "PR #5845", "PR #5872", "PR #6008", "PR #6048", "PR #6056", "PR #6096", "PR #6102", "PR #6202", "PR #6208", "PR #6273", "PR #6293", "PR #6311", "PR #6339", "PR #6343", "PR #6468", "PR #6523", "PR #6613", "PR #6555", "PR #6650", .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, "PR #6704", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", "PR #6696", vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, "PR #6674", docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, "PR #6818", "PR #6830", "PR #6884", tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #6947"]
sources: ["PR #3422", "PR #5074", "PR #5255", "PR #5310", "PR #5402", "PR #5524", "PR #5543", "PR #5670", "PR #5713", "PR #5780", "PR #5823", "PR #5836", "PR #5957", "PR #5976", docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, "PR #5845", "PR #5872", "PR #6008", "PR #6048", "PR #6056", "PR #6096", "PR #6102", "PR #6202", "PR #6208", "PR #6273", "PR #6293", "PR #6311", "PR #6339", "PR #6343", "PR #6468", "PR #6523", "PR #6613", "PR #6555", "PR #6650", .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, "PR #6704", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", "PR #6696", vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, "PR #6674", docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, "PR #6818", "PR #6830", "PR #6884", tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #6947", "PR #7225"]
confidence: high
---

Expand Down Expand Up @@ -76,3 +76,10 @@ confidence: high
random BF16 weights,只证明 construction、routing 与 control-flow,不证明真实 checkpoint 加载、
输出质量、显存或吞吐。^[PR #5823]


## OMNI-CI-2h — 后台 retry 回归必须等待可观察恢复,而不是固定 sleep

- 触发:测试 orchestrator/duplex reaper 等“一次失败后继续循环”的后台任务,或在慢清理/异常日志路径上用 wall-clock 窗口断言重试次数。
- 强制:由成功重试路径设置 `asyncio.Event`(或等价信号),以有界 `wait_for` 等待恢复;在 `finally` 发 shutdown 并有界 await 任务。对 immediate 与 delayed 首次失败做参数化,使 delayed 超过旧观察窗口仍稳定。
- 禁止:`sleep(35ms)` 后断言 `calls >= 2`;依赖日志或调度速度填满重试窗口;漏关后台 task。
- 验收:immediate/delayed 均在超时前看到第二次成功调用;禁用 retry 时测试应 timeout 而非假绿。^[PR #7225]
6 changes: 3 additions & 3 deletions knowledge/repos/vllm-omni/ci/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ confidence: high

只有 `OMNI-CI-数字字母` 是可审计规则 ID。

相关专题:[CI 并行测试与 engine fixture 合同](rules-test-fixtures.md)(`OMNI-CI-2b`, `OMNI-CI-2c`, `OMNI-CI-2d`)。
相关专题:[CI 并行测试与 engine fixture 合同](rules-test-fixtures.md)(`OMNI-CI-2b`, `OMNI-CI-2c`, `OMNI-CI-2d`, `OMNI-CI-2h`)。

## Direct 代码快速入口

Expand All @@ -23,15 +23,15 @@ confidence: high
| `NIGHTLY`/`WEEKLY`/`NON_CRITICAL`、L1/L4/L5、coverage、scheduled bootstrap | `OMNI-CI-1c` | `upload_pipeline.py` → bootstrap YAML → ready/merge/nightly/weekly YAML → coverage helper |
| Qwen3-TTS Base/CustomVoice ready/merge collection 或 source dependencies | [OMNI-CI-1d](rules-tts.md#omni-ci-1d-qwen3-tts-base-的-dummy-ready-oracle-与-real-weight-merge-coverage-必须分离) | CUDA/AMD ready+merge YAML → Qwen3-TTS e2e markers |
| pre-commit、SPDX、shellcheck、stability marker | `OMNI-CI-2a` | `.pre-commit-config.yaml`、`.buildkite/**`、`tools/**` |
| xdist、共享 worker、helper 拆分/启动回滚、下载 fixture、进程池 | `OMNI-CI-2b` | `tests/conftest.py`、`tests/helpers/{client,clean,runtime,stage_config}.py`、`tests/model_tests/**` |
| xdist、共享 worker、helper 拆分/启动回滚、下载 fixture、进程池、后台 retry 回归 | `OMNI-CI-2b`、`OMNI-CI-2h` | `tests/conftest.py`、`tests/helpers/{client,clean,runtime,stage_config}.py`、`tests/model_tests/**` |
| 重模型 cold start、共享 engine/server fixture、sleep/wake | `OMNI-CI-2c` | `tests/entrypoints/test_omni_sleep_mode.py`、OmniServer fixture scope/lock |
| diffusion tiny builder、model settings、alignment exclusion、重模型 OOM | `OMNI-CI-2d` | `tests/model_tests/diffusion/{diff_model_builders,model_settings,test_alignment}.py` → common offline tests |
| perf baseline、hardware label、DFX result artifact、assert-baseline | `OMNI-CI-3a` | `tests/dfx/conftest.py`、`tests/dfx/perf/scripts/run_benchmark.py`、`run_diffusion_benchmark.py`、`tests/dfx/perf/tests/**` |
| ASR transcript、文本相似度或词汇归一化 | `OMNI-CI-3d` | `tests/helpers/media.py::{preprocess_text, cosine_similarity_text}` → `tests/helpers/tests/test_media.py` |
| Qwen3-TTS perf/CI task↔checkpoint 配对、Base speaker-embedding miss-EOS retry | [OMNI-CI-3g](rules-tts.md)、[OMNI-CI-3f](rules-tts.md) | `tests/dfx/perf/tests/**` TTS JSON → serving validation;`test_qwen3_tts_speaker_embedding` |
| scheduled CUDA mirror secret env、`HF_TOKEN` alias、GPU preset expansion | `OMNI-CI-1f` | `upload_pipeline.py` GPU preset expansion → pod env / step commands |

| AMD/ROCm CI timeout、quarantine、diagnostic hang、memory assertion、Qwen3-TTS argv 或 Qwen3-Omni sleep/abort control plane | [OMNI-CI-2f](rules-amd.md#omni-ci-2f-amd-ci-stabilization-必须保留有界执行与测量信号) | `.buildkite/amd/**` → `tests/buildkite/test_amd_pipeline.py` / `ci/qwen3_omni_moe_colocate_async.yaml` → target test |
| AMD/ROCm CI timeout、quarantine、diagnostic hang、memory assertion、Qwen3-TTS argv 或 Qwen3-Omni sleep/abort control plane | [OMNI-CI-2f](rules-amd.md#omni-ci-2f-amd-ci-stabilization-必须保留有界执行与测量信号)、[OMNI-CI-2g](rules-amd.md) | `.buildkite/amd/**` → `tests/buildkite/test_amd_pipeline.py` / `ci/qwen3_omni_moe_colocate_async.yaml` → target test |

## OMNI-CI-1a — 硬件 lane 必须真实收集并执行目标路径

Expand Down
6 changes: 3 additions & 3 deletions knowledge/repos/vllm-omni/components/configuration/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,13 @@ sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/]
| 遇到什么 | 查看哪里 |
|---|---|
| 理解配置从 deploy、CLI、默认 factory 到 structured/legacy config 的稳定边界 | [配置构造架构](architecture.md) |
| 根据 PR 描述直达 strict schema、pipeline sampling constraints、deploy/topology、composable strategy、global engine args 的 owner 校验或显存配置的规则组与第一批源码 | [配置开发门禁与代码地图](rules.md) |
| 根据 PR 描述直达 strict schema、pipeline sampling constraints、deploy/topology、composable strategy、global engine args 的 owner 校验、CLI-only 负向 alias 或显存配置的规则组与第一批源码 | [配置开发门禁与代码地图](rules.md) |

| 对象存储 URI、HF cache snapshot 或空 `config.json` 的模型名称回退与 pipeline 路由 | [模型引用路由规则](rules-model-reference-routing.md) |
| stage full-payload transport capability、topology-owned projection 或 deploy/CLI override rejection | [stage transport capability](rules-stage-transport.md) |
| diffusion Ulysses SymmMem transport flag 的 deploy/CLI/default-stage projection | [diffusion parallel transport rules](rules-diffusion-parallel-transport.md) |
| diffusion attention shorthand、structured config 与 global override 的优先级和最终规范表示 | [diffusion attention precedence](rules-diffusion-attention-precedence.md) |
| legacy stage engine args 的 copy boundary、connector/default injection 或 repeated build mutation | [legacy engine-args isolation](rules-legacy-engine-args.md) |
| legacy stage engine args 的 copy boundary、connector/default injection、repeated build mutation、global engine args 的 owner 校验或 CLI-only 负向 alias | [legacy engine-args isolation](rules-legacy-engine-args.md) |
| 审计配置来源和多层加工 | [config audit](config-audit-plain-language.md) |
| 设计配置归一化与 parity 验证 | [config normalization parity](config-normalization-parity.md) |
| 查询 `Omni()` 初始化参数 | [Omni init args](omni-init-args.md) |
Expand All @@ -52,4 +52,4 @@ sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/]
| 添加新模型和注册点 | [adding a model](adding-a-model.md) |
| composable strategy 的 axis、并行拓扑、load balancing owner 与 resolve_omni_config 统一解析 | [并行拓扑合同](rules-parallel-topology.md) |

| 冻结 topology、辅助 stage 注入与模型专用 deploy profile | [topology 与部署 profile](rules-topology-profiles.md) |
| 冻结 topology、辅助 stage 注入、模型专用 deploy profile 与 LLM `additional_config` 投影 | [topology 与部署 profile](rules-topology-profiles.md) |
Loading
Loading