diff --git a/doc/architecture/SPEC/contract.md b/doc/architecture/SPEC/contract.md index 3625bfc94..24abf7c08 100644 --- a/doc/architecture/SPEC/contract.md +++ b/doc/architecture/SPEC/contract.md @@ -1,8 +1,8 @@ # contract.py —— 规范 - + -`LOC ~185 · 旧版跨仓库契约兼容层 · refactor-status: compatibility-shim` +`旧版跨仓库契约兼容层 · refactor-status: compatibility-shim` ## 职责 本 copilot 的**旧版对外消费契约兼容层**。新宿主只允许 import @@ -13,7 +13,7 @@ ## 公开契约（`__all__`） `SDK_API_VERSION` / `DIRECT_API_VERSION` / `STRICT_API_VERSION` / -`QUALITY_API_VERSION` / `KNOWLEDGE_API_VERSION`（当前前四者为 `"1.0.0"`，Knowledge 为 `"1.1.0"`，且都从 `sdk.v1.models` +`QUALITY_API_VERSION` / `KNOWLEDGE_API_VERSION`（SDK、Direct、Quality 为 `"1.0.0"`，Strict 为 `"1.2.0"`，Knowledge 为 `"1.1.0"`，且都从 `sdk.v1.models` 取唯一值）。`capabilities(max_strict_workers=1, supports_file_locking=True) -> dict` 委托 SDK typed handshake 再投影为兼容 dict；它包含 distribution/SDK/Direct/Strict/Quality/Knowledge 版本、resource @@ -22,10 +22,10 @@ revision、supported repositories，以及 `supports_expected_head`、 `supports_idempotent_strict_start`、`supports_knowledge_curation`、 `max_strict_workers`。其余兼容导出包括 `build_review_result(run_dir) -> dict`（结构化评审 结果：`contract_version`、`run_id`、`state`、`reviewed_head_sha`、`verdict`、 -`summary_markdown`、`comments`、`finding_dispositions`、 +`summary_markdown`、`comments`、`finding_dispositions`、`findings`、 `stale`/`expected_head_sha`/`actual_head_sha`、`diagnostics`）；`unknown_run_result(run_id)`（显式 `state: unknown`，绝不抛错 —— 丢响应和还在跑必须可区分）； -`sanitize_comments` 与 `COMMENT_FIELDS`；以及从 `direct_routing` 再导出的 +`sanitize_comments` 与 `COMMENT_FIELDS`，以及 `finding_id` / `build_findings`；另有从 `direct_routing` 再导出的 `direct_knowledge_routes` / `direct_execution_budget` / `direct_completion_result` / `direct_mandatory_review_guides` / 完整的 `direct_review_plan`。 @@ -51,6 +51,14 @@ revision、supported repositories，以及 `supports_expected_head`、 凭据——每个候选一条 `{anchor, disposition, declared}`，按 `DISPOSITION_FIELDS` 白名单投影，被撤下的评论正文本身绝不过界。 消费方因此无需把散文里的取舍和手上的列表对账（#141）。 +- `findings` 按发布顺序对每条最终评论输出 + `{finding_id, severity, anchor, head_sha}`。identity 取文件路径与完整评论文本 + （合并空白并 casefold）的 SHA-256 前 16 个 hex，不包含行号；anchor 是 + `{path, line}`，无文件时为 `None`。severity 使用最终发布值，head 绑定 + `pr_head_sha`。保留重复 identity，不与按 anchor 记账的 disposition 做 join， + 避免同位置的撤下候选污染已发布 finding。评审前失败的 run 返回空列表。 +- 当前没有 carried finding 的显式复查生产链，因此不返回 `carried_from`、 + `head_recheck` 或 `findings_missing_carried`；不能把结果中缺少旧 finding 当作已修复。 - 本模块自身保持仓库中立；仓库专属的 Direct 路由表住在 `direct_routing.py`（见其页）。 - 能力身份的权威实现在 `sdk.v1.get_capabilities`；这里不维护第二份版本或 @@ -70,9 +78,9 @@ revision、supported repositories，以及 `supports_expected_head`、 新的 Direct helper → 在 `direct_routing` 实现、在这里再导出。 ## 测试 -`test_contract.py`（23 例：verdict 字段化、评论白名单、stale 即终局事实、 +`test_contract.py`（覆盖：verdict 字段化、评论白名单、stale 即终局事实、 早死 run 的降级结果、unknown-run 显式化、能力上报、import 方向、 -本模块仓库中立）。 +本模块仓库中立、finding identity/最终 severity/head 绑定、重复保留与 disposition 隔离）。 ## 重构备注 新模块（PR2 mixed-mode contract 拆分）。保持它薄：任何"顺手在这里实现" diff --git a/doc/architecture/SPEC/sdk.md b/doc/architecture/SPEC/sdk.md index be4bcc673..4d309f786 100644 --- a/doc/architecture/SPEC/sdk.md +++ b/doc/architecture/SPEC/sdk.md @@ -1,6 +1,6 @@ # sdk/ —— 规范 - + `Python SDK v1 · 跨仓库唯一 typed 边界 · refactor-status: ok` @@ -51,6 +51,11 @@ - **Strict created 语义真实**：SDK 从 core 的 `(run_id, created)` 读取；命中 idempotency key 的重试返回 `created=False` 且绝不再次入队。review depth 只经 policy allowlisted `params` 传入。 +- **Strict findings 绑定发布结果**：`get_result` 的 `findings` 对最终发布集合逐条返回 + `finding_id`、最终 `severity`、`anchor` 与 `head_sha`，保持顺序及重复 identity。 + identity 以文件路径与完整归一化文本生成，行号变动不改变 identity；它不与 + `finding_dispositions` 按 anchor 拼接。当前没有 carried-finding recheck，缺失旧 + finding 不代表已修复；不得承诺 `head_recheck` 或 `findings_missing_carried` 字段。 - **知识规则由 provider 唯一定义**：catalog 只暴露当前仓库 owner 与 general 的 owner rule page document ID —— 每个 owner 的 `rules.md` 入口页及其 `type: rule` 的 `rules-.md` 专题页（`catalog_entries()` 附每页 @@ -76,8 +81,8 @@ push、开 PR 或 schedule。ReviewBot 必须向 `KnowledgeCurator` 传 dedicated work checkout，并继续拥有重试、artifact 与 fork publication；SDK 也绝不写 packaged knowledge tree。 -- SDK、Direct、Quality API 版本常量为 `1.0.0`；Strict 为 `1.1.0` - （结果新增 `finding_dispositions`，#141），Knowledge 为 `1.1.0` +- SDK、Direct、Quality API 版本常量为 `1.0.0`；Strict 为 `1.2.0` + （1.1 新增 `finding_dispositions`；1.2 新增 `findings`），Knowledge 为 `1.1.0` （1.1 向后兼容地新增 `catalog_entries()` / `KnowledgeCatalogEntry` 与容量预检）， distribution 为 `0.2.0`；`Capabilities.knowledge_api_version` 与 `supports_knowledge_curation` 组成 ReviewBot 的 paired-release 握手，避免只按 diff --git a/knowledge/general/bug/guides/di-calculation.md b/knowledge/general/bug/guides/di-calculation.md index 79acaf5f3..7e40e324a 100644 --- a/knowledge/general/bug/guides/di-calculation.md +++ b/knowledge/general/bug/guides/di-calculation.md @@ -1,7 +1,7 @@ --- title: "DI 计算说明" created: 2026-08-07 -updated: 2026-08-07 +updated: 2026-09-22 type: guide tags: [general, bug] sources: [] @@ -17,9 +17,9 @@ sources: [] | Bug 优先级分类 | 划分标准 | DI 值 | SLO | |---|---|---|---| | critical | 高优先级 | 10 | 1 天 | -| high priority | CI Failure | 3 | 3 天 | -| medium priority | 中优先级 | 1 | 5 天 | -| low priority | 低优先级 | 0.1 | 10 天 | +| high priority | CI Failure | 3 | 5 天 | +| medium priority | 中优先级 | 1 | 10 天 | +| low priority | 低优先级 | 0.1 | 14 天 | | invalid | 非问题 | 0 | / | ## 其他规则 diff --git a/knowledge/repos/vllm-omni/benchmark/_index.md b/knowledge/repos/vllm-omni/benchmark/_index.md index de20709b7..314f10e28 100644 --- a/knowledge/repos/vllm-omni/benchmark/_index.md +++ b/knowledge/repos/vllm-omni/benchmark/_index.md @@ -1,7 +1,7 @@ --- title: "vLLM-Omni 性能与 Profiling" created: 2026-07-10 -updated: 2026-09-21 +updated: 2026-09-22 type: index tags: [vllm-omni, benchmark] sources: [benchmarks/tts/bench_tts.py, benchmarks/tts/model_configs.yaml, "PR #6522", "PR #6634", "PR #6818", vllm_omni/benchmarks/data_modules/omniinteract_dataset.py, vllm_omni/benchmarks/omniinteract.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/patch/test_patch.py, vllm_omni/benchmarks/duplex/, vllm_omni/entrypoints/cli/benchmark/omni_duplex_eval.py, vllm_omni/experimental/fullduplex/client.py] @@ -21,7 +21,7 @@ sources: [benchmarks/tts/bench_tts.py, benchmarks/tts/model_configs.yaml, "PR #6 | 遇到什么 | 查看哪里 | |---|---| -| 新增/审查 perf JSON、warmup 覆盖、realtime artifact、Hub dataset fallback、per-request timeout 或 `/v1/videos` 轮询预算、最终性能复测与控制漂移 | [benchmark rules](rules.md) | +| 新增/审查 perf JSON、warmup 覆盖、realtime artifact、Hub dataset fallback、per-request timeout 或 `/v1/videos` 轮询预算、最终性能复测与控制漂移 | [benchmark rules](rules.md) 新增核对：BENCH-1f2、BENCH-1g。 | | 查看旧 benchmark 总览 | [overview](overview.md) | | 运行 AR graph、Hunyuan 或 MiniCPM-o Omni-DuplexEval benchmark | [benchmark guides](guides/_index.md) | | 调查 profiling、模型加载和性能验证错误 | [benchmark incidents](incidents/_index.md) | diff --git a/knowledge/repos/vllm-omni/benchmark/rules.md b/knowledge/repos/vllm-omni/benchmark/rules.md index d8146763f..abb515052 100644 --- a/knowledge/repos/vllm-omni/benchmark/rules.md +++ b/knowledge/repos/vllm-omni/benchmark/rules.md @@ -1,10 +1,10 @@ --- title: "vLLM-Omni Benchmark 规则" created: 2026-09-05 -updated: 2026-09-21 +updated: 2026-09-22 type: rule tags: [vllm-omni, benchmark] -sources: ["PR #7648", "PR #6817", tests/dfx/perf/scripts/run_benchmark.py, tests/benchmarks/test_omniinteract.py, "PR #7130", "PR #7259"] +sources: ["PR #7648", "PR #6817", tests/dfx/perf/scripts/run_benchmark.py, tests/benchmarks/test_omniinteract.py, "PR #7130", "PR #7259", "PR #7624", "PR #7504"] confidence: high --- @@ -72,3 +72,17 @@ confidence: high determinism 或 eager 数值等价证明。 - 验收：产物能追溯各 arm 的真实 source/config/environment，报告候选差值和控制漂移； 稳态统计可从相同筛选重算，正确性另由对应 parity 测试支持。 ^[PR #7648] + +## BENCH-1f2 — benchmark 结果必须始终导出样本计数，包括零 + +- 触发：修改 `benchmarks/patch` 的 result dict，或 DFX 对 `num_tpot_samples` 等计数字段的读取。 +- 强制：`calculate_metrics()` 已经产出的 `num_ttft_samples`、`num_tpot_samples`、`num_itl_samples`、`num_audio_ttfp_samples`、`num_audio_rtf_samples` 必须写入结果，包括 0，且与 percentile 是否被选中无关。JSON 往返后字段仍在。缺键是 `None`，不是 0；DFX 在 finite `mean_tpot_ms` 下仍会因 `isinstance(None, int)` 失败。 +- 禁止：只在选中 TPOT/ITL percentile 时写出计数；用均值存在代替样本数字段；改 DFX 判定去迁就缺字段的 JSON。 +- 验收：真实聚合加 JSON round-trip，分别覆盖 measured/unmeasured TPOT 与选中/未选中 percentile；省略 `num_tpot_samples` 而保留 finite mean 的结果必须被 baseline 拒绝。^[PR #7624] + +## BENCH-1g — duplex/eval 媒体解码必须用捆绑 PyAV，不得依赖 host ffmpeg + +- 触发：修改 `omni-duplex-eval` 或同类 benchmark 的视频时长、帧抽取、音频 PCM 解码，或重新引入 `ffmpeg`/`ffprobe` subprocess。 +- 强制：媒体路径经 PyAV（`av`）打开/seek/decode/resample；结果不得依赖主机安装的 ffmpeg 版本。帧抽取语义对齐“首个 PTS ≥ 目标时刻”；非 WAV 音频经 `AudioResampler` 并在循环后 `resample(None)` flush 尾部样本。 +- 禁止：`subprocess` 调用 host `ffmpeg`/`ffprobe` 作为默认路径；把“本机 ffmpeg 能跑”当作可复现证据；在已知会挂起的旧 HEVC decoder 上无超时地阻塞 generate 阶段。 +- 验收：覆盖 duration/JPEG/PCM 合同与失败 raise；至少用曾触发 host ffmpeg 4.4.2 挂起的样本证明不再无限忙等。^[PR #7504] diff --git a/knowledge/repos/vllm-omni/ci/_index.md b/knowledge/repos/vllm-omni/ci/_index.md index f6012e79c..0a694940a 100644 --- a/knowledge/repos/vllm-omni/ci/_index.md +++ b/knowledge/repos/vllm-omni/ci/_index.md @@ -1,7 +1,7 @@ --- title: "vLLM-Omni CI" created: 2026-07-10 -updated: 2026-09-21 +updated: 2026-09-22 type: index tags: [vllm-omni, ci] sources: [.buildkite/cuda/pipeline.yml, docs/contributing/ci/test_system_overview.md, tests/diffusion/quantization/test_svdquant_config.py, tests/diffusion/quantization/test_svdquant_linear.py, tests/diffusion/quantization/test_svdquant_tp_loading.py, tests/diffusion/quantization/test_wan_autoround_mxfp4.py, tests/e2e/offline_inference/test_wan21_autoround_mxfp4.py, "PR #5544", "PR #6162", "PR #6170", "PR #6303", "PR #6390", "PR #6613", .buildkite/cuda/test-nightly.yml, tests/e2e/online_serving/run_minicpmo_realtime_duplex_server_vad.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/platforms/npu/test_diffusion_attn_backend_selector.py, "PR #6054", .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, tests/e2e/online_serving/test_hunyuan_video_15_expansion.py, tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json, tests/dfx/perf/tests/test_hunyuanvideo15_i2v_vllm_omni.json, "PR #6349", .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, tests/helpers/clean.py, "PR #6704", tests/model_executor/models/minicpmo_4_5/test_pipeline.py, tests/model_executor/stage_input_processors/test_minicpmo_4_5_async_chunk.py, "PR #5464", "PR #6730", "PR #6745", "PR #6727", tests/diffusion/quantization/test_quantization_quality.py, "PR #5831", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", tests/diffusion/models/minimax_h3/test_minimax_h3_quantization_quality.py, "PR #6742", "PR #6650", pyproject.toml, tests/helpers/mark.py, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, "PR #6174", .buildkite/cuda/test-weekly.yml, tests/e2e/offline_inference/test_dots_tts_expansion.py, "PR #6556", tests/e2e/online_serving/minimax_h3/] @@ -209,11 +209,15 @@ sources: [.buildkite/cuda/pipeline.yml, docs/contributing/ci/test_system_overvie | 遇到什么 | 查看哪里 | |---|---| -| 审查硬件 lane（含 MiniMax-H3 DLO DP2 ready smoke 的证据边界）、回归 fence、CI 工具供应链、ASR 文本比较或 xdist/shared fixture、Buildkite 失败日志与运行时差异 | [CI rules](rules.md) | -| AMD/ROCm timeout、quarantine、memory signal、single-card diffusion job 的 multi-card marker 排除、Qwen3-TTS argv 或 Qwen3-Omni control-plane fixture | [AMD/ROCm CI rules](rules-amd.md) | +| 审查硬件 lane（含 MiniMax-H3 DLO DP2 ready smoke 的证据边界）、回归 fence、CI 工具供应链、ASR 文本比较或 xdist/shared fixture、Buildkite 失败日志与运行时差异 | [CI rules](rules.md) 新增核对：OMNI-CI-1g、OMNI-CI-1h。 | +| AMD/ROCm timeout、quarantine、memory signal、single-card diffusion job 的 multi-card marker 排除、Qwen3-TTS argv 或 Qwen3-Omni control-plane fixture | [AMD/ROCm CI rules](rules-amd.md) 新增核对：OMNI-CI-2i、OMNI-CI-2i2、OMNI-CI-2i3、OMNI-CI-2i4。 | | CUDA L4 Kubernetes preset、GPU-count shard、resource/retry policy | [L4 Kubernetes CI rules](rules-l4-k8s.md) | | Whisper 转写 helper 的 GPU 首选、16 GiB 门槛、CPU fallback 或 CUDA ready/merge source dependency | [Whisper 转写 CI 规则](rules-whisper-transcription.md) | | 查看仓库特有 CI 陷阱 | [CI guides](guides/_index.md) | | 调查历史 CI 失败 | [CI incidents](incidents/_index.md) | - [CI 并行测试与 engine fixture 合同](rules-test-fixtures.md)：`OMNI-CI-2b`, `OMNI-CI-2c`, `OMNI-CI-2d`, `OMNI-CI-2h`。 + +- [rules-review-validation](rules-review-validation.md) 新增核对：VOMNI-CI-1a、VOMNI-CI-1b。 + +- [release-gates](release-gates.md)：维护者分工与发布政策。 diff --git a/knowledge/repos/vllm-omni/ci/release-gates.md b/knowledge/repos/vllm-omni/ci/release-gates.md new file mode 100644 index 000000000..3fd9a632f --- /dev/null +++ b/knowledge/repos/vllm-omni/ci/release-gates.md @@ -0,0 +1,34 @@ +--- +title: "发布规范" +created: 2026-08-28 +updated: 2026-09-22 +type: guide +tags: [vllm-omni, ci] +sources: ["https://github.com/JiusiServe/InferMatrixCopilot/pull/70"] +--- + +# 发布规范 + +vLLM-Omni 版本 **release 前**必须满足以下门禁。标注 `(Guide)` 的项按当期迭代指南解释与举证；其余为硬性门槛。 +相关入口：[CI 索引](_index.md)、[DI 计算说明](../../../general/bug/guides/di-calculation.md)、[benchmark](../benchmark/_index.md)。 + +## Release 前门禁 + +| # | 要求 | 类型 | 说明 | +|---|---|---|---| +| 1 | UT coverage meets this iteration requirement | Guide | 单元测试覆盖率达到当期迭代要求；具体阈值与举证方式以当期 Guide 为准 | +| 2 | Performance regression < 10% | Guide | 性能回退小于 10%；对比基线、指标与证据口径以当期 Guide 为准 | +| 3 | Latest CUDA L1 & L2 & L3 & L4 & L5 (excluding Non-critical) pass rate = 100% | Gate | 最新一轮 CUDA L1/L2/L3/L4/L5 中，排除 Non-critical 后通过率必须为 100% | +| 4 | Latest NPU L1 & L2 & L4 pass rate = 100% | Gate | 最新一轮 NPU L1/L2/L4 通过率必须为 100% | +| 5 | Requirement completion rate > 85% | Gate | 需求完成率大于 85% | +| 6 | Remaining DI < 30 | Gate | 剩余 DI 小于 30；DI 计算见 [DI 计算说明](../../../general/bug/guides/di-calculation.md) | +| 7 | No remaining critical issues | Gate | 无未关闭的 critical 问题 | +| 8 | All remaining bugs have assignees | Gate | 所有剩余 bug 均已指派负责人 | + +## 验收记录 + +发布前在发布说明或 checklist 中逐项给出： + +- 是否通过（Pass / Fail / N/A） +- 证据链接（CI run、benchmark 报告、需求看板、DI 统计、issue 列表） +- Guide 项写明当期 Guide 版本或链接 diff --git a/knowledge/repos/vllm-omni/ci/rules-amd.md b/knowledge/repos/vllm-omni/ci/rules-amd.md index 7422b5415..96e5e0c7d 100644 --- a/knowledge/repos/vllm-omni/ci/rules-amd.md +++ b/knowledge/repos/vllm-omni/ci/rules-amd.md @@ -1,10 +1,10 @@ --- title: "AMD/ROCm CI 规则" created: 2026-09-05 -updated: 2026-09-10 +updated: 2026-09-22 type: rule tags: [vllm-omni, ci] -sources: ["PR #6704", "PR #6830", "PR #6884", .buildkite/amd/, tests/helpers/clean.py, tests/helpers/stage_config.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #7234"] +sources: ["PR #6704", "PR #6830", "PR #6884", .buildkite/amd/, tests/helpers/clean.py, tests/helpers/stage_config.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #7234", "PR #6966", "PR #7395", "PR #6978", "PR #7398"] confidence: high --- @@ -42,3 +42,31 @@ confidence: high - 强制：`mi300_1` 一类单卡 model job 的 pytest marker 必须 `not (cards_2 or … or cards_8)`，同时保留无 `cards_1` 的 legacy 单卡用例；真正需要双卡的用例（如 LTX2 Ulysses parity）改到已有双卡 lane，并声明 `rocm` 资源与 `device_count >= world_size` 早失败。 - 禁止：让 `cards_2+` 测试在单卡 worker 上 spawn rank1→GPU1 导致 `invalid device ordinal`；用邻近 green shard 宣称 multi-GPU routing 已修好。 - 验收：pipeline argv/collection 断言单卡 job 排除 multi-card markers、双卡 job 收集目标文件；硬件 marker helper 覆盖 ROCm 声明。^[PR #7234] + +## OMNI-CI-2i — AMD bootstrap 必须按 ready/merge-test 标签选择 L2/L3 suite + +- 触发：修改 `.buildkite/amd` bootstrap、`select_test_suites.py`、AMD PR label 路由，或 skip-ci 对 AMD suite 的过滤。 +- 强制：`ready` 选 L2（ready suite），`merge-test` 选 L3（merge suite）；两标签同时存在时合并多 suite 且共享一次 image build；`DEBUG_TEST_YAML` 优先；main 继续 L3；无 tier 标签的 PR 可保留 legacy ready fallback。PR labels 精确匹配且失败时 fail closed；skip-ci 必须对已选 L2/L3 独立过滤。 +- 禁止：凡 PR 一律上传 ready suite；用子串匹配 labels；在仓库侧假装已改变 Buildkite 外部 trigger 条件；把 `nightly-test` 当成已有 AMD L4 覆盖。 +- 验收：selector/bootstrap 单测覆盖 ready-only、merge-only、both、debug override、main、label fetch failure 与 per-suite skip-ci；日志报告实际 `TEST_SPECS`。^[PR #6966] + +## OMNI-CI-2i2 — ROCm Dockerfile 必须与 CI 的 vLLM release 对齐并在构建期 canary + +- 触发：修改 `docker/Dockerfile.rocm` 的 `BASE_IMAGE` / `VLLM_VERSION_OR_COMMIT_HASH` / `USE_NIGHTLY_BUILD`、`.buildkite/amd` 的 image build 命令，或 AMD 运行时因缺失 vLLM API 失败。 +- 强制：默认 `BASE_IMAGE` tag 与可选 source rebuild 的 `VLLM_VERSION_OR_COMMIT_HASH` 都必须等于 `docker/Dockerfile.ci` 的 `VLLM_BASE_TAG`；AMD build 不得 `--build-arg` 覆盖这两个默认。可选 nightly 重装之后、业务层拷贝之前，必须用当前 Omni 依赖的 vLLM API 做 image-build canary，构建失败优于整 lane runtime 失败。 +- 禁止：只升 Omni 代码而留下过期 ROCm base；把 canary 推到测试阶段；用自定义 source pin 却不更新 `tests/buildkite/test_rocm_dockerfile.py`；把 Docker pin 误当成非 Docker 安装会自动装上匹配 vLLM。 +- 验收：静态回归断言 ROCm base/source ref 跟踪 CI tag、默认 `USE_NIGHTLY_BUILD=0`、AMD build 不覆盖上述 arg、canary 位于 nightly 块之后；故意错位 pin 必须失败。^[PR #7395] + +## OMNI-CI-2i3 — AMD nightly suite 必须可显式选中且不受 L2/L3 skip-ci 误杀 + +- 触发：修改 AMD bootstrap/`select_test_suites`、`NIGHTLY_TESTS` YAML、`nightly-test` label，或 skip-ci 对 suite spec 的过滤。 +- 强制：`nightly` 映射 `NIGHTLY_TESTS:test-amd-nightly.yml`；`main+NIGHTLY=1` 或 PR `nightly-test` 选中它；可与 ready/merge 组合共享一次 image build。`NIGHTLY_TESTS:*` 在 skip-ci 过滤中原样保留，docs-only 也不得剥掉显式/scheduled nightly。burn-in 叶子保持 `NonBlocking` 直至另有 gate。 +- 禁止：把 `nightly-test` 当成无 suite 的噪声 label；用 L2/L3 diff gate 静默丢掉已选 nightly；把实验 nightly 阈值外推为 CUDA H100 基线。 +- 验收：渲染 `DEBUG_TEST_YAML=nightly`、`NIGHTLY=1`、组合 label 与 docs-only+nightly，断言 suite spec 与子 pipeline 叶子集合。^[PR #6978] + +## OMNI-CI-2i4 — AMD entrypoints GPU ready job 必须单次 pytest、有界超时并检测进程泄漏 + +- 触发：修改 `.buildkite/amd/test-amd-ready.yml` 的 entrypoints/R2-02 GPU coverage、artifact 根路径，或 teardown 进程快照比较。 +- 强制：`mi300_1` NonBlocking job 设 `VLLM_WORKER_MULTIPROC_METHOD=spawn`，marker 排除 `cards_2`–`cards_8`，只跑一次 verbose pytest（禁止第二趟 `--collect-only`）。内层 `timeout` 短于 Buildkite step，为 teardown/artifact 留窗口。artifact 根在 Buildkite checkout；前后 `ps` 快照用 PID+start-time 身份比较，泄漏则 fail closed。零 collection 不得被允许通过。 +- 禁止：依赖 PID-only 比较掩盖 reuse；把 CUDA pipeline 定义当作 ROCm ready 覆盖；用邻近 green shard 宣称本 job 合同已满足。 +- 验收：结构断言单次 pytest、超时信封、marker、六类 artifact 与 `process-cleanup` PASS；真实 MI300 跑通 selected node。^[PR #7398] diff --git a/knowledge/repos/vllm-omni/ci/rules-review-validation.md b/knowledge/repos/vllm-omni/ci/rules-review-validation.md new file mode 100644 index 000000000..2a0174913 --- /dev/null +++ b/knowledge/repos/vllm-omni/ci/rules-review-validation.md @@ -0,0 +1,35 @@ +--- +title: "CI 选择器与质量验证规则" +created: 2026-09-22 +updated: 2026-09-22 +type: rule +tags: [vllm-omni, ci] +sources: ["PR #5695", "PR #5696", "PR #5436"] +--- + +# CI 选择器与质量验证规则 + +## VOMNI-CI-1a — 选择器改动必须闭合到实际收集的测试集合 + +- 触发：新增、拆分、收窄或移动 Buildkite step，或修改 path、label、pytest `-m`、 + hardware marker、队列、卡数及非默认目标分支。 +- 强制：冻结 PR 的真实 target base；分别在 base/head 收集测试，并把 + `path/label → pytest selector → test marker → hardware/queue/card count` 逐项对齐； + 每个被移除的平台或测试都必须有明确归类。 +- 禁止：从文件名推断收集集合；在 `main` 上验证一个目标为 release/challenge 分支的 + diff；把未解释的掉测、平台移除或 label 变化当作纯 CI 重排。 +- 验收：保存 base/head 的 collection diff，head 中每个选中测试都能到达匹配的 runner， + 每个掉测都有 `moved`、`intentionally removed` 或 `still covered elsewhere` 结论，并用 + 实际 target branch 跑一次选择器 smoke。 ^[PR #5695] ^[PR #5696] + +## VOMNI-CI-1b — 平台专有质量 oracle 必须来自同一硬件 case + +- 触发：按 CUDA/ROCm/NPU 或具体硬件选择 golden、评分器、阈值或质量 job。 +- 强制：golden provenance、测试输入、模型 revision、执行参数、marker、队列和实际硬件 + 属于同一个 case；平台分支必须显式选择 oracle，未命中的平台保持原合同。 +- 禁止：用另一硬件的本地输出生成 golden；用无法解释的阈值放宽吸收平台漂移；修改 + 一个平台时顺带改变其他平台的 baseline。 +- 验收：目标 CI runner 产出的 artifact 可复现提交 golden 的 digest 和评分，目标平台 + 命中专属 oracle，至少一个未改平台 control 仍选择原 oracle；阈值证据同时满足 + [DIFF-3a](../components/diffusion/rules.md#diff-3a-质量阈值必须由完全相同的测试-case-产生)。 + ^[PR #5436] diff --git a/knowledge/repos/vllm-omni/ci/rules.md b/knowledge/repos/vllm-omni/ci/rules.md index aa4ac9453..0da9345c8 100644 --- a/knowledge/repos/vllm-omni/ci/rules.md +++ b/knowledge/repos/vllm-omni/ci/rules.md @@ -1,10 +1,10 @@ --- title: "vLLM-Omni CI 规则" created: 2026-08-23 -updated: 2026-09-21 +updated: 2026-09-22 type: rule tags: [vllm-omni, ci] -sources: ["PR #7648", "PR #3422", "PR #5074", "PR #5255", "PR #5310", "PR #5402", "PR #5524", "PR #5543", "PR #5670", "PR #5713", "PR #5780", "PR #5823", "PR #5836", "PR #5957", "PR #5976", docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, "PR #5845", "PR #5872", "PR #6008", "PR #6048", "PR #6056", "PR #6096", "PR #6102", "PR #6202", "PR #6208", "PR #6273", "PR #6293", "PR #6311", "PR #6339", "PR #6343", "PR #6468", "PR #6523", "PR #6613", "PR #6555", "PR #6650", .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, "PR #6704", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", "PR #6696", vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, "PR #6674", docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, "PR #6818", "PR #6830", "PR #6884", tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #6947", "PR #7118"] +sources: ["PR #7648", "PR #3422", "PR #5074", "PR #5255", "PR #5310", "PR #5402", "PR #5524", "PR #5543", "PR #5670", "PR #5713", "PR #5780", "PR #5823", "PR #5836", "PR #5957", "PR #5976", docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, "PR #5845", "PR #5872", "PR #6008", "PR #6048", "PR #6056", "PR #6096", "PR #6102", "PR #6202", "PR #6208", "PR #6273", "PR #6293", "PR #6311", "PR #6339", "PR #6343", "PR #6468", "PR #6523", "PR #6613", "PR #6555", "PR #6650", .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, "PR #6704", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", "PR #6696", vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, "PR #6674", docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, "PR #6818", "PR #6830", "PR #6884", tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #6947", "PR #7118", "PR #6597", "PR #7371", "PR #7534"] confidence: high --- @@ -284,3 +284,17 @@ PR 报告的 collection=1 和 local two-card B300 184.97 s 是该精确 workload 运行时可用的隔离验证，不能取消生产测试门禁或隐藏 skip 原因。 - 验收：修复绑定当前 head，明确本地实际执行数、skip 与平台边界；重触发后核对新 build 的 SHA 和状态。queued/running 只能报告已触发，不能报告 CI 通过。 ^[PR #7648] + +## OMNI-CI-1g — PR 与 post-merge L3 必须按 registry 做 diff 过滤 + +- 触发：修改 `ci_source_file_dependencies.yml`、`upload_pipeline.py`、merge bootstrap 或 pipeline 的 `source_file_dependencies`。 +- 强制：key/list 由 registry 展开为路径前缀，按 PR/commit diff 保留命中步骤；无 key 的步骤保留，上传前剥离 uploader-only 字段。模型路径用共享 anchor，job key 另列自己的脚本/pytest 路径，不从 command 猜。PR-label 与 post-merge main L3 都过滤；仅显式 `--all`/`--e2e`（scheduled nightly/weekly）保留全量。无 job-key 命中时，被上传的 pipeline YAML 或 `source_filter_fallback` 命中可 keep-all；已有 job-key 命中时正常过滤优先。 +- 禁止：仅因 `BUILDKITE_BRANCH=main` 就 keep-all；普通 main merge bootstrap 默认传 `--all`；把未注册 key 交给 Buildkite，或用旧模型族 label 代替有效 tier label。 +- 验收：覆盖 key 展开/未注册拒绝、无关 diff 丢弃、无 key 保留、fallback 优先级；main 的 `force_all=False/e2e_only=False` 返回 diff，显式全量返回 `None`，普通 merge bootstrap 不带 `--all`。^[PR #6597] ^[PR #7371] + +## OMNI-CI-1h — xdist online 测试不得用全局 GPU 占用做跨 worker 屏障 + +- 触发：修改 `tests/model_tests/diffusion` 等 online/xdist suite、`cleanup_test_environment`，或在并行 worker 上等待“整卡显存回落阈值”。 +- 强制：`cleanup_test_environment` 观察的是整卡占用，不是当前 worker 自有分配。在 `is_xdist_worker` 下，online diffusion 等并行 suite 必须禁用/no-op 该全局 cleanup，避免 server teardown 变成跨 worker 屏障；`OmniServer` 仍拆除自己的子进程树。非 xdist 运行保留更广的 cleanup。 +- 禁止：在 xdist 下等待 sibling worker 的分配消失才结束 teardown；把全局阈值等待说成单测资源隔离；或删除非 xdist 路径上仍需要的环境清理。 +- 验收：xdist worker 断言 cleanup 被 monkeypatch 为 no-op；非 worker 控制路径仍调用真实 cleanup；并行 online lane 不因全局 GPU wait 放大 wall time。^[PR #7534] diff --git a/knowledge/repos/vllm-omni/components/_index.md b/knowledge/repos/vllm-omni/components/_index.md index 66813ad8f..2d89dfbdf 100644 --- a/knowledge/repos/vllm-omni/components/_index.md +++ b/knowledge/repos/vllm-omni/components/_index.md @@ -24,3 +24,5 @@ sources: [] 需要解释稳定数据流时再进入对应 owner 的 `architecture.md`，不要把 architecture 当成 review 的默认前置阅读。 + +- [comfyui](comfyui/_index.md)：comfyui owner 规则。 diff --git a/knowledge/repos/vllm-omni/components/comfyui/_index.md b/knowledge/repos/vllm-omni/components/comfyui/_index.md new file mode 100644 index 000000000..d91077c96 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/comfyui/_index.md @@ -0,0 +1,21 @@ +--- +title: "ComfyUI" +created: 2026-09-22 +updated: 2026-09-22 +type: index +tags: [vllm-omni, components, serving] +sources: ["PR #5756"] +--- + +# ComfyUI + +## 什么时候查这里 + +- 修改 ComfyUI video node/client 的模式选择、reference 组合和 multipart 编码。 +- 服务端校验回到 [Serving](../serving/_index.md)；checkpoint 能力回到对应模型 owner。 + +## 目录内容 + +| 遇到什么 | 查看哪里 | +|---|---| +| video mode、reference 组合、请求字段一致性 | [ComfyUI 规则](rules.md)：`COMFY-1a` | diff --git a/knowledge/repos/vllm-omni/components/comfyui/rules.md b/knowledge/repos/vllm-omni/components/comfyui/rules.md new file mode 100644 index 000000000..485b5e52a --- /dev/null +++ b/knowledge/repos/vllm-omni/components/comfyui/rules.md @@ -0,0 +1,17 @@ +--- +title: "ComfyUI 规则" +created: 2026-09-22 +updated: 2026-09-22 +type: rule +tags: [vllm-omni, components, serving] +sources: ["PR #5756"] +--- + +# ComfyUI 规则 + +## COMFY-1a — video mode 与 multipart 字段必须共享当前能力矩阵 + +- 触发：ComfyUI video node/client 修改 frame、image/audio/video references 或 mode。 +- 强制：先按当前服务端和目标 checkpoint 的能力矩阵校验组合，再确定唯一 mode；node 与 client 使用同一矩阵，按服务端字段名和输入顺序编码 multipart。新增 first/last frame 或 reference 组合必须同步两端。 +- 禁止：按字段遍历顺序覆盖 mode；把早期 MiniMax H3 的 reference 限制推广到所有模型或后续功能。 +- 验收：每个当前支持的组合断言最终 mode、multipart key/count/order；相邻非法组合在发送前失败。^[PR #5756] diff --git a/knowledge/repos/vllm-omni/components/configuration/_index.md b/knowledge/repos/vllm-omni/components/configuration/_index.md index 6d6ab8a55..5e9d78845 100644 --- a/knowledge/repos/vllm-omni/components/configuration/_index.md +++ b/knowledge/repos/vllm-omni/components/configuration/_index.md @@ -1,7 +1,7 @@ --- title: "vLLM-Omni Configuration" created: 2026-07-16 -updated: 2026-09-21 +updated: 2026-09-22 type: index tags: [vllm-omni, components, config] sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/] @@ -35,13 +35,13 @@ sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/] | 遇到什么 | 查看哪里 | |---|---| | 理解配置从 deploy、CLI、默认 factory 到 structured/legacy config 的稳定边界 | [配置构造架构](architecture.md) | -| 根据 PR 描述直达 strict schema、pipeline sampling constraints、deploy/topology、composable strategy、global engine args 的 owner 校验、CLI-only 负向 alias 或显存配置的规则组与第一批源码、环境变量 inventory | [配置开发门禁与代码地图](rules.md) | +| 根据 PR 描述直达 strict schema、pipeline sampling constraints、deploy/topology、composable strategy、global engine args 的 owner 校验、CLI-only 负向 alias 或显存配置的规则组与第一批源码、环境变量 inventory | [配置开发门禁与代码地图](rules.md) 新增核对：CONF-3e。 | | 对象存储 URI、HF cache snapshot 或空 `config.json` 的模型名称回退与 pipeline 路由 | [模型引用路由规则](rules-model-reference-routing.md) | | stage full-payload transport capability、topology-owned projection 或 deploy/CLI override rejection | [stage transport capability](rules-stage-transport.md) | | diffusion Ulysses SymmMem transport flag 的 deploy/CLI/default-stage projection | [diffusion parallel transport rules](rules-diffusion-parallel-transport.md) | | diffusion attention shorthand、structured config 与 global override 的优先级和最终规范表示 | [diffusion attention precedence](rules-diffusion-attention-precedence.md) | -| legacy stage engine args 的 copy boundary、connector/default injection、repeated build mutation、global engine args 的 owner 校验或 CLI-only 负向 alias | [legacy engine-args isolation](rules-legacy-engine-args.md) | +| legacy stage engine args 的 copy boundary、connector/default injection、repeated build mutation、global engine args 的 owner 校验或 CLI-only 负向 alias | [legacy engine-args isolation](rules-legacy-engine-args.md) 新增核对：VOMNI-CFG-1u。 | | 审计配置来源和多层加工 | [config audit](config-audit-plain-language.md) | | 设计配置归一化与 parity 验证 | [config normalization parity](config-normalization-parity.md) | | 查询 `Omni()` 初始化参数 | [Omni init args](omni-init-args.md) | diff --git a/knowledge/repos/vllm-omni/components/configuration/rules-legacy-engine-args.md b/knowledge/repos/vllm-omni/components/configuration/rules-legacy-engine-args.md index 8135708c2..ab7171abe 100644 --- a/knowledge/repos/vllm-omni/components/configuration/rules-legacy-engine-args.md +++ b/knowledge/repos/vllm-omni/components/configuration/rules-legacy-engine-args.md @@ -1,10 +1,10 @@ --- title: "legacy engine args 投影隔离" created: 2026-09-05 -updated: 2026-09-10 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, config] -sources: ["PR #6783", vllm_omni/engine/stage_init_utils.py, tests/engine/test_build_engine_args_no_mutate.py, "PR #5929", "PR #7237"] +sources: ["PR #6783", vllm_omni/engine/stage_init_utils.py, tests/engine/test_build_engine_args_no_mutate.py, "PR #5929", "PR #7237", "PR #7390"] confidence: high --- @@ -31,3 +31,10 @@ confidence: high - 强制：把该 alias 标为非 stage 字段；在 `log_stats` 等 canonical 值已由 launcher/`__init__` 消费后，于 resolver 边界 `pop` 残余 alias，使 structured ownership 看不到它。标准与 headless 启动都必须消费。 - 禁止：让 residual `disable_log_stats` 落入 `resolve_omni_config` / `VllmOmniConfig` 并报 “no structured config owner”；为通过校验而放宽真正的 stage engine-arg ownership。 - 验收：断言字段在 `_NON_STAGE_ENGINE_CLI_FIELDS` 且不在 global stage CLI fields；engine 与 headless 解析后 kwargs/`cli_overrides` 不含该 alias；带 alias 的 TTS/structured pipeline 仍能构造。^[PR #7237] + +## VOMNI-CFG-1u — 用户可见的 upstream ModelConfig CLI 必须有 typed stage owner + +- 触发：文档/示例使用的全局 ModelConfig CLI（如 `served_model_name`、`allowed_local_media_path`、`allowed_media_domains`、`max_logprobs`、`logprobs_mode`、`mm_processor_kwargs`、`mm_processor_cache_type`、`hf_token`、`hf_config_path`、`generation_config`、`override_generation_config`、`enable_prompt_embeds`）在 `from_pipeline_config` 报无 owner，或新增同类 upstream ModelConfig 输入。 +- 强制：把字段同时加入 `_ModelEngineOverrides` 与 `OmniStageModelConfig`，类型跟随 vLLM `EngineArgs`，默认 `None`，使 `_project_omni_config_fields` 只投影用户显式值，并经 typed/legacy engine-args 到达各 stage。不得把这些 ModelConfig 输入塞进 `_NON_STAGE_ENGINE_CLI_FIELDS` 来绕过 ownership。 +- 禁止：仅因 legacy `build_stage_runtime_overrides` 仍能透传就让 structured ownership 拒绝文档命令；把尚未归属的非 ModelConfig 开关（如 `enable_lora`/`speculative_config`）一并放行。 +- 验收：对至少两个 pipeline key 参数化断言显式全局值进入每个 stage 的 `model_config`，并用 typed `build_engine_args_dict_from_omni_stage_config` 读回同一集合；字段集合 census 必须包含这些 owner。^[PR #7390] diff --git a/knowledge/repos/vllm-omni/components/configuration/rules.md b/knowledge/repos/vllm-omni/components/configuration/rules.md index 9ded6e5ea..ed98f4216 100644 --- a/knowledge/repos/vllm-omni/components/configuration/rules.md +++ b/knowledge/repos/vllm-omni/components/configuration/rules.md @@ -1,10 +1,10 @@ --- title: "vLLM-Omni 配置开发门禁" created: 2026-07-16 -updated: 2026-09-21 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, config] -sources: ["PR #7648", "claude-workflow-starter-private@296ea45", "PR #4281", "PR #5031", "PR #5073", "PR #5671", "PR #5678", "zuiho-kai/claude-workflow-starter@c217fc6", vllm_omni/config/model.py, vllm_omni/config/stage_config.py, vllm_omni/config/config_factory.py, vllm_omni/config/omni_config.py, vllm_omni/config/composable_parallel/, vllm_omni/deploy/qwen3_omni_moe.yaml, vllm_omni/engine/stage_init_utils.py, tests/config/test_config_factory.py, tests/engine/test_arg_utils.py, tests/engine/test_stage_engine_args.py, "PR #4795", "PR #5842", "PR #6082", "PR #6156", "PR #5741", "PR #6068", "PR #4765", "PR #5666", "PR #4222", "PR #5604", "PR #6293", "PR #6094", "vllm_omni/diffusion/data.py", "PR #6050", "PR #6322", "vllm_omni/config/pipeline_registry.py", "vllm_omni/diffusion/models/pi0_pipeline_config.py", "vllm_omni/diffusion/models/pi0/pipeline_pi0.py", "PR #5048", "PR #6458", "PR #6102", "PR #6308", "PR #6182", "PR #4820", "PR #6619", "PR #6680", "PR #6422", vllm_omni/deploy/higgs_multimodal_qwen3.yaml] +sources: ["PR #7648", "claude-workflow-starter-private@296ea45", "PR #4281", "PR #5031", "PR #5073", "PR #5671", "PR #5678", "zuiho-kai/claude-workflow-starter@c217fc6", vllm_omni/config/model.py, vllm_omni/config/stage_config.py, vllm_omni/config/config_factory.py, vllm_omni/config/omni_config.py, vllm_omni/config/composable_parallel/, vllm_omni/deploy/qwen3_omni_moe.yaml, vllm_omni/engine/stage_init_utils.py, tests/config/test_config_factory.py, tests/engine/test_arg_utils.py, tests/engine/test_stage_engine_args.py, "PR #4795", "PR #5842", "PR #6082", "PR #6156", "PR #5741", "PR #6068", "PR #4765", "PR #5666", "PR #4222", "PR #5604", "PR #6293", "PR #6094", "vllm_omni/diffusion/data.py", "PR #6050", "PR #6322", "vllm_omni/config/pipeline_registry.py", "vllm_omni/diffusion/models/pi0_pipeline_config.py", "vllm_omni/diffusion/models/pi0/pipeline_pi0.py", "PR #5048", "PR #6458", "PR #6102", "PR #6308", "PR #6182", "PR #4820", "PR #6619", "PR #6680", "PR #6422", vllm_omni/deploy/higgs_multimodal_qwen3.yaml, "PR #7655"] --- # vLLM-Omni 配置开发门禁 @@ -236,3 +236,10 @@ sources: ["PR #7648", "claude-workflow-starter-private@296ea45", "PR #4281", "PR 比较描述为接受任意 truthy 值。 - 验收：`tests/config/test_environment_variables.py` 的静态读取分类、文档和数量检查通过， 并确认启用值到达真实 consumer；重命名后旧名称无未解释读取。 ^[PR #7648] + +## CONF-3e — 已删除的 dots.tts 与 Dynin-Omni 不得残留注册面 + +- 触发：改 `pipeline_registry.py`、`environment_variable_inventory.py` 或 `vllm_omni/deploy/`，或试图恢复 `dots_tts` / `dynin_omni`。 +- 强制：pipeline import 与 `OMNI_PIPELINES` key、各 inventory 桶中的 `DOTS_TTS_*` / `DYNIN_*`，以及默认、CI、multiconnector deploy YAML 必须同时缺席。共享注释不得指向已删实现。 +- 禁止：只删模型代码却留下 key、env 名或 YAML；rebase 时恢复 `DOTS_TTS_PIPELINE` 或 `DYNIN_OMNI_PIPELINE`；再把已删除模型的旧 deploy pin 当作在运合同。 +- 验收：registry 无这两项 key，deploy 目录无对应 YAML，inventory 无上述名称。^[PR #7655] diff --git a/knowledge/repos/vllm-omni/components/diffusion/_index.md b/knowledge/repos/vllm-omni/components/diffusion/_index.md index a377eac6f..a65f4fa69 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/_index.md +++ b/knowledge/repos/vllm-omni/components/diffusion/_index.md @@ -1,7 +1,7 @@ --- title: "Diffusion" created: 2026-07-10 -updated: 2026-09-21 +updated: 2026-09-22 type: index tags: [vllm-omni, components, diffusion] sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/attention/backends/flashinfer_attn.py] @@ -37,20 +37,20 @@ sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/ | CustomOp XPU platform dispatch、PyTorch-native fallback、CUDA contract 与非同义 override 边界 | [CustomOp dispatch rules](rules-custom-op-dispatch.md) | | runtime temporary 与 loader-scoped parameter dtype、shared RMSNorm accuracy | [tensor dtype rules](rules-tensor-dtype.md) | | Wan VAE height/width spatial reshard、empty tail、attention extent | [Wan spatial-shard rules](rules-wan-spatial-shard.md) | -| multi-DiT、dotted `_dit_modules`、loader-to-offloader handoff 与跨 Cache-DiT/compile/LoRA/offload lifecycle、batch sampling-key 的 provided 标志 | [component lifecycle rules](rules-component-lifecycle.md) | +| multi-DiT、dotted `_dit_modules`、loader-to-offloader handoff 与跨 Cache-DiT/compile/LoRA/offload lifecycle、batch sampling-key 的 provided 标志 | [component lifecycle rules](rules-component-lifecycle.md) 新增核对：DIFF-2ag。 | -| PEFT 与 distilled LoRA、startup fusion、delta/key/alpha、Qwen/Wan transformer mapping、PEFT suspend/resume | [LoRA rules](rules-lora.md) | +| PEFT 与 distilled LoRA、startup fusion、delta/key/alpha、Qwen/Wan transformer mapping、PEFT suspend/resume | [LoRA rules](rules-lora.md) 新增核对：DIFF-2ag2、DIFF-2ag3。 | -| Blackwell FA4、显式/auto CUDA backend、per-role mask/SP pad；local FlashAttention deterministic opt-in、role-aware masked cross-attention、NPU packed mask-free/laser fallback、TRTLLM packed-padding 与 config propagation、equal-rank SP auto-pad fast path、等长 batch padding mask 省略 | [attention rules](rules-attention.md) | +| Blackwell FA4、显式/auto CUDA backend、per-role mask/SP pad；local FlashAttention deterministic opt-in、role-aware masked cross-attention、NPU packed mask-free/laser fallback、TRTLLM packed-padding 与 config propagation、equal-rank SP auto-pad fast path、等长 batch padding mask 省略 | [attention rules](rules-attention.md) 新增核对：DIFF-1ai3、DIFF-1aj2、DIFF-1ai4、DIFF-1d2。 | | SP auto-padding、`mask_sp_padding`、dense/varlen 与 advanced UAA 边界 | [SP padding rules](rules-sp-padding.md) | -| video/audio mux、DLO DP wave、RPC reply ownership/result release、result queue、async pump、SHM ownership、async-output wait 与 shutdown、mid-stream interaction handler | [output/runtime rules](rules-output-lifecycle.md) | +| video/audio mux、DLO DP wave、RPC reply ownership/result release、result queue、async pump、SHM ownership、async-output wait 与 shutdown、mid-stream interaction handler | [output/runtime rules](rules-output-lifecycle.md) 新增核对：DIFF-1ai、DIFF-1aj、DIFF-1ai2。 | | distilled continuous sigma schedule、boundary/step 语义与 modality shift | [sigma schedule rules](sigma-schedules.md) | | HunyuanImage3 Distil CFG、MeanFlow special-token / scheduler-r 合同 | [HunyuanImage3 HY3-8d](../../models/hunyuan-image3/rules.md) | | diffusion step 与 request/continuous batching；DIFF-1af BAGEL image step wave、packed state、geometry/CFG admission 与 fallback cleanup | [step and batching](step-and-batching.md) | | request-wave admission coalescing、stable window、deadline 与 finite config | [admission wait rules](rules-admission-wait.md) | -| paged KV/cache 预算、native/backend/platform 闭环、GQA/Ring/Ulysses layout、FlashInfer plan、能力 metadata、cache_backend None 规范化 | [paged cache 与系统运行时规则](rules-system-runtime.md) | +| paged KV/cache 预算、native/backend/platform 闭环、GQA/Ring/Ulysses layout、FlashInfer plan、能力 metadata、cache_backend None 规范化 | [paged cache 与系统运行时规则](rules-system-runtime.md) 新增核对：DIFF-4ad。 | | Scheduler-managed diffusion KV 的请求控制面、Hunyuan layout 与未实现边界 | [paged KV control plane](paged-kv-control-plane.md) | | Cache-DiT、TeaCache 和 prefix cache | [cache acceleration](cache-acceleration.md) | @@ -59,4 +59,4 @@ sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/ | checkpoint remap、HSDP/FSDP、final-layout HWR、registered HWR mmap DLO transport、component quantization、在线量化加载与 TorchAO `.bin` index fallback | [checkpoint 与加载合同](rules-checkpoint-loading.md) | -- [Diffusion 平台 kernel 与设备合同](rules-platform-kernels.md)：`DIFF-1t`, `DIFF-1u`, `DIFF-1v`, `DIFF-1w`。 +- [Diffusion 平台 kernel 与设备合同](rules-platform-kernels.md) 新增核对：DIFF-1ai5。 diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-attention.md b/knowledge/repos/vllm-omni/components/diffusion/rules-attention.md index 7ebc0620a..07e7502f4 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-attention.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-attention.md @@ -1,10 +1,10 @@ --- title: "Diffusion attention 规则" created: 2026-09-02 -updated: 2026-09-10 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5543", "PR #5866", "PR #5887", "PR #5891", "PR #5897", "PR #5997", "PR #6000", "PR #6037", "PR #6518", "PR #6563", "PR #6724", "PR #6909", docs/design/feature/skip_softmax.md, docs/user_guide/diffusion/attention_backends.md, docs/user_guide/diffusion/attention_backends/trtllm.md, docs/user_guide/diffusion/attention_backends/fastvideo_vsa.md, docs/user_guide/diffusion/attention_backends/rainfusion.md, vllm_omni/config/omni_config.py, vllm_omni/config/stage_config.py, vllm_omni/diffusion/attention/backends/abstract.py, vllm_omni/diffusion/attention/backends/fastvideo_vsa.py, vllm_omni/diffusion/attention/backends/flash_attn.py, vllm_omni/diffusion/attention/backends/rainfusion_attn.py, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/diffusion_kv/paged_attention_adapter.py, vllm_omni/diffusion/models/minimax_h3/denoise_loop.py, vllm_omni/diffusion/models/minimax_h3/packed_sequence.py, vllm_omni/diffusion/data.py, vllm_omni/engine/arg_utils.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/cli/serve.py, vllm_omni/platforms/cuda/platform.py, vllm_omni/platforms/npu/platform.py, tests/config/test_omni_config.py, tests/diffusion/attention/test_fastvideo_vsa.py, tests/diffusion/attention/test_flash_attn.py, tests/diffusion/attention/test_attention_config.py, tests/diffusion/attention/test_piecewise_attn.py, tests/diffusion/attention/test_rainfusion_plan.py, tests/diffusion/attention/test_ulysses_uaa.py, tests/diffusion/diffusion_kv/test_paged_attention_adapter.py, tests/diffusion/models/minimax_h3/test_minimax_h3_packing.py, tests/diffusion/cache/test_teacache_extractors.py, "PR #5500", "vllm_omni/diffusion/models/ltx2/ltx2_transformer.py", "PR #6070", "vllm_omni/diffusion/attention/backends/cudnn_attn.py", "PR #5614", "PR #5194", "vllm_omni/diffusion/models/hidream_o1_image/hidream_o1_image_transformer.py", "vllm_omni/diffusion/models/hidream_o1_image/pipeline_hidream_o1_image.py", "PR #6181", "vllm_omni/diffusion/cache/teacache/extractors.py", "vllm_omni/diffusion/models/longcat_image/pipeline_longcat_image.py", "vllm_omni/diffusion/models/longcat_image/pipeline_longcat_image_edit.py", "PR #5717", "PR #6871"] +sources: ["PR #5543", "PR #5866", "PR #5887", "PR #5891", "PR #5897", "PR #5997", "PR #6000", "PR #6037", "PR #6518", "PR #6563", "PR #6724", "PR #6909", docs/design/feature/skip_softmax.md, docs/user_guide/diffusion/attention_backends.md, docs/user_guide/diffusion/attention_backends/trtllm.md, docs/user_guide/diffusion/attention_backends/fastvideo_vsa.md, docs/user_guide/diffusion/attention_backends/rainfusion.md, vllm_omni/config/omni_config.py, vllm_omni/config/stage_config.py, vllm_omni/diffusion/attention/backends/abstract.py, vllm_omni/diffusion/attention/backends/fastvideo_vsa.py, vllm_omni/diffusion/attention/backends/flash_attn.py, vllm_omni/diffusion/attention/backends/rainfusion_attn.py, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/diffusion_kv/paged_attention_adapter.py, vllm_omni/diffusion/models/minimax_h3/denoise_loop.py, vllm_omni/diffusion/models/minimax_h3/packed_sequence.py, vllm_omni/diffusion/data.py, vllm_omni/engine/arg_utils.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/cli/serve.py, vllm_omni/platforms/cuda/platform.py, vllm_omni/platforms/npu/platform.py, tests/config/test_omni_config.py, tests/diffusion/attention/test_fastvideo_vsa.py, tests/diffusion/attention/test_flash_attn.py, tests/diffusion/attention/test_attention_config.py, tests/diffusion/attention/test_piecewise_attn.py, tests/diffusion/attention/test_rainfusion_plan.py, tests/diffusion/attention/test_ulysses_uaa.py, tests/diffusion/diffusion_kv/test_paged_attention_adapter.py, tests/diffusion/models/minimax_h3/test_minimax_h3_packing.py, tests/diffusion/cache/test_teacache_extractors.py, "PR #5500", "vllm_omni/diffusion/models/ltx2/ltx2_transformer.py", "PR #6070", "vllm_omni/diffusion/attention/backends/cudnn_attn.py", "PR #5614", "PR #5194", "vllm_omni/diffusion/models/hidream_o1_image/hidream_o1_image_transformer.py", "vllm_omni/diffusion/models/hidream_o1_image/pipeline_hidream_o1_image.py", "PR #6181", "vllm_omni/diffusion/cache/teacache/extractors.py", "vllm_omni/diffusion/models/longcat_image/pipeline_longcat_image.py", "vllm_omni/diffusion/models/longcat_image/pipeline_longcat_image_edit.py", "PR #5717", "PR #6871", "PR #7185", "PR #7324", "PR #7047", "PR #5691"] confidence: high --- @@ -217,3 +217,37 @@ confidence: high - 强制：仅当 `seq_lengths` 存在真实 padding（至少一档短于 `max(seq_lengths)`）时才物化 `(batch, max_len)` bool mask；单样本或全员等长 batch 必须返回 `None`，让 backend 直接走 dense path。all-true CUDA mask 会触发 `torch.any(~mask)` 的 device→host sync，不能当作“无害 no-op”。 - 禁止：无条件 `new_zeros` + 填 True；把 all-true device mask 交给 FlashAttention 只为“形状完整”；把省略 mask 说成改变了可变长 batch 的有效 token 集合。 - 验收：等长与单样本断言 helper/`forward` 得到 `None`；可变长断言 mask shape 与 True/False 边界；不得用一次端到端加速数字代替 sync 合同。^[PR #6871] + +## DIFF-1ai3 — 显式 Hub FlashAttention 必须在选择时解析构建并失败可见 + +- 触发：修改 `FLASH_ATTN_HUB` / `FLASH_ATTN_3_HUB` 选择、`flash_attn_hub._load_hub_module` / `_get_hub_module`、`kernels` pin，或依赖 Hub FA 的 accuracy/serving 路径。 +- 强制：显式 Hub backend 在 platform 返回 path 前必须走与执行相同的 loader：对 `kernels-community/flash-attn2` 或 `flash-attn3` 依次尝试 `get_kernel(..., version=1)` 再 `version=2`；任一命中即缓存。variant miss 或加载失败必须 `RuntimeError`，说明无兼容 torch/CUDA build，并提示 `kernels>=0.16.1`（含 stable-ABI 匹配）或改选其他 backend。`requirements` 与 LTX/NATTEN remediation 文案中的 `kernels` / `transformers` floor 必须与该 loader 所需 companion pin 同步（当前为 `kernels==0.16.1` 与 `transformers>=5.13.0,<5.15`）。 +- 禁止：显式 Hub 选择在 miss 后静默落到 native/SDPA；preflight 只试 `version=1` 或再试无 version 的第三态；把 accuracy 探针在 Hub 不可用时双方改跑 SDPA 仍当作 FA3-vs-FA3 门禁；把 import/`has_flash_attn` 成功误当成 Hub build 已解析。 +- 验收：CPU 覆盖 v1 miss→v2 hit 的版本序列、双 Hub backend 的 variant miss 均 raise、显式选择无 silent fallback；真实 torch/CUDA 环境再证明 stable-ABI 或 exact-tag build 可加载。不得用 SDPA 噪声 SSIM 解释 Hub miss。^[PR #7185] + +## DIFF-1aj2 — NPU dense FlashAttention 在 `causal=True` 时必须物化因果掩码 + +- 触发：修改 `FlashAttentionImpl.forward_fa_npu`、NPU MindIE-SD / `npu_fusion_attention` dense path，或 `self.causal` 语义。 +- 强制：`causal=True` 且无显式 mask 时，必须走 bottom-right 对齐的压缩/物化因果 mask（`Sq!=Skv` 时按 `key_pos <= query_pos + (Skv-Sq)`），不能把 `attn_mask=None` 交给非因果 dense op。显式 keep-mask 须与因果条件组合后 contiguous 下发；`causal=False` 保持原无 mask 行为。 +- 禁止：依赖 MindIE-SD 隐式因果参数；让因果请求静默变成双向 attention；只靠“推理未崩”证明因果语义。 +- 验收：CPU/mock 覆盖 `Sq==Skv`、更短/更长 query、显式 mask 组合与非因果 `None`；目标 NPU 用已知因果模型对照。^[PR #7324] + +## DIFF-1ai4 — Ring Attention 的 `valid_kv_length` 是全局前缀，且不得与 causal 并用 + +- 触发：修改 Ring Flash/PyTorch attention、`AttentionMetadata.valid_kv_length`、`ring_kv_block_valid_length`，或向 ring forward 传入前缀裁剪语义。 +- 强制：`valid_kv_length` 表示整条全局 K/V 序列的 contiguous 有效前缀（`0 < L <= block_size * world_size`）；每个 circulated block 按全局 `block_rank` 映射为本地 `max(0, min(block_size, L - block_start))`，再 `k/v[:, :local]`。`None` 表示整块有效。本地长度为 0 时跳过该 step 的 FA/LSE 更新，不得对空 K/V 调 kernel。 +- 强制：`causal=True` 且 `valid_kv_length is not None` 必须立即 `ValueError`。裁短 circulated K/V 会缩短 `seqlen_k` 而 query 仍是 padded 长度，FlashAttention causal 对角线 bottom-right 对齐会静默错位；caller 须先 unpad 再进 ring，或使用 `causal=False`。 +- 禁止：把 `valid_kv_length` 当成“本 rank 本地长度”直接切片；在 causal ring 路径上静默 trim；用 bool/非 int 冒充长度；或把 trim 当成慢路径而不是 unsupported 组合。 +- 验收：覆盖全局前缀落在首块/中块/末块、越界/非 int fail-closed、零长 block 跳过，以及 causal+valid_kv_length 拒绝；local dense backend 仍可直接消费同一全局前缀语义。^[PR #7047] + +## DIFF-1d2 — packed varlen 边界是原子合同，所有并行路径都必须保留 + +- 触发：producer 或 backend 读写 `cu_seqlens_q`、`cu_seqlens_k`、`max_seqlen_q`、 + `max_seqlen_k`，或把带 padding 的 packed 输入送入 Ring/Ulysses/hybrid attention。 +- 强制：四个字段一起生产、转发和消费；每条启用的 parallel/backend 路径必须把相同 + sequence boundary 交给 kernel，无法表达时在执行前明确拒绝。 +- 禁止：只提供部分 metadata；在 Ring 路径丢弃边界；假设已分配但 backend 忽略的 + `attn_mask` 能阻止真实 token 关注 padding 或下一条 packed sequence。 +- 验收：缺任一字段立即失败；两个不同长度样本的 packed 结果分别与独立运行对齐且无 + cross-sequence attention；local 与每个声明支持的 Ring/Ulysses 组合做数值 parity， + 不支持的组合有 fail-fast 测试。 ^[PR #5691] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md b/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md index 8740242d3..a83d56a34 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md @@ -1,10 +1,10 @@ --- title: "Diffusion component lifecycle 规则" created: 2026-09-02 -updated: 2026-09-09 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5720", "PR #5853", "PR #5882", "PR #5884", "PR #6486", "PR #6591", vllm_omni/diffusion/cache/base.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/models/interface.py, vllm_omni/diffusion/offloader/module_collector.py, vllm_omni/diffusion/offloader/startup.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/interface.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/models/sana_video/test_cache_offload.py, tests/diffusion/test_diffusion_model_runner.py, tests/diffusion/test_diffusion_scheduler.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_recipes.py", "PR #6072", "PR #7078"] +sources: ["PR #5720", "PR #5853", "PR #5882", "PR #5884", "PR #6486", "PR #6591", vllm_omni/diffusion/cache/base.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/models/interface.py, vllm_omni/diffusion/offloader/module_collector.py, vllm_omni/diffusion/offloader/startup.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/interface.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/models/sana_video/test_cache_offload.py, tests/diffusion/test_diffusion_model_runner.py, tests/diffusion/test_diffusion_scheduler.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_recipes.py", "PR #6072", "PR #7078", "PR #7047"] confidence: high --- @@ -126,3 +126,11 @@ residency 留在模型 owner。规则入口与其他共享机制仍见 [Diffusio - 强制：凡 pipeline 用 “omitted vs explicit” provided 标志做 batch-local gate 的字段，都必须进入 request-mode 与 step-mode 的 sampling-params key；数值相等但 provided 不同的请求不得合批。当前合同要求 `guidance_scale_2_provided` 与数值 `guidance_scale_2` 一并参与 key。 - 禁止：只比较 auto-fill 后的数值；假设同数值即同语义；只更新 request scheduler 而漏掉 step scheduler；把 H3 等 single-request guard 当作可合批模型的例外模板。 - 验收：用真实 key builder 覆盖 omitted `guidance_scale_2` 与显式同值、数值不同，以及两套 scheduler 的 key 分离；并断言 pipeline 首请求 gate 不会跨不同 provided 语义共享 batch。^[PR #7078] + +## DIFF-2ag — pipeline 自管 staging 的组件不得进入 always-resident discovery 列表 + +- 触发：修改 `_vae_modules` / `ModuleDiscovery`、layerwise/HSDP/model-level offload 的 component 选择，或 pipeline 自己在 encode/decode 边界调用 `load_to_device`/`offload_to_cpu`。 +- 强制：`SupportsComponentDiscovery` 的 `_vae_modules` 语义是“VAE(s) always on GPU”。凡由 pipeline 自行互斥 staging、且不能交给 offloader/loader 搬迁的 VAE，必须从 discovery 列表省略（声明为空），使 layerwise/HSDP/model-level 路径都看不到它们；需要保留的 `pipeline.vae` 别名只服务 registry/patch，不算 discovery 声明。 +- 强制：因此退出 discovery 的组件，model-level CPU offload 入口不得再从 discovery 推导 stage 集，必须显式列出仍由 model-level 管理的 encoder/VAE 等；on-demand offload plan 同步丢掉已省略组件。 +- 禁止：一边在 pipeline 里互斥 staging，一边把同一 VAE 放进 `_vae_modules` 让 offloader/loader 再拉回 device；或用后端特殊分支掩盖 discovery 合同冲突。 +- 验收：discovery/offload plan 断言 pipeline-staged VAE 不出现；其余仍声明 VAE 的 pipeline 合同不变；model-level 显式 stage 列表仍覆盖应有组件。^[PR #7047] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md b/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md index e73e11ae0..d0278cb3e 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md @@ -1,10 +1,10 @@ --- title: "Diffusion LoRA 规则" created: 2026-09-02 -updated: 2026-09-09 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #2783", docs/user_guide/diffusion/lora.md, vllm_omni/config/omni_config.py, vllm_omni/config/stage_config.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/lora/loader.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/lora/layers/base_linear.py, vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2_i2v.py, vllm_omni/diffusion/utils/tf_utils.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/cli/serve.py, tests/diffusion/lora/test_loader.py, tests/diffusion/lora/test_lora_manager.py, tests/entrypoints/test_async_omni_diffusion_config.py, "PR #5500", "vllm_omni/diffusion/models/ltx2/ltx2_adapter_parser.py", "vllm_omni/diffusion/models/ltx2/ltx2_phase_adapter.py", "PR #6070", "PR #6476", "PR #6550", vllm_omni/diffusion/models/minimax_h3/lora.py, "PR #6268", benchmarks/kernels/benchmark_diffusion_lora_expand.py, tests/diffusion/lora/test_base_linear.py, "PR #7195"] +sources: ["PR #2783", docs/user_guide/diffusion/lora.md, vllm_omni/config/omni_config.py, vllm_omni/config/stage_config.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/lora/loader.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/lora/layers/base_linear.py, vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2_i2v.py, vllm_omni/diffusion/utils/tf_utils.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/cli/serve.py, tests/diffusion/lora/test_loader.py, tests/diffusion/lora/test_lora_manager.py, tests/entrypoints/test_async_omni_diffusion_config.py, "PR #5500", "vllm_omni/diffusion/models/ltx2/ltx2_adapter_parser.py", "vllm_omni/diffusion/models/ltx2/ltx2_phase_adapter.py", "PR #6070", "PR #6476", "PR #6550", vllm_omni/diffusion/models/minimax_h3/lora.py, "PR #6268", benchmarks/kernels/benchmark_diffusion_lora_expand.py, tests/diffusion/lora/test_base_linear.py, "PR #7195", "PR #7349", "PR #5907"] confidence: high --- @@ -137,3 +137,24 @@ confidence: high - 强制：deactivate 只 `suspend_lora()`（保存并清零 `_diffusion_lora_active_slices`），保留 stacked A/B；同一 adapter_id 与同一 rounded scale 再激活时 `resume_lora()` 复原掩码，禁止整层 `reset_lora`+rebind。`_suspended_adapter_id` 在真实 reset、rank 驱动的 `create_lora_weights`、新 bind 开始、以及 remove 被 suspend 的 adapter 时必须清空；不同 adapter 或不同 scale 仍走完整 rebind。 - 禁止：把 suspend 当成可丢弃的 upload；resume 覆盖后注册的新 layer 的 inactive mask；在失败 bind 后仍保留旧 suspended 身份。 - 验收：CPU 覆盖 suspend/resume、同 id+scale 不 re-upload、不同 scale/id 走 rebind、remove suspended 触发 reset、中途失败后 inactive，以及 add_adapter 后新 layer 不被错误 resume。^[PR #7195] + +## DIFF-2ag2 — 绑定零层的 diffusion LoRA adapter 必须失败而不是静默 no-op + +- 触发：修改 `DiffusionLoRAManager._bind_adapter_weights` / `_activate_adapter`、 + binding 记账，或 pipeline 可选 `_validate_diffusion_lora_binding` hook。 +- 强制：无论 pipeline 是否提供 binding validator，绑定过程都必须无条件记录实际命中的 + module 名；`bound_lora_names` 为空时立即 `ValueError`，错误信息同时列出 expected + target modules 与 adapter 收到的 module 名。既有 activate 的 try/except 必须 reset 全部 + wrapper 并保持 inactive，不得留下“已激活但等于 base”的状态。 +- 禁止：仅在有 validator 时才记账；让零绑定激活返回成功或只打 DEBUG；把 diffusers 命名 + checkpoint 对不上 engine layout 的情况当成有效 adapter。 +- 验收：CPU 回归用故意不匹配的 `target_modules` 断言激活抛错、消息含收到的模块名、 + active id 仍为 `None` 且无 `set_lora`；有 validator 的模型路径仍覆盖完整 binding + completeness。^[PR #7349] + +## DIFF-2ag3 — LoRA/ModelOpt 默认只保留通用组件名与通用 fused 映射 + +- 触发：修改 `DiffusionLoRAManager` 默认扫描组件、ModelOpt `DEFAULT_PACKED_MODULES_MAPPING`，或 pipeline/model 的 `_lora_components` / `_dit_modules` / `packed_modules_mapping`。 +- 强制：manager 默认只扫描通用 diffusers 名 `transformer`、`transformer_2`、`unet`；模型专用属性（如 Bagel 的 `bagel`）必须由 pipeline 经 `_dit_modules` 或 `_lora_components` 声明。ModelOpt 默认只保留通用 attention 融合（`to_qkv`、`add_kv_proj`）；模型专用融合（如 Z-Image `w13`）必须写在模型 `packed_modules_mapping` 并由 adapter 合并。 +- 禁止：把 `dit`/`bagel`/`w13` 等模型私有名硬编码进共享默认；在未扫描真实 denoiser 时宣称 adapter 已生效（零层绑定、输出等同 base）。 +- 验收：Bagel/同类测试 pipeline 必须显式 `_lora_components` 才能发现层；Z-Image 映射在模型侧声明后仍可解析；去掉声明后不得靠框架默认偷跑。^[PR #5907] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-output-lifecycle.md b/knowledge/repos/vllm-omni/components/diffusion/rules-output-lifecycle.md index 12223046b..e3ce91e77 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-output-lifecycle.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-output-lifecycle.md @@ -1,10 +1,10 @@ --- title: "Diffusion output 与 multiprocess runtime 规则" created: 2026-09-02 -updated: 2026-09-08 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5550", "PR #5864", "PR #5885", "PR #5978", "PR #6750", vllm_omni/diffusion/diffusion_engine.py, vllm_omni/diffusion/executor/multiproc_executor.py, vllm_omni/diffusion/inline_stage_diffusion_client.py, vllm_omni/diffusion/io_support.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/output_formatter.py, vllm_omni/diffusion/ipc.py, vllm_omni/diffusion/sched/request_scheduler.py, vllm_omni/diffusion/stage_diffusion_proc.py, vllm_omni/diffusion/utils/media_utils.py, vllm_omni/diffusion/worker/diffusion_worker.py, tests/diffusion/test_async_output_timeout.py, tests/diffusion/test_async_output_worker.py, tests/diffusion/test_diffusion_engine.py, tests/diffusion/test_diffusion_engine_cleanup.py, tests/diffusion/test_diffusion_ipc.py, tests/diffusion/test_inline_stage_diffusion_client.py, tests/diffusion/test_ipc_async.py, tests/diffusion/test_multiproc_engine_concurrency.py, tests/diffusion/test_result_pump.py, tests/diffusion/test_stage_diffusion_proc.py, tests/entrypoints/openai_api/test_video_server.py, "PR #6023", "PR #5983", "PR #4222", "PR #6094", "PR #6255", "PR #6288", "PR #6308", "PR #6499", "PR #6749", "PR #6847", "PR #6953", vllm_omni/diffusion/data.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/models/diffusers_adapter/pipeline_utils.py, tests/diffusion/test_diffusion_output_formatter.py, tests/diffusion/test_diffusion_plugin_hooks.py, tests/entrypoints/openai_api/test_video_pipeline_capability.py, "PR #6294"] +sources: ["PR #5550", "PR #5864", "PR #5885", "PR #5978", "PR #6750", vllm_omni/diffusion/diffusion_engine.py, vllm_omni/diffusion/executor/multiproc_executor.py, vllm_omni/diffusion/inline_stage_diffusion_client.py, vllm_omni/diffusion/io_support.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/output_formatter.py, vllm_omni/diffusion/ipc.py, vllm_omni/diffusion/sched/request_scheduler.py, vllm_omni/diffusion/stage_diffusion_proc.py, vllm_omni/diffusion/utils/media_utils.py, vllm_omni/diffusion/worker/diffusion_worker.py, tests/diffusion/test_async_output_timeout.py, tests/diffusion/test_async_output_worker.py, tests/diffusion/test_diffusion_engine.py, tests/diffusion/test_diffusion_engine_cleanup.py, tests/diffusion/test_diffusion_ipc.py, tests/diffusion/test_inline_stage_diffusion_client.py, tests/diffusion/test_ipc_async.py, tests/diffusion/test_multiproc_engine_concurrency.py, tests/diffusion/test_result_pump.py, tests/diffusion/test_stage_diffusion_proc.py, tests/entrypoints/openai_api/test_video_server.py, "PR #6023", "PR #5983", "PR #4222", "PR #6094", "PR #6255", "PR #6288", "PR #6308", "PR #6499", "PR #6749", "PR #6847", "PR #6953", vllm_omni/diffusion/data.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/models/diffusers_adapter/pipeline_utils.py, tests/diffusion/test_diffusion_output_formatter.py, tests/diffusion/test_diffusion_plugin_hooks.py, tests/entrypoints/openai_api/test_video_pipeline_capability.py, "PR #6294", "PR #6615", "PR #7126", "PR #7198"] confidence: high --- @@ -176,3 +176,44 @@ confidence: high - 强制：runner 与 pipeline 共有同一 `InteractionCoordinator`；仅对已注册 modality 构造 handler；`enqueue` 按 modality 路由；`apply_at_chunk_boundary` 以稳定顺序推进（prompt 优先，再及其他 modality）；per-request session 落在 `StepRequestState.interaction_sessions`，chunk ACK 元数据写入 `interaction_chunk_metadata`。未注册 modality 必须 fail closed。当前公开行为保持 prompt-only，直到另有 PR 注册并验收非 prompt handler。 - 禁止：在 denoise 中途旁路 coordinator 直接改 `prompt_embeds`；在非 chunk 边界静默应用 interaction；未注册就接受 camera/`multi_modal_data`；或把内部 handler 重构当成用户 API 已扩展。 - 验收：覆盖 prompt enqueue→chunk-boundary apply→ACK event id、未注册 modality 拒绝，以及无 session 的 modality 跳过；不得用本框架落地证明 camera/MM 路径已可用。^[PR #6294] + +## DIFF-1ai — 迁移后的视频输出必须以 typed `media` 为唯一能力信号 + +- 触发：修改 `DiffusionOutput`、device-side video postprocess、`video_output_transport`、 + runner 预 D2H 准备、worker IPC media envelope 或 engine finalizer 路由。 +- 强制：迁移动画输出以 `DiffusionOutput.media`（`DiffusionMediaOutput`）为唯一能力信号， + 与 legacy `output` 以及 model-specific `post_process_func` 互斥。runner 在 normalize/ + request-split 之后、异步 D2H 之前完成 request-owned 准备：合格视频转 contiguous uint8 + `BTHWC`，或按策略保留已标 transport-ready 的 float；IPC 只接受 transport-ready 的 + 版本化 `diffusion_media_v1` envelope；engine 走通用 media finalizer，绕过 legacy + model postprocessor。`to_cpu=True` 前 media 必须已 prepared。feature 默认关闭，经 + `video_output_transport.enable_device_postprocess` 显式开启。合同违规 fail loud； + OOM 可回退到 owned float 并告警，其他 runtime 错误仍传播。未迁移模型保持 legacy path。 +- 禁止：对已迁移路径同时填充 `media` 与 legacy `output`；未准备就 D2H/to_cpu；把 HTTP + sink、codec 策略或未迁移模型静默当作同一 Phase-1 合同。 +- 验收：正负覆盖互斥字段、未 prepared `to_cpu`、IPC 拒收未就绪 media、opt-in 开关与 + OOM float fallback；至少一条迁移模型（如 WAN2.2 非 latent）走 typed path，一条未迁移 + 模型仍走 legacy。^[PR #6615] + +## DIFF-1aj — multiproc worker shutdown 必须升至 SIGKILL 并保留幸存者供重试 + +- 触发：修改 `_ExecutorShutdownCleaner`、diffusion worker process join/terminate/kill、 + worker monitor 与显式 `shutdown()` 的竞态，或 cleaner 持有的 process 引用生命周期。 +- 强制：cleanup 按有界三阶段推进：cooperative join → `terminate`+join → `kill`+join； + 信号/join 的 `OSError` 不得中断 peer cleanup。最终仍存活的进程必须记 error（含 name/pid） + 并保留在 cleaner 上，供下一次显式 shutdown 重试，不得在第一次失败后丢弃引用。 + cleaner 用非阻塞锁，防止 monitor 与显式 shutdown 并发操作同一批 `Process`；二次进入 + 直接返回。`broadcast_mq` 在 enqueue 成败后都要清空；result-pump/pending-future 清理 + 仍在 `finally`。 +- 禁止：只做 graceful+SIGTERM 后假定已退出；失败后清空 process 列表导致无法再 reap； + 把存活进程当作 GPU 内存已回收或 HWR/服务自动重启已完成。 +- 验收：真实 CPU child（含忽略 SIGTERM）在 queue 失败时经 SIGKILL 退出并被 reap；合作 + 退出路径 exit 0；覆盖 shared deadline、signal/join OSError 续跑、幸存者保留与并发 + cleaner。^[PR #7126] + +## DIFF-1ai2 — 复合 mid-stream interaction 必须原子入队，非懒会话在 chunk0 前物化 + +- 触发：修改 `InteractionCoordinator.enqueue_parts`、`lazy_initialize_session`、camera/`multi_modal_data` 与 prompt 同事件入队、`synchronized_monotonic_time`，或 chunk-boundary apply 的 media/latent 计数。 +- 强制：含多 modality 的同一事件先全部 `validate_payload`，再入队；任一路不支持或非法不得部分改队列。`received_at`/boundary 时钟经 rank 同步，使 USP/SP 分片共享同一到达时刻。`lazy_initialize_session=False` 的 handler（如 camera）必须在首个 denoise chunk 前 `maybe_prepare_initial_session`，即使尚无 client enqueue。chunk apply 区分 `num_media_frames` 与 `num_latent_frames`，需要者才 `peek_chunk_media`。 +- 禁止：先入队再校验导致半更新；把未注册 camera 当已可用；懒会话与必须预创建会话混用同一 skip 逻辑；用未同步的 per-rank `time.monotonic()` 驱动跨分片时间线。 +- 验收：复合 prompt+camera 原子成功/失败、未注册拒绝、非懒会话在 chunk0 前存在、介质计数字段传播。^[PR #7198] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md b/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md index 9a7c5b079..d1dc9a120 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md @@ -1,10 +1,10 @@ --- title: "Diffusion 平台 kernel 与设备合同" created: 2026-07-20 -updated: 2026-09-07 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #4341", "PR #5001", "PR #5087", "PR #5088", "PR #5136", "PR #5255", "PR #5344", "PR #5543", "PR #5720", "PR #5737", "PR #5764", "PR #5801", "PR #5802", "PR #5838", "PR #5839", "PR #5848", "PR #5872", "PR #5881", "PR #5896", "PR #5981", "PR #6094", "PR #6102", "PR #6279", "PR #6385", "PR #6445", "PR #6651", "PR #6722", "PR #5831", "PR #6989", vllm_omni/diffusion/attention/backends/flashinfer_attn.py, vllm_omni/diffusion/attention/backends/ring/ring_kernels.py, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/distributed/hsdp.py, vllm_omni/diffusion/executor/multiproc_executor.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/model_loader/diffusers_loader.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/offloader/, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/quantization/component_config.py, vllm_omni/quantization/factory.py, tests/diffusion/attention/test_attention_sp.py, tests/diffusion/attention/test_ulysses_uaa.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/offloader/test_distributed_layerwise_backend.py, tests/diffusion/test_diffusion_config_propagation.py, tests/diffusion/test_multiproc_engine_concurrency.py, "PR #4755", "PR #5990", "vllm_omni/diffusion/layers/fused_qk_norm_rope.py", "vllm_omni/diffusion/cache/teacache/extractors.py", "vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py", "tests/diffusion/layers/test_fused_qk_norm_rope.py", "PR #6165", "PR #5677", "vllm_omni/diffusion/quantization/hsdp_fp8.py", "tests/diffusion/quantization/test_hsdp_fp8.py", "PR #4845", "PR #6173", "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_components.py", "vllm_omni/diffusion/model_loader/hub_prefetch.py", "PR #5910", "PR #5676", "PR #6125", "PR #6320", "PR #5877", "vllm_omni/diffusion/vllm_config.py", "PR #6283", "vllm_omni/diffusion/layers/activation.py", "PR #6281", "vllm_omni/diffusion/attention/ops/minimax_h3_modulation.py", "PR #6130", "PR #6364", "PR #6073", "PR #4820", vllm_omni/diffusion/attention/backends/fastvideo_vsa.py, "PR #6150", "PR #6410", tests/diffusion/layers/test_activation.py, tests/diffusion/layers/test_fused_qk_norm_rope_npu.py] +sources: ["PR #4341", "PR #5001", "PR #5087", "PR #5088", "PR #5136", "PR #5255", "PR #5344", "PR #5543", "PR #5720", "PR #5737", "PR #5764", "PR #5801", "PR #5802", "PR #5838", "PR #5839", "PR #5848", "PR #5872", "PR #5881", "PR #5896", "PR #5981", "PR #6094", "PR #6102", "PR #6279", "PR #6385", "PR #6445", "PR #6651", "PR #6722", "PR #5831", "PR #6989", vllm_omni/diffusion/attention/backends/flashinfer_attn.py, vllm_omni/diffusion/attention/backends/ring/ring_kernels.py, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/distributed/hsdp.py, vllm_omni/diffusion/executor/multiproc_executor.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/model_loader/diffusers_loader.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/offloader/, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/quantization/component_config.py, vllm_omni/quantization/factory.py, tests/diffusion/attention/test_attention_sp.py, tests/diffusion/attention/test_ulysses_uaa.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/offloader/test_distributed_layerwise_backend.py, tests/diffusion/test_diffusion_config_propagation.py, tests/diffusion/test_multiproc_engine_concurrency.py, "PR #4755", "PR #5990", "vllm_omni/diffusion/layers/fused_qk_norm_rope.py", "vllm_omni/diffusion/cache/teacache/extractors.py", "vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py", "tests/diffusion/layers/test_fused_qk_norm_rope.py", "PR #6165", "PR #5677", "vllm_omni/diffusion/quantization/hsdp_fp8.py", "tests/diffusion/quantization/test_hsdp_fp8.py", "PR #4845", "PR #6173", "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_components.py", "vllm_omni/diffusion/model_loader/hub_prefetch.py", "PR #5910", "PR #5676", "PR #6125", "PR #6320", "PR #5877", "vllm_omni/diffusion/vllm_config.py", "PR #6283", "vllm_omni/diffusion/layers/activation.py", "PR #6281", "vllm_omni/diffusion/attention/ops/minimax_h3_modulation.py", "PR #6130", "PR #6364", "PR #6073", "PR #4820", vllm_omni/diffusion/attention/backends/fastvideo_vsa.py, "PR #6150", "PR #6410", tests/diffusion/layers/test_activation.py, tests/diffusion/layers/test_fused_qk_norm_rope_npu.py, "PR #6982"] confidence: high --- @@ -40,3 +40,9 @@ confidence: high - 禁止：只用 `x.is_cpu` 作为 Triton 门禁；让 NPU、XPU、MUSA 或其他非 CUDA tensor 进入 raw Triton launch；认为传入 row stride 就足以支持非连续列；对错误布局静默读写；或用 mock/wiring 结果宣称真实 kernel 数值 parity。 - 验收：CPU/mock 覆盖 CUDA、NPU、XPU、MUSA、unsupported dtype/shape、空行和 contiguous/strided 输入，断言非 CUDA 路径不启动 Triton 且 NPU 保留 native RMSNorm；CUDA 目标硬件以 FP32 reference 核对值、dtype 和容差，并单独记录 kernel 数量/延迟。当前 PR 未新增该模块测试且 NPU CI 不覆盖 H3，在补齐前不得称为生产级跨平台支持。^[PR #6281] +## DIFF-1ai5 — fused_qk_norm_rope 的 interleaved 模式与 token gate 必须分模式闭合 + +- 触发：修改共享 `fused_qk_norm_rope`、packed RoPE table、`interleaved` 分派、Boogu/MiniMax consumer，或 `VLLM_OMNI_FUSED_QK_NORM_ROPE_MIN_TOKENS`。 +- 强制：`interleaved=False` 保持 half-split `(d, d+rotary_dim/2)` 与既有 MiniMax-H3 fused geometry（`head_dim==128`、`rotary_dim==96`）；`interleaved=True` 使用相邻对 `(2i, 2i+1)`，CUDA Triton 仅在 even `rotary_dim <= head_dim <= 256` 时融合，否则 eager。consumer 的 packed `[cos|sin]` table 必须匹配所选 pairing，不得把 half-split 与 adjacent-pair frequency 混用。融合路径的 token 门槛由 op 级 env 覆盖；短于门槛时整条 eager 链必须 bit-exact。新增 env 必须进入 public inventory 与 docs。 +- 禁止：把 interleaved 默认塞进 MiniMax half-split path；无门槛强制融合；只改一个 consumer 却不更新共享 op 合同或 inventory。 +- 验收：分别覆盖 half-split 与 interleaved 的 fused/eager 数值、geometry 拒绝、门槛上下切换，以及 env 缺省/覆盖；Boogu 集成断言 table 去重与 `[B*S, D]` 复用。^[PR #6982] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md b/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md index 6bec90c78..93a17b36e 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md @@ -1,10 +1,10 @@ --- title: "Diffusion paged cache 与系统运行时规则" created: 2026-09-03 -updated: 2026-09-09 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5255", "PR #5344", "PR #5543", "PR #5838", "PR #6094", "PR #6102", "PR #6385", "PR #6340", "PR #6714", "PR #6814", "PR #6563", "PR #5716", "PR #6786", vllm_omni/diffusion/attention/, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/attention/parallel/ring_kernels.py, vllm_omni/diffusion/diffusion_kv/, vllm_omni/diffusion/distributed/cfg_parallel.py, vllm_omni/diffusion/distributed/parallel_state.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/platforms/interface.py, vllm_omni/platforms/npu/platform.py, tests/diffusion/diffusion_kv/, tests/diffusion/distributed/test_cfg_parallel.py, tests/diffusion/attention/test_piecewise_attn.py, tests/diffusion/attention/test_ulysses_uaa.py, "PR #5491", "PR #5194", "vllm_omni/diffusion/data.py", "vllm_omni/diffusion/utils/hf_utils.py", "PR #7041", "PR #6463", "PR #6844"] +sources: ["PR #5255", "PR #5344", "PR #5543", "PR #5838", "PR #6094", "PR #6102", "PR #6385", "PR #6340", "PR #6714", "PR #6814", "PR #6563", "PR #5716", "PR #6786", vllm_omni/diffusion/attention/, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/attention/parallel/ring_kernels.py, vllm_omni/diffusion/diffusion_kv/, vllm_omni/diffusion/distributed/cfg_parallel.py, vllm_omni/diffusion/distributed/parallel_state.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/platforms/interface.py, vllm_omni/platforms/npu/platform.py, tests/diffusion/diffusion_kv/, tests/diffusion/distributed/test_cfg_parallel.py, tests/diffusion/attention/test_piecewise_attn.py, tests/diffusion/attention/test_ulysses_uaa.py, "PR #5491", "PR #5194", "vllm_omni/diffusion/data.py", "vllm_omni/diffusion/utils/hf_utils.py", "PR #7041", "PR #6463", "PR #6844", "PR #7548"] confidence: high --- @@ -226,3 +226,10 @@ confidence: high - 强制：一次 `generate()` 驱动整段 rollout：`prepare_encode` 只跑一次，随后多轮 `denoise_step`/`step_scheduler`，由 `post_decode` 产出每个 AR chunk。request-mode 与 stepwise 必须共用同一份 block/DMD math，禁止维护两套可漂移实现。runner 仅为实现该合同的 pipeline 打开 `step_execution`；绑定 runner-owned KV 时 `session_id == request_id`；错误路径 fail-closed 释放 session，完成或 scheduler abort 时退役。持有 AR-Diffusion paged KV 的 stage 必须保持 `max_num_seqs=1`。 - 禁止：用多次 `generate()`/tick 冒充同一 session；在 denoise step 内提交本应属于 `post_decode` 的 clean-x0 KV commit；或把逐步 camera/prompt mid-request interaction 写成已由本合同覆盖。 - 验收：覆盖 stepwise 与 request-mode 的共享 math 边界、session bind/release、abort/completion 退役，以及 `max_num_seqs=1` 拓扑；request-scoped camera script 等模型字段另由模型 owner 验收，不得外推为通用 mid-request interaction。^[PR #6844] + +## DIFF-4ad — 无法合成观测的 pipeline 必须退出通用文本 warmup + +- 触发：修改 `DiffusionEngine.run_startup_warmup`、`dummy_run_num_frames`，或新增必须携带非文本观测（如 `robot_obs`）的 diffusion pipeline。 +- 强制：通用 startup warmup 是文本、两步请求。pipeline 无法从该请求合成所需观测时，必须声明 `dummy_run_num_frames: ClassVar[int] = 0`，使 `run_startup_warmup` 不提交请求、也不因此 `close`。模型自有 warmup（如 AR-Diffusion 的 `ar_diffusion_warmup_requests`）保持可用。能接受文本 dummy 的模型仍走两步通用 warmup。 +- 禁止：把 DreamZero/LingBot 的 opt-out 扩成所有模型跳过 warmup；用一步 shortcut 让非法文本请求穿过观测检查；opt-out 后仍断言 engine 被关闭。 +- 验收：经真实 registry 查找，DreamZero 与 LingBot 的 startup 不调用 `add_req_and_wait_for_response` 且不 `close`；其余模型的两步 dummy 合同不变。^[PR #7548] diff --git a/knowledge/repos/vllm-omni/components/distributed/_index.md b/knowledge/repos/vllm-omni/components/distributed/_index.md index 0c98510e6..34df5642b 100644 --- a/knowledge/repos/vllm-omni/components/distributed/_index.md +++ b/knowledge/repos/vllm-omni/components/distributed/_index.md @@ -1,7 +1,7 @@ --- title: "Distributed（跨 stage 通信与数据搬运）" created: 2026-07-16 -updated: 2026-09-06 +updated: 2026-09-22 type: index tags: [vllm-omni, components, distributed] sources: [vllm_omni/distributed/omni_connectors/, vllm_omni/distributed/omni_coordinator/, docs/design/feature/disaggregated_inference.md] @@ -53,5 +53,5 @@ sources: [vllm_omni/distributed/omni_connectors/, vllm_omni/distributed/omni_coo | 已修过的 connector/端口产品坑 | [connector pitfalls](connector-pitfalls.md) | | 选择和配置 connector backend | [connector backends](connector-backends.md) | | 跨 stage `async_chunk` 流式语义 | [async chunk](async-chunk.md) | -| TP KV receive consensus、chunk boundary、active-window 合同与 Mooncake TCP write completion | [distributed rules](rules.md) | +| TP KV receive consensus、chunk boundary、active-window 合同与 Mooncake TCP write completion | [distributed rules](rules.md) 新增核对：DIST-1l、DIST-1l2、DIST-1l3、DIST-1l4。 | diff --git a/knowledge/repos/vllm-omni/components/distributed/rules.md b/knowledge/repos/vllm-omni/components/distributed/rules.md index d693a2d07..f938ba54b 100644 --- a/knowledge/repos/vllm-omni/components/distributed/rules.md +++ b/knowledge/repos/vllm-omni/components/distributed/rules.md @@ -1,10 +1,10 @@ --- title: "Distributed 传输规则" created: 2026-08-05 -updated: 2026-09-08 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, distributed] -sources: ["PR #5744", "PR #5976", "PR #6001", "PR #6089", "PR #6834", vllm_omni/diffusion/distributed/parallel_state.py, tests/diffusion/distributed/test_expert_parallel_layout.py, vllm_omni/distributed/omni_connectors/adapter.py, vllm_omni/distributed/omni_connectors/kv_transfer_manager.py, vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py, vllm_omni/distributed/omni_connectors/transfer_adapter/base.py, vllm_omni/worker/omni_connector_model_runner_mixin.py, tests/distributed/omni_connectors/test_kv_recv_tp_consensus.py, tests/distributed/omni_connectors/test_chunk_transfer_adapter.py, tests/worker/test_omni_connector_mixin.py, "PR #5146", "PR #6021", "PR #6033", "PR #6360", "PR #6406", "PR #6626", "PR #6529", "PR #7136"] +sources: ["PR #5744", "PR #5976", "PR #6001", "PR #6089", "PR #6834", vllm_omni/diffusion/distributed/parallel_state.py, tests/diffusion/distributed/test_expert_parallel_layout.py, vllm_omni/distributed/omni_connectors/adapter.py, vllm_omni/distributed/omni_connectors/kv_transfer_manager.py, vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py, vllm_omni/distributed/omni_connectors/transfer_adapter/base.py, vllm_omni/worker/omni_connector_model_runner_mixin.py, tests/distributed/omni_connectors/test_kv_recv_tp_consensus.py, tests/distributed/omni_connectors/test_chunk_transfer_adapter.py, tests/worker/test_omni_connector_mixin.py, "PR #5146", "PR #6021", "PR #6033", "PR #6360", "PR #6406", "PR #6626", "PR #6529", "PR #7136", "PR #6889", "PR #6093", "PR #7870", "PR #7166"] confidence: high --- @@ -142,3 +142,31 @@ confidence: high - 强制：CUDA Mooncake 依赖必须钉在包含 acknowledged TCP writes 的版本（`0.3.12` 起）；`batch_transfer_sync_write` 返回成功后，destination 内存必须已更新到可读状态，不能只表示本地 send 入队。回归必须在 receiver 被挂起时证明 completion 不会提前返回，恢复后再断言目的端字节一致。 - 禁止：降级到无 ACK 的 TCP 实现却不更新测试合同；把 RDMA 路径的 completion 语义外推到 TCP；或在 completion 前启动依赖 destination 内容的零拷贝消费。 - 验收：强制 TCP 的挂起-恢复测试覆盖「停止期间不完成、恢复后成功且字节匹配」；依赖声明与该测试同步变更。^[PR #7136] + +## DIST-1l — parked async-chunk sender 收到终态 update 必须先投递 terminal chunk + +- 触发：修改 `OmniSchedulerMixin.add_request`、`OmniChunkTransferAdapter.save_async` / sender cleanup，或 realtime/async-chunk 在 `WAITING_FOR_STREAMING_REQ` 上的终态竞态。 +- 强制：当终态 update（`resumable=False`）落在已 parked 且 `has_active_sender` 的 session 上时，不得只走 upstream 的本地 `FINISHED_ABORTED`。必须先 `force_request_finished` 入队携带 `finished` 的 request-terminal chunk（必要时推进 `_omni_segment_generation` 以通过 dedup），再本地 finish。`terminal_pending` 必须挡住丢弃该 chunk 的 cleanup，直到 connector 接受。 +- 禁止：假设 orchestrator 的一次性 StreamingUpdate 对下游 receiver 足够；在 `finish_requests` 之后才尝试 enqueue terminal；让 cleanup 在 terminal 上线前回收 generation，导致下游卡在 `WAITING_FOR_CHUNK` 直至输入超时。 +- 验收：覆盖 parked+active-sender 终态、非 sender/非 parked 回退、`force_request_finished` 顺序、terminal_pending 与 deferred cleanup；证明下游经 payload `finished` 结束，而不是仅靠 600s stall。^[PR #6889] + +## DIST-1l2 — NIXL connector 超时后必须保留 native 资源直到终端态 + +- 触发：新增/修改 `NixlConnector`、NIXL READ、ZMQ claim/`XFER_DONE`、lease 或 receive timeout。 +- 强制：NIXL 无远端 READ 取消时，超时/放弃后仍保留 buffer、registration、descriptor、transfer handle 与 remote agent，直到 native 终端态；`close()` 等待后再释放。源 registration 有界 lease（`lease_seconds` / `VLLM_OMNI_NIXL_LEASE_S`）并由 reaper 回收。跨 stage 请求须把 producer endpoint/`payload_sender_info` 经 scheduler handle 传到 consumer。 +- 禁止：超时后立刻 deregister 或丢弃仍可能被 DMA 写入的内存；把 vLLM P/D block-table NIXL 接口当作 Omni 任意 payload connector；依赖未声明的固定 `zmq_port` 却不经 resolver。 +- 验收：覆盖超时保留、lease 回收、metadata claim/ack，以及 `OmniChunkRecvHandle.payload_sender_info` 贯通；缺 NIXL 依赖时 lazy import 为 None。^[PR #6093] + +## DIST-1l3 — NIXL 所有权测试必须用 reliable_claim_queries 隔离 claim 查询 + +- 触发：修改 NIXL connector 中断言 terminal 或 deferred read ownership、`_wait_for_transfer` 或 `entered.wait` 的测试，或改变 metadata claim 的获取方式。 +- 强制：这类所有权用例必须挂上 `reliable_claim_queries`。fixture 仍覆盖 metadata 解析、wire 编解码、claim 生成和 producer claim 状态机，但不得用一次有界（10 ms）真实 ZMQ query 取得 claim。真实 ZMQ discovery 只由同模块 handshake 测试覆盖。 +- 禁止：让所有权断言依赖 CI 下可能超时的短 query，使 `consumer.get()` 在进入 transfer wait 前返回；或放宽 `entered.wait` 来掩盖未挂 fixture。 +- 验收：`direct` 与 `done`/`error`/`timeout`/`unknown` 在 fixture 下重复通过；所有权用例源码不再走有界 claim query。handshake 测试仍覆盖真实 ZMQ discovery。^[PR #7870] + +## DIST-1l4 — 原生 Mooncake AR→DiT KV 必须关 async_chunk 并按物理页序释放 + +- 触发：部署或修改 `kv_transfer_config`/`MooncakeConnector`、AR→DiT 原生 KV handoff、`OmniARScheduler` 的 block free，或与 legacy `OmniKVTransferManager` 并存的路径。 +- 强制：存在 stage `kv_transfer_config` 时必须 `async_chunk=False`。原生路径由 scheduler 拥有 diffusion KV 页；connector 失败 fail-close diffusion engine 后再拆页。非 caching P/D 池释放时按递增 `block_id` 归还，便于相邻页合并；deferred free 存逆序以匹配 vLLM drain。原生 transfer 边界用已确认 computed tokens，排除 async output placeholders；需要时短暂改写 request status/token 计数并在 `finally` 恢复。 +- 禁止：`async_chunk=True` 搭配原生 KV transfer；把 eviction-priority free 顺序当成无缓存池的物理布局；用含 placeholder 的 token 计数作为 transfer 边界；connector 失败后仍继续拆页当成功。 +- 验收：配置拒绝 async_chunk+kv_transfer；物理序 free；confirmed-token transfer 边界；失败 fail-close。^[PR #7166] diff --git a/knowledge/repos/vllm-omni/components/host-weight-runtime/_index.md b/knowledge/repos/vllm-omni/components/host-weight-runtime/_index.md index 3a9cfd56a..380d7976b 100644 --- a/knowledge/repos/vllm-omni/components/host-weight-runtime/_index.md +++ b/knowledge/repos/vllm-omni/components/host-weight-runtime/_index.md @@ -1,7 +1,7 @@ --- title: "Host Weight Runtime" created: 2026-08-23 -updated: 2026-09-04 +updated: 2026-09-22 type: index tags: [vllm-omni, components] sources: ["PR #6419", "PR #6486", "PR #6591", "PR #6692", vllm_omni/host_weight_runtime/] @@ -24,4 +24,4 @@ confidence: high | 遇到什么 | 查看哪里 | |---|---| -| typed failure、lease/store concurrency、single-take carrier、restore transaction，以及 hardened artifact 的 cleanup/quarantine/publication、deny-marker 或 durability I/O failure | [rules](rules.md) | +| typed failure、lease/store concurrency、single-take carrier、restore transaction，以及 hardened artifact 的 cleanup/quarantine/publication、deny-marker 或 durability I/O failure | [rules](rules.md) 新增核对：HWR-1f。 | diff --git a/knowledge/repos/vllm-omni/components/host-weight-runtime/rules.md b/knowledge/repos/vllm-omni/components/host-weight-runtime/rules.md index 4be763028..59d4018fe 100644 --- a/knowledge/repos/vllm-omni/components/host-weight-runtime/rules.md +++ b/knowledge/repos/vllm-omni/components/host-weight-runtime/rules.md @@ -1,10 +1,10 @@ --- title: "Host Weight Runtime 规则" created: 2026-08-23 -updated: 2026-09-04 +updated: 2026-09-22 type: rule tags: [vllm-omni, components] -sources: ["PR #6419", "PR #6427", "PR #6445", "PR #6486", "PR #6591", "PR #6692", vllm_omni/host_weight_runtime/] +sources: ["PR #6419", "PR #6427", "PR #6445", "PR #6486", "PR #6591", "PR #6692", vllm_omni/host_weight_runtime/, "PR #7128"] confidence: high --- @@ -124,3 +124,18 @@ confidence: high artifact miss。 - 验收：覆盖 capability/budget gate、mid-registration rollback、PyTorch pinned-source verification、 unregister retry 与 retained-owner GC，并断言 successful direct path 无 private staging allocation。^[PR #6591] + +## HWR-1f — domain-init 锁等待必须受 `WaitPolicy` 单调截止约束 + +- 触发：修改 `FilesystemHostWeightStore` 构造、`domain-init.lock`、`WaitPolicy` 向 store + 的传递，或 `HostWeightRuntime.from_config()` 的初始化路径。 +- 强制：store 构造必须把 runtime `WaitPolicy` 传入，并用 `time.monotonic() + + coordination_timeout_seconds` 作为 `domain-init.lock` 的 deadline；超时转为 retryable + `DOMAIN_UNAVAILABLE`，preferred 可 fallback，required 失败。超时 waiter 不得解锁 owner， + 也不得改写 owner 的 domain metadata。初始化预算与 resolve/publish 协调预算分开；这不是 + 硬启动 deadline，也不取消进行中的 filesystem I/O/production。争用结束后新 runtime 可重试。 +- 禁止：构造期 `FileLock(..., deadline=None)` 无限阻塞，使 preferred 也无法 fallback；把 + coordination timeout 说成可取消的同步 producer 总时限。 +- 验收：外部持锁下分别覆盖 preferred fallback 与 required failure、typed + `DOMAIN_UNAVAILABLE`、owner 锁与 metadata 保持、deadline 前释放可成功，以及释放后新 + runtime 的 `LOCAL_HIT` 恢复。^[PR #7128] diff --git a/knowledge/repos/vllm-omni/components/model-executor/_index.md b/knowledge/repos/vllm-omni/components/model-executor/_index.md index b06c63245..bfb910340 100644 --- a/knowledge/repos/vllm-omni/components/model-executor/_index.md +++ b/knowledge/repos/vllm-omni/components/model-executor/_index.md @@ -1,7 +1,7 @@ --- title: "Model Executor" created: 2026-07-10 -updated: 2026-09-06 +updated: 2026-09-22 type: index tags: [vllm-omni, components, model-executor] sources: [] @@ -39,7 +39,7 @@ sources: [] | runtime info、跨 stage payload、batch 与 request RNG 合同 | [跨 stage bridge 与 batch 合同](rules-bridge-batch.md) | | loader 的 dtype/config 获取、fused shard 与多模块 checkpoint 载入 | [loader 合同](rules-loader-contract.md) | | shared image example envelope 与 model_extras 参数声明 | [image task envelope 合同](rules-image-task-envelope.md) | -| Omni 输出类型与字段/复制合同 | [输出类型合同](rules-output-contract.md) | +| Omni 输出类型与字段/复制合同 | [输出类型合同](rules-output-contract.md) 新增核对：EXEC-7c、EXEC-7d。 | | 采样循环不变量、热路径缓存、AR 音频侧路、preprocess phase/one-token prefill | [运行时热路径合同](rules-runtime-hot-paths.md) | -| NPU runner、ROCm 分页注意力、NPU 模型补丁 | [平台后端合同](rules-platform-backends.md) | +| NPU runner、ROCm 分页注意力、NPU 模型补丁 | [平台后端合同](rules-platform-backends.md) 新增核对：EXEC-10f、EXEC-10f2。 | diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md b/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md index ac58b57da..395202c60 100644 --- a/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md @@ -1,10 +1,10 @@ --- title: "Omni 输出类型合同" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, model-executor] -sources: ["PR #5146", "PR #6152", vllm_omni/outputs/] +sources: ["PR #5146", "PR #6152", vllm_omni/outputs/, "PR #7448", "PR #7608"] confidence: high --- @@ -27,3 +27,17 @@ confidence: high - 验收：用真实 `RequestOutput` 与 `OmniRequestOutput` 做属性 parity 测试；覆盖新字段默认值为 `None`，以及 `ec_transfer_params`、`num_cache_creation_tokens` 等非空值经 `from_stage_output()` 后保持不变。 ^[PR #6152] 相关执行流见 [model-executor architecture](architecture.md)；跨 stage 合同见 [bridge/batch 规则](rules-bridge-batch.md)。 + +## EXEC-7c — 已知 sample-rate 键必须按最新快照合并，不得当生成内容累积 + +- 触发：修改 `MultimodalPayload.merged_with`、DELTA/CUMULATIVE 音频输出合并，或 `sr`/`sample_rate`/`audio_sample_rate` 的分区与巩固。 +- 强制：这三个键是 metadata 快照，不是 waveform 内容。合并前先从 incoming 捕获快照，再写入 metadata 分区一次；缺失键保留既有值。tensor/scalar 表示切换时必须从 tensors 分区移除同名残留。audio/latent 内容继续按既有策略累积。 +- 禁止：把 sample-rate 追加成 list 直到 consolidate；让非终态 DELTA 输出保留增长的标量历史；在两边分区同时留下陈旧值。 +- 验收：覆盖整数/标量/向量/metadata tensor、表示切换、缺键保留、self-merge、长 DELTA/CUMULATIVE 流与 abort 刷新；断言快照始终非 list 且 `payload[key] is payload.to_dict()[key]`。^[PR #7448] + +## EXEC-7d — 张量累积策略必须按 (modality, key) 解析，codec 键不得套用波形默认 + +- 触发：修改 `get_accumulation_strategy`、`MultimodalPayload.consolidate_tensors`、`_consolidate_tensor_list`，或 AUDIO 模态下非波形键（如 `codes.audio`/`codes.ref`）的巩固。 +- 强制：`consolidate_tensors` 接收 modality，并对每个 key 调用 `get_accumulation_strategy(modality, key)`。流水线通过 `register_key_accumulation_strategy` 注册覆盖；Qwen3-TTS 必须将 `codes.audio`→`CONCAT_DIM0`、`codes.ref`→`REPLACE`。`CONCAT_LAST` 失败仍可 flatten 后拼接；其他策略失败必须带 key 名 raise，不得静默 keep-last。 +- 禁止：整 payload 共用单一 modality 默认；把 codec-frame 矩阵当 `CONCAT_LAST` 波形；把非 `audio` 键的 concat 失败吞成 keep-last。 +- 验收：合成张量证明 `codes.audio` 沿 dim0 拼满、`codes.ref` 只保留一份，以及错误策略 raise；真实波形键仍走 `CONCAT_LAST`。^[PR #7608] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md b/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md index 836798696..e98949e3e 100644 --- a/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md @@ -1,10 +1,10 @@ --- title: "平台后端合同" created: 2026-09-04 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, model-executor] -sources: ["PR #5886", "PR #6061", "PR #6096", vllm_omni/platforms/, "PR #5604", "PR #6293", "PR #5571", "vllm_omni/platforms/xpu/platform.py", "PR #5569", "vllm_omni/platforms/xpu/utils.py", "PR #5048", "PR #6350", "PR #6102", "PR #6563", "PR #6054", vllm_omni/platforms/npu/platform.py, tests/platforms/npu/test_diffusion_platform.py, tests/platforms/npu/test_diffusion_attn_backend_selector.py, "PR #6674", vllm_omni/platforms/npu/worker/npu_ar_model_runner.py, vllm_omni/platforms/npu/worker/npu_generation_model_runner.py, vllm_omni/platforms/npu/worker/npu_model_runner.py] +sources: ["PR #5886", "PR #6061", "PR #6096", vllm_omni/platforms/, "PR #5604", "PR #6293", "PR #5571", "vllm_omni/platforms/xpu/platform.py", "PR #5569", "vllm_omni/platforms/xpu/utils.py", "PR #5048", "PR #6350", "PR #6102", "PR #6563", "PR #6054", vllm_omni/platforms/npu/platform.py, tests/platforms/npu/test_diffusion_platform.py, tests/platforms/npu/test_diffusion_attn_backend_selector.py, "PR #6674", vllm_omni/platforms/npu/worker/npu_ar_model_runner.py, vllm_omni/platforms/npu/worker/npu_generation_model_runner.py, vllm_omni/platforms/npu/worker/npu_model_runner.py, "PR #7301", "PR #7393"] confidence: high --- @@ -114,3 +114,17 @@ confidence: high state；只在 start 处 gate、却无条件同步/导出，或将 gate 本身当作 NPU 性能结果。 - 验收：AR/generation 都覆盖 enabled+timed、disabled+timed、enabled+not-timed 及 per-output disable， 断言 start/sync/export 一致。PR #6674 没有这些专项测试，属于后续验收要求。^[PR #6674] + +## EXEC-10f — XPU W8A16 FP8 linear 必须 flatten-then-reshape 恢复 N-D 输出 + +- 触发：修改 `vllm_omni/platforms/xpu/patch.py`、`XPUOmniPlatform` 初始化补丁，或 XPU 上 diffusion `--quantization fp8` 的 ScaledMM 路径。 +- 强制：对 `XPUW8A16FP8LinearKernel.apply_weights` 在平台 init 幂等 patch：先按 `(*x.shape[:-1], out_features)` 记录输出形，把 activation reshape 为 2-D 再调用原实现，最后 `.view` 回 N-D。与其他 ScaledMM/W8A8 合同一致；kernel 不可用时 skip。 +- 禁止：把 3-D diffusion activation 直接交给 fake 恒返回 2-D 的 `fp8_gemm_w8a16`；把该 patch 宣称为 LLM 2-D 路径修复或上游永久替代（upstream 修好后应变 no-op）。 +- 验收：XPU 上 FLUX/Qwen-Image 等 FP8 diffusion 能完成 dummy warmup 与生成；2-D LLM 路径不受影响。^[PR #7301] + +## EXEC-10f2 — AR runner 在 async PP 下必须配对 sampled-token broadcast 的 send + +- 触发：修改 `GPUARModelRunner.sample_tokens`、async scheduling + `pipeline_parallel_size > 1`，或继承/覆盖上游 `_pp_receive_prev_sampled_token_ids_to_input_batch` / `_pp_broadcast_prev_sampled_token_ids`。 +- 强制：非最终 PP rank 在 connector-only 分支 posted 的 async broadcast receive，必须由最终 rank 在 `_update_states_after_model_execute` 之后立刻执行匹配 send。守卫与上游一致：`use_async_scheduling` 且 `not broadcast_pp_output` 且 `pp.world_size > 1` 且 `pp.is_last_rank`。 +- 禁止：只继承 receive 而不补 send；在 `broadcast_pp_output=True`（external_launcher 已广播 logits/输出）时重复发送 sampled ids；把死锁归因于必须关闭 Thinker `async_scheduling` 而不修复配对集体通信。 +- 验收：ledger 或真实多进程 PP group 证明非最终 rank 的 receive 仅因最终 rank 的 Omni send 完成；PP1 与 `broadcast_pp_output` 路径不变。缺 send 时不得把“设 `async_scheduling: false`”当作长期合同。^[PR #7393] diff --git a/knowledge/repos/vllm-omni/components/scheduler/_index.md b/knowledge/repos/vllm-omni/components/scheduler/_index.md index 0ad145ca6..2ef1a8f31 100644 --- a/knowledge/repos/vllm-omni/components/scheduler/_index.md +++ b/knowledge/repos/vllm-omni/components/scheduler/_index.md @@ -1,7 +1,7 @@ --- title: "Scheduler（AR/生成请求调度）" created: 2026-07-16 -updated: 2026-09-06 +updated: 2026-09-22 type: index tags: [vllm-omni, components, scheduler] sources: [vllm_omni/core/sched/omni_ar_scheduler.py, vllm_omni/core/sched/omni_generation_scheduler.py, vllm_omni/core/sched/omni_scheduler_mixin.py, vllm_omni/core/sched/output.py, vllm_omni/core/prefix_cache.py, docs/design/module/ar_runtime.md, docs/design/module/archive/ar_module.md] @@ -40,3 +40,5 @@ sources: [vllm_omni/core/sched/omni_ar_scheduler.py, vllm_omni/core/sched/omni_g |---|---| | 理解调度器继承链、KV transfer 与 prefix cache 语义 | [architecture](architecture.md) | | 按 PR 描述直达 prefix cache、token budget、upstream 接口或 side-stream 首批源码 | [rules / Direct 代码快速入口](rules.md#direct-代码快速入口) | + +- [rules-shared-lifecycle](rules-shared-lifecycle.md) 新增核对：SCHED-6a2。 diff --git a/knowledge/repos/vllm-omni/components/scheduler/rules-shared-lifecycle.md b/knowledge/repos/vllm-omni/components/scheduler/rules-shared-lifecycle.md new file mode 100644 index 000000000..3fc8e01b0 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/scheduler/rules-shared-lifecycle.md @@ -0,0 +1,24 @@ +--- +title: "Scheduler shared lifecycle 规则" +created: 2026-09-22 +updated: 2026-09-22 +type: rule +tags: [vllm-omni, components, scheduler] +sources: ["PR #5461"] +--- + +# Scheduler shared lifecycle 规则 + +## SCHED-6a2 — AR 与 generation 的共享生命周期只能有一个 canonical 实现 + +- 触发：把 AR/generation scheduler 的 I/O、输出包装、finish 或统计逻辑移入 + `OmniSchedulerMixin`，或修改该 mixin 及任一 scheduler 的 shared hook。 +- 强制：两类 scheduler 共用 pending connector/chunk 输入处理、wait queue restore、 + `OmniSchedulerOutput` 包装、失败 KV load 的 terminal output、finished IDs、队列清理、 + stats/events 和 finish cleanup；`NewRequestData` 转成 Omni 结构时逐字段无损，并保留已经 + 是 Omni 类型的 fast path。AR 的 synthetic-abort 等差异策略必须作为显式参数留在调用点。 +- 禁止：把共享生命周期复制回两个 subclass；重建 `NewRequestData` 时只挑当前已知字段； + 在 mixin 内按 scheduler 类型隐式猜差异策略；失败 KV load 只改状态而不发 terminal output。 +- 验收：同一组 contract 测试分别驱动 AR 与 generation 输入路径；断言 base dataclass 的 + 每个字段均被转交、现有 Omni entry 保持 identity、失败 KV load 终止输出、finished ID/ + stats/events/cleanup 一致，并单独验证显式 abort policy。 ^[PR #5461] diff --git a/knowledge/repos/vllm-omni/components/serving/_index.md b/knowledge/repos/vllm-omni/components/serving/_index.md index 767ed0d89..cc5c17f6f 100644 --- a/knowledge/repos/vllm-omni/components/serving/_index.md +++ b/knowledge/repos/vllm-omni/components/serving/_index.md @@ -1,7 +1,7 @@ --- title: "Serving" created: 2026-07-10 -updated: 2026-09-06 +updated: 2026-09-22 type: index tags: [vllm-omni, components, serving] sources: [] @@ -30,15 +30,15 @@ sources: [] |---|---| | 理解入口到 engine 的边界，选择 unified HTTP/WS endpoint、区分 wire protocol，或确认 PersonaPlex/JoyVL 是否是独立 server | [architecture 与 API route owner](architecture.md) | | 根据 PR 描述直达 request contract、stage sampling constraints、streaming format、artifact readiness、factory lifecycle 或 metrics 的规则组与第一批源码 | [rules 与代码地图](rules.md) | -| Prometheus family、pipeline waiting/running gauge、stage/replica snapshot 或 collector lifecycle | [metrics 生命周期规则](rules-metrics.md) | -| upstream launcher shutdown、renderer warmup 与多 replica compile-cache 隔离 | [upstream 兼容规则](rules-upstream-compat.md) | -| stage/replica 死亡、request-local failure、readiness/liveness 或驱逐后 cleanup | [fault-isolation rules](rules-fault-isolation.md) | +| Prometheus family、pipeline waiting/running gauge、stage/replica snapshot 或 collector lifecycle | [metrics 生命周期规则](rules-metrics.md) 新增核对：SERV-2g。 | +| upstream launcher shutdown、renderer warmup 与多 replica compile-cache 隔离 | [upstream 兼容规则](rules-upstream-compat.md) 新增核对：SERV-7f、SERV-7e、SERV-7g、SERV-7h。 | +| stage/replica 死亡、request-local failure、readiness/liveness 或驱逐后 cleanup | [fault-isolation rules](rules-fault-isolation.md) 新增核对：SERV-5u。 | | batched chat frontend fan-out、choice cardinality、whole-batch error/cancellation | [batch chat rules](rules-batch-chat.md) | | engine startup/shutdown、stage lifecycle、acknowledged abort 的 shutdown race、duplex stage prewarm/CFG companion，或 opt-in event-driven orchestration / reader-poller reconcile / final-output drain | [engine 生命周期规则](rules-engine-lifecycle.md) | | duplex stage-keyed segment state、跨 stage boundary/metadata 隔离 | [duplex 编排规则](rules-duplex-orchestration.md) | -| adapter-owned full-duplex session、deferred cognition/gate、barge-in cancellation、response ownership、`to_thread` worker drain、close/flush/session cleanup，或 startup warmup/realtime admission | [session lifecycle 规则](rules-session-lifecycle.md) | +| adapter-owned full-duplex session、deferred cognition/gate、barge-in cancellation、response ownership、`to_thread` worker drain、close/flush/session cleanup，或 startup warmup/realtime admission | [session lifecycle 规则](rules-session-lifecycle.md) 新增核对：SERV-6j。 | | acknowledged abort 的 RPC ACK、queue/router shutdown race、cancellation cleanup 或 backpressure/timeout 保持 | [acknowledged abort 生命周期规则](rules-abort-lifecycle.md) | | assembled FastAPI route 覆盖、method/path ownership 或 request-body forwarding | [app assembly 规则](rules-app-assembly.md) | -| stage config→EngineArgs projection、explicit `devices`、TP/local-DP/PP、replica layout 或 worker 创建前的 layout guard | [stage 启动与设备布局规则](rules-stage-startup.md) | -| 公开请求字段的校验、来源冲突、alias/extras 归一与 consumer view | [请求输入合同](rules-request-input.md) | -| speech 输出采样率 capability、resample、streaming header/flush | [speech 输出规则](rules-speech-output.md) | +| stage config→EngineArgs projection、explicit `devices`、TP/local-DP/PP、replica layout 或 worker 创建前的 layout guard | [stage 启动与设备布局规则](rules-stage-startup.md) 新增核对：SERV-12c、SERV-8b。 | +| 公开请求字段的校验、来源冲突、alias/extras 归一与 consumer view | [请求输入合同](rules-request-input.md) 新增核对：SERV-4r、SERV-4r2、SERV-4r3、SERV-4r4、SERV-4s、SERV-4r5、SERV-4i2。 | +| speech 输出采样率 capability、resample、streaming header/flush | [speech 输出规则](rules-speech-output.md) 新增核对：SERV-9c。 | diff --git a/knowledge/repos/vllm-omni/components/serving/rules-fault-isolation.md b/knowledge/repos/vllm-omni/components/serving/rules-fault-isolation.md index 425813d54..b29cde600 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-fault-isolation.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-fault-isolation.md @@ -1,10 +1,10 @@ --- title: "Serving replica fault-isolation 规则" created: 2026-09-02 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #4583", "PR #6170", vllm_omni/engine/orchestrator.py, vllm_omni/engine/stage_pool.py, vllm_omni/entrypoints/async_omni.py, vllm_omni/entrypoints/omni_base.py, tests/dfx/reliability/test_reliability_qwen3_omni.py, tests/engine/test_orchestrator_error_handling.py, tests/entrypoints/test_omni_entrypoints.py] +sources: ["PR #4583", "PR #6170", vllm_omni/engine/orchestrator.py, vllm_omni/engine/stage_pool.py, vllm_omni/entrypoints/async_omni.py, vllm_omni/entrypoints/omni_base.py, tests/dfx/reliability/test_reliability_qwen3_omni.py, tests/engine/test_orchestrator_error_handling.py, tests/entrypoints/test_omni_entrypoints.py, "PR #6759"] confidence: high --- @@ -59,3 +59,10 @@ confidence: high 发送 error 后以 abort/close-duplex-session cleanup 回收该 request；普通非-duplex 路径继续抛出， 不能扩大 catch 范围而吞没共享 orchestrator 的编程错误。验收要证明坏 duplex handoff 不终止 orchestrator，且同一 engine 的后续 request 仍可完成。^[PR #6170] + +## SERV-5u — 异步 video job 必须等 scheduler 准入才标 in_progress，DELETE 须先有界 abort engine + +- 触发：修改 `POST /v1/videos` 异步 job 状态机、`DELETE /v1/videos/{id}`，或 diffusion `emit_request_lifecycle` / request-started 控制面事件。 +- 强制：job 保持 `queued`，直到 scheduler 首次准入并经 opt-in lifecycle marker 通知；不得在 background task 启动时就标 `in_progress`。DELETE 先对 engine request 做有界 abort（`VLLM_OMNI_ABORT_TIMEOUT`，默认 2s），再取消 frontend task，并再等一小段 cleanup 以免 orchestrator 卡住；abort 后重读 job，使并发完成的 artifact 仍被删除。 +- 禁止：只取消 asyncio task 而留下 engine 侧扩散；把 task 启动当作已开始推理；让 DELETE 在 abort 上无限挂起。 +- 验收：queued-until-admission、in-progress DELETE abort、abort timeout、完成后 DELETE 清 artifact，以及 `generate()` cancel 的有界 cleanup。^[PR #6759] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-metrics.md b/knowledge/repos/vllm-omni/components/serving/rules-metrics.md index 2bb56fe14..5b1cc6c19 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-metrics.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-metrics.md @@ -1,10 +1,10 @@ --- title: "Serving metrics 生命周期规则" created: 2026-09-05 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #3576", "PR #4755", "PR #6549", vllm_omni/engine/async_omni_engine.py, vllm_omni/engine/orchestrator.py, vllm_omni/entrypoints/async_omni.py, vllm_omni/entrypoints/omni_base.py, tests/metrics/test_prometheus.py] +sources: ["PR #3576", "PR #4755", "PR #6549", vllm_omni/engine/async_omni_engine.py, vllm_omni/engine/orchestrator.py, vllm_omni/entrypoints/async_omni.py, vllm_omni/entrypoints/omni_base.py, tests/metrics/test_prometheus.py, "PR #6561"] confidence: high --- @@ -38,3 +38,19 @@ confidence: high - 禁止：把 orchestrator 已 dispatch、但仍在 stage scheduler queue 的请求报作 running；分散复制公式；让负值、过期/竞态 snapshot 或 engine waiting 大于 dispatched 产生负 gauge；在 final request 仍在 `request_states` 时丢掉既有 `total - 1` 防止 waiting 卡为 1 的语义。 - 限制：该实现求的是各 `(stage, replica)` 最新 waiting snapshot 的**和**，并不按 request ID 去重；代码和 PR #6549 的 3-dispatched/2-queued 测试只证明单一 queue 的拆分，未证明同一 pipeline request 不会同时落在多个 snapshot。因此该值是受 clamp 保护的 scheduler-queue occupancy，不应声称是精确的全局唯一请求数。PR #6549 唯一 review thread 仅确认 `engines_` 名称用于区分 frontend 与 engine waiting，未处理该 cardinality 问题。 - 验收：用真实 Prometheus scrape 覆盖 3 个已 dispatch、其中 2 个 engine scheduler waiting，精确断言 running=1、waiting=2；同时覆盖 arrival、普通/最终 stage result、cleanup、replica removal 和 snapshot 的负值/超 dispatched 值，确认既有 finalization race 语义不变。^[PR #6549] + +## SERV-2g — metrics 去重集合必须挂在请求 state 并随其释放 + +- 触发：修改 `OmniBase` 的 result/message metrics 去重、`ClientRequestState`、abort 后 + request state 保留，或 `_log_summary_and_cleanup`。 +- 强制：`id(msg)` / `id(result)` 去重集合必须是每个 `ClientRequestState` 上的 + `consumed_metric_message_ids`，由 `_handle_output_message` / `_process_single_result` + 写入；不得再维护与 request 并行的实例级 `dict[request_id, set]`。abort 可按既有合同 + 暂时保留 `request_states`，但去重集合只能活在该 state 上，并由 `generate()` 正常 + cleanup / `_log_summary_and_cleanup` 与 state 一并释放。 +- 禁止：在 `OmniBase` 上保留 request-keyed metrics 旁路 map；abort 路径留下跨请求生命周期 + 的去重条目；让同一 message 在 abort-then-cleanup 前后被累计两次，或 cleanup 后实例上仍 + 能按 request_id 读到残留 map。 +- 验收：生产 handler 写入 state 集合后 abort，断言实例属性无独立 request-keyed metrics + map；cleanup 后 `request_states` 与任何 request-keyed map 均无该 id；同一 message 重放 + 只累计一次。^[PR #6561] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-request-input.md b/knowledge/repos/vllm-omni/components/serving/rules-request-input.md index 495c86590..b796591a5 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-request-input.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-request-input.md @@ -1,10 +1,10 @@ --- title: "请求输入合同" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #3805", "PR #5374", "PR #5885", "PR #6598", vllm_omni/data_entry_keys.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/openai/, vllm_omni/entrypoints/omni_base.py, vllm_omni/engine/orchestrator.py, vllm_omni/inputs/, tests/engine/test_async_omni_engine_input.py, tests/engine/test_orchestrator_error_handling.py, tests/entrypoints/test_omni_entrypoints.py, tests/entrypoints/openai_api/test_invalid_audio_speech.py, tests/entrypoints/openai_api/test_serving_speech.py, "PR #5181", "PR #6182"] +sources: ["PR #3805", "PR #5374", "PR #5885", "PR #6598", vllm_omni/data_entry_keys.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/openai/, vllm_omni/entrypoints/omni_base.py, vllm_omni/engine/orchestrator.py, vllm_omni/inputs/, tests/engine/test_async_omni_engine_input.py, tests/engine/test_orchestrator_error_handling.py, tests/entrypoints/test_omni_entrypoints.py, tests/entrypoints/openai_api/test_invalid_audio_speech.py, tests/entrypoints/openai_api/test_serving_speech.py, "PR #5181", "PR #6182", "PR #6963", "PR #4741", "PR #4167", "PR #7447", "PR #7459", "PR #7463", "PR #5691"] confidence: high --- @@ -139,3 +139,58 @@ engine 生命周期见 [engine 生命周期规则](rules-engine-lifecycle.md)； - 强制：transform 只操作 Stage-0 copy；原始 prompt 保留给 downstream，并在 transform 前后保持同一 global request ID。只把已处理的 metadata 合并回原始视图，先移除 transform-owned stale keys。临时目录只通过内部 `REQUEST_ARTIFACT_DIRS_KEY` 交给 request state；preprocess、companion build、enqueue 前失败立即回收，admit 后由 orchestrator 在所有 terminal cleanup 路径回收，且内部 key 不得进入 stage payload。 - 禁止：把 transformed prompt 当作 downstream 原始媒体、让旧 prepared descriptor 跨请求复用、在 ownership 已交给 orchestrator 后由 frontend 提前删除，或让异常路径泄漏转码目录。 - 验收：覆盖 transform replacement/copy、request ID 与 metadata merge、成功 terminal/abort、preprocess/companion/enqueue 异常以及无 artifact control；断言 downstream 看见原始媒体+允许的 processed meta，目录恰好由当前 owner 回收，内部 key 未传输。^[PR #5885] + +## SERV-4r — video 参考图必须在 RGB 转换前执行像素上限并映射解码器炸弹 + +- 触发：修改 `/v1/videos` 的 `image_reference` / `input_reference(s)`、MiniMax H3 multipart 图片上传，或 `video_api_utils._decode_image_bytes` / `_validate_image_pixel_limit`。 +- 强制：`Image.open` 后、`convert("RGB")` 前按 header 尺寸检查 `width * height`；当 `VLLM_MAX_IMAGE_PIXELS > 0` 且超出时拒绝。`=0` 禁用检查，恰等于上限放行。Pillow `DecompressionBombError` 必须映射为专用像素上限 client error（HTTP 400），不得落入泛化 “not a valid image” 后再当 video 重试。H3 multipart 与共享 decode helper 共用同一合同。 +- 禁止：先全量 decode/convert 再检像素；把 decoder bomb 误判为可 video-fallback 的无效图；用 500 或静默接受放大资源占用。 +- 验收：覆盖超限在 convert 前失败、disabled/inclusive 边界、bomb→专用 400、以及 H3 multipart 字段拒绝。^[PR #6963] + +## SERV-4r2 — `/v1/audio/generate` 必须在协议层拒绝空 prompt 与越界采样字段 + +- 触发：修改 `OpenAICreateAudioGenerateRequest`、audio generate invalid-param 可靠性测试，或该入口的 prompt/采样字段限界。 +- 强制：`input` 以 field validator 拒绝空串与纯空白；`audio_length` 要求 `gt=0`；`guidance_scale` 限界 `ge=0, le=1000`；`num_inference_steps` 限界 `ge=1, le=1000`。非法值必须在 entrypoint Pydantic 校验失败，不得拖到 engine 超时或 HTTP 200。 +- 禁止：依赖 engine-level 检查代替协议拒绝；把越界 steps/guidance 留在 `_INT64_MAX` 一类无业务上界；用 skip mark 掩盖未接线的 invalid-param case。 +- 验收：空/空白 input、负/零 `audio_length`、负/过大 guidance、零/负/过大 steps 均返回字段定位的 4xx，并与 invalid-param 测试期望一致。^[PR #4741] + +## SERV-4r3 — `vllm serve --omni` 必须在校验期要求显式 model + +- 触发：修改 `OmniServeCommand.validate`、`--omni` 启动路径、`model`/`model_tag`/`explicit_keys`，或部署 YAML 与 checkpoint 来源关系。 +- 强制：`--omni` 下 model 只能来自非空 positional `model_tag`，或出现在 `explicit_keys` 中的非空 `--model`。`--deploy-config` / stage YAML 只携带 per-stage engine args，checkpoint 始终由 `args.model` 注入，不得凭 YAML 存在判定“已提供 model”。空/空白 model 视为未提供，并在 validate 期以明确 `ValueError` 失败。 +- 禁止：依赖 vLLM `ModelConfig` 默认 `Qwen/Qwen3-0.6B` 进入 Omni/diffusion 路由；把默认模型缺失伪装成深层 diffusion-registry `Model class … not found`；让仅 `--deploy-config` 或空 `--model "$MODEL"` 绕过守卫。 +- 验收：`["serve","--omni"]`、仅 deploy-config、空/空白 positional/`--model` 均 raise “requires an explicit model”；positional、`--model`、以及显式 model+deploy-config 不得触发该守卫。^[PR #4167] + +## SERV-4r4 — 图片 generations 与 edits 的 `output_compression` 必须同一闭环 + +- 触发：修改 `/v1/images/generations` 或 edits 的协议字段、handler 到 `encode_image_base64_with_compression` 的转发。 +- 强制：`ImageGenerationRequest` 声明 `output_compression`（0–100，默认 100）并原样传入 encoder；PNG 映射为 compress_level（100→0，1→9），jpeg/webp 映射为 quality。缺省字节合同与 edits 一致。 +- 禁止：只在 edits/`Form` 接受该字段而 generations 静默丢弃；用 HTTP 200 或非默认 format 冒充 compression 已生效。 +- 验收：L1 断言 PNG 100/1 与等价 compress_level 同大小且 1 更小，JPEG 高低 compression 体积方向正确；handler 用非默认值断言 encoder kwargs。^[PR #7447] + +## SERV-4s — 内存中的 image `file` 响应必须按固定块异步产出 + +- 触发：修改 `ImageGenerationResponse.stream_response`、`response_format=file`，或把已物化 PNG/ZIP 交给 Starlette `StreamingResponse`。 +- 强制：对已在内存中的单图/ZIP 使用异步迭代器按固定块（如 64 KiB）`memoryview` 切片产出；保留既有 headers/`Content-Length` 与完整 body 字节。 +- 禁止：把 `io.BytesIO` 直接交给 `StreamingResponse`（按行/`0x0A` 切分且同步 iterable 会触发 per-fragment threadpool）；为“流式”再无意义地按换行拆二进制。 +- 验收：构造含多处 `0x0A` 的 PNG/ZIP，断言 ASGI body 哈希不变、非空 frame 数约为 `ceil(size/chunk)` 而非 newline 次数。^[PR #7459] + +## SERV-4r5 — pipeline 要求的 `stop_token_ids` 必须与 caller/deploy 停止集相加去重 + +- 触发：修改 `merge_sampling_constraints`、`OmniBase._apply_sampling_constraints`、pipeline `sampling_constraints.stop_token_ids`，或 deploy `default_sampling_params.stop_token_ids`。 +- 强制：标量约束仍由 pipeline 覆盖同名 caller/deploy 字段；惟 `stop_token_ids` 例外——先取 caller/deploy 列表，再追加 pipeline 要求的 ID，并以 `dict.fromkeys` 保序去重。legacy `yaml_extras` 与 structured config、以及 request 期重建 `SamplingParams`，必须共用同一 merge helper，且不 mutate 调用方对象。 +- 禁止：用整表 `constraints` 覆盖把 caller 自定义 stop 抹掉；只在 YAML merge 或只在 request apply 一侧实现相加；或让模型必需 terminator（如 MiniCPM-o TTS 边界）依赖重复 YAML/环境变量拷贝。 +- 验收：覆盖 deploy 已有 stop + pipeline 新增 stop 的并集、caller mapping/dataclass/msgspec 重建、immutability，以及至少一条真实 pipeline 的 Stage-0 必需 stop 进入全部 shipping deploy。^[PR #7463] + +## SERV-4i2 — multipart artifact 在 I/O 前验合同、在所有终态释放 + +- 触发：公开请求接收一个或多个 `UploadFile`、URL media，或把输入物化后交给后台任务。 +- 强制：在 read/download/persist/decode 前完成 dispatcher、模型 capability 与 + source/task matrix 校验；流式执行 count、per-file 和 aggregate byte 上限，超限返回 + 413；只清理由当前请求创建的 artifact，并保留输入顺序直到最终 consumer 完成。 +- 禁止：把无限 multipart 先复制到临时目录再校验；让非法组合进入 engine 后变成 sync + 500 或 async failed job；只在成功路径或 audio 分支 cleanup。 +- 验收：sync/async 都覆盖冲突 400、超限中途 413、setup 失败、generation 失败、timeout、 + cancellation、missing-job/early-return 与成功；无效请求未调用持久化或 engine，所有已创建 + 路径在每个终态消失，用户提供的非 owned 路径不被删除。MiniMax H3 的当前矩阵见 + [MiniMax H3 media 规则](../../models/minimax-h3/rules-media.md)。 ^[PR #5691] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-session-lifecycle.md b/knowledge/repos/vllm-omni/components/serving/rules-session-lifecycle.md index e7e3fe7ef..ae456085f 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-session-lifecycle.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-session-lifecycle.md @@ -1,10 +1,10 @@ --- title: "Serving session lifecycle 规则" created: 2026-09-05 -updated: 2026-09-06 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #6537", "PR #6354", vllm_omni/config/stage_config.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/engine/duplex/adapter.py, vllm_omni/engine/duplex/runtime.py, vllm_omni/experimental/fullduplex/mage_vl/adapter.py, vllm_omni/experimental/fullduplex/mage_vl/serving/backend.py, vllm_omni/experimental/fullduplex/mage_vl/serving/server.py, tests/e2e/features/fullduplex/test_mage_vl_adapter.py, tests/e2e/features/fullduplex/test_mage_vl_serving.py, tests/e2e/online_serving/nemotron_voicechat_realtime_duplex.py] +sources: ["PR #6537", "PR #6354", vllm_omni/config/stage_config.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/engine/duplex/adapter.py, vllm_omni/engine/duplex/runtime.py, vllm_omni/experimental/fullduplex/mage_vl/adapter.py, vllm_omni/experimental/fullduplex/mage_vl/serving/backend.py, vllm_omni/experimental/fullduplex/mage_vl/serving/server.py, tests/e2e/features/fullduplex/test_mage_vl_adapter.py, tests/e2e/features/fullduplex/test_mage_vl_serving.py, tests/e2e/online_serving/nemotron_voicechat_realtime_duplex.py, "PR #7363"] confidence: high --- @@ -44,3 +44,17 @@ confidence: high slot；因 warmup failure 拒绝服务或永久阻塞 real client；或在 shutdown 遗留 background warmup task。 - 验收：覆盖 disabled、health-then-self-WS ordering、warmup tag 绕过 admission gate、explicit close、 success/failure 均 set event、real-client 120-second escape hatch，以及 shutdown cancellation。^[PR #6354] + +## SERV-6j — 视频预热缓存不得在帧被逐出后回写，取消重启不得插入固定延迟 + +- 触发：修改 streaming video WebSocket 的 `frame_pil_cache` / prewarm decode、有界 + `frame_buffer` 逐出，或 cancel-then-restart 上一 query 后的等待逻辑。 +- 强制：prewarm 完成（成功 PIL 或 `_BAD_FRAME`）写入缓存前，必须再次确认该 b64 仍在 + 当前 `frame_buffer`；已逐出则丢弃结果，不得把帧重新插入 cache。取消上一 query 并 + await task/`abort()` 后，不得再 `asyncio.sleep(0.1)`；Orchestrator abort ACK 已是完成 + 信号，墙钟等待不是额外 completion 证明。 +- 禁止：让 decode 竞态把已逐出帧钉回 cache 造成会话期泄漏；用固定 sleep 冒充 abort + 排空；把去掉 sleep 说成已消除更深的 EngineCore GPU race。 +- 验收：`max_frames=1` 下延迟 decode 时，被逐出帧的 PIL 在会话仍打开时可被回收； + `max_frames=2` 时仍缓存到 teardown。中断连续 query 时记录的 sleep 请求为空，且旧 + generator 关闭与 abort 完成发生在下一次 generate 之前。^[PR #7363] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-speech-output.md b/knowledge/repos/vllm-omni/components/serving/rules-speech-output.md index 78bed551b..082b4301e 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-speech-output.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-speech-output.md @@ -1,10 +1,10 @@ --- title: "Speech 输出采样率规则" created: 2026-09-05 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #6553", docs/serving/speech_api.md, vllm_omni/entrypoints/openai/audio_utils_mixin.py, vllm_omni/entrypoints/openai/protocol/audio.py, vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/entrypoints/openai/tts_adapters/base.py, vllm_omni/entrypoints/openai/tts_adapters/qwen3_tts.py, tests/entrypoints/openai_api/test_audio_format.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/e2e/online_serving/test_qwen3_tts_customvoice_expansion.py] +sources: ["PR #6553", docs/serving/speech_api.md, vllm_omni/entrypoints/openai/audio_utils_mixin.py, vllm_omni/entrypoints/openai/protocol/audio.py, vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/entrypoints/openai/tts_adapters/base.py, vllm_omni/entrypoints/openai/tts_adapters/qwen3_tts.py, tests/entrypoints/openai_api/test_audio_format.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/e2e/online_serving/test_qwen3_tts_customvoice_expansion.py, "PR #7499"] confidence: high --- @@ -20,3 +20,10 @@ confidence: high - 禁止：upsample、16k/stereo/general adapter support，或据 server-side resampling 声称降低模型生成 latency/cost。 - 验收：Qwen native/8k、other adapter reject、batch override、streaming chunk continuity/tail/source mismatch/ first-audio source-sr failure。CPU/DSP 与单一 expansion fixture 不证明跨硬件音质、TTFB 或带宽收益。^[PR #6553] + +## SERV-9c — 稀疏音频非流式累积必须走 adapter OutputPolicy，不得按模型名分支 + +- 触发：新增/修改 TTS adapter 的非流式输出、`OutputPolicy.accumulate_nonstreaming`，或 `serving_speech._generate_audio_bytes` 对 per-step delta 的拼接。 +- 强制：MOSS-TTS-Nano、Gepard 等 `async_chunk=false` 的稀疏音频模型由 adapter 在 `PreparedRequest.output_policy` 声明 `accumulate_nonstreaming`。serving 用 `request_id` 键存 policy，仅非流式路径在准备成功后写入，并在累积器 `pop` 消费；优先使用 FINAL_ONLY 已拼接波形，否则 `torch.cat` delta。streaming 永不读写该表。 +- 禁止：在 `serving_speech.py` 增加 Gepard/MOSS 等模型名分支；在可失败步骤前写入 policy 导致泄漏；让并发请求共享可变 policy 槽位。 +- 验收：adapter 设/不设 flag 的非流式拼接与 sentinel-only final；并发非流式隔离；streaming 路径无 policy 条目。^[PR #7499] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-stage-startup.md b/knowledge/repos/vllm-omni/components/serving/rules-stage-startup.md index 2e5eb0086..30594fd74 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-stage-startup.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-stage-startup.md @@ -1,10 +1,10 @@ --- title: "Stage 启动与设备布局规则" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #6050", "PR #5445", "Issue #5003", "PR #5742", vllm_omni/engine/stage_init_utils.py, tests/engine/test_stage_device_layout.py] +sources: ["PR #6050", "PR #5445", "Issue #5003", "PR #5742", vllm_omni/engine/stage_init_utils.py, tests/engine/test_stage_device_layout.py, "PR #7328", "PR #7292"] confidence: high --- @@ -33,3 +33,17 @@ confidence: high - 强制：对声明 `runtime.devices` 的每个 LLM stage，在 `create_engine_config` 或 worker/executor 创建前，以同一公式验证并拆分设备：每个本地 replica 需要 `tensor_parallel_size × data_parallel_size_local × pipeline_parallel_size`；`data_parallel_size_local` 未设置时才回退到全局 DP，值为 `0` 的 head process 不做本地设备验证。显式列表可为单 replica template（恰为每副本大小）或完整 pool（该大小 × `num_replicas`）；未声明 `devices` 时保留 launcher 分配语义。仅当去掉 TP 后设备数恰好有效，错误才说明 top-level TP 会广播到全部 stage，并要求在每个 stage override 中一并设置 TP 与 `devices`；其他 TP/DP/PP/replica 不匹配给通用维度说明。 - 禁止：按集群全局 DP 校验本地 `runtime.devices`；让 guard 与 replica splitter 使用不同的每副本宽度；在 engine config/worker 启动后才暴露 `local rank ... out of bounds`；把 PP 或其他布局错误一律归因于 top-level TP；因为没有显式 devices 而拒绝 vLLM 的设备分配。 - 验收：覆盖 TP 广播的单卡 stage 在 engine-config/executor 前失败、匹配 TP/device 通过、未声明 devices 跳过、local-DP 小于 global-DP 与 local-DP=0、PP/local-DP 加多副本的 template 和 full-pool 拆分，以及 PP-only mismatch 不含 TP 专属 workaround。^[Issue #5003] ^[PR #5742] + +## SERV-12c — 串行 LLM 副本初始化必须按物理 GPU 重叠连通分量分组 + +- 触发：修改 `stage_runtime` 的 replica init group、`parse_physical_device_ids`、`device_overlap_group_keys`、`acquire_device_locks` / `spawn_device_lock`，或多 stage 共享/重叠 `runtime.devices` 的启动路径。 +- 强制：本地串行 LLM replica 按已解析物理 device **集合**的连通分量分组：集合相交（含 `{0,1}` 与 `{0}`、`"1,0"` 与 `"0,1"`、传递链）共用同一 `device-group:` 并顺序初始化；不相交集合才并行。非整数 visibility（UUID/MIG）或 unresolved 解析为 `None` 时，该批串行 LLM 收拢为单一 `device-group:*`。diffusion / remote / `parallel_stage_init` 保持既有 per-replica override key。 +- 禁止：仅用精确 device **字符串**分组，使重叠但不等的集合进入并行线程后，同进程对已持有 inode 的 `flock(LOCK_EX|LOCK_NB)` 自相争并在仍持有 `spawn_device_lock` 时空转到 `stage_init_timeout`。 +- 验收：覆盖重叠/传递重叠同组、不相交异组、非整数 devices 收拢；多 stage `0,1`+`0` 启动不得因 device-lock timeout 无锁继续。^[PR #7328] + +## SERV-8b — 全局 replica launch lock 只覆盖 spawn 与 runtime.env，READY 等待必须释放 + +- 触发：修改 `StageRuntime._initialize_local_llm_replica`、`_replica_launch_lock`、`parallel_stage_init`，或多 GPU 多副本启动超时。 +- 强制：全局 launch lock 仅包裹进程 spawn 与 process-global `runtime.env` overlay；launch context 的 READY 等待必须在锁外执行，使不同 device group 可并行完成 load/profile/compile/capture。同 GPU 仍由既有 per-device exclusive lock 串行到 READY；`ExitStack` 管理 launch context 并传播 READY 失败清理。 +- 禁止：默认路径把 lock 持有到 READY（跨 GPU 串行化整段 init）；为加速而削弱同设备 EX lock；改变 timeout/deploy 阈值来掩盖串行启动。 +- 验收：不同设备并发启动回归必须在未改代码时失败、修复后通过；同设备顺序与 READY 失败后的 resource/device-lock 清理仍成立。^[PR #7292] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md b/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md index 25074b5ab..e5127c74b 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md @@ -1,10 +1,10 @@ --- title: "Serving upstream 兼容规则" created: 2026-09-02 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #5976", "PR #5957", vllm_omni/engine/stage_engine_startup.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/entrypoints/utils.py, vllm_omni/request.py, tests/engine/test_stage_engine_startup_cache_env.py, tests/config/test_endpoint_policy.py, "PR #5036", "PR #6642", "PR #6773", "PR #6707", vllm_omni/config/endpoint_policy.py] +sources: ["PR #5976", "PR #5957", vllm_omni/engine/stage_engine_startup.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/entrypoints/utils.py, vllm_omni/request.py, tests/engine/test_stage_engine_startup_cache_env.py, tests/config/test_endpoint_policy.py, "PR #5036", "PR #6642", "PR #6773", "PR #6707", vllm_omni/config/endpoint_policy.py, "PR #6051", "PR #7426", "PR #5647"] confidence: high --- @@ -55,3 +55,40 @@ confidence: high `tests/config/test_endpoint_policy.py`；API-server 路径仍仅为其 `ImportError` fallback 的静态合同。 PR 描述报告该 suite `4 passed`，批准 review 的边界是 static review、merge tree 与 CI，且明确未执行 untrusted fork code；因此不能扩展为完整 vLLM 版本兼容或端到端声明。^[PR #6707, merged 2026-09-02] + +## SERV-7f — OpenPI `nd` 标记必须同时接受 vLLM-native 与 msgpack-numpy 的 `kind` + +- 触发：修改 `entrypoints/openpi/connection.py` 的 `_decode_vllm_numpy_marker` / `_unpack_numpy`，或 OpenPI/DreamZero 观测 payload 解码。 +- 强制：`kind` 仍为区分 array marker 与普通 mapping 的必填字段。plain ndarray 的 `kind` 接受 `""`（msgpack-numpy 包）或 dtype kind 字符（vLLM-native）；二者任一匹配即通过。`kind == "V"`（structured）必须在调用 `np.dtype(type)` 之前拒绝，因该方言的 `type` 是 descriptor list。outbound 仍发 openpi-client 标记。 +- 禁止：要求 `kind == dtype.kind` 而拒绝空 kind；对 structured marker 让 `np.dtype` 抛出不透明 `TypeError`；把 msgpack-numpy **标量**（可省略 `kind`）误当成必须解码的 ndarray marker。 +- 验收：分别覆盖 vLLM-native `kind`、手写 `kind=b""`、真实 `msgpack_numpy.packb` 观测 round-trip、structured `kind=V` 拒绝，以及无 `kind` 的用户 dict 原样保留。^[PR #6051] + +## SERV-7e — realtime/video-stream 音频事件必须使用 `response.output_audio.*` + +- 触发：修改 realtime WebSocket、video stream serving、示例客户端或协议文档中的音频 delta/done 事件名。 +- 强制：服务端发出的增量与终态音频事件类型必须是 `response.output_audio.delta` / `response.output_audio.done`（及配套 transcript 命名），与已迁移的 OpenAI realtime 合同一致；示例与测试断言同一集合。 +- 禁止：在任一生产发送路径残留 legacy `response.audio.delta` / `response.audio.done`；只改文档/示例而漏改 `realtime_connection` 或 `serving_video_stream`/`video_stream_base`。 +- 验收：duplex/realtime/video-stream 测试收集到的音频事件类型集合等于 `response.output_audio.*`，且不接受仅 legacy 名作为成功合同。^[PR #7426] + +## SERV-7g — 公开 serve 入口退役必须在所有 producer 上 fail closed + +- 触发：删除或改名 serve CLI option、wrapper/chart key 或公开配置文件入口。 +- 强制：parser、headless builder、脚本、chart、测试和公开 serving 文档在同一批迁移到 + canonical replacement；已删除的外部 key 必须显式拒绝并给出迁移目标。当前 serve + 部署 YAML 的唯一公开 flag 是 `--deploy-config`。 +- 禁止：删除是有意 breaking change 时又加静默 alias/fallback；wrapper 接受但忽略旧值； + 留下仍会发出旧 flag 的可运行命令。 +- 验收：replacement 可解析且到达最终 consumer，旧 option parser 失败；standard/headless + 发出同一 canonical 值，wrapper/chart 同时有正向与旧 key 拒绝测试，公开 serve 调用扫描 + 不再命中退役 flag。 ^[PR #5647] + +## SERV-7h — 兼容性按 ingress 分类，不能按同名 symbol 推断 + +- 触发：公开 CLI 已移除，但 Python/direct API、offline example 或 resolver 仍保留同名字段。 +- 强制：分别列出 public CLI、wrapper/chart、offline example、Python/direct 和内部 resolver； + public 行为由 Serving owner 决定，仍保留的 loader/schema 语义交给 + [Configuration rules](../configuration/rules.md) 并记录明确 follow-up 边界。 +- 禁止：因为仓库仍出现 `stage_configs_path` 就恢复公开 serve flag；也不能因为 public parser + 已删除就宣称所有内部兼容路径已移除。 +- 验收：公开 rejection 与 canonical forwarding 测试通过；每个暂留内部/direct 路径有独立 + compatibility test，直到后续迁移显式删除。 ^[PR #5647] diff --git a/knowledge/repos/vllm-omni/docs/_index.md b/knowledge/repos/vllm-omni/docs/_index.md index 4769b15a8..ed85f4a97 100644 --- a/knowledge/repos/vllm-omni/docs/_index.md +++ b/knowledge/repos/vllm-omni/docs/_index.md @@ -1,7 +1,7 @@ --- title: "vLLM-Omni 文档" created: 2026-07-10 -updated: 2026-09-15 +updated: 2026-09-22 type: index tags: [vllm-omni, docs] sources: ["PR #5715", "PR #6029", "PR #6858", .claude/skills/readme.md, README.md, docs/configuration/README.md, docs/contributing/README.md, docs/getting_started/installation/README.md, docs/getting_started/installation/gpu/cuda.inc.md, docs/getting_started/installation/npu/npu.inc.md] @@ -38,3 +38,5 @@ sources: ["PR #5715", "PR #6029", "PR #6858", .claude/skills/readme.md, README.m commit tag，文档称只保留 newest 14。各 tag/image 的 live availability 仍须发布时核验。^[PR #6858] - README 将 full-duplex realtime serving 标为 experimental；公开说明、验收和支持承诺 不能把它写成 stable feature。 + +- [rules](rules.md) 新增核对：DOCS-1a。 diff --git a/knowledge/repos/vllm-omni/docs/rules.md b/knowledge/repos/vllm-omni/docs/rules.md new file mode 100644 index 000000000..f3400cbae --- /dev/null +++ b/knowledge/repos/vllm-omni/docs/rules.md @@ -0,0 +1,23 @@ +--- +title: "vLLM-Omni 文档规则" +created: 2026-09-22 +updated: 2026-09-22 +type: rule +tags: [vllm-omni, docs] +sources: ["PR #5715"] +--- + +# vLLM-Omni 文档规则 + +## DOCS-1a — release version 与支持声明必须作为一个文档集合更新 + +- 触发：发布 vLLM-Omni 版本，更新 vLLM compatibility、安装命令、CUDA/NPU 镜像或支持 + 模型列表。 +- 强制：同一变更同步核对 root README 的发布公告/节奏、configuration 的 upstream 链接、 + installation compatibility、CUDA package/image pin、NPU branch/image pin 和 supported-model + 声明；每处版本都来自本次发布矩阵，而不是从相邻示例复制。 +- 禁止：只更新首页版本而保留旧安装 pin；CUDA 与 NPU 文档指向不同发布代际；链接文字 + 已更新但目标仍是旧 upstream；用一次发布的具体版本号写成永久规则。 +- 验收：对文档树做 repository-wide 的旧版本/旧分支扫描并逐个归类预期保留项；严格模式 + 构建完整文档，检查内部链接、includes 与导航；从每个公开安装入口回读到同一兼容矩阵。 + ^[PR #5715] diff --git a/knowledge/repos/vllm-omni/models/bagel/_index.md b/knowledge/repos/vllm-omni/models/bagel/_index.md index f7b9b6310..62b6cb2a2 100644 --- a/knowledge/repos/vllm-omni/models/bagel/_index.md +++ b/knowledge/repos/vllm-omni/models/bagel/_index.md @@ -1,7 +1,7 @@ --- title: "BAGEL（统一模型多形态部署参照）" created: 2026-07-21 -updated: 2026-09-05 +updated: 2026-09-22 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #6359", "PR #7049", vllm_omni/model_executor/models/bagel/, vllm_omni/diffusion/models/bagel/, vllm_omni/deploy/bagel.yaml] @@ -44,7 +44,7 @@ sources: ["PR #6359", "PR #7049", vllm_omni/model_executor/models/bagel/, vllm_o | 遇到什么 | 查看哪里 | 说明 | |---|---|---| | KV 桥接、3 路 CFG、MoT、变体拓扑 | [architecture](architecture.md) | AR→DiT 数据流与 reviewer 陷阱 | -| CFG 分支 position ID 打包、BAGEL-3 step wave | [rules](rules.md) | 1-D RoPE 与 Lance mRoPE 的序列轴合同；BAGEL 专有 step/packing、有效几何与 fallback 清理 | +| CFG 分支 position ID 打包、BAGEL-3 step wave | [rules](rules.md) | 1-D RoPE 与 Lance mRoPE 的序列轴合同；BAGEL 专有 step/packing、有效几何与 fallback 清理 新增核对：BAGEL-4。 | ## 配置与 checkpoint 差异 diff --git a/knowledge/repos/vllm-omni/models/bagel/rules.md b/knowledge/repos/vllm-omni/models/bagel/rules.md index 478792595..c504b5431 100644 --- a/knowledge/repos/vllm-omni/models/bagel/rules.md +++ b/knowledge/repos/vllm-omni/models/bagel/rules.md @@ -1,10 +1,10 @@ --- title: "BAGEL 实现规则" created: 2026-09-02 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #5775", "PR #5884", "PR #6359", "PR #7049", vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/diffusion_engine.py, vllm_omni/diffusion/models/bagel/bagel_transformer.py, vllm_omni/diffusion/models/bagel/pipeline_bagel.py, vllm_omni/diffusion/models/lance/lance_transformer.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/test_diffusion_engine_dummy_run.py, tests/diffusion/models/bagel/test_step_execution.py] +sources: ["PR #5775", "PR #5884", "PR #6359", "PR #7049", vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/diffusion_engine.py, vllm_omni/diffusion/models/bagel/bagel_transformer.py, vllm_omni/diffusion/models/bagel/pipeline_bagel.py, vllm_omni/diffusion/models/lance/lance_transformer.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/test_diffusion_engine_dummy_run.py, tests/diffusion/models/bagel/test_step_execution.py, "PR #7287"] confidence: high --- @@ -93,3 +93,11 @@ step request abort/finish 后，runner 必须先清理 request state、paged dif step request finish/abort 后的 explicit text full-forward fallback。schedule regression 必须断言 `N < 2` 在 complete 和 step image path 均被拒绝，且不将 denoise update 数写成 `N`；dummy-run regression 必须对 request 和 step 两种 mode 都断言两个 inference steps。^[PR #6359] ^[PR #7049] + +## BAGEL-4 — 显式输出画布优先于 KV image_shape 与源图尺寸 + +**适用范围**：`/v1/images/edits` 与 BAGEL img2img/`get_bagel_pre_process_func`/`_forward_single` 的 height/width 决议，含 two-stage 的 `kv_metadata["image_shape"]`。 + +**合同**：画布优先级为 **显式请求 > `kv_metadata["image_shape"]` > 输入图像推导**。`size=auto` 在 API 层解析为源尺寸后，必须用 `height_not_provided`/`width_not_provided` 标记非意图画布；preprocessor 对派生画布同样置位。显式边长按 latent stride 向下对齐并告警；超过 checkpoint 上限以 `OmniClientError`（HTTP 400）拒绝。源图仍可为 VAE/ViT prefill 做 stride 对齐，但不得覆盖已请求的输出画布。非流式 generation/edit 响应的 `size` 必须来自实际生成图像。 + +**验收**：显式 `size` 不被源图或 AR 发布的 KV shape 覆盖；`size=auto` 与纯离线 img2img 保持派生对齐；越界拒绝；响应 `size` 与解码图像一致。^[PR #7287] diff --git a/knowledge/repos/vllm-omni/models/cosmos3/_index.md b/knowledge/repos/vllm-omni/models/cosmos3/_index.md index d4a07f0c8..2a95ee543 100644 --- a/knowledge/repos/vllm-omni/models/cosmos3/_index.md +++ b/knowledge/repos/vllm-omni/models/cosmos3/_index.md @@ -1,7 +1,7 @@ --- title: "Cosmos3" created: 2026-07-20 -updated: 2026-09-05 +updated: 2026-09-22 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #4657", "PR #5001", "PR #5634", "PR #6920", docs/features/session_state_manager.md, recipes/cosmos3/Cosmos3-Nano.md, vllm_omni/diffusion/models/cosmos3/, vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py, vllm_omni/experimental/world_models/adapters/state_cosmos3_adapter.py, vllm_omni/platforms/rocm/platform.py, tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py] diff --git a/knowledge/repos/vllm-omni/models/cosmos3/rules.md b/knowledge/repos/vllm-omni/models/cosmos3/rules.md index 2b1008121..821c9aeb4 100644 --- a/knowledge/repos/vllm-omni/models/cosmos3/rules.md +++ b/knowledge/repos/vllm-omni/models/cosmos3/rules.md @@ -1,10 +1,10 @@ --- title: "Cosmos3 规则" created: 2026-07-20 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #4657", "PR #5001", "PR #5634", "PR #6049", docs/features/session_state_manager.md, recipes/cosmos3/Cosmos3-Nano.md, vllm_omni/diffusion/models/cosmos3/, vllm_omni/model_extras/cosmos3.py, vllm_omni/model_extras/registry.py, vllm_omni/experimental/world_models/adapters/state_cosmos3_adapter.py, vllm_omni/platforms/rocm/platform.py, tests/diffusion/models/cosmos3/test_session_memory_equivalence.py, tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py, "PR #6107", "PR #5614", "PR #6325", "PR #6913", "PR #6920"] +sources: ["PR #4657", "PR #5001", "PR #5634", "PR #6049", docs/features/session_state_manager.md, recipes/cosmos3/Cosmos3-Nano.md, vllm_omni/diffusion/models/cosmos3/, vllm_omni/model_extras/cosmos3.py, vllm_omni/model_extras/registry.py, vllm_omni/experimental/world_models/adapters/state_cosmos3_adapter.py, vllm_omni/platforms/rocm/platform.py, tests/diffusion/models/cosmos3/test_session_memory_equivalence.py, tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py, "PR #6107", "PR #5614", "PR #6325", "PR #6913", "PR #6920", "PR #7427"] confidence: high --- @@ -159,3 +159,10 @@ confidence: high 4×GB300 multi-chunk transfer（Ulysses 4、VAE patch parallelism 4、tiling，1280×720、101 frames、24 FPS、 53-frame chunks、5 conditional frames、2 steps），确认完成且 metadata/video 为 101 frames at 24 FPS。 该证据不构成广泛 parity、质量或性能声明。^[PR #6920] + +## COSMOS-1c — Distilled Cosmos3 必须在加载前拒绝 CFG parallel + +- 触发：scheduler `_class_name` 判定 distilled Cosmos3、修改 `cfg_parallel_size` admission，或 distilled 路径的 `guidance_scale` / `negative_prompt` 语义。 +- 强制：在加载 tokenizer/组件之前，若 distilled 且 `cfg_parallel_size > 1`，立即 `ValueError`，并指引 `--cfg-parallel-size 1` 与 `--ulysses-degree` 做多卡。distilled 路径强制 `guidance_scale=1.0`；调用方显式请求其他值时 `warning_once`，并说明 `negative_prompt` 不影响生成。 +- 禁止：静默忽略 CFGP 导致无收益的重复实例；先加载权重再报并行非法；把 distilled 的 guidance override 写成仍消费 negative branch。 +- 验收：参数化覆盖 distilled/non-distilled × `cfg_parallel_size`/`ulysses_degree`，断言 distilled+CFGP>1 在组件加载前失败；guidance resolver 覆盖显式非 1.0 的 warning 与强制 1.0。^[PR #7427] diff --git a/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md b/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md index 71a29853b..fc987e266 100644 --- a/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md +++ b/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md @@ -1,7 +1,7 @@ --- title: "CosyVoice3（单架构双 stage,RAS 合并停止,TRT 加速）" created: 2026-07-21 -updated: 2026-09-05 +updated: 2026-09-22 type: index tags: [vllm-omni, models] sources: ["PR #5673", "PR #5869", "PR #6424", "PR #6955", vllm_omni/data_entry_keys.py, vllm_omni/model_executor/models/cosyvoice3/, vllm_omni/deploy/cosyvoice3.yaml, vllm_omni/model_executor/stage_input_processors/cosyvoice3.py, vllm_omni/transformers_utils/configs/cosyvoice3.py] @@ -51,7 +51,7 @@ sources: ["PR #5673", "PR #5869", "PR #6424", "PR #6955", vllm_omni/data_entry_k | 遇到什么 | 查看哪里 | 说明 | |---|---|---| | RAS 停止机制、双交接注册、TRT 门 | [architecture](architecture.md) | 数据流与 reviewer 陷阱 | -| CosyVoice3 TensorRT CFM 的 stream handoff、allocator lifetime、context pool、plan 并发发布、tmp 所有权或 cleanup failure | [rules](rules.md) | `COSYVOICE3-1a/1b` 的顺序、发布边界与验收 | +| CosyVoice3 TensorRT CFM 的 stream handoff、allocator lifetime、context pool、plan 并发发布、tmp 所有权或 cleanup failure | [rules](rules.md) | `COSYVOICE3-1a/1b` 的顺序、发布边界与验收 新增核对：COSYVOICE3-2a、COSYVOICE3-1c。 | ## 配置与 checkpoint 差异 diff --git a/knowledge/repos/vllm-omni/models/cosyvoice3/rules.md b/knowledge/repos/vllm-omni/models/cosyvoice3/rules.md index 985107088..a8e302f23 100644 --- a/knowledge/repos/vllm-omni/models/cosyvoice3/rules.md +++ b/knowledge/repos/vllm-omni/models/cosyvoice3/rules.md @@ -1,10 +1,10 @@ --- title: "CosyVoice3 规则" created: 2026-09-04 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, models] -sources: ["PR #5673", "PR #6955", vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py, vllm_omni/model_executor/models/cosyvoice3/flow_estimator_trt.py, tests/model_executor/models/cosyvoice3/test_cosyvoice3_components.py, benchmarks/tts/benchmark_cosyvoice3_trt_streams.py] +sources: ["PR #5673", "PR #6955", vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py, vllm_omni/model_executor/models/cosyvoice3/flow_estimator_trt.py, tests/model_executor/models/cosyvoice3/test_cosyvoice3_components.py, benchmarks/tts/benchmark_cosyvoice3_trt_streams.py, "PR #6896", "PR #7521"] confidence: high --- @@ -50,3 +50,17 @@ confidence: high 证明 replace 失败保留旧 plan、cleanup 失败保留 replace error、partial write 清理 owned tmp、 collision 保留 foreign tmp，以及两条同步 publisher 使用不同 source path 且仅留下完整 final plan。 ^[PR #6955] + +## COSYVOICE3-2a — deploy 必须在 name 推断失败时显式绑定 `pipeline: cosyvoice3` + +- 触发：修改 `vllm_omni/deploy/cosyvoice3.yaml`、CosyVoice3 的 `StageConfigFactory` / deploy 解析，或依赖 HF cache `models--…/snapshots/` 且 root `config.json` 无识别元数据的启动路径。 +- 强制：shipping deploy 必须声明 `pipeline: cosyvoice3`（或等价显式 pipeline key），使 hash-named snapshot 在 `try_infer_model_type` / name-fallback 失败时仍解析为已注册的两 LLM stage（talker + code2wav），而不是 default diffusion stage。 +- 禁止：仅依赖 basename/hash 推断；让空 `config.json` snapshot 落入 Diffusers `model_index.json` 查找并在启动时报 `Diffusers pipeline index not found`。 +- 验收：以空 `config.json` 的 `snapshots/<40-hex>` 布局回归 `create_from_model` 与 legacy stage 路径，断言 `pipeline_config.model_type == "cosyvoice3"`、两 stage 均为 LLM（非 DIFFUSION）且 model 路径保留。^[PR #6896] + +## COSYVOICE3-1c — 流式 HiFT 必须在有界 mel 窗上增量计算并携带相位 + +- 触发：修改 CosyVoice3 `_stream_hift_from_feat`、`CausalHiFTGenerator`、`SineGen`/`SineGen2` 的 `phase_acc`，或 streaming chunk 的 noise/F0 cache。 +- 强制：每个 chunk 只在有界 mel 窗上计算（末 64 帧再加上 F0 感受野历史），不得对累积全谱重跑。谐波相位用共享的边界 `phase_acc` 带入下一窗。noise 必须是固定、按位置索引的 buffer，不能用随 chunk 变化的 `torch.randn_like`。F0 predictor 的左因果卷积要吃 window 左侧的真实历史，不能零填充；这段 margin 属于 `cache_state`，在进入 SineGen/decode 前切掉，且短于感受野的 chunk 不得饿死下一次调用。finalize 除 `conv_pre_look_right` 外还要释放 predictor 自己扣下的 `trim`，否则流末约 3 个 mel 帧丢失。 +- 禁止：用 PCM16 逐字节相等当正确性门槛（亚 LSB 的再结合误差会翻边界样本）；只测未触发 voiced 的随机权重；把 `scipy`/`s3tokenizer` 放回 dev extra——它们是 base install 的硬依赖，scipy 下限保持 `>=1.11.0`。 +- 验收：相对全量重算，float64 应逐位一致；float32 用 `assert_close(atol=1e-6, rtol=1e-5)`。同时覆盖 SineGen（22050 Hz）与 SineGen2（24000 Hz）、强制 voiced，以及均匀 chunk 和单 mel 帧的小块。^[PR #7521] diff --git a/knowledge/repos/vllm-omni/models/higgs-audio/_index.md b/knowledge/repos/vllm-omni/models/higgs-audio/_index.md index 5f1e4ec32..175214992 100644 --- a/knowledge/repos/vllm-omni/models/higgs-audio/_index.md +++ b/knowledge/repos/vllm-omni/models/higgs-audio/_index.md @@ -1,7 +1,7 @@ --- title: "Higgs-Audio（V2/V3 双谱系 TTS;higgs_multimodal_qwen3 即 V3）" created: 2026-07-21 -updated: 2026-09-05 +updated: 2026-09-22 type: index tags: [vllm-omni, models] sources: ["PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/higgs_audio_v2/, vllm_omni/model_executor/models/higgs_audio_v3/, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_tokenizer.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py, vllm_omni/transformers_utils/configs/higgs_audio_v3.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py, tests/e2e/online_serving/test_higgs_audio_v3.py] @@ -53,7 +53,7 @@ sources: ["PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py | 遇到什么 | 查看哪里 | 说明 | |---|---|---| | 谱系差异表、delay-pattern 生命周期、流式窗口数学 | [architecture](architecture.md) | 数据流与 reviewer 陷阱 | -| V3 request RNG、decode state、voice-clone prompt sentinel 或 chunked prefill substitution | [rules](rules.md) | `HIGGS-1a`、`HIGGS-2a` | +| V3 request RNG、decode state、voice-clone prompt sentinel 或 chunked prefill substitution | [rules](rules.md) | `HIGGS-1a`、`HIGGS-2a` 新增核对：HIGGS-3a。 | ## 配置与 checkpoint 差异（v2 vs v3 一览） diff --git a/knowledge/repos/vllm-omni/models/higgs-audio/rules.md b/knowledge/repos/vllm-omni/models/higgs-audio/rules.md index 38ba1001d..9b03dc24e 100644 --- a/knowledge/repos/vllm-omni/models/higgs-audio/rules.md +++ b/knowledge/repos/vllm-omni/models/higgs-audio/rules.md @@ -1,10 +1,10 @@ --- title: "Higgs-Audio 规则" created: 2026-09-05 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_tokenizer.py, vllm_omni/worker/gpu_model_runner.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py, tests/e2e/online_serving/test_higgs_audio_v3.py] +sources: ["PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_tokenizer.py, vllm_omni/worker/gpu_model_runner.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py, tests/e2e/online_serving/test_higgs_audio_v3.py, "PR #7076"] confidence: high --- @@ -54,3 +54,10 @@ confidence: high 一部分的 chunked prefill 和 legacy `-100` caller 注入正确 code rows；serving smoke 证明相同 路径重写时 cache salt 仍变化。H100 voice-clone E2E 未在该 PR 的 lanes 运行，不能把 unit coverage 宣称为 end-to-end 证据。^[PR #7065] + +## HIGGS-3a — 共享 reference-code encode 必须屏蔽请求取消并以 task 身份退休 + +- 触发：修改 Higgs-Audio V3 `_resolve_higgs_audio_v3_ref_codes`、artifact-keyed inflight map。 +- 强制：同一 `artifact_key` 的 cold encode 只创建一个共享 `asyncio.Task`；creator 与 waiter 都必须 `await asyncio.shield(task)`，请求取消不得取消共享 encode。inflight 条目只能在 task `done_callback` 里用 identity guard 弹出；未取消任务须 `t.exception()` 消费失败以免 “exception never retrieved”。成功路径仍 clone 后返回，cache 写入保持既有合同。 +- 禁止：直接 `await task` 让 caller cancellation 传入共享 flight；在 creator 的 `finally` 里按请求退出撤销 inflight（会让仍在跑的 task 失位并可能重复 encode）。 +- 验收：确定性 CPU 回归覆盖 creator+waiter 取消后幸存者仍从同一 encode 完成、全取消后失败路径无 loop exception handler 且 inflight 清空；与 MOSS single-flight 生命周期对齐但不改 cache key/容量/API。^[PR #7076] diff --git a/knowledge/repos/vllm-omni/models/hunyuan-image3/_index.md b/knowledge/repos/vllm-omni/models/hunyuan-image3/_index.md index 0747cc236..c071e9f82 100644 --- a/knowledge/repos/vllm-omni/models/hunyuan-image3/_index.md +++ b/knowledge/repos/vllm-omni/models/hunyuan-image3/_index.md @@ -1,7 +1,7 @@ --- title: "HunyuanImage3" created: 2026-07-10 -updated: 2026-09-05 +updated: 2026-09-22 type: index tags: [vllm-omni, models, hunyuan-image3] sources: ["PR #5541", "PR #6563", "PR #4048", vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py, vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py, vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_tokenizer.py, vllm_omni/diffusion/models/hunyuan_image3/request_layout.py] diff --git a/knowledge/repos/vllm-omni/models/hunyuan-image3/rules.md b/knowledge/repos/vllm-omni/models/hunyuan-image3/rules.md index 9e1f4a20b..7f27b296c 100644 --- a/knowledge/repos/vllm-omni/models/hunyuan-image3/rules.md +++ b/knowledge/repos/vllm-omni/models/hunyuan-image3/rules.md @@ -1,10 +1,10 @@ --- title: "HunyuanImage3 开发规则" created: 2026-07-13 -updated: 2026-09-08 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, hunyuan-image3] -sources: [incidents/painterly/_index.md, hf-alignment-pitfalls.md, vllm_omni/diffusion/models/hunyuan_image3/prompt_utils.py, vllm_omni/model_extras/hunyuan_image3.py, vllm_omni/model_extras/registry.py, "PR #6094", "vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py", "PR #6306", "PR #6102", "PR #6563", "PR #4048", vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py, vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_tokenizer.py, vllm_omni/diffusion/models/hunyuan_image3/request_layout.py, tests/diffusion/models/hunyuan_image3/test_hunyuan_image3_step_execution.py, tests/diffusion/models/hunyuan_image3/test_image_kv_cache_manager.py, "PR #7021"] +sources: [incidents/painterly/_index.md, hf-alignment-pitfalls.md, vllm_omni/diffusion/models/hunyuan_image3/prompt_utils.py, vllm_omni/model_extras/hunyuan_image3.py, vllm_omni/model_extras/registry.py, "PR #6094", "vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py", "PR #6306", "PR #6102", "PR #6563", "PR #4048", vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py, vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_tokenizer.py, vllm_omni/diffusion/models/hunyuan_image3/request_layout.py, tests/diffusion/models/hunyuan_image3/test_hunyuan_image3_step_execution.py, tests/diffusion/models/hunyuan_image3/test_image_kv_cache_manager.py, "PR #7021", "PR #6111"] --- # HunyuanImage3 开发规则 @@ -170,3 +170,10 @@ sources: [incidents/painterly/_index.md, hf-alignment-pitfalls.md, vllm_omni/dif - 强制：公开 `get_expert_mapping()` 必须返回 flat `list[tuple[str, str, int, str]]`（非 MoE 返回 `[]`），供上游 loader 消费；Hunyuan checkpoint 的 `gate_proj`/`up_proj` → `gate_and_up_proj` 分片 remapping 只能放在独立的 `_get_expert_weights_remapping()`，由 `load_weights` 显式取用。AR 与 diffusion transformer 两侧同一合同。 - 禁止：让 `get_expert_mapping()` 再返回 `(mapping, remapping)` 二元组，或把 Omni-local remapping 塞进上游期望的 public API。 - 验收：对 `HunyuanModel` 与 `HunyuanImage3Model` 断言 public mapping 为 flat list、local remapping 字典独立，并覆盖非 MoE 空返回。^[PR #7021] + +## HY3-10a — 文本-only chat 必须按请求标为 comprehension，不得走图像 stage 强制 + +- 触发：修改 HunyuanImage3 AR `sample()`、`engine_output_type="latent"` 的 generation 部署上的 chat completions、`SamplingParams.extra_args["ar_task_mode"]`，或 AR runner 向 custom sampler 转发 `extra_args`。 +- 强制：任务模式是 per-request 属性。serving 对 modalities 仅为 text 的 chat 在普通 `SamplingParams` 上 `setdefault("ar_task_mode", "comprehension")`，不覆盖调用方显式值，也不标记 image/audio/video 输出请求。声明 `model_sampler_wants_extra_args` 的模型必须经共享 dispatcher 按 `req_ids` 顺序收到每行 `extra_args`。comprehension 行只做无状态图像脚手架 token 屏蔽（含 ``/``/``/`` 等），禁止 `→` 等 generation stage 强制。 +- 禁止：仅凭引擎级 `engine_output_type` 让文本答案走图像 stage 转移；把 DiT scaffold token 漏进文本答案；破坏既有两参数 custom sampler 合同。 +- 验收：混合 comprehension/generation batch、生产 chat 入口的 text-only 路由、GPU/NPU 转发与未标记行的 generation 行为保持不变。^[PR #6111] diff --git a/knowledge/repos/vllm-omni/models/lingbot-world/_index.md b/knowledge/repos/vllm-omni/models/lingbot-world/_index.md index 3b3c62fa7..a48335f51 100644 --- a/knowledge/repos/vllm-omni/models/lingbot-world/_index.md +++ b/knowledge/repos/vllm-omni/models/lingbot-world/_index.md @@ -1,7 +1,7 @@ --- title: "LingBot World" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-22 type: index tags: [vllm-omni, models, diffusion] sources: [vllm_omni/diffusion/models/lingbot_world/] @@ -50,4 +50,4 @@ confidence: high | 遇到什么 | 查看哪里 | |---|---| -| 该模型的硬门禁规则 | 尚未沉淀；由逐 commit 同步命中该 owner 时在 `rules.md` 建立 | +| 该模型的硬门禁规则 | [session lifecycle 规则](rules.md)：`LBW-1a`、`LBW-1b` 新增核对：LBW-1a、LBW-1b。 | diff --git a/knowledge/repos/vllm-omni/models/lingbot-world/rules.md b/knowledge/repos/vllm-omni/models/lingbot-world/rules.md new file mode 100644 index 000000000..7e0e13991 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/lingbot-world/rules.md @@ -0,0 +1,31 @@ +--- +title: "LingBot World session lifecycle 规则" +created: 2026-09-22 +updated: 2026-09-22 +type: rule +tags: [vllm-omni, models, diffusion] +sources: ["PR #6533", "PR #6838", vllm_omni/diffusion/models/lingbot_world/pipeline.py] +--- + +# LingBot World session lifecycle 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批 live 源码 | +|---|---|---| +| streaming VAE、session reset/close、admission bytes | `session-decode`：`LBW-1a` | `lingbot_world/pipeline.py` session hooks → `experimental/ar_diffusion/streaming_decode.py` | +| condition block、encoder history、temporal RoPE | `condition-history`：`LBW-1b` | `lingbot_world/pipeline.py` condition state → `lingbot_world/transformer.py` RoPE → model admission | + +## LBW-1a — AR 流式 VAE decode 状态必须按 session 持有并计入 admission + +- 触发：修改 LingBot/Wan 系 AR-Diffusion `post_decode`、streaming VAE decode、`SupportsStreamingDecode`，或 `model_owned_state_bytes_per_session`。 +- 强制：每个 `session_id`（与 `request_id` 同键）持有独立 `StreamingDecodeState`；chunk 间复用同一 temporal cache；`reset_ar_diffusion_session`/`close_ar_diffusion_session` 一并释放。`model_owned_state_bytes_per_session` 必须计入 decoder 按分辨率声明的常驻字节，不能只算 image condition。会走 VAE tiled decode 的 shape 不得冒充可跨 chunk 线程 cache。 +- 禁止：把 temporal cache 留在共享 VAE 模块上跨 session 覆写；块级独立 decode 却声称 timeline 连续；漏报 decode state 导致 admission 低估显存。 +- 验收：覆盖跨 chunk 连续性、session 隔离、release、非流式/tiling fallback，以及 admission 字节随 H×W 缩放不随 session 长度增长。^[PR #6533] + +## LBW-1b — AR 条件编码历史必须会话持有、双份计入并在终止边界释放 + +- 触发：修改 AR-Diffusion realtime/stepwise 条件编码、Wan VAE encoder cache、session admission 字节预算，或 temporal RoPE 超出预计算表。 +- 强制：跨 block 推进时保留 causal encoder history（committed + in-flight），每 session 只驻留当前 condition block 与有界 cache；admission 必须计入两份 encoder history 与 streaming-decode 字节。reset/close 必须释放 encoder cache。超出预计算 RoPE 表的 temporal 位置按绝对位置即时算 cos/sin，不得扩张常驻表。条件编码要求 unpatched、非 tiled Wan encoder。 +- 禁止：用固定像素/latent 帧上限冒充无界 realtime；只预算 self-KV 而漏算 encoder cache；在失败 block 后提交 pending history；把本变更写成已解决共享 paged-KV 长度上限。 +- 验收：因果条件递进、session 隔离/清理、失败不提交、内存会计、tick/stepwise 一致，以及 RoPE 越界且 cache 尺寸固定。^[PR #6838] diff --git a/knowledge/repos/vllm-omni/models/ltx2/_index.md b/knowledge/repos/vllm-omni/models/ltx2/_index.md index 5980435df..71533768b 100644 --- a/knowledge/repos/vllm-omni/models/ltx2/_index.md +++ b/knowledge/repos/vllm-omni/models/ltx2/_index.md @@ -1,7 +1,7 @@ --- title: "LTX-2 家族（含 LTX-2.3/2.5）" created: 2026-07-16 -updated: 2026-09-05 +updated: 2026-09-22 type: index tags: [vllm-omni, models, ltx2] sources: ["PR #6847", vllm_omni/diffusion/models/ltx2/, vllm_omni/diffusion/registry.py, recipes/LTX/LTX-2.md, recipes/LTX/LTX-2.5.md] @@ -38,4 +38,4 @@ sources: ["PR #6847", vllm_omni/diffusion/models/ltx2/, vllm_omni/diffusion/regi | 遇到什么 | 查看哪里 | |---|---| | 模型结构、serving 方式与已有性能/精度证据 | [architecture](architecture.md) | -| LTX-2.5 DiffVAE 选择、artifact、并行或最小几何约束 | [LTX-2.5 decoder rules](rules.md) | +| LTX-2.5 DiffVAE 选择、artifact、并行或最小几何约束 | [LTX-2.5 decoder rules](rules.md) 新增核对：LTX-4、LTX25-3。 | diff --git a/knowledge/repos/vllm-omni/models/ltx2/rules.md b/knowledge/repos/vllm-omni/models/ltx2/rules.md index 6f2f149b4..819e00d6d 100644 --- a/knowledge/repos/vllm-omni/models/ltx2/rules.md +++ b/knowledge/repos/vllm-omni/models/ltx2/rules.md @@ -1,10 +1,10 @@ --- title: "LTX-2.5 DiffVAE 规则" created: 2026-09-04 -updated: 2026-09-05 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, ltx2, diffusion] -sources: ["PR #6189", "PR #7000", "PR #7020", recipes/LTX/LTX-2.5.md, requirements/common.txt, docs/user_guide/diffusion/attention_backends/huggingface_hub.md, vllm_omni/diffusion/models/ltx2/ltx2_components.py, vllm_omni/diffusion/models/ltx2/ltx2_conditioning.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder_distributed.py, vllm_omni/diffusion/models/ltx2/ltx2_latents.py, vllm_omni/diffusion/models/ltx2/ltx2_request.py, vllm_omni/diffusion/models/ltx2/ltx2_runtime.py, tests/diffusion/models/ltx2/test_ltx25_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_output_cuda.py, tests/diffusion/models/ltx2/test_ltx2_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_vae.py] +sources: ["PR #6189", "PR #7000", "PR #7020", recipes/LTX/LTX-2.5.md, requirements/common.txt, docs/user_guide/diffusion/attention_backends/huggingface_hub.md, vllm_omni/diffusion/models/ltx2/ltx2_components.py, vllm_omni/diffusion/models/ltx2/ltx2_conditioning.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder_distributed.py, vllm_omni/diffusion/models/ltx2/ltx2_latents.py, vllm_omni/diffusion/models/ltx2/ltx2_request.py, vllm_omni/diffusion/models/ltx2/ltx2_runtime.py, tests/diffusion/models/ltx2/test_ltx25_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_output_cuda.py, tests/diffusion/models/ltx2/test_ltx2_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_vae.py, "PR #7231", "PR #7350"] confidence: high --- @@ -71,3 +71,17 @@ confidence: high 对比。有效 `[0, 1]` 的 `do_normalize=False` 输入须与 unclipped legacy path 比较；另测 intentional out-of-range defensive clamp。FP32 presentation bytes 必须 exact，BF16/FP16 相对 legacy FP32 presentation 最多相差一个 uint8 level；官方 similarity 仅是本 PR 所报 LTX matrix 的限定证据。^[PR #7000] + +## LTX-4 — LTX vocoder 的 CUDA 路径必须 scoped 启用并恢复 `cudnn.deterministic` + +- 触发：修改 `ltx2_runtime._run_ltx_vocoder` / `_deterministic_ltx_vocoder`、BWE vocoder 调用，或 accuracy reference runner 的官方 vocoder forward。 +- 强制：仅在 `device_type == "cuda"` 时进入 context：将 `torch.backends.cudnn.deterministic` 设为 `True`，成功与异常退出都恢复调用前值；不得改动其他 cuDNN 开关。保留既有 BWE FP32 autocast / MPS float 合同。官方 reference runner 对 pinned BWE vocoder 施加同一 scoped pin，使两侧音频可稳定对照。 +- 禁止：把 process-wide deterministic 永久打开；假设并发 worker 可安全争用该全局 flag；把偶发相对-L2 过线当作已消除算法漂移。 +- 验收：CPU 覆盖初始 True/False 与注入异常后均恢复，且 `enabled`/`benchmark`/`allow_tf32` 等其他设置不变；相似度/固定 mel 实验只作有界证据，须在相同硬件与序列化执行假设下解读。^[PR #7231] + +## LTX25-3 — DiffVAE pointwise fusion 与 TileLang FNA 必须按 SM 独立门禁 + +- 触发：修改 `resolve_ltx2_vae_operators`、fusion/FNA compute-capability allowlist，或 LTX DiffVAE 加速准入。 +- 强制：pointwise fusion 与 TileLang FNA 分两个 capability 集合独立判定；已验证的 SM100/SM103 可开自校验 fusion（首用 bit-exact，失败回退普通数学），FNA 在未单独验证前保持 SM90-only。不得用单一 allowlist 绑死两条路径。 +- 禁止：因 FNA 未验证而关闭已合格 SM 上的 fusion；或把 fusion 资格静默推广成 FNA/NATTEN schedule 已验证。 +- 验收：平台单测覆盖 SM90 fusion+FNA、SM100/103 fusion-only、以及更低/未列 SM 全关。^[PR #7350] diff --git a/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md b/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md index f16b0fbc7..c87ac8ae3 100644 --- a/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md +++ b/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md @@ -1,7 +1,7 @@ --- title: "MammothModa2（Preview/Dev 的 AR→DiT 与 AR-only 拓扑）" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-22 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #6694", vllm_omni/model_executor/models/mammoth_moda2/, vllm_omni/diffusion/models/mammoth_moda2/, vllm_omni/model_executor/stage_input_processors/mammoth_moda2.py, vllm_omni/deploy/mammoth_moda2.yaml, vllm_omni/deploy/mammoth_moda2_ar.yaml] diff --git a/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md index d4b5cca86..f26c872ff 100644 --- a/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md +++ b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md @@ -1,10 +1,10 @@ --- title: "MammothModa2 规则" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, model-executor, diffusion] -sources: ["PR #6694", vllm_omni/model_extras/mammothmodal2_preview.py, vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py, vllm_omni/model_executor/stage_input_processors/mammoth_moda2.py, vllm_omni/diffusion/models/mammoth_moda2/pipeline_mammothmoda2_dit.py, tests/model_extras/test_model_extras.py, tests/e2e/offline_inference/test_mammoth_moda2_expansion.py] +sources: ["PR #6694", vllm_omni/model_extras/mammothmodal2_preview.py, vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py, vllm_omni/model_executor/stage_input_processors/mammoth_moda2.py, vllm_omni/diffusion/models/mammoth_moda2/pipeline_mammothmoda2_dit.py, tests/model_extras/test_model_extras.py, tests/e2e/offline_inference/test_mammoth_moda2_expansion.py, "PR #7102"] confidence: high --- @@ -50,3 +50,10 @@ confidence: high empty visual condition、extra-body 参数透传和 AR-only text control。PR 作者报告的 54 个 CPU tests 与单次 L20X Preview T2I PNG smoke 只证明该提交环境的路径可运行；golden-image E2E 仍因既有 issue skip，不能据此声称质量、吞吐、Dev T2I 或跨硬件能力。^[PR #6694] + +## MAMMO-1c — AR→DiT `full_hidden_states` 必须保持源 FP16/BF16 直至 DiT 消费 + +- 触发：修改 `ar2dit`、`serialize_additional_information` / EngineCore payload，或 DiT `_split_ar_conditions` 的条件行选取与最终 `.to(device, dtype)`。 +- 强制：跨 stage 传输保持 AR 源 dtype（FP16/BF16）的 contiguous `full_hidden_states`；EngineCore/OmniSerializer 以 raw bytes 保留 bfloat16，不得为“过 numpy 边界”无条件 `.float()` 扩成 FP32。DiT 在选完 text/image 条件行之后，才在既有最终转换点落到 model device/dtype。 +- 禁止：在 bridge 或 condition split 处提前 host-side float32 加倍 staging 与 H2D；把精度优化扩成 request-end streaming / Layer-2 lifecycle 设计；或假定旧 FP32 wire 注释仍有效。 +- 验收：FP16/BF16 经 serialize/deserialize 往返 dtype 与数值不变；condition split 输出保持 transfer dtype 直至最终 cast；connector 级 BF16 round-trip 覆盖。^[PR #7102] diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md index 20ac364fc..9712b8554 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md @@ -1,7 +1,7 @@ --- title: "MiniCPM-o 4.5" created: 2026-07-20 -updated: 2026-09-06 +updated: 2026-09-22 type: index tags: [vllm-omni, models, model-executor] sources: ["PR #3642", "PR #5382", "PR #5524", "PR #5638", "PR #5792", "PR #5869", "PR #6056", "PR #6154", "PR #6170", "PR #6318", "PR #6404", "PR #6678", "PR #6628", tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/engine/duplex/test_duplex_deploy_config.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, vllm_omni/experimental/fullduplex/client.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, vllm_omni/platforms/npu/models/minicpmo_4_5_code2wav.py, "PR #6082", "PR #6587"] @@ -34,7 +34,7 @@ Step-Audio2 的 token2wav。engine cache/profile 与 fallback 门禁见 MCPMO-1c 四份 bundled deploy 在 CUDA 上另默认开启 HiFT graph：只 capture pre-iSTFT 子图，按 connector chunk/cache shape 预捕并限量 lazy capture，且不服从 stage `enforce_eager`；非 CUDA 回 eager。 HiFT/CFM 的 shape、显存、并发、整代退休与部分-graph 性能证据边界见 -[Code2Wav CUDA graph 规则](rules-cuda-graphs.md)。 +[Code2Wav CUDA graph 规则](rules-cuda-graphs.md)。 新增核对：MCPMO-1k。 Thinker 的 Whisper/APM audio encoder 仍构造 dense `[B,1,T,T]` mask；chunk mask 已用 broadcasted query/key index 代替逐 row Python fill，但不改变 chunk/left-context/lookahead @@ -51,7 +51,7 @@ Online serving CI 的 core suite 固定 async chunk，expansion 分开覆盖 syn fixture 不再覆盖 eager/KV cap 并固定 active-speech turn window；验证证据边界见 MCPMO-5b。 描述直达源码与模型专有门禁见 [rules](rules.md#direct-代码快速入口)；并发 duplex 的 ragged -Talker chunk、full-attention 容量 rollover、shipping profile 与 chat 隔离见 [native duplex 规则](rules-duplex.md) +Talker chunk、full-attention 容量 rollover、shipping profile 与 chat 隔离见 [native duplex 规则](rules-duplex.md) 新增核对：MCPMO-4j。 ；Code2Wav state、batch cap 与 NPU residual limits 见 [Code2Wav 并发批处理规则](rules-code2wav-batching.md)。新模型语义验证见 [model validation](../../review/guides/model-validation.md)。 @@ -69,3 +69,5 @@ Realtime video 不以「每 200 ms append」定义 frame cadence：首个约 103 [Model Executor rules](../../components/model-executor/rules.md)。 - [MiniCPM-o 4.5 Code2Wav 运行时合同](rules-code2wav-runtime.md)：`MCPMO-1f`, `MCPMO-1g`, `MCPMO-1h`, `MCPMO-1i`。 + +- [rules-loader](rules-loader.md) 新增核对：MCPMO-5a2。 diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md index 816847257..e2b3e1cdd 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md @@ -1,10 +1,10 @@ --- title: "MiniCPM-o 4.5 Code2Wav CUDA graph 规则" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, model-executor] -sources: [vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, "PR #5869", "PR #6082", "PR #6587"] +sources: [vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, "PR #5869", "PR #6082", "PR #6587", "PR #7416"] confidence: high --- @@ -26,3 +26,11 @@ confidence: high - 强制：warmup 或 capture 发生异常时，在该 wrapper 的剩余生命周期内保持禁用并退休其自有 generation；不能在可能脏的 capture stream/allocator 状态继续捕获。无法由 key 重建的 dtype 仅标记该 shape eager，不得禁用其他可捕获 shape；capture admission 不得依赖 device free-memory。维护固定字段的 calls/hits/captures/flushes/eager 累计 telemetry。 - 禁止：捕获整个 `_estimator_step` 或手写替代 graph target；使用无界 shape cache、逐图 eviction、跨 owner 的 process-wide cleanup，或在 nested capture 中重放；捕获失败后伪装成功；把 CFM graph 的配置、性能或 parity 结论外推到 HiFT、encoder、TRT 或其他模型。 - 验收：CUDA 测试覆盖 uncached/cached shape 的 graph/eager 数值 parity、`None` cache parity、lazy capture、整代 flush 后 HiFT replay parity、capture/warmup failure 后不再 capture、unsupported dtype shape-local eager 与 `max_graphs <= 0` eager；CPU/mock 覆盖 active capture/non-CUDA、free-memory 非 gating、telemetry 和 inert non-CUDA memory reporting，并确认 deploy 配置值实际到达 estimator。^[PR #6082] ^[PR #6587] + +## MCPMO-1k — CFM graph 的变长 chunk 必须先对齐帧桶，padding 不得进入 cache + +- 触发：修改 MiniCPM-o 4.5 Code2Wav 的 CFM decode chunk、`cfm_graph_bucket_frames`、reference-audio 长度，或 attention/CNN cache 边界。 +- 强制：捕获前把 mel 帧 pad 到 `bucket_frames` 的倍数，输出再裁回有效帧，使 chunk 长度落在同一 grid。每份开启 CFM graph 的 shipping profile 必须显式写 `cfm_graph_bucket_frames`（当前 16），不能靠代码默认。`bucket_frames <= 1`、graphs 已禁用、ragged valid-lengths，或 pad 会越过 decoder noise capacity 时不 pad。 +- 强制：padded 列必须被真实 `attn_mask` 排除；零 key 仍会计入 softmax 分母。成为下一 chunk 的 key 或左上下文之前，清零 attention cache 的 padded 列，以及各 block CNN cache 中来自 padding 的尾部。speaker 向量会铺满每一帧，清零必须作用在 clone 上。conditioning pad 复制最后一帧有效值。reference audio 先转 mono、抗混叠重采样到 24 kHz，再按 `ref_audio_max_seconds`（默认 6s）截断或补零，使各请求共享一个 L0；默认 prompt 走同一路径，`soundfile` 的 `(samples, channels)` 先转置。 +- 禁止：让首尾 chunk 或 `plan_token2wav_encode_slices` 的任意长度各自成为 graph key；只清零 `x`/`mu`/`cond` 而留下非零 speaker 列；把桶化 parity 外推到 HiFT、TRT 或其他模型。 +- 验收：覆盖 pad/trim、noise overflow 不 pad、attn_mask 与 cache 清零、reference 归一化后的单一 L0，以及四份 CFM-graph profile 都带上 `cfm_graph_bucket_frames`。^[PR #7416] diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md index 261f292e9..80024a380 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md @@ -1,10 +1,10 @@ --- title: "MiniCPM-o 4.5 native duplex 规则" created: 2026-09-04 -updated: 2026-09-08 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #6318", "PR #6346", "PR #6404", "PR #6458", "PR #6619", "PR #6630", "PR #6678", "PR #6626", "PR #6767", "PR #6821", vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, vllm_omni/deploy/minicpmo_4_5_2gpu.yaml, vllm_omni/deploy/minicpmo_4_5_3gpu.yaml, vllm_omni/deploy/minicpmo_4_5_8x4090.yaml, examples/online_serving/minicpmo/realtime_duplex_demo.py, vllm_omni/experimental/fullduplex/client.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/session.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/runtime_adapter.py, vllm_omni/entrypoints/duplex/runtime_bridge.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/experimental/fullduplex/video_stacking.py, tests/config/test_config_factory.py, tests/engine/duplex/test_duplex_deploy_config.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, tests/entrypoints/openai_api/test_duplex_handler.py, tests/examples/test_minicpmo_realtime_duplex_simple_demo.py, "PR #6529", vllm_omni/entrypoints/duplex/protocol.py, vllm_omni/entrypoints/duplex/realtime_state.py, tests/entrypoints/openai/test_duplex_protocol.py, "PR #6799"] +sources: ["PR #6318", "PR #6346", "PR #6404", "PR #6458", "PR #6619", "PR #6630", "PR #6678", "PR #6626", "PR #6767", "PR #6821", vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, vllm_omni/deploy/minicpmo_4_5_2gpu.yaml, vllm_omni/deploy/minicpmo_4_5_3gpu.yaml, vllm_omni/deploy/minicpmo_4_5_8x4090.yaml, examples/online_serving/minicpmo/realtime_duplex_demo.py, vllm_omni/experimental/fullduplex/client.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/session.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/runtime_adapter.py, vllm_omni/entrypoints/duplex/runtime_bridge.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/experimental/fullduplex/video_stacking.py, tests/config/test_config_factory.py, tests/engine/duplex/test_duplex_deploy_config.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, tests/entrypoints/openai_api/test_duplex_handler.py, tests/examples/test_minicpmo_realtime_duplex_simple_demo.py, "PR #6529", vllm_omni/entrypoints/duplex/protocol.py, vllm_omni/entrypoints/duplex/realtime_state.py, tests/entrypoints/openai/test_duplex_protocol.py, "PR #6799", "PR #7416"] confidence: high --- @@ -113,3 +113,10 @@ confidence: high - 强制：默认遵循 checkpoint 声明的 `full_attention`；在该模式下，只要「下一 condition + codec-generation reserve」仍落在由 runtime `max_model_len` 与 `tts_config.max_position_embeddings` 导出的有效上限内，就必须保留已累积的 Talker KV/history 并追加；仅在严格溢出时才重建受支持的 one-previous-condition 窗口，释放旧 KV/encoder 与 connector watermark，重置异步 replacement state，并让请求重新进入 admission。精确贴合上限时允许 append，之后继续累积直至下次溢出。显式 `sliding_recompute` 仍在每个 condition 边界重算；正整数 `meta.next_stage_prompt_len` 是 scheduler 的权威 placeholder 长度，合法 Talker condition 即使缺少 legacy `ids.prompt` 也必须先 append 真实 prompt 再推进 sequence/lifecycle。 - 禁止：把无条件 per-condition recompute 当作生产默认；硬编码 `4096` 当作有效上限；在未溢出时丢弃可保留的 Talker history；或把本路径写成已支持物理 KV preemption 后的增量 full-attention replay。 - 验收：覆盖 fit/exact-fit/overflow、producer-owned 与 receiver-owned sequencing 兼容、`next_stage_prompt_len` 驱动的 append，以及显式 `sliding_recompute` 仍按边界重算；不得用 capacity-triggered rollover 的通过外推为已修复无关 perf 回归。^[PR #6799] + +## MCPMO-4j — shipping profile 不得再写入正的 active_stream_window + +- 触发：修改四份 `minicpmo_4_5*.yaml` 的 `active_stream_window`，或 `ChunkTransferAdapter` 的有界 `_promote_active_streams` 路径。 +- 强制：这些 profile 省略该键，使默认 `0` 关闭 limiter，adapter 留在 unlimited path。配置测试断言 `active_stream_window == 0`，而不是 `== max_sessions`。8×4090 的并发上限仍由 stage `max_num_seqs=1` 承担，不靠窗口 1。 +- 禁止：为恢复 first-packet 并发把正窗口写回这四份 profile；把窗口等于 session capacity 的旧断言写回测试。任何正值都会离开 unlimited path。 +- 验收：四份 base profile 展开后窗口为 0，且默认 serve 不进入 K-bounded promote。^[PR #7416] diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-loader.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-loader.md new file mode 100644 index 000000000..b724f9e96 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-loader.md @@ -0,0 +1,17 @@ +--- +title: "MiniCPM-o 4.5 multimodal loader 规则" +created: 2026-09-22 +updated: 2026-09-22 +type: rule +tags: [vllm-omni, models] +sources: ["PR #7384"] +--- + +# MiniCPM-o 4.5 multimodal loader 规则 + +## MCPMO-5a2 — `SupportsMultiModal` 实现必须显式覆盖 `embed_multimodal` + +- 触发：Omni 模型类实现/继承 `get_multimodal_embeddings`，或对齐 vLLM V1 encoder profiling / runtime 的 `embed_multimodal` 调用。 +- 强制：concrete 类必须 override `embed_multimodal` 并委托到现有 `get_multimodal_embeddings`（或等价实现）；不得依赖 Protocol stub 的默认 `None`。同目录 sibling wrapper 已委托时，LLM/核心类必须保持同一合同。 +- 禁止：只实现旧 `get_multimodal_embeddings` 却让 V1 profiling/runtime 落入 stub；把远处的 `NoneType` 启动失败当成权重/registry 问题。 +- 验收：无权重构造 + monkeypatch 断言 `embed_multimodal` 转发 kwargs 并返回 embeddings；去掉 override 时测试必须看到 stub 的 `None`。^[PR #7384] diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md index 907b76e3a..65134626d 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md @@ -1,10 +1,10 @@ --- title: "MiniCPM-o 4.5 规则" created: 2026-07-20 -updated: 2026-09-10 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #3642", "PR #5165", "PR #5382", "PR #5524", "PR #5638", "PR #5792", "PR #5869", "PR #6056", "PR #6154", "PR #6170", "PR #6318", "PR #6828", tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py, vllm_omni/benchmarks/data_modules/seed_tts_dataset.py, vllm_omni/benchmarks/data_modules/seed_tts_eval.py, vllm_omni/benchmarks/patch/patch.py, vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/experimental/fullduplex/client.py, vllm_omni/entrypoints/duplex/chat_fallback.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/vad.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_audio_chunk_mask.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_code2wav_batching.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_pipeline.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, tests/model_executor/models/minicpmo_4_5/test_vision_flash_attention.py, "PR #6082", "PR #5604", "PR #6274", "PR #6346", "PR #6397", "PR #6406", "PR #6458", "PR #6587", "PR #6619", "PR #6757", "PR #6529", "PR #6772", vllm_omni/entrypoints/duplex/protocol.py, "PR #7071", "PR #5730"] +sources: ["PR #3642", "PR #5165", "PR #5382", "PR #5524", "PR #5638", "PR #5792", "PR #5869", "PR #6056", "PR #6154", "PR #6170", "PR #6318", "PR #6828", tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py, vllm_omni/benchmarks/data_modules/seed_tts_dataset.py, vllm_omni/benchmarks/data_modules/seed_tts_eval.py, vllm_omni/benchmarks/patch/patch.py, vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/experimental/fullduplex/client.py, vllm_omni/entrypoints/duplex/chat_fallback.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/vad.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_audio_chunk_mask.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_code2wav_batching.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_pipeline.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, tests/model_executor/models/minicpmo_4_5/test_vision_flash_attention.py, "PR #6082", "PR #5604", "PR #6274", "PR #6346", "PR #6397", "PR #6406", "PR #6458", "PR #6587", "PR #6619", "PR #6757", "PR #6529", "PR #6772", vllm_omni/entrypoints/duplex/protocol.py, "PR #7071", "PR #5730", "PR #7344", "PR #7517"] confidence: high --- @@ -300,3 +300,17 @@ confidence: high - 强制：embedding items 的 `modality` 必须是 `"audio"`（不得从 image 类 copy 成 `"image"`）。单条 `audio_embeds` 张量为 `(s, h)` 时，placeholder 长度取 `len(single_audio_embeds)` / 前导维 `s`，再经 `get_audio_len_by_num_chunks`；不得 `sum(map(len, …))` 把 hidden size 累进 token 数。 - 禁止：以 `modality="image"` 注册 audio embedding；让 `(10, 4096)` 请求约 40960 个 placeholder 并撞 `max_model_len`。 - 验收：modality 断言为 audio；经真实 `_get_prompt_updates` 覆盖 1s/3s/30s 与不同 `h`，placeholder 只随 `s` 变化。^[PR #5730] + +## MCPMO-2g — MiniCPM-o structured content 必须无分隔符拼接，对齐 native omni chat + +- 触发：修改 MiniCPM-o 4.5 chat template、online serving `--chat-template`、image/audio/video 多模态 content parts，或 `minicpmo45_native.jinja` 打包。 +- 强制：structured content 按 native `chat(omni_mode=True)` 直接拼接 text/media parts，不插入额外换行/分隔符；显式 text 空白保留。模板须作为 package data 发布，online/recipe 显式启用并设 `chat-template-content-format=openai`；image→speech 一致性用例保持 image-first。 +- 禁止：依赖通用 string-content 渲染在 media 与 text 间加换行；只改测试不改 shipping template/package data。 +- 验收：CPU 模板回归覆盖 structured parts；online image-to-text-audio 一致性在目标硬件通过。^[PR #7344] + +## MCPMO-3f — Thinker forward 必须返回裸 hidden_states tensor + +- 触发：修改 `MiniCPMO45OmniLLMForConditionalGeneration.forward`、Omni wrapper 的 shape 归一化，或 AR/`GPUModelRunner` 对 thinker 输出的消费。 +- 强制：thinker `forward` 只返回裸 `hidden_states` `torch.Tensor`；`inputs_embeds is None` 时只构建一次 embeddings。wrapper 仅当 `ndim==3 and shape[0]==1` 时 squeeze batch 维，得到 `(seq_len, H)`，并保留 2D `(1, H)` decode 行。 +- 禁止：返回 `(text_inputs_embeds, hidden_states)` tuple；用 `added_batch_dim` 误 squeeze 单 token 的 `(1, H)`；让 runner 用 `[0]` 把 embeddings 当最终 hidden 算 logits。 +- 验收：断言返回类型/值、单次 embedding 构建，以及 `(1,H)/(N,H)/(1,1,H)/(1,N,H)` 归一化。^[PR #7517] diff --git a/knowledge/repos/vllm-omni/models/minimax-h3/_index.md b/knowledge/repos/vllm-omni/models/minimax-h3/_index.md index b654a7d81..be59e1239 100644 --- a/knowledge/repos/vllm-omni/models/minimax-h3/_index.md +++ b/knowledge/repos/vllm-omni/models/minimax-h3/_index.md @@ -1,7 +1,7 @@ --- title: "MiniMax H3" created: 2026-08-05 -updated: 2026-09-05 +updated: 2026-09-22 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #5703", "PR #5706", "PR #5709", "PR #5720", "PR #5723", "PR #5737", "PR #5740", "PR #5752", "PR #5756", "PR #5764", "PR #5779", "PR #5785", "PR #5801", "PR #5824", "PR #5829", "PR #5837", "PR #5840", "PR #5850", "PR #5857", "PR #5881", "PR #5885", "PR #5891", "PR #5896", "PR #5914", "PR #5946", "PR #5972", "PR #5978", "PR #5991", "PR #5863", "PR #6476", "PR #6555", "PR #6556", .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-ready.yml, .buildkite/cuda/test-merge.yml, apps/ComfyUI-vLLM-Omni/comfyui_vllm_omni/, docs/design/architecture_overview.md, docs/models/supported_models.md, docs/user_guide/quantization/fp8.md, vllm_omni/config/omni_config.py, vllm_omni/diffusion/attention/backends/flash_attn.py, vllm_omni/diffusion/attention/backends/rainfusion_attn.py, vllm_omni/diffusion/attention/backends/trtllm_attn.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/teacache/, vllm_omni/diffusion/forward_context.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/models/minimax_h3/, vllm_omni/model_executor/models/minimax_h3/, vllm_omni/model_executor/stage_input_processors/minimax_h3.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/utils/hf_utils.py, vllm_omni/entrypoints/omni_base.py, vllm_omni/platforms/npu/platform.py, vllm_omni/platforms/rocm/platform.py, vllm_omni/quantization/int8_config.py, recipes/MiniMaxAI/MiniMax-H3.md, recipes/MiniMaxAI/MiniMax-H3-4090.md, recipes/MiniMaxAI/MiniMax-H3-5090.md, recipes/MiniMaxAI/MiniMax-H3-MUSA.md, recipes/MiniMaxAI/MiniMax-H3-NPU.md, recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md, recipes/MiniMaxAI/MiniMax-H3-RTX-PRO-5000.md, recipes/MiniMaxAI/MiniMax-H3-RTX-PRO-6000.md, tests/diffusion/attention/test_rainfusion_plan.py, tests/diffusion/attention/test_trtllm_attn.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_teacache_extractors.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/models/minimax_h3/, tests/diffusion/quantization/test_int8_config.py, tests/e2e/accuracy/minimax_h3/, tests/e2e/features/comfyui/test_comfyui_integration.py, tests/e2e/online_serving/minimax_h3/, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, vllm_omni/entrypoints/openai/video_api_utils.py, "PR #6213", "PR #6742", "PR #6666", "PR #6824", "PR #6714"] @@ -154,10 +154,10 @@ execution，不能代替 retained accuracy gate、Buildkite H100 result 或其 ## 审查入口 H3 input matrix/media ingress，以及 text-encoder completeness、online FP8 的 component namespace、loader 顺序、joint quality 与 offload 边界见 -[media rules](rules-media.md)（含 `text_conditioning/v1` schema）与 [MiniMax H3 rules](rules.md#direct-代码快速入口) +[media rules](rules-media.md)（含 `text_conditioning/v1` schema）与 [MiniMax H3 rules](rules.md#direct-代码快速入口) 新增核对：MMH3-2r、MMH3-2s、MMH3-2t。 ；checkpoint transform、quantized loader 与 text-encoder fused-source 完整性正文见 [loading rules](rules-loading.md)；DLO、consumer/H100/ROCm -部署和硬件证据正文见 [deployment rules](rules-deployment.md)；conditioned VAE 确定性、modular task 选择与 request 级 Cache-DiT/TeaCache/sigma schedule/Turbo LoRA 生命周期、alias metadata 同构与 Turbo artifact 合同见 [缓存与任务生命周期规则](rules-cache-task.md)。 +部署和硬件证据正文见 [deployment rules](rules-deployment.md)；conditioned VAE 确定性、modular task 选择与 request 级 Cache-DiT/TeaCache/sigma schedule/Turbo LoRA 生命周期、alias metadata 同构与 Turbo artifact 合同见 [缓存与任务生命周期规则](rules-cache-task.md)。 新增核对：MMH3-2r2。 H3 VAE decoder 的 model-local eager dispatch、remote-code eligibility、exactness guards、selective @@ -165,3 +165,5 @@ FP16 materialization、keyframe encode 的 cuDNN/TF32 pin 与 compile/spatial-pa 。 Qwen3-VL text encoder 的 cuDNN SDPA process-global-state restore、encoder-rank/VAE boundary 和 CPU-only validation limit 见 [encoder state rules](rules-encoder-state.md)。 + +- [rules-quality](rules-quality.md) 新增核对：MMH3-4a2。 diff --git a/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md b/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md index 969a0c607..b69e459eb 100644 --- a/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md +++ b/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md @@ -1,10 +1,10 @@ --- title: "MiniMax H3 缓存与任务生命周期规则" created: 2026-09-04 -updated: 2026-09-08 +updated: 2026-09-22 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #5703", "PR #5720", "PR #5810", "PR #5837", "PR #5840", "PR #5853", "PR #5991", "PR #6476", "PR #6550", "PR #6666", "PR #6714", "PR #6909", vllm_omni/diffusion/models/minimax_h3/batched_packing.py, vllm_omni/diffusion/models/minimax_h3/fasth3.py, vllm_omni/diffusion/models/minimax_h3/lora.py, vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py, vllm_omni/diffusion/models/minimax_h3/npu/lora.py, vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py, vllm_omni/diffusion/models/minimax_h3/quality_policy.py, vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/entrypoints/openai/video_api_utils.py, tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py, tests/diffusion/models/minimax_h3/test_minimax_h3_fasth3.py, tests/diffusion/models/minimax_h3/test_minimax_h3_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_native_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_parallel.py, tests/diffusion/models/minimax_h3/test_minimax_h3_step_execution.py, tests/entrypoints/openai_api/test_video_api_utils.py, "PR #7162", "PR #7062"] +sources: ["PR #5703", "PR #5720", "PR #5810", "PR #5837", "PR #5840", "PR #5853", "PR #5991", "PR #6476", "PR #6550", "PR #6666", "PR #6714", "PR #6909", vllm_omni/diffusion/models/minimax_h3/batched_packing.py, vllm_omni/diffusion/models/minimax_h3/fasth3.py, vllm_omni/diffusion/models/minimax_h3/lora.py, vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py, vllm_omni/diffusion/models/minimax_h3/npu/lora.py, vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py, vllm_omni/diffusion/models/minimax_h3/quality_policy.py, vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/entrypoints/openai/video_api_utils.py, tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py, tests/diffusion/models/minimax_h3/test_minimax_h3_fasth3.py, tests/diffusion/models/minimax_h3/test_minimax_h3_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_native_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_parallel.py, tests/diffusion/models/minimax_h3/test_minimax_h3_step_execution.py, tests/entrypoints/openai_api/test_video_api_utils.py, "PR #7162", "PR #7062", "PR #7838"] confidence: high --- @@ -231,3 +231,10 @@ Cache-DiT、TeaCache、distilled sigma schedule 与 Turbo LoRA 的生命周期 - 强制：`--lora-path` 指向单文件，或只含一个可解析 Turbo artifact 的目录；多候选必须 fail closed。`ref2v` 只能挂在 Ref2VA-only server；combined/`transformers_ref` 拓扑不得绑定只注入 `transformer` 的 Turbo adapter。ComfyUI 融合 QKV/SwiGLU 导出按名拒绝。 - 禁止：继续要求单一 `4step_v1.0_768p` 文件名或 `alpha==128`；在目录里静默挑选；把未声明 alpha 当 rank；或让 Ref2VA 请求跑到未蒸馏 DiT 的 few-step schedule。 - 验收：覆盖多 artifact 文件名矩阵、缺/错 alpha、多文件目录拒绝、task/steps/shift 负向门禁、Ref2VA-only vs combined 拒绝，以及 Diffusers-layout packing；不得把单一 artifact 通过外推为全矩阵质量证据。^[PR #7062] + +## MMH3-2r2 — 长视频 continuation 必须按全局时间线切窗并同步 AV 边界 + +- 触发：修改 MiniMax-H3 `long_video`/`continuation_*`、`audio_mode=lock_source`、窗口 RoPE `temporal_offset`、`continuation_prompts`，或 fused QK-norm 的 token 索引宽度。 +- 强制：默认 latent-tail continuation 只在 Ref2VA request execution（非 step）下启用；window/overlap 必须落在 `17n+5` 网格且 window∈[107,345]、overlap