diff --git a/knowledge/repos/vllm-omni/components/configuration/_index.md b/knowledge/repos/vllm-omni/components/configuration/_index.md index 82179a175..31b9f81c4 100644 --- a/knowledge/repos/vllm-omni/components/configuration/_index.md +++ b/knowledge/repos/vllm-omni/components/configuration/_index.md @@ -49,5 +49,6 @@ sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/] | 审查或新增 process 环境变量、stage `env` 和安全诊断输出 | [environment-variable contract](environment-variable-contract.md) | | 计算多 stage 显存预算 | [deployment memory budget](deployment-memory-budget.md) | | 添加新模型和注册点 | [adding a model](adding-a-model.md) | -| composable strategy 的 axis、并行拓扑与 load balancing owner | [并行拓扑合同](rules-parallel-topology.md) | +| composable strategy 的 axis、并行拓扑、load balancing owner 与 resolve_omni_config 统一解析 | [并行拓扑合同](rules-parallel-topology.md) | + | 冻结 topology、辅助 stage 注入与模型专用 deploy profile | [topology 与部署 profile](rules-topology-profiles.md) | diff --git a/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md b/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md index 3f40a2214..a98d68317 100644 --- a/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md +++ b/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md @@ -1,10 +1,10 @@ --- title: "并行拓扑合同" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-09-09 type: rule tags: [vllm-omni, components, config] -sources: [vllm_omni/config/composable_parallel/, vllm_omni/config/config_factory.py, vllm_omni/config/stage_config.py, "PR #5531"] +sources: [vllm_omni/config/composable_parallel/, vllm_omni/config/config_factory.py, vllm_omni/config/stage_config.py, "PR #5531", "PR #5140"] confidence: high --- @@ -64,3 +64,10 @@ confidence: high - 禁止：在配置解析期把省略的 DP 固定为 1 后据此生成不足的设备列表；接受与 WORLD 推导值不一致的显式 DP；允许 HSDP 与 TP、普通 DP、PP 或 EP 组合；只用 `num_gpus` 做校验却不传播到 stage 配置。 - 验收：覆盖 direct、structured 和 async stage 路径；`num_gpus=8`、TP=2、SP=2 必须得到 DP=2、WORLD=8 和 devices `0`–`7`，省略 DP 的配置在运行时解析前仍保持未指定；覆盖 HSDP mesh 不匹配及不支持组合，并确认都在进程组创建前失败。^[PR #5531] + +## CONF-4f — 生产启动必须经 resolve_omni_config 统一解析 + +- 触发：修改 `AsyncOmniEngine`、headless stage 启动、`StageConfigFactory` 公开入口，或 registry/deploy/CLI 到最终 stage 的 source selection。 +- 强制：标准与 headless 生产路径只调用 `resolve_omni_config()` 获得 `OmniConfigResolution`（structured config、runtime compatibility stages、strategy-derived LB、effective deploy path）；factory 的 create/legacy 方法是 resolver 与测试的下层 primitive，engine/entrypoint 不得绕过 resolver 直接选源。通用单 stage diffusion 仅在 registry 解析失败后作为 fallback；legacy `stage_configs_path`/`stage_configs` 必须显式报错。 +- 禁止：engine 侧二次合成 default diffusion stage 或重复 merge LoRA/attention/quant/cache/profiler；把 OmegaConf-compatible stages 宣传为稳定公共 ABI；静默恢复已删除的 stage-configs-path 行为。 +- 验收：同一 model+deploy+override 在标准与 headless 路径得到等价 topology 与 LB policy；覆盖 registry 命中、generic diffusion fallback、以及拒绝已删除 legacy 参数。^[PR #5140] diff --git a/knowledge/repos/vllm-omni/components/diffusion/_index.md b/knowledge/repos/vllm-omni/components/diffusion/_index.md index ef27ae866..febae7c0b 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/_index.md +++ b/knowledge/repos/vllm-omni/components/diffusion/_index.md @@ -37,8 +37,10 @@ sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/ | CustomOp XPU platform dispatch、PyTorch-native fallback、CUDA contract 与非同义 override 边界 | [CustomOp dispatch rules](rules-custom-op-dispatch.md) | | runtime temporary 与 loader-scoped parameter dtype、shared RMSNorm accuracy | [tensor dtype rules](rules-tensor-dtype.md) | | Wan VAE height/width spatial reshard、empty tail、attention extent | [Wan spatial-shard rules](rules-wan-spatial-shard.md) | -| multi-DiT、dotted `_dit_modules`、loader-to-offloader handoff 与跨 Cache-DiT/compile/LoRA/offload lifecycle | [component lifecycle rules](rules-component-lifecycle.md) | -| PEFT 与 distilled LoRA、startup fusion、delta/key/alpha、Qwen/Wan transformer mapping | [LoRA rules](rules-lora.md) | +| multi-DiT、dotted `_dit_modules`、loader-to-offloader handoff 与跨 Cache-DiT/compile/LoRA/offload lifecycle、batch sampling-key 的 provided 标志 | [component lifecycle rules](rules-component-lifecycle.md) | + +| PEFT 与 distilled LoRA、startup fusion、delta/key/alpha、Qwen/Wan transformer mapping、PEFT suspend/resume | [LoRA rules](rules-lora.md) | + | Blackwell FA4、显式/auto CUDA backend、per-role mask/SP pad；local FlashAttention deterministic opt-in、role-aware masked cross-attention、NPU packed mask-free/laser fallback、TRTLLM packed-padding 与 config propagation | [attention rules](rules-attention.md) | | SP auto-padding、`mask_sp_padding`、dense/varlen 与 advanced UAA 边界 | [SP padding rules](rules-sp-padding.md) | | video/audio mux、DLO DP wave、RPC reply ownership/result release、result queue、async pump、SHM ownership、async-output wait 与 shutdown | [output/runtime rules](rules-output-lifecycle.md) | @@ -46,7 +48,8 @@ sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/ | HunyuanImage3 Distil CFG、MeanFlow special-token / scheduler-r 合同 | [HunyuanImage3 HY3-8d](../../models/hunyuan-image3/rules.md) | | diffusion step 与 request/continuous batching；DIFF-1af BAGEL image step wave、packed state、geometry/CFG admission 与 fallback cleanup | [step and batching](step-and-batching.md) | | request-wave admission coalescing、stable window、deadline 与 finite config | [admission wait rules](rules-admission-wait.md) | -| paged KV/cache 预算、native/backend/platform 闭环、GQA/Ring/Ulysses layout、FlashInfer plan、能力 metadata | [paged cache 与系统运行时规则](rules-system-runtime.md) | +| paged KV/cache 预算、native/backend/platform 闭环、GQA/Ring/Ulysses layout、FlashInfer plan、能力 metadata、cache_backend None 规范化 | [paged cache 与系统运行时规则](rules-system-runtime.md) | + | Scheduler-managed diffusion KV 的请求控制面、Hunyuan layout 与未实现边界 | [paged KV control plane](paged-kv-control-plane.md) | | Cache-DiT、TeaCache 和 prefix cache | [cache acceleration](cache-acceleration.md) | | TP/PP/SP/CFG/VAE/HSDP 等并行策略 | [parallelism](parallelism.md) | diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md b/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md index 9793522b6..8740242d3 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md @@ -1,10 +1,10 @@ --- title: "Diffusion component lifecycle 规则" created: 2026-09-02 -updated: 2026-09-05 +updated: 2026-09-09 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5720", "PR #5853", "PR #5882", "PR #5884", "PR #6486", "PR #6591", vllm_omni/diffusion/cache/base.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/models/interface.py, vllm_omni/diffusion/offloader/module_collector.py, vllm_omni/diffusion/offloader/startup.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/interface.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/models/sana_video/test_cache_offload.py, tests/diffusion/test_diffusion_model_runner.py, tests/diffusion/test_diffusion_scheduler.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_recipes.py", "PR #6072"] +sources: ["PR #5720", "PR #5853", "PR #5882", "PR #5884", "PR #6486", "PR #6591", vllm_omni/diffusion/cache/base.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/models/interface.py, vllm_omni/diffusion/offloader/module_collector.py, vllm_omni/diffusion/offloader/startup.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/interface.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/models/sana_video/test_cache_offload.py, tests/diffusion/test_diffusion_model_runner.py, tests/diffusion/test_diffusion_scheduler.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_recipes.py", "PR #6072", "PR #7078"] confidence: high --- @@ -119,3 +119,10 @@ residency 留在模型 owner。规则入口与其他共享机制仍见 [Diffusio retry 与 required no-retry；断言 partial backend disable、carrier/lease close、fresh path bypasses HWR and checkpoint mmap；另注入 unregistration failure，断言 retryable retention、backend drop 和 distributed teardown ordering。^[PR #6486] ^[PR #6591] + +## DIFF-2ad — batch-first sampling gate 的 provided 标志必须进入 RequestBatchSamplingParamsKey + +- 触发：修改 `RequestBatchSamplingParamsKey`、`_build_sampling_params_key`，或 pipeline 从 batch 首请求读取 `*_provided` 再决定 guidance/image-guidance 行为。 +- 强制：凡 pipeline 用 “omitted vs explicit” provided 标志做 batch-local gate 的字段，都必须进入 request-mode 与 step-mode 的 sampling-params key；数值相等但 provided 不同的请求不得合批。当前合同要求 `guidance_scale_2_provided` 与数值 `guidance_scale_2` 一并参与 key。 +- 禁止：只比较 auto-fill 后的数值；假设同数值即同语义；只更新 request scheduler 而漏掉 step scheduler；把 H3 等 single-request guard 当作可合批模型的例外模板。 +- 验收：用真实 key builder 覆盖 omitted `guidance_scale_2` 与显式同值、数值不同，以及两套 scheduler 的 key 分离；并断言 pipeline 首请求 gate 不会跨不同 provided 语义共享 batch。^[PR #7078] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md b/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md index a026c50e9..e73e11ae0 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-lora.md @@ -1,10 +1,10 @@ --- title: "Diffusion LoRA 规则" created: 2026-09-02 -updated: 2026-09-05 +updated: 2026-09-09 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #2783", docs/user_guide/diffusion/lora.md, vllm_omni/config/omni_config.py, vllm_omni/config/stage_config.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/lora/loader.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/lora/layers/base_linear.py, vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2_i2v.py, vllm_omni/diffusion/utils/tf_utils.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/cli/serve.py, tests/diffusion/lora/test_loader.py, tests/diffusion/lora/test_lora_manager.py, tests/entrypoints/test_async_omni_diffusion_config.py, "PR #5500", "vllm_omni/diffusion/models/ltx2/ltx2_adapter_parser.py", "vllm_omni/diffusion/models/ltx2/ltx2_phase_adapter.py", "PR #6070", "PR #6476", "PR #6550", vllm_omni/diffusion/models/minimax_h3/lora.py, "PR #6268", benchmarks/kernels/benchmark_diffusion_lora_expand.py, tests/diffusion/lora/test_base_linear.py] +sources: ["PR #2783", docs/user_guide/diffusion/lora.md, vllm_omni/config/omni_config.py, vllm_omni/config/stage_config.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/lora/loader.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/lora/layers/base_linear.py, vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py, vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2_i2v.py, vllm_omni/diffusion/utils/tf_utils.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/cli/serve.py, tests/diffusion/lora/test_loader.py, tests/diffusion/lora/test_lora_manager.py, tests/entrypoints/test_async_omni_diffusion_config.py, "PR #5500", "vllm_omni/diffusion/models/ltx2/ltx2_adapter_parser.py", "vllm_omni/diffusion/models/ltx2/ltx2_phase_adapter.py", "PR #6070", "PR #6476", "PR #6550", vllm_omni/diffusion/models/minimax_h3/lora.py, "PR #6268", benchmarks/kernels/benchmark_diffusion_lora_expand.py, tests/diffusion/lora/test_base_linear.py, "PR #7195"] confidence: high --- @@ -22,10 +22,13 @@ confidence: high | Wan dual transformer、high/low-noise file order、partial load | `DIFF-2n` | `WanLoraLoaderMixin` → `get_transformer_from_pipeline(name)` → `transformer`/`transformer_2` | | LoRA linear active output slices、inference expand accumulation、autograd/dtype fallback | `DIFF-2ab` | `diffusion/lora/layers/base_linear.py::DiffusionBaseLinearLayerWithLoRA.apply` | | legacy dynamic adapter hook、packed/stacked binding、DLO sidecar residency、activation rollback | `DIFF-2x` | `DiffusionLoRAManager::{_load_adapter,_bind_adapter_weights,_activate_adapter}` → model pipeline hook | +| PEFT deactivate/suspend、slice-mask resume、同 adapter+scale 再激活不 re-upload | `DIFF-2ae` | `DiffusionLoRAManager` deactivate/activate → `DiffusionBaseLinearLayerWithLoRA::{suspend_lora,resume_lora}` | + | 审查组 | 什么时候触发 | 规则 ID | |---|---|---| -| `checkpoint-distributed` | diffusion LoRA backend、checkpoint、融合与 transformer mapping | `DIFF-2l`, `DIFF-2m`, `DIFF-2n`, `DIFF-2x` | +| `checkpoint-distributed` | diffusion LoRA backend、checkpoint、融合与 transformer mapping | `DIFF-2l`, `DIFF-2m`, `DIFF-2n`, `DIFF-2x`, `DIFF-2ae` | + ## DIFF-2l — backend 是 startup 配置，不是 request 字段 @@ -127,3 +130,10 @@ confidence: high - 强制：DLO owns base-weight streaming only。DLO-enabled manager 在 wrapper replacement 时必须要求可用的 resident-buffer protocol、将 request-switchable A/B sidecars 放在 compute device，且 wrapper 因更大 rank 重新分配后必须重放该 device placement；缺 protocol fail closed。非 DLO path 不得获得此 placement side effect。 - 禁止：将已识别的专用 artifact 悄悄退回 generic loader；按名称而非真实绑定确认成功；将一个模型的 packed layout/scale 规则泛化给所有 diffusion 模型；失败后继续声明旧 adapter active；把 sidecars 加入 DLO host shards/collective，或靠每 block transfer 伪装为 DLO support。 - 验收：CPU regressions 覆盖 model hook/fallback、binding completeness、mid-loop failure 后 reset 与旧 adapter retry；DLO mock 覆盖 replacement 的 compute-device placement、缺 resident protocol 的 fail-closed 以及 rank-driven reallocation 后 A/B buffer 仍在该 device。模型 integration 另覆盖 artifact identity、target/shape rejection、packed QKV/FFN binding 和实际 request sampling contract。^[PR #6476] ^[PR #6550] + +## DIFF-2ae — PEFT 去激活必须 suspend 切片掩码并保留 uploaded 权重 + +- 触发：修改 `DiffusionLoRAManager` 的 deactivate/activate、`DiffusionBaseLinearLayerWithLoRA` 的 active-slice mask，或混合 base/adapted 请求切换路径。 +- 强制：deactivate 只 `suspend_lora()`（保存并清零 `_diffusion_lora_active_slices`），保留 stacked A/B；同一 adapter_id 与同一 rounded scale 再激活时 `resume_lora()` 复原掩码，禁止整层 `reset_lora`+rebind。`_suspended_adapter_id` 在真实 reset、rank 驱动的 `create_lora_weights`、新 bind 开始、以及 remove 被 suspend 的 adapter 时必须清空；不同 adapter 或不同 scale 仍走完整 rebind。 +- 禁止：把 suspend 当成可丢弃的 upload；resume 覆盖后注册的新 layer 的 inactive mask；在失败 bind 后仍保留旧 suspended 身份。 +- 验收：CPU 覆盖 suspend/resume、同 id+scale 不 re-upload、不同 scale/id 走 rebind、remove suspended 触发 reset、中途失败后 inactive，以及 add_adapter 后新 layer 不被错误 resume。^[PR #7195] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md b/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md index 238972ef7..2fb28e1c4 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-system-runtime.md @@ -1,10 +1,10 @@ --- title: "Diffusion paged cache 与系统运行时规则" created: 2026-09-03 -updated: 2026-09-05 +updated: 2026-09-09 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5255", "PR #5344", "PR #5543", "PR #5838", "PR #6094", "PR #6102", "PR #6385", "PR #6340", "PR #6714", "PR #6814", "PR #6563", "PR #5716", "PR #6786", vllm_omni/diffusion/attention/, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/attention/parallel/ring_kernels.py, vllm_omni/diffusion/diffusion_kv/, vllm_omni/diffusion/distributed/cfg_parallel.py, vllm_omni/diffusion/distributed/parallel_state.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/platforms/interface.py, vllm_omni/platforms/npu/platform.py, tests/diffusion/diffusion_kv/, tests/diffusion/distributed/test_cfg_parallel.py, tests/diffusion/attention/test_piecewise_attn.py, tests/diffusion/attention/test_ulysses_uaa.py, "PR #5491", "PR #5194", "vllm_omni/diffusion/data.py", "vllm_omni/diffusion/utils/hf_utils.py"] +sources: ["PR #5255", "PR #5344", "PR #5543", "PR #5838", "PR #6094", "PR #6102", "PR #6385", "PR #6340", "PR #6714", "PR #6814", "PR #6563", "PR #5716", "PR #6786", vllm_omni/diffusion/attention/, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/attention/parallel/ring_kernels.py, vllm_omni/diffusion/diffusion_kv/, vllm_omni/diffusion/distributed/cfg_parallel.py, vllm_omni/diffusion/distributed/parallel_state.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/platforms/interface.py, vllm_omni/platforms/npu/platform.py, tests/diffusion/diffusion_kv/, tests/diffusion/distributed/test_cfg_parallel.py, tests/diffusion/attention/test_piecewise_attn.py, tests/diffusion/attention/test_ulysses_uaa.py, "PR #5491", "PR #5194", "vllm_omni/diffusion/data.py", "vllm_omni/diffusion/utils/hf_utils.py", "PR #7041"] confidence: high --- @@ -205,3 +205,10 @@ confidence: high - 强制：Scheduler 只拥有 logical block allocation、generation 和 `DiffusionKVMetadata`；每个 Worker rank 消费自己的 `KVCacheConfig`，拥有物理 KV tensors、native `BlockTables` 和 request/sequence/context rows。安装 metadata 前必须校验 group/count/range/capacity，并以 staged append、apply 和失败 rollback 原子发布；重复 generation 幂等，冲突或过期快照拒绝，BlockTables 变更使已准备的 attention batch 失效，终止时先清 Worker rows 而不释放 Scheduler blocks。request-mode runner 负责从 snapshot 构造并激活 prefill/denoise row metadata；model 只提供 Q/K/V layout 与 `full_attn_spans`。 - 禁止：Worker 自行分配或释放 Scheduler-owned logical blocks；把部分 row 安装、native append/apply 异常或 stale snapshot 当作成功；从 padding 猜异构 prefix 的逻辑布局；在清理或 wake refresh 后继续复用旧的 native metadata/buffer view。 - 验收：CPU/mock 覆盖 rank-local config、请求/上下文 row mapping、重复/过期/冲突 generation、非法 block 与容量、append/apply rollback、幂等清理和 wake refresh；GPU/NPU 另验证非连续 BlockTables、32Q/8KV GQA、异构 prefix 与 piecewise attention。Hunyuan paged request execution 不证明 `denoise_step`、cross-request prefix reuse、AR KV、negative CFG、Ring 或 AllGather。^[PR #6102] ^[PR #6563] + +## DIFF-4ac — 声明为 str 的 diffusion cache_backend 必须在 kwargs 边界把显式 None 规范成 none + +- 触发：修改 `normalize_omni_diffusion_kwargs`、`OmniDiffusionConfig.cache_backend`，或 pipeline/runner 对 cache backend 的 admission 校验。 +- 强制：字段声明为 `str` 且 canonical “无 cache” sentinel 为 `"none"` 时，key 缺失才允许走 env/`"none"` 默认；调用方传入显式 `None`（如 CLI `default=None`）必须在 kwargs 边界规范成 `"none"`，且不得回落到 `DIFFUSION_CACHE_BACKEND` 查找。下游校验可把残留 `None` 当 `"none"` 作 defense-in-depth，但不能代替边界规范化。 +- 禁止：只在 key 缺失时默认、让显式 `None` 泄漏成 typed `str` 字段上的 `None`；把 `"none"` 与 env 覆盖混为一谈；把某一 pipeline 的 `None` 特判当成唯一修复点。 +- 验收：`from_kwargs`/`create_default_diffusion` 对 `cache_backend=None` 断言最终为 `"none"`；至少一个严格 `("none", …)` 校验路径覆盖 `None` 与 `"none"` 均可通过，未知 backend 仍拒绝。^[PR #7041] diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md index af17bbcad..4ae05fa88 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md @@ -1,10 +1,10 @@ --- title: "MiniCPM-o 4.5 规则" created: 2026-07-20 -updated: 2026-09-06 +updated: 2026-09-09 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #3642", "PR #5165", "PR #5382", "PR #5524", "PR #5638", "PR #5792", "PR #5869", "PR #6056", "PR #6154", "PR #6170", "PR #6318", "PR #6828", tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py, vllm_omni/benchmarks/data_modules/seed_tts_dataset.py, vllm_omni/benchmarks/data_modules/seed_tts_eval.py, vllm_omni/benchmarks/patch/patch.py, vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/experimental/fullduplex/client.py, vllm_omni/entrypoints/duplex/chat_fallback.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/vad.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_audio_chunk_mask.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_code2wav_batching.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_pipeline.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, tests/model_executor/models/minicpmo_4_5/test_vision_flash_attention.py, "PR #6082", "PR #5604", "PR #6274", "PR #6346", "PR #6397", "PR #6406", "PR #6458", "PR #6587", "PR #6619", "PR #6757", "PR #6529", "PR #6772", vllm_omni/entrypoints/duplex/protocol.py] +sources: ["PR #3642", "PR #5165", "PR #5382", "PR #5524", "PR #5638", "PR #5792", "PR #5869", "PR #6056", "PR #6154", "PR #6170", "PR #6318", "PR #6828", tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py, vllm_omni/benchmarks/data_modules/seed_tts_dataset.py, vllm_omni/benchmarks/data_modules/seed_tts_eval.py, vllm_omni/benchmarks/patch/patch.py, vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/experimental/fullduplex/client.py, vllm_omni/entrypoints/duplex/chat_fallback.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/vad.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_audio_chunk_mask.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_code2wav_batching.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_pipeline.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, tests/model_executor/models/minicpmo_4_5/test_vision_flash_attention.py, "PR #6082", "PR #5604", "PR #6274", "PR #6346", "PR #6397", "PR #6406", "PR #6458", "PR #6587", "PR #6619", "PR #6757", "PR #6529", "PR #6772", vllm_omni/entrypoints/duplex/protocol.py, "PR #7071"] confidence: high --- @@ -20,6 +20,8 @@ confidence: high |---|---| | version/registry、processor、remote code | `MCPMO-1a`、`MCPMO-2a` | | Whisper/APM 或 SigLIP attention | `MCPMO-2b`、`MCPMO-2c` | +| audio_feature_lens ragged slices、encoder batch TensorShape | `MCPMO-2e` | + | TTS/Code2Wav/TRT、CUDA Graph | `MCPMO-1b`–`MCPMO-1e`；[CUDA graphs](rules-cuda-graphs.md) | | batch、runtime info、reference-audio handoff | `MCPMO-3a`–`MCPMO-3e`；[Code2Wav batching](rules-code2wav-batching.md) | | native duplex、sampling budget、shipping profile、playback ACK | `MCPMO-4a`–`MCPMO-4h`；[duplex](rules-duplex.md) | @@ -283,3 +285,10 @@ confidence: high 共享 bridge/batch 规则见 [Model Executor rules](../../components/model-executor/rules.md)； 公开入口完整性见 [model adaptation guardrails](../../review/guides/model-adaptation-guardrails.md)。 + +## MCPMO-2e — audio_feature_lens 的切片维必须允许跨样本 ragged + +- 触发：修改 `MiniCPMOAudioFeatureInputs.audio_feature_lens` 的 `TensorShape`、encoder batch 合批，或 `get_audio_hidden_states`/`get_audio_embedding_streaming` 对 per-audio lens 的消费。 +- 强制：`audio_feature_lens` 声明 `TensorShape("bn", "s", dynamic_dims={"s"})`，允许同一 encoder batch 内不同音频有不同 slice 数（如 >30s 与 ≤30s）；消费侧继续 `hstack` + 按 `len(audio_feature_lens_raw[i])` 逐音频处理。仅放松 `"s"`；`"bn"` 与其他非动态维仍严格校验。 +- 禁止：因 list 内 lens 长度不一致在 schema 层抛 `inconsistent shapes` 并拖垮 Stage-0 engine；把 ragged 误当成非法输入。 +- 验收：直接构造混合 slice 数（如 3 vs 2）与同 slice 数两例，断言均可构造且不改变既有同长路径。^[PR #7071] diff --git a/knowledge/repos/vllm-omni/models/minimax-h3/_index.md b/knowledge/repos/vllm-omni/models/minimax-h3/_index.md index fe203354c..07217ddb4 100644 --- a/knowledge/repos/vllm-omni/models/minimax-h3/_index.md +++ b/knowledge/repos/vllm-omni/models/minimax-h3/_index.md @@ -159,6 +159,7 @@ text-encoder fused-source 完整性正文见 [loading rules](rules-loading.md) 部署和硬件证据正文见 [deployment rules](rules-deployment.md)；conditioned VAE 确定性、modular task 选择与 request 级 Cache-DiT/TeaCache/sigma schedule/Turbo LoRA 生命周期见 [缓存与任务生命周期规则](rules-cache-task.md)。 H3 VAE decoder 的 model-local eager dispatch、remote-code eligibility、exactness guards、selective -FP16 materialization 与 compile/spatial-parallel fallback 见 [VAE eager-ops rules](rules-vae-ops.md)。 +FP16 materialization、keyframe encode 的 cuDNN/TF32 pin 与 compile/spatial-parallel fallback 见 [VAE eager-ops rules](rules-vae-ops.md) +。 Qwen3-VL text encoder 的 cuDNN SDPA process-global-state restore、encoder-rank/VAE boundary 和 CPU-only validation limit 见 [encoder state rules](rules-encoder-state.md)。 diff --git a/knowledge/repos/vllm-omni/models/minimax-h3/rules-vae-ops.md b/knowledge/repos/vllm-omni/models/minimax-h3/rules-vae-ops.md index a2b17db66..40ab805c0 100644 --- a/knowledge/repos/vllm-omni/models/minimax-h3/rules-vae-ops.md +++ b/knowledge/repos/vllm-omni/models/minimax-h3/rules-vae-ops.md @@ -1,10 +1,10 @@ --- title: "MiniMax H3 VAE eager-ops rules" created: 2026-09-05 -updated: 2026-09-05 +updated: 2026-09-09 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #6607", vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/models/minimax_h3/ops/README.md, vllm_omni/diffusion/models/minimax_h3/ops/vae/__init__.py, vllm_omni/diffusion/models/minimax_h3/ops/vae/dispatch.py, vllm_omni/diffusion/models/minimax_h3/ops/vae/qk_norm_rope.py, vllm_omni/diffusion/models/minimax_h3/ops/vae/scaled_residual.py, tests/diffusion/models/minimax_h3/test_minimax_h3_vae_ops.py] +sources: ["PR #6607", vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/models/minimax_h3/ops/README.md, vllm_omni/diffusion/models/minimax_h3/ops/vae/__init__.py, vllm_omni/diffusion/models/minimax_h3/ops/vae/dispatch.py, vllm_omni/diffusion/models/minimax_h3/ops/vae/qk_norm_rope.py, vllm_omni/diffusion/models/minimax_h3/ops/vae/scaled_residual.py, tests/diffusion/models/minimax_h3/test_minimax_h3_vae_ops.py, "PR #7191"] confidence: high --- @@ -34,3 +34,10 @@ confidence: high rejection、idempotence、selective FP16 and FP16 SwiGLU path、compile/SP/original-forward fallback； target GPU 另以 direct Q/K、residual 与 complete decoded tensor 的 `torch.equal` 对 reference 验 exactness。性能报告必须绑定 exact device/software/workload，不能代替持续 CI gate。^[PR #6607] + +## MMH3-4d — GPU keyframe VAE encode 必须 pin 并恢复 deterministic cuDNN TF32 + +- 触发：修改 `MiniMaxH3VideoVAE.encode_image`、keyframe encode 的 cuDNN/TF32 设置，或该路径与后续 denoiser 的数值敏感边界。 +- 强制：仅在 CUDA 设备上进入 scoped context：`cudnn.enabled=True`、`benchmark=False`、`deterministic=True`、`allow_tf32=True`；成功与异常退出都必须恢复调用方原 backend 状态。非 CUDA 设备 no-op。该 pin 只包裹 keyframe encode，不改变非 keyframe 路径或 accuracy 阈值。 +- 禁止：依赖进程 ambient cuDNN algorithm 选择；把 pin 做成进程级永久开关；把单次卡组差异描述成已证明的跨 SKU/driver 完全确定性。 +- 验收：CPU/mock 覆盖 enter/exit 四标志 pin 与 restore、encode 成功与失败路径都离开 context；真实 GPU accuracy gate 另绑定固定拓扑与 workload，不得用本 context 单测代替。^[PR #7191] diff --git a/knowledge/repos/vllm-omni/models/minimax-h3/rules.md b/knowledge/repos/vllm-omni/models/minimax-h3/rules.md index 08f14d73a..b8e67f7eb 100644 --- a/knowledge/repos/vllm-omni/models/minimax-h3/rules.md +++ b/knowledge/repos/vllm-omni/models/minimax-h3/rules.md @@ -1,10 +1,10 @@ --- title: "MiniMax H3 规则" created: 2026-09-02 -updated: 2026-09-05 +updated: 2026-09-09 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #5703", "PR #5706", "PR #5720", "PR #5737", "PR #5764", "PR #5779", "PR #5801", "PR #5824", "PR #5829", "PR #5836", "PR #5837", "PR #5840", "PR #5853", "PR #5881", "PR #5891", "PR #5896", "PR #5991", "PR #5997", "PR #6000", "PR #6040", "PR #6724", benchmarks/diffusion/backends.py, benchmarks/diffusion/diffusion_benchmark_serving.py, docs/design/feature/skip_softmax.md, docs/user_guide/diffusion/attention_backends.md, docs/user_guide/diffusion/attention_backends/trtllm.md, vllm_omni/config/model.py, vllm_omni/config/omni_config.py, vllm_omni/diffusion/attention/backends/rainfusion_attn.py, vllm_omni/diffusion/attention/backends/trtllm_attn.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/cache/teacache/, vllm_omni/diffusion/forward_context.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/models/minimax_h3/encoder.py, vllm_omni/diffusion/models/minimax_h3/lora.py, vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py, vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py, vllm_omni/diffusion/models/minimax_h3/quality_policy.py, vllm_omni/diffusion/models/minimax_h3/time_request.py, vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/offloader/module_residency.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/utils/hf_utils.py, vllm_omni/entrypoints/omni_base.py, vllm_omni/quantization/int8_config.py, tests/dfx/perf/scripts/run_diffusion_benchmark.py, tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json, tests/diffusion/attention/test_rainfusion_plan.py, tests/diffusion/attention/test_trtllm_attn.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/cache/test_teacache_extractors.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py, tests/diffusion/models/minimax_h3/test_minimax_h3_dlo_lifecycle.py, tests/diffusion/models/minimax_h3/test_minimax_h3_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_parallel.py, tests/diffusion/models/minimax_h3/test_minimax_h3_quality_policy.py, tests/diffusion/models/minimax_h3/test_minimax_h3_quantization.py, tests/diffusion/models/minimax_h3/test_minimax_h3_quantization_quality.py, tests/diffusion/offloader/test_module_residency.py, tests/diffusion/quantization/test_int8_config.py, recipes/MiniMaxAI/MiniMax-H3.md, recipes/MiniMaxAI/MiniMax-H3-5090.md, recipes/MiniMaxAI/MiniMax-H3-MUSA.md, recipes/MiniMaxAI/MiniMax-H3-NPU.md, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/engine/async_omni_engine.py, "PR #5915", "PR #5990", "vllm_omni/diffusion/layers/fused_qk_norm_rope.py", "vllm_omni/diffusion/cache/teacache/extractors.py", "tests/diffusion/layers/test_fused_qk_norm_rope.py", "PR #6061", "vllm_omni/platforms/npu/models/minimax_h3.py", "vllm_omni/platforms/npu/platform.py", "tests/diffusion/models/minimax_h3/test_minimax_h3_qwen3vl_rope.py", "PR #6167", "PR #6173", "PR #6283", "PR #6281", "PR #6878", "vllm_omni/diffusion/attention/ops/minimax_h3_modulation.py", "PR #6476", "PR #6526", "PR #6410", vllm_omni/diffusion/models/minimax_h3/denoise_loop.py] +sources: ["PR #5703", "PR #5706", "PR #5720", "PR #5737", "PR #5764", "PR #5779", "PR #5801", "PR #5824", "PR #5829", "PR #5836", "PR #5837", "PR #5840", "PR #5853", "PR #5881", "PR #5891", "PR #5896", "PR #5991", "PR #5997", "PR #6000", "PR #6040", "PR #6724", benchmarks/diffusion/backends.py, benchmarks/diffusion/diffusion_benchmark_serving.py, docs/design/feature/skip_softmax.md, docs/user_guide/diffusion/attention_backends.md, docs/user_guide/diffusion/attention_backends/trtllm.md, vllm_omni/config/model.py, vllm_omni/config/omni_config.py, vllm_omni/diffusion/attention/backends/rainfusion_attn.py, vllm_omni/diffusion/attention/backends/trtllm_attn.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/cache/teacache/, vllm_omni/diffusion/forward_context.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/models/minimax_h3/encoder.py, vllm_omni/diffusion/models/minimax_h3/lora.py, vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py, vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py, vllm_omni/diffusion/models/minimax_h3/quality_policy.py, vllm_omni/diffusion/models/minimax_h3/time_request.py, vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/offloader/module_residency.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/utils/hf_utils.py, vllm_omni/entrypoints/omni_base.py, vllm_omni/quantization/int8_config.py, tests/dfx/perf/scripts/run_diffusion_benchmark.py, tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json, tests/diffusion/attention/test_rainfusion_plan.py, tests/diffusion/attention/test_trtllm_attn.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/cache/test_teacache_extractors.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py, tests/diffusion/models/minimax_h3/test_minimax_h3_dlo_lifecycle.py, tests/diffusion/models/minimax_h3/test_minimax_h3_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_parallel.py, tests/diffusion/models/minimax_h3/test_minimax_h3_quality_policy.py, tests/diffusion/models/minimax_h3/test_minimax_h3_quantization.py, tests/diffusion/models/minimax_h3/test_minimax_h3_quantization_quality.py, tests/diffusion/offloader/test_module_residency.py, tests/diffusion/quantization/test_int8_config.py, recipes/MiniMaxAI/MiniMax-H3.md, recipes/MiniMaxAI/MiniMax-H3-5090.md, recipes/MiniMaxAI/MiniMax-H3-MUSA.md, recipes/MiniMaxAI/MiniMax-H3-NPU.md, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/engine/async_omni_engine.py, "PR #5915", "PR #5990", "vllm_omni/diffusion/layers/fused_qk_norm_rope.py", "vllm_omni/diffusion/cache/teacache/extractors.py", "tests/diffusion/layers/test_fused_qk_norm_rope.py", "PR #6061", "vllm_omni/platforms/npu/models/minimax_h3.py", "vllm_omni/platforms/npu/platform.py", "tests/diffusion/models/minimax_h3/test_minimax_h3_qwen3vl_rope.py", "PR #6167", "PR #6173", "PR #6283", "PR #6281", "PR #6878", "vllm_omni/diffusion/attention/ops/minimax_h3_modulation.py", "PR #6476", "PR #6526", "PR #6410", vllm_omni/diffusion/models/minimax_h3/denoise_loop.py, "PR #6876"] confidence: high --- @@ -21,7 +21,9 @@ confidence: high | media ingress、reference tasks、split text encoder | `MMH3-2a`、`MMH3-2b`、`MMH3-2l`；[media](rules-media.md) | | modular task、cache policy、schedules、LoRA/FastH3 | `MMH3-2c`–`MMH3-2n`；[cache/task](rules-cache-task.md) | | DLO、hardware recipes、CI/perf evidence | `MMH3-3a`–`MMH3-3m`；[deployment](rules-deployment.md) | -| VAE kernels、materialization、fallback | `MMH3-4c`；[VAE ops](rules-vae-ops.md) | +| VAE kernels、materialization、keyframe encode determinism、fallback | `MMH3-4c`、`MMH3-4d`；[VAE ops](rules-vae-ops.md) | +| NPU text-MLP packed SwiGLU、quant dispatch、DiT activation | `MMH3-4a`、`MMH3-4b` | + ## 完整代码路由 @@ -198,9 +200,10 @@ confidence: high ## MMH3-4a — H3 Qwen3-VL NPU 文本 MLP 必须使用 packed gate/up 与融合 SwiGLU - 触发：修改 MiniMax H3 Qwen3-VL text encoder 的 MLP、`gate_up_proj` packing、NPU fused operator 或对应 platform override。 -- 强制：NPU 路径使用一次 `F.linear(x, self.gate_up_proj.weight)` 生成 packed gate/up，再调用 `torch_npu.npu_swiglu(gate_up, dim=-1)`，最后复用既有 `down_proj`；gate/up 顺序和结果须等价于 `F.silu(gate) * up`，且补丁只作用于 NPU 目标 text MLP，CPU/CUDA 保持原路径。 -- 禁止：在 NPU 上切分 packed 权重后发起两次 `F.linear` 或使用未融合的 SiLU 乘法；只直接调用 helper 而不验证生产注册链；把该优化扩展到 H3 DiT 或其他模型。 -- 验收：NPU 测试以多个 packed shape 对照 split projection reference，并从真实 diffusion runner 初始化入口断言目标 `MiniMaxH3Qwen3VLTextMLP.forward` 已替换且重复初始化幂等；未通过生产入口的 helper 数值测试不能作为 hook 生效证据。^[PR #6167] +- 强制：NPU 路径调用一次 `self.gate_up_proj(x)`（或等价模块 `__call__`）生成 packed gate/up，再调用 `torch_npu.npu_swiglu(gate_up, dim=-1)`，最后复用既有 `down_proj`；模块调用保留 linear 层的量化 dispatch（online INT8 的 `quant_method.apply`），gate/up 顺序和结果须等价于 `F.silu(gate) * up`，且补丁只作用于 NPU 目标 text MLP，CPU/CUDA 保持原路径。 +- 禁止：`F.linear(x, self.gate_up_proj.weight)` 直接读 weight 绕过量化 dispatch（INT8 `matB` 会落入不支持的普通 matmul）；在 NPU 上切分 packed 权重后发起两次 `F.linear` 或使用未融合的 SiLU 乘法；只直接调用 helper 而不验证生产注册链；把该优化扩展到 H3 DiT 或其他模型。 +- 验收：NPU 测试以多个 packed shape 对照 split projection reference，并从真实 diffusion runner 初始化入口断言目标 `MiniMaxH3Qwen3VLTextMLP.forward` 已替换且重复初始化幂等；NPU regression 另断言 `gate_up_proj` 以模块被调用且 packed 输出进入融合 SwiGLU，全局 `--quantization int8` 的 H3 生成路径不得再出现 `DT_INT8`/`aclnnMatmul` matB 失败；未通过生产入口的 helper 数值测试不能作为 hook 生效证据。^[PR #6167] ^[PR #6876] + ## MMH3-4b — H3 DiT 的 packed SwiGLU 必须通过本地 activation 接入 diff --git a/knowledge/repos/vllm-omni/models/qwen-omni/_index.md b/knowledge/repos/vllm-omni/models/qwen-omni/_index.md index 0e48f9b7e..8db116c44 100644 --- a/knowledge/repos/vllm-omni/models/qwen-omni/_index.md +++ b/knowledge/repos/vllm-omni/models/qwen-omni/_index.md @@ -60,4 +60,5 @@ sources: ["PR #5073", "PR #5671", "PR #5687", "PR #5976", "PR #6284", "PR #4322" | 遇到什么 | 查看哪里 | |---|---| | stage 拓扑、代际差异与官方性能优化结论 | [architecture](architecture.md) | -| Qwen3-Omni Thinker MRoPE、CUDA compilation custom-op boundary、AWQ/compressed-tensors 名称映射、MoE backend default/override、固定种子音频回归、audio-encoder head/TP divisibility，或 Qwen2.5 code2wav soft-fail/旧 speech helper | [Qwen-Omni rules](rules.md) | +| Qwen3-Omni Thinker MRoPE、CUDA compilation custom-op boundary、AWQ/compressed-tensors 名称映射、MoE backend default/override、固定种子音频回归、audio-encoder head/TP divisibility、Qwen2.5 code2wav soft-fail/旧 speech helper，或 code predictor 的 RoPE theta/rope_parameters 读取 + | [Qwen-Omni rules](rules.md) | diff --git a/knowledge/repos/vllm-omni/models/qwen-omni/rules.md b/knowledge/repos/vllm-omni/models/qwen-omni/rules.md index 83d81c91c..db1db3c98 100644 --- a/knowledge/repos/vllm-omni/models/qwen-omni/rules.md +++ b/knowledge/repos/vllm-omni/models/qwen-omni/rules.md @@ -1,10 +1,10 @@ --- title: "Qwen-Omni 规则" created: 2026-09-04 -updated: 2026-09-05 +updated: 2026-09-09 type: rule tags: [vllm-omni, models, qwen-omni] -sources: ["PR #5687", "PR #6284", "PR #6449", "PR #4322", "PR #6748", "PR #6886", "PR #7019", vllm_omni/config/pipeline_registry.py, vllm_omni/deploy/qwen3_omni_moe.yaml, vllm_omni/deploy/qwen3_omni_moe_thinking.yaml, vllm_omni/engine/stage_init_utils.py, vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni.py, vllm_omni/model_executor/models/qwen3_omni/quantization.py, vllm_omni/model_executor/models/qwen3_omni/qwen3_omni.py, vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_moe_thinker.py, vllm_omni/quantization/component_config.py, tests/config/test_config_factory.py, tests/diffusion/quantization/test_component_routing.py, tests/engine/test_stage_engine_args.py, tests/model_executor/models/qwen3_omni/test_qwen3_omni_quantization.py] +sources: ["PR #5687", "PR #6284", "PR #6449", "PR #4322", "PR #6748", "PR #6886", "PR #7019", vllm_omni/config/pipeline_registry.py, vllm_omni/deploy/qwen3_omni_moe.yaml, vllm_omni/deploy/qwen3_omni_moe_thinking.yaml, vllm_omni/engine/stage_init_utils.py, vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni.py, vllm_omni/model_executor/models/qwen3_omni/quantization.py, vllm_omni/model_executor/models/qwen3_omni/qwen3_omni.py, vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_moe_thinker.py, vllm_omni/quantization/component_config.py, tests/config/test_config_factory.py, tests/diffusion/quantization/test_component_routing.py, tests/engine/test_stage_engine_args.py, tests/model_executor/models/qwen3_omni/test_qwen3_omni_quantization.py, "PR #7228"] confidence: high --- @@ -14,7 +14,8 @@ confidence: high | PR 描述信号 | 规则 | |---|---| -| MRoPE、quantization mapping、pipeline key、audio TP、code2wav 或 backend defaults | `QOMNI-1a`–`QOMNI-1f` | +| MRoPE、quantization mapping、pipeline key、audio TP、code2wav、backend defaults 或 code-predictor RoPE theta | `QOMNI-1a`–`QOMNI-1g` | + ## QOMNI-1a — Qwen3-Omni Thinker MRoPE 必须保留 CUDA custom-op 边界 @@ -78,3 +79,10 @@ confidence: high - 强制：仅当 `model_arch` 为 `Qwen3OmniMoeForConditionalGeneration` 且已解析的 `moe_backend` 为缺失或 `auto` 时，finalization 才默认写入 `triton`；任何显式 backend 必须原样保留。随仓库发布的 `qwen3_omni_moe.yaml` 必须在 Thinker stage 0 和 Talker stage 1 都显式 pin `moe_backend: triton`；Code2Wav 不继承该模型专属 MoE 设置。 - 禁止：通过 `VLLM_USE_FLASHINFER_MOE_FP16` 或其他 process-wide 环境变量选择 backend；把默认值写成覆盖用户/deploy 显式选择；只改 deploy YAML 却不覆盖 legacy 与 typed builder 的最终 engine args。 - 验收：以 Qwen3-Omni default 和一个显式 backend control，分别经 `build_legacy_engine_args_dict` 与 `build_engine_args_dict_from_omni_stage_config` 断言最终 `moe_backend`；默认两路径均为 `triton`，显式值不变。helper 单测只能补充分支覆盖，不能代替 builder-level assertion。PR 只提供 compile/Ruff/diff 检查及因缺少 `torch` 未运行的 targeted pytest，故不构成 GPU 稳定性、性能或音频质量证据。^[PR #7019] + +## QOMNI-1g — 共享 code predictor 的 RoPE theta 必须先读 rope_parameters + +- 触发：修改 `qwen3_code_predictor._RotaryEmbedding`、Qwen3-Omni Talker code predictor config 反序列化，或 Transformers 对 `rope_theta`/`rope_parameters` 的布局。 +- 强制：构造 `inv_freq` 时优先 `config.rope_parameters["rope_theta"]`（若 mapping 存在），否则回退 top-level `rope_theta`，再默认 `10000.0`。Qwen3-Omni（Transformers 5.10+ 常把 checkpoint 的 `1_000_000` 只放在 nested 字段）与仍带 top-level 的 Qwen3-TTS/legacy 必须共用该顺序。 +- 禁止：只读 top-level 导致静默回退 `10000`；把 nested 优先写成破坏 TTS top-level 兼容的唯一来源。 +- 验收：参数化覆盖 nested-only、nested 优先于冲突 top-level、默认 `10000`，以及 TTS/legacy top-level（含删除 `rope_parameters`）重建正确 `inv_freq`。^[PR #7228]