From 61c25dea11f943952cc426b93e031642ba4efe11 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Mon, 19 Oct 2026 13:33:37 +0000 Subject: [PATCH] knowledge(vllm-omni): 4 change(s) from refine --- .../vllm-omni/components/diffusion/_index.md | 1 + .../diffusion/rules-memory-unit-labels.md | 19 +++++++++++++++++++ .../components/model-executor/_index.md | 1 + .../model-executor/rules-moss-codec-perf.md | 19 +++++++++++++++++++ .../vllm-omni/models/lingbot-video/rules.md | 13 +++++++++++-- .../repos/vllm-omni/models/voxcpm2/rules.md | 13 +++++++++++-- 6 files changed, 62 insertions(+), 4 deletions(-) create mode 100644 knowledge/repos/vllm-omni/components/diffusion/rules-memory-unit-labels.md create mode 100644 knowledge/repos/vllm-omni/components/model-executor/rules-moss-codec-perf.md diff --git a/knowledge/repos/vllm-omni/components/diffusion/_index.md b/knowledge/repos/vllm-omni/components/diffusion/_index.md index b83ba271..0bd66ae4 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/_index.md +++ b/knowledge/repos/vllm-omni/components/diffusion/_index.md @@ -84,3 +84,4 @@ shared RMSNorm 的empty-input与fused精度边界沿 tensor dtype rules 的DIFF- - [Diffusion request batch 的 pipeline 合同](rules-request-batch.md)。 - [Diffusion JIT CUDA 构建 include 顺序规则](rules-jit-cuda-includes.md) - [AR-Diffusion 实时模型集成规则](rules-ar-realtime-integration.md) +- [Diffusion runner peak memory 日志单位规则](rules-memory-unit-labels.md) diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-memory-unit-labels.md b/knowledge/repos/vllm-omni/components/diffusion/rules-memory-unit-labels.md new file mode 100644 index 00000000..0f91d039 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-memory-unit-labels.md @@ -0,0 +1,19 @@ +--- +title: "Diffusion runner peak memory 日志单位规则" +created: 2026-10-19 +updated: 2026-10-19 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #8865"] +--- + +# Diffusion runner peak memory 日志单位规则 + +## VLLM-OMNI-PR8865-MEM-UNIT-LABELS — runner peak memory 日志必须按除数标注 GiB 并冻结 peak_memory_mb 命名 + +- 触发：修改 `vllm_omni/diffusion/worker/diffusion_model_runner.py` 中 `_sample_peak_memory_mb` 的 per-request peak memory 日志行，或该方法的返回字段 `peak_memory_mb` 的命名与语义。 +- 强制：该日志行中按 `1024**3` 换算的 reserved、allocated、pool overhead 数值必须标注 GiB，并复用 `vllm.utils.mem_utils.format_gib` 生成字符串，与 vLLM 其余二进制尺寸输出一致；pool overhead 百分比守卫直接以 bytes 相除（`pool_overhead_bytes / peak_reserved_bytes`），不得经过换算后的 GB 浮点中间量。`peak_memory_mb` 按公开字段名冻结：其值实为 `1024**2` 换算的 MiB，名字与单位的不匹配在 `_sample_peak_memory_mb` 的 docstring 中说明（`Named _mb to match the public peak_memory_mb field`）。 +- 禁止：把该方法中按 `1024**3` 换算出的数字标成 GB；以“修正单位”为由重命名 `peak_memory_mb`/`_sample_peak_memory_mb` 或改变其返回值；在纯标签/docstring 改动中顺手改变数值、返回值或字段名。 +- 验收：日志三个分项均以 GiB 标注且数值等于对应 bytes 按 `1024**3` 换算；百分比由 bytes 直接计算；`peak_memory_mb` 字段名、返回语义与既有消费方保持不变。^[PR #8865] + + diff --git a/knowledge/repos/vllm-omni/components/model-executor/_index.md b/knowledge/repos/vllm-omni/components/model-executor/_index.md index 895a44d0..6cdfdd15 100644 --- a/knowledge/repos/vllm-omni/components/model-executor/_index.md +++ b/knowledge/repos/vllm-omni/components/model-executor/_index.md @@ -64,3 +64,4 @@ sources: [] - [Model Executor — Dynamo/cudagraph compile guard rules](rules-compile-guard.md) - [Model Executor — INC/AutoRound 量化配置规则](rules-inc-autoround.md) - [Model Executor — Orchestration Loop Rules](rules-orchestration-loop.md) +- [MOSS codec 解码性能与数值规则](rules-moss-codec-perf.md) diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-moss-codec-perf.md b/knowledge/repos/vllm-omni/components/model-executor/rules-moss-codec-perf.md new file mode 100644 index 00000000..42744d38 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-moss-codec-perf.md @@ -0,0 +1,19 @@ +--- +title: "MOSS codec 解码性能与数值规则" +created: 2026-10-19 +updated: 2026-10-19 +type: rule +tags: [vllm-omni, components, model-executor] +sources: ["PR #8002"] +--- + +# MOSS codec 解码性能与数值规则 + +## VLLM-OMNI-PR8002-MOSS-CODEC-PERF — MOSS codec 解码优化必须绑定缓存生命周期与逐设备数值证据 + +- 触发：修改 `vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py` 中 transformer 层的 attention bias（`attn_bias`）构造与跨层共享、`_cached_attn_bias` 缓存，或 `_sa_block`/`_ff_block` 的 residual+layer_scale 融合（如 `torch.addcmul`）。 +- 强制：跨层共享的 attention bias 由 block 内第一层计算并显式经 `attn_bias` 参数传给后续层；`_cached_attn_bias` 必须在 forward 结束时清空（含异常中断路径），不得跨 step 持有 tensor。融合算子必须以实际模块类型守卫（`isinstance(..., MossAudioTokenizerLayerScale)`），对 `nn.Identity()`（`layer_scale=None`）回退原 mul+add 路径。数值主张逐设备/精度分别取证、按实测覆盖面表述：bit-identical 主张须在真实 checkpoint 上以 `torch.equal` 验证（本例 mask 共享覆盖 CPU fp32、NPU fp32/bf16）；改变累加顺序的融合（如 addcmul，非 bit-identical）须在真实 checkpoint 上逐设备/精度报告 SNR 或 bit-identity（本例 CPU fp32 SNR 107.1 dB、NPU fp32 bit-identical、NPU bf16 SNR 31.5 dB）。“误差低于该 dtype 自身量化噪声”是更强的额外主张：只须在作出该主张的设备/精度上实测论证，且不得推广为每个设备/精度的统一门槛（本例仅 NPU bf16：融合 vs 手写差值 SNR 31.5 dB，高于 bf16-vs-fp32 的 26.6/27.3 dB 基线；CPU fp32 与 NPU fp32 未做、也不要求做该比较）。 +- 禁止：让缓存 bias 在 forward 异常、streaming 每步或 streaming 退出后残留；不设回退地对所有配置启用融合；把改变累加顺序的 `addcmul` 声称为 bit-identical；仅用 CI 小 codec fixture 的数值结果代表真实模型；把“低于量化噪声”的比较当作所有设备/精度的必备证据，或将单一 dtype 上验证过的噪声底结论外推到未测设备/精度。 +- 验收：CPU 测试（`tests/model_executor/models/moss_tts/test_audio_tokenizer_v2.py`，`core_model and cpu`）断言非流式、流式、execution-context 驱动、层内抛异常与多步 streaming 后 `_cached_attn_bias is None`；数值等价的 NPU 变体在 `test_audio_tokenizer_v2_npu.py`（`core_model and npu`）覆盖；真实 checkpoint 的逐设备测量结果（本例 CPU fp32、NPU fp32/bf16）写入 PR，噪声底一类结论只限实测过的设备/精度。^[PR #8002] + + diff --git a/knowledge/repos/vllm-omni/models/lingbot-video/rules.md b/knowledge/repos/vllm-omni/models/lingbot-video/rules.md index 808e4f84..e486dd13 100644 --- a/knowledge/repos/vllm-omni/models/lingbot-video/rules.md +++ b/knowledge/repos/vllm-omni/models/lingbot-video/rules.md @@ -1,10 +1,10 @@ --- title: "LingBot-Video 规则" created: 2026-08-10 -updated: 2026-10-06 +updated: 2026-10-19 type: rule tags: [vllm-omni, models, diffusion] -sources: [vllm_omni/diffusion/models/lingbot_video/request_utils.py, vllm_omni/diffusion/models/lingbot_video/image_condition.py, vllm_omni/diffusion/models/lingbot_video/pipeline_lingbot_video.py, vllm_omni/entrypoints/openai/serving_video.py, vllm_omni/model_extras/lingbot_video.py, vllm_omni/model_extras/registry.py, examples/offline_inference/text_to_image/text_to_image.py, examples/offline_inference/image_to_video/image_to_video.py, tests/diffusion/models/lingbot_video/test_request_utils.py, tests/diffusion/models/lingbot_video/test_image_condition.py, tests/diffusion/models/lingbot_video/test_pipeline_lingbot_video.py, tests/entrypoints/openai_api/test_image_server.py, tests/entrypoints/openai_api/test_video_server.py, "PR #5311", "PR #5976", "PR #6049", "Issue #5883", "PR #7037"] +sources: [vllm_omni/diffusion/models/lingbot_video/request_utils.py, vllm_omni/diffusion/models/lingbot_video/image_condition.py, vllm_omni/diffusion/models/lingbot_video/pipeline_lingbot_video.py, vllm_omni/entrypoints/openai/serving_video.py, vllm_omni/model_extras/lingbot_video.py, vllm_omni/model_extras/registry.py, examples/offline_inference/text_to_image/text_to_image.py, examples/offline_inference/image_to_video/image_to_video.py, tests/diffusion/models/lingbot_video/test_request_utils.py, tests/diffusion/models/lingbot_video/test_image_condition.py, tests/diffusion/models/lingbot_video/test_pipeline_lingbot_video.py, tests/entrypoints/openai_api/test_image_server.py, tests/entrypoints/openai_api/test_video_server.py, "PR #5311", "PR #5976", "PR #6049", "Issue #5883", "PR #7037", "PR #8651"] --- # LingBot-Video 规则 @@ -97,3 +97,12 @@ sources: [vllm_omni/diffusion/models/lingbot_video/request_utils.py, vllm_omni/d - 验收：同时覆盖 formatter 的 image/video key、T2I 单帧 shape、TI2V input cardinality、最终 image size limit、frame-count precedence 和 n>1 的 400；FP32/BF16 与 omitted/pt/np 的黑、中灰、白须穿过真实 API frame conversion 后保持原 `[0,1]` 值，不能只测全白；bare pt 与 latent identity control 保持。^[PR #7037] PR 所报 H200 单次耗时、峰值显存与 bitwise parity 缺少本知识树可复跑的命令/产物，不能作为性能或精度 gate。 + +## VLLM-OMNI-PR8651-LBV-DEVICE-AUTOCAST — autocast 的 device_type 必须取自真实设备，cuDNN 布局与权重 dtype 门各守其位 + +- 触发：修改 `vllm_omni/diffusion/models/lingbot_video/` 中 `lingbot_video_transformer.py` 的 `apply_rotary_emb`、`pipeline_lingbot_video.py` 的 `_transformer_autocast`/`_decode_latents`、`image_condition.py` 的 `encode_clean_image_latent` 的 autocast/half-precision 路径或 `channels_last_3d` 布局，或在 `docs/models/supported_models.md` 为 `LingBotVideoPipeline` 勾选新硬件列。 +- 强制：autocast 上下文的 `device_type` 必须来自真实张量/模块 device：`apply_rotary_emb` 的禁用保护写 `torch.amp.autocast(device_type=x.device.type, enabled=False)`——硬编码 `"cuda"` 的禁用保护在 XPU 等设备上是无效 no-op。各路径的启用条件归属不同：`encode_clean_image_latent` 在所有非 CPU 设备上启用 bf16 autocast（`enabled=vae_device.type != "cpu"`），不检查权重 dtype；`_transformer_autocast` 与 `_decode_latents` 以 `device.type == "cpu"` 为唯一设备排除条件，且 autocast dtype 另有 fp16/bf16 权重 dtype 门。`channels_last_3d` 是 cuDNN VAE 路径，仅当 `vae_device.type == "cuda"` 且 latents 为 5D 时应用；`torch._grouped_mm` 保持 CUDA-only。支持矩阵新勾选硬件（本例 XPU）必须与上述 device 合同同批落地。 +- 禁止：把 autocast 的 device_type 硬编码为 `"cuda"`；用 `device.type != "cuda"` 把 XPU 等非 CUDA 加速卡排除在 half-precision 之外；在非 CUDA 设备上套用 `channels_last_3d`；声称 encode 路径存在权重 dtype 门或要求未来改动为其添加证据中不存在的门；未按 device 合同验证就勾选支持矩阵。 +- 验收：按路径核对启用条件——encode：非 CPU（含 fp32 权重）开启、CPU 关闭；transformer/decode：非 CPU 且 fp16/bf16 权重才开启；CUDA 5D decode 使用 `channels_last_3d` 而其他设备布局不变；`_grouped_mm` 仍 CUDA-only；支持矩阵勾选与代码合同一致。^[PR #8651] + + diff --git a/knowledge/repos/vllm-omni/models/voxcpm2/rules.md b/knowledge/repos/vllm-omni/models/voxcpm2/rules.md index e4000949..ea8077aa 100644 --- a/knowledge/repos/vllm-omni/models/voxcpm2/rules.md +++ b/knowledge/repos/vllm-omni/models/voxcpm2/rules.md @@ -1,10 +1,10 @@ --- title: "VoxCPM2 请求噪声规则" created: 2026-09-22 -updated: 2026-10-06 +updated: 2026-10-19 type: rule tags: [vllm-omni, models] -sources: ["PR #7866", "PR #7010"] +sources: ["PR #7866", "PR #7010", "PR #8311"] --- # VoxCPM2 请求噪声规则 @@ -29,3 +29,12 @@ sources: ["PR #7866", "PR #7010"] - 强制：三字段只从 `_VoxCPM2RuntimeConfig` 消费，默认 `10`、`2.0`、`1.0`；`inference_timesteps` 钳制到至少 `2`，`cfg_value` 必须有限，否则初始化失败，`cfg_cutoff_ratio` 夹至 `[0,1]`。所有 solver 路径使用同一已解析值。 - 禁止：让 CFG-Zero* 的一阶输入跳过全部 estimator 并返回原始噪声；让 NaN/inf CFG 流入 solver；在 talker 或 graph path 保留另一套硬编码默认。有限负 CFG 当前仍可解析，不能把未实现的非负门禁写成已有合同。 - 验收：覆盖负/0/1/2/4/10 timesteps、非有限 CFG、cutoff 两端以及不同 batch size；断言 estimator 调用数、有限且非原始噪声的输出、输入 noise 未被修改，以及省略配置的默认保持。^[PR #7010] + +## VLLM-OMNI-PR8311-VOXCPM2-DEFAULT-SYNC — VoxCPM2 运行时默认值翻转必须与 deploy YAML 的 hf_overrides pin 对账 + +- 触发：修改 `_VoxCPM2RuntimeConfig`（`vllm_omni/model_executor/models/voxcpm2/runtime_config.py`）的运行时开关默认值（如 `enable_batched_fsq_fusion`、`enable_batched_prefill_tail` 由 False 改 True），或增删 `vllm_omni/deploy/voxcpm2.yaml` 对应 stage `hf_overrides` 中的同名运行时字段。 +- 强制：`voxcpm2.yaml` 的 `hf_overrides` 值优先于 dataclass 默认，且两者可以有意不同（如 `unified_decode_graph_max_batch_size` 在 YAML pin 为 8、dataclass 默认 64）。翻转 dataclass 默认时必须在同一 PR 对账 YAML 是否 pin 该字段：本例两个批处理开关在翻转默认的同时以 `true` 显式写入 YAML；对已 pin 的字段，只改 dataclass 不会改变 bundled 部署的实际取值。把批处理/融合开关由默认关闭改为默认开启属于改变执行路径的决定，须附目标硬件实测收益（本例 Ascend 910B3 TTFP -103ms）与音频质量回归（WAV 输出无退化），且单硬件结果不得外推为全平台承诺。 +- 禁止：只改 `runtime_config.py` 或只改 YAML 而未对账另一侧的最终生效值；把 dataclass 默认与 YAML pin 的有意分歧（如 `unified_decode_graph_max_batch_size` 64 vs 8）当作必须消除的错误；以 kernel dispatch 开销降低的推断代替基准数据；默认开启改变执行路径的开关而不做输出质量验证。 +- 验收：逐字段核对翻转后 dataclass 默认与 YAML `hf_overrides` 的最终取值符合意图（未 pin 字段吃到新默认，已 pin 字段显式更新或保持 pin），并随 PR 提交开启前后的 TTFP/延迟基准与音频质量检查结果。^[PR #8311] + +