Skip to content

knowledge(vllm-omni): 1 change(s) from intake #360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tzhouam merged 1 commit into from
Oct 13, 2026
Merged

knowledge(vllm-omni): 1 change(s) from intake #360

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions knowledge/repos/vllm-omni/rules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "vLLM-Omni 硬门禁"
created: 2026-07-10
updated: 2026-10-12
updated: 2026-10-13
type: rule
tags: [vllm-omni]
sources: [benchmark/incidents/_index.md, ci/incidents/_index.md, git/incidents/_index.md, remote/incidents/_index.md, "PR #8014", "PR #8786", "PR #8770"]
sources: [benchmark/incidents/_index.md, ci/incidents/_index.md, git/incidents/_index.md, remote/incidents/_index.md, "PR #8014", "PR #8786", "PR #8770", "PR #8645"]
---

# vLLM-Omni 硬门禁
Expand Down Expand Up @@ -125,3 +125,12 @@ sources: [benchmark/incidents/_index.md, ci/incidents/_index.md, git/incidents/_
- 验收：`vllm_omni/platforms/npu/profiler.py` 的 `_create_profiler` 中，`torch_npu.profiler.profile(...)` 调用包含 `record_shapes=profiler_config.torch_profiler_record_shapes`。^[PR #8770]

<!-- kb:rule status=active since=v0.30.0 -->

## VLLM-OMNI-PR8645-DUPLEXEVAL-BENCH — Omni-DuplexEval 并入 bench serve：独占 run 目录、计时后评分与取消语义

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Route the rule through the benchmark owner

Every trigger in this section is confined to benchmark code, CLI wiring, tests, and documentation, so the existing knowledge/repos/vllm-omni/benchmark/rules.md is the nearest owner. Keeping it at repository root both exposes a feature-specific contract to unrelated vLLM work and leaves Direct benchmark reviews unable to retrieve it: knowledge/repos/vllm-omni/_routes.yaml has no benchmark owner, and Direct does not use the root page as a fallback once that route table exists. Move the rule under the benchmark owner and add the corresponding route.

AGENTS.md reference: AGENTS.md:L12-L17

Useful? React with 👍 / 👎.


- 触发：改动 `vllm_omni/benchmarks/duplex/omni_duplex_eval_serving.py`、`omni_duplex_eval_runner.py`、`omni_duplex_eval_judge.py`，或 `vllm_omni/entrypoints/cli/benchmark/`（cli_args.py / serve.py）中 `omni-duplex-eval` 数据集与 `--duplex-eval-*` 旗标在 `vllm bench serve` 的接线，或 `tests/benchmarks/duplex/test_omni_duplex_eval_serving.py` 与 `docs/cli/bench/serve.md` 的对应小节。
- 强制：`--duplex-eval-response-root` 仅是父目录，每次调用必须在其下创建独占 `run-*` 子目录（使并发/QPS 扫描不覆盖旧结果），实际路径写日志并存入保存结果的 `omni_duplex_eval.response_root`；只有成功的测量会话在基准计时结束后发布 `<split>/<id>.json` 与 `<id>.meta.json` 工件，工件错误记入 `omni_duplex_eval` 摘要而不丢弃服务测量。可选评分（`--duplex-eval-evaluate` 及 judge 旗标）复用既有独立评测器且必须在性能计时结束后运行，按 total/evaluated/skipped/failed 报告覆盖：零 evaluated 记为 `failed` 而非合法准确率，warmup 永不评分，judge 失败不得使已完成的性能测量作废。会话语义：一个视频即一个会话/请求；正常 barge-in 取消是回合结局而非样本失败，其已收集文本仍可交给 judge；关闭会话前排空观察到的活跃响应，关闭边界产生的响应记入元数据 `unfinished_response_ids`，由结果的 `omni_duplex_eval.published_with_unfinished_responses` 计数、非零时告警；服务器错误、失败响应与排空/关闭超时仍使样本失败。媒体准备在各并发槽的 worker 线程内懒惰执行并在请求完成时释放解码输入，参考音频只编码一次共享；准备不计入单会话时延但计入总时长与吞吐；`--omni-request-timeout-s`（默认 900 秒）只覆盖准备后的完整会话。

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Split independent invariants into separate rule IDs

This single ID and lifecycle footer cover independent contracts for run-directory allocation, artifact publication, judge coverage, cancellation/draining, media-decoding concurrency, and request timeout. doc/knowledge/contributing/page-rules.md:127 requires one stable ID per behavioral invariant; otherwise a reviewer cannot mark, replace, or retire one facet without treating the entire bundle as satisfied or obsolete. Split these constraints into separately auditable benchmark rule IDs.

Useful? React with 👍 / 👎.

- 禁止：把工件直接写入 response-root 根层或复用/覆盖既有 `run-*` 目录；在计时结束前评分、给 warmup 评分，或把零 evaluated 报成有效准确率；因 judge 失败丢弃性能结果；把正常 barge-in 取消计为样本失败；在事件循环上同步解码全部媒体或令驻留解码片段数突破 `--max-concurrency`。
- 验收：`tests/benchmarks/duplex/test_omni_duplex_eval_serving.py` 通过；同一 response-root 下重复调用生成互不相同的 `run-*` 目录；保存结果含 `omni_duplex_eval.response_root`、带覆盖计数的 `omni_duplex_eval.accuracy` 与 `published_with_unfinished_responses` 字段。^[PR #8645]

<!-- kb:rule status=active since=v0.30.0 -->

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove the active rule until PR 8645 merges

The source PR #8645 is still Open, whereas src/infermatrix_copilot/kb_service/intake.py defines merged_pr events as upstream-merged PRs and requires current-head validation of path and behavior claims. Publishing its branch-only files, flags, and test as status=active since=v0.30.0 admits an unmerged proposal into the live reviewer contract, so reviews against upstream main can demand interfaces that do not exist there. Remove or defer this rule until the upstream merge is observed.

Useful? React with 👍 / 👎.

Loading