Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
b32f114
Rework Strict review for the official model: deep passes, per-comment…
tzhouam Aug 13, 2026
81e4b6f
Eval harness: pipelined gen+judge orchestrator, cursor-agent harness …
tzhouam Aug 14, 2026
b3a0f07
Eval data: wave-2 holdout arms, judgments, and consolidated model-com…
tzhouam Aug 14, 2026
31701ae
MoA harness members: mixture members may ride a provider-registry har…
tzhouam Aug 14, 2026
25f4ccb
Audit: resolve relative tool paths against the session cwd
tzhouam Aug 14, 2026
a9e650b
Cursor-backend eval campaign: composer/grok arms, MoA replicate, cons…
tzhouam Aug 14, 2026
3c9305f
Results: normalized view — each arm ÷ its in-matchup baseline
tzhouam Aug 14, 2026
3047242
Results: fix two stale hand-written rows, stamp table INTERIM
tzhouam Aug 14, 2026
4c0986b
Strict review v14/v15: distilled duties, docs pass, second round, arc…
tzhouam Aug 15, 2026
bbf162a
Eval: waves 3-4 frozen holdouts + holdout3/holdout4 harness wiring
tzhouam Aug 15, 2026
1d28450
Eval data: v14/v15 campaign arms, judgments, wave-3/4 baselines, results
tzhouam Aug 15, 2026
893d745
Cheap-seats model routing + per-pass backend role-split
tzhouam Aug 15, 2026
d5d6193
Providers: integrate M2/M3 backends (claude-code, codex) from PR #82
tzhouam Aug 15, 2026
e3cdc17
Eval data: wave-4 v15 replicate 2, role-split + v16 probes, v16 gate
tzhouam Aug 15, 2026
36cbec5
Paired item-clustered statistics; wave-5 draw + pre-registration
tzhouam Aug 15, 2026
0aa5229
Campaign report v14-v16 + document the two supported review configs
tzhouam Aug 15, 2026
22e100b
Wave-5 pre-registered measurement: the recall deficit is real (-.049)
tzhouam Aug 16, 2026
f55aa55
Routed seats: preserve routing on retry, fall back loudly, gate the arm
tzhouam Aug 16, 2026
ae1b6f1
Conversion fixes: the reasoning existed, the routing lost it
tzhouam Aug 16, 2026
f9383d0
Pre-register the v17 conversion-fix probe before generating it
tzhouam Aug 16, 2026
11e9318
Probe: disclose smoke reuse and the dedupe defect left unfixed on pur…
tzhouam Aug 16, 2026
b718c9b
v17 conversion-fix probe: the fixes did not close the recall gap (-.050)
tzhouam Aug 16, 2026
17f549c
Amend the v17 probe: the first run was an unintended vendor mixture
tzhouam Aug 17, 2026
cabee89
Relabel the first v17 run as v17-moa in the data and the results table
tzhouam Aug 17, 2026
4725020
Correct the MoA contamination magnitude: ~10%, not 24%
tzhouam Aug 17, 2026
ebf075e
Corrected v17ds run halted: DeepSeek 402 Insufficient Balance at 7/15…
tzhouam Aug 17, 2026
3fb41b8
v17 val, both backends: Composer 2.5 reaches recall parity where Deep…
tzhouam Aug 17, 2026
a7ccbbf
Corrected v17 probe completes: parity on spent splits; yesterday's co…
tzhouam Aug 17, 2026
17b5e15
DeepSeek Harness (dsh) provider: the fourth backend, and the first AP…
tzhouam Aug 17, 2026
8ba5e95
Composer 2.5 on v17 train: the api/DeepSeek core is the champion
tzhouam Aug 17, 2026
c89b6a7
Render defects the pr4893 forensics found, plus dsh on native tools
tzhouam Aug 17, 2026
e2cfcaf
Three eval-process gates, one per incident that cost a measurement today
tzhouam Aug 17, 2026
dd97643
dsh: bound the session ourselves -- the harness has no step cap
tzhouam Aug 17, 2026
cff2d4f
Three doc linters, because nothing checked documentation at all
tzhouam Aug 17, 2026
8fee3d3
Two false capability claims the SPEC rewrite caught
tzhouam Aug 17, 2026
9647b1b
Refactor doc/ into a signposted tree, and fix what the move broke
tzhouam Aug 17, 2026
31e5c72
Wire the doc gates into CI, blocking
tzhouam Aug 17, 2026
fe35baf
v19 signal density: the ledger dedupe was inert, and it cost Composer…
tzhouam Aug 17, 2026
7bbb35b
Collapse path identifiers to basenames when scoring shared subjects
tzhouam Aug 17, 2026
4c8d67e
v20: merge duplicate findings to their richest statement instead of d…
tzhouam Aug 17, 2026
c5c5697
v21: give each finding a claim headline and its evidence a quote block
tzhouam Aug 17, 2026
fb54f94
v21 fix: emit a headline only when there IS a claim and an argument
tzhouam Aug 18, 2026
59579ed
Revert v21: the finding-headline restructure measurably costs recall
tzhouam Aug 18, 2026
82d682f
v22: require the causal mechanism in a finding, not just the flag
tzhouam Aug 18, 2026
1e40b25
Revert v22: requiring per-finding mechanism depth costs recall
tzhouam Aug 18, 2026
7d4055d
v23: add a breadth sweep lens to the round-1 ensemble
tzhouam Aug 18, 2026
4e3b324
Revert v23 sweep lens: coverage rose, recall did not
tzhouam Aug 18, 2026
89e915f
Correct the pr4970 figures in 4e3b324, and record leave-one-out
tzhouam Aug 19, 2026
17b77c3
Carry forward the PR #71 RFC from the remote branch, with its rollout…
tzhouam Aug 19, 2026
d7cd6b9
Fix the CI breakage from the knowledge/ to doc/knowledge/ move
tzhouam Aug 19, 2026
e12f5fc
Add the vs-Opus-5 performance chart used in the PR description
tzhouam Aug 19, 2026
2449ac1
Land the in-flight doc/knowledge refactor for the authoring pages
tzhouam Aug 19, 2026
d228b0a
Point the doc-link checker at the moved templates tree, and finish CL…
tzhouam Aug 19, 2026
6f66ed2
Land the rest of the doc/ refactor so the tree is self-consistent
tzhouam Aug 19, 2026
91919a5
Give CI full git history so the SPEC freshness check can work
tzhouam Aug 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
23 changes: 22 additions & 1 deletion .env.template
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,34 @@ CLAUDE_CODE_ATTREBUTION_HEADER=0
# api = the raw-API keys above
# cursor = Cursor subscription via the cursor-agent CLI (no API key;
# the CLI must be installed and logged in)
# claude-code / codex = declared, ship in M2/M3
# claude-code = Claude subscription/API via the claude CLI (M2, shipped)
# codex = ChatGPT subscription via the codex CLI (M3; needs a login)
STRICT_BACKEND=api
# STRICT_BACKEND_MODEL= # model id inside the harness (optional)
# STRICT_BACKEND_CONCURRENCY=2 # concurrent harness sessions
# STRICT_BACKEND_CLI= # binary path override (else PATH lookup)
# STRICT_BACKEND_TIMEOUT_S=1800 # per-session wall-clock ceiling

# --- Review model routing (measured on the wave-4/5 gates; see
# --- eval/dataset/results/model_comparison.md before changing these) -------
# Cheap seats: the planner's gray-zone call and the tool-less coverage/
# residue promotion do not carry the precision result, so a cheap model is
# the right default. NOTE a reasoning model needs headroom — the planner
# silently failed for two campaigns at a 400-token ceiling because thinking
# consumed it before any JSON appeared.
# REVIEW_PLANNER_MODEL=deepseek-v4-flash
# REVIEW_PROMOTION_MODEL=deepseek-v4-flash
#
# Per-pass backends: map a review pass to "provider:model" to run that seat
# on a different engine. Measured configurations (same gate, same judge):
# (unset) — all-DS "v15": the precision-leaning config, ~$1/item.
# adversary+round2 on claude-code — "v16": the recall-leaning config; the
# only one that reproduced the GOLD latent-gap catch inside the pipeline.
# Costs 2 harness sessions/item on top of the DS core.
# investigator/docs/round2 on cursor — subscription-priced breadth; it
# measured WORSE precision and no recall gain. Not recommended.
# REVIEW_LENS_BACKENDS={"adversary": "claude-code:claude-fable-5", "round2": "claude-code:claude-fable-5"}

# Repos: JSON map of repo name -> local path (machine-specific; .env is gitignored,
# which is why absolute paths belong HERE and never in a committed file).
REPO_PATHS={"vllm-omni": "/path/to/vllm-omni", "afd-plugin": "/path/to/afd-plugin"}
Expand Down
27 changes: 26 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: tests

# The suite is ~485 offline cases with no network and no paid LLM calls, so it can
# run on every push. Until this existed it ran only when someone remembered to type
# `pytest` (doc/CODE_TOUR.md documents it as a manual step) — tests that nothing runs
# `pytest` (doc/architecture/CODE_TOUR.md documents it as a manual step) — tests that nothing runs
# protect nothing.
#
# NOTE: this workflow is a SIGNAL, not a merge gate. Making it blocking needs a branch
Expand All @@ -21,7 +21,16 @@ jobs:
suite:
runs-on: ubuntu-latest
steps:
# Full history, not the default depth-1 shallow clone. check_spec_freshness
# dates each source file with `git log -1 -- <path>`, and in a shallow clone
# the single commit has no parent, so git reports EVERY tracked file as
# created in it. All 62 SPEC pages then read as modified today and the
# strict check fails wholesale. It only ever passed because the tip commit
# happened to predate the pages' verified-against: dates; the first commit
# made on a later day broke all 62 at once.
- uses: actions/checkout@v4
with:
fetch-depth: 0

- uses: actions/setup-python@v5
with:
Expand Down Expand Up @@ -74,6 +83,22 @@ jobs:
", ".join(_RESOURCE_MARKERS))
PY

# Doc integrity. These exist because the doc/ refactor measured the damage
# they prevent: 29 in-code citations of doc/ paths that no linter checked,
# and 44 of 45 SPEC pages silently behind their own source.
- name: Doc linters
run: |
python tools/check_doc_links.py
python tools/check_doc_citations.py

# BLOCKING as of the P5 rewrite: all 92 modules have a SPEC page and all
# 62 pages carry a `verified-against:` date. --strict additionally fails
# on a page whose source changed after that date, or one that dropped its
# marker — which is the whole point: 44 of 45 pages had silently fallen
# behind before this check existed.
- name: SPEC coverage and freshness
run: python tools/check_spec_freshness.py --strict

# Written, tested, and until now invoked only from prose (AGENTS.md, DEVELOPMENT.md).
- name: Knowledge linters
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/vllm-omni-release-audit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
- .github/workflows/vllm-omni-release-audit.yml
- adapters/vllm_omni/manifest.yaml
- adapters/vllm_omni/release_baseline.yaml
- doc/VLLM_OMNI_RELEASE_MAINTENANCE.md
- doc/contributing/release-maintenance.md
- plugins/infermatrix-copilot/skills/imupdate/**
- knowledge/repos/vllm-omni/**
- tools/audit_vllm_omni_release.py
Expand Down
22 changes: 13 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,32 @@

## Knowledge edits

`knowledge/` is a vendored adaptation of
`zuiho-kai/claude-workflow-starter`. Before adding, moving, or deleting any
knowledge page:
Before adding, moving, or deleting any knowledge page:

1. Read `doc/PLAN-knowledge-reorg.md`, `knowledge/CONTRIBUTING.md`, then exactly
the one linked contribution topic that matches the change. The plan's
owner-scoped inventory and detail-retention rules remain authoritative for
this vendored tree.
1. Read `doc/knowledge/CONTRIBUTING.md`, then exactly the one linked contribution
topic that matches the change. The current contract is that entry page plus
`doc/knowledge/contributing/`, `doc/knowledge/SCHEMA.md` and the two validators;
`doc/archive/PLAN-knowledge-reorg.md` is archived history, not the directory
contract. Keep working owner-scoped and merge synonymous conclusions union-first.
2. Route each conclusion by both purpose and code owner. PR-learning and review-
experience intake may produce only executable rules in the nearest owner's
`rules.md`; model-specific rules stay with that model owner. Raw PR material
is temporary input outside the repository and must be deleted after the
owner-scoped rule batch passes validation. Never add PR case/history/result/
incident pages or collapse heterogeneous owners into one catch-all page.
3. Treat `knowledge/SCHEMA.md` as an additive metadata overlay only. Existing
3. Treat `doc/knowledge/SCHEMA.md` as an additive metadata overlay only. Existing
`incidents/`, `history/`, and `results/` files are legacy material, not an
allowed destination for new PR learning. Generated evaluation cases, hidden
labels, predictions, judgments, and run reports belong under `eval/`, not in
the product knowledge tree.
4. Never persist raw evidence pages or replay outputs. Update the nearest
`_index.md` for every current rules page.
5. Run both validators after the complete owner-scoped batch:
5. Editing `knowledge/repos/vllm-omni/` also touches a machine contract the two
validators cannot see: `owner_documents` entry pages, in-body SHA pins and
upstream `sources:` are reconciled by `tools/audit_vllm_omni_release.py`, and
any PR touching that slice runs it in `enforce` mode
(`doc/knowledge/contributing/validation.md`).
6. Run both validators after the complete owner-scoped batch:

```powershell
python knowledge/tools/check_knowledge_tree.py
Expand Down
24 changes: 13 additions & 11 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
| Git、PR、rebase | [general/git](knowledge/general/git/_index.md) | [Git](knowledge/repos/vllm-omni/git/_index.md) / [rebase](knowledge/repos/vllm-omni/rebase/_index.md) |
| Benchmark | [general/benchmark](knowledge/general/benchmark/_index.md) | [vLLM-Omni benchmark](knowledge/repos/vllm-omni/benchmark/_index.md) |
| 远端验证 | [general/remote](knowledge/general/remote/_index.md) | [vLLM-Omni remote](knowledge/repos/vllm-omni/remote/_index.md) |
| 新增或整理知识 | [knowledge/CONTRIBUTING](knowledge/CONTRIBUTING.md) | 先确定最近的 owner |
| 新增或整理知识 | [knowledge/CONTRIBUTING](doc/knowledge/CONTRIBUTING.md) | 先确定最近的 owner |

vLLM-Omni 的代码 owner:

Expand All @@ -31,7 +31,7 @@ vLLM-Omni 的代码 owner:
| queue、token budget、prefix cache、调度 | [scheduler](knowledge/repos/vllm-omni/components/scheduler/rules.md) |
| 明确的模型或 registry key | [models](knowledge/repos/vllm-omni/models/_index.md) |

完整知识入口仍在 [knowledge/README](knowledge/README.md)。只有 owner 不明确时才看
完整知识入口仍在 [知识树地图](doc/knowledge/tree-map.md)。只有 owner 不明确时才看
[组件职责表](knowledge/repos/vllm-omni/components/_index.md);不要默认打开事故记录、
history、results 或所有 `_index.md`。

Expand All @@ -49,12 +49,12 @@ src/infermatrix_copilot/
playbooks/ Strict/Autonomous 工作流定义
adapters/ 发布包中的仓库配置
knowledge/ 通用、仓库、组件和模型知识
doc/SPEC/ 与 src 文件对应的约束和职责
doc/architecture/SPEC/ 与 src 文件对应的约束和职责
test/ 离线测试
```

整体架构先看 [CODE_TOUR](doc/CODE_TOUR.md),修改具体源码前再看
[file-level SPEC](doc/SPEC/README.md) 中对应文件。
整体架构先看 [CODE_TOUR](doc/architecture/CODE_TOUR.md),修改具体源码前再看
[file-level SPEC](doc/architecture/SPEC/README.md) 中对应文件。

## 常见改动从哪里开始

Expand All @@ -65,7 +65,7 @@ test/ 离线测试
| 模型、Key、Base URL | `config.py`、`llm.py` | `test_llm_providers.py`、`test_tier_split.py` |
| 工作流步骤 | `playbooks/*.yaml`、`engine/steps/` | 对应 step 测试和 playbook 加载测试 |
| 新仓库支持 | `adapters/<repo>/`、`knowledge/repos/<repo>/` | adapter 测试和知识检查 |
| 更新 vLLM-Omni 知识 | `/imupdate` 维护流程 | [release maintenance](doc/VLLM_OMNI_RELEASE_MAINTENANCE.md) |
| 更新 vLLM-Omni 知识 | `/imupdate` 维护流程 | [release maintenance](doc/contributing/release-maintenance.md) |
| 安装或发布包 | `scripts/install_mcp.py`、`pyproject.toml` | installer 测试和 wheel 内容检查 |

## 本地开发
Expand Down Expand Up @@ -117,9 +117,11 @@ required status check —— 这是仓库设置,任何一次提交都做不到

## 进一步阅读

- [设计](doc/DESIGN.md)
- [知识来源和同步边界](doc/KNOWLEDGE.md)
- [扩展知识库](doc/EXTENDING-KNOWLEDGE.md)
- [MCP 安装与接口](doc/MCP.md)
- [Autonomous workflow](docs/autonomous-workflow.md)
- [指南](doc/GUIDE.md) —— 操作者使用 + 维护者开发,外加 playbook / step / tool
清单和性能对比(`CODE_TOUR` 的结构视角对照版)
- [设计](doc/architecture/DESIGN.md)
- [知识来源和同步边界](doc/architecture/KNOWLEDGE.md)
- [扩展知识库](doc/knowledge/writing.md)
- [MCP 安装与接口](doc/guide/mcp.md)
- [Autonomous workflow](doc/guide/autonomous-workflow.md)
- [评测](eval/README.md)
Loading
Loading