Repository navigation
[RFC] Hardware-aware optimization goals throughout the RSI workflow #143
Description
Goal and acceptance
Before candidate search, InferRSI should establish traceable optimization goals based on available hardware × model × workload × user requirements, then carry those goals through RSI diagnosis, search, validation, stopping, and learning.
Hardware determines feasibility and optimization headroom; application requirements determine what to optimize. A GPU model alone cannot select an application SLO, and every platform should not be required to achieve the same speedup.
Status: Proposed. This RFC proposes workflow and contract extensions. It does not represent an implemented capability, initiate hardware experiments, or change the goals or verdicts of existing acceptance issues.
Background and existing capabilities
Inspected main on 2026-10-25: 65147a03. Recheck main before implementation.
infra/inspection.pyalready combines repository, Python package, device, and visibility information. The CUDA adapter reads device models, drivers, and topology. Discovering an environment does not establish that a model or execution mode works in it.RunSpecalready has metric, direction, min_improvement_pct, workload/model/device, and budget fields.harness/evaluation.pyevaluates the primary metric, quality command, and integrity. Its current single-primary-metric comparison is not a general goal contract covering multiple workloads and constraints.- Existing
methods/performance-ceiling-analysis,methods/optimization-search-policy, andworkflows/rsi-evidencecan be reused. The missing connection is between environment facts, headroom analysis, and goal freezing, with each iteration consuming the same contract.
Related RFC responsibilities: #30 addresses Campaign Plan organization; this proposal adds the derivation and consumption rules for Hardware → Optimization goal. #131 addresses optimization direction selection after bottleneck diagnosis; #137 addresses goal progress and next actions across sessions. This proposal does not assume those proposals are implemented and does not introduce another plan database or agent runtime. It also reuses the performance loop in #15 and task-specific benchmark selection in #60.
1. Derive verifiable goals from the hardware environment
1.1 Environment profile and workload
Extend the existing inspection/platform adapters with structured facts, retaining their source, collection time, and reasons for missing information:
| Layer | Information to record | Effect on goals |
|---|---|---|
| Accelerator | Backend, architecture/SKU, actual allocated count, available memory per device, precision and kernel support, power/clock constraints, sharing or partitioning state | Whether the model and target workload fit; compute, bandwidth, and capacity limits |
| Interconnect | Actual device mapping, PCIe/NVLink/other links, P2P capability, NUMA, and network for multi-node execution | Communication costs of TP/PP/EP, stage placement, and offloading |
| Host | CPU cores and quotas, NUMA, RAM, and relevant storage/network constraints | Whether scheduling, preprocessing, weight loading, or data movement enters the critical path |
| Software | Driver/runtime/compiler/framework versions, backend and actual execution path, relevant compilation and graph state | Record installed, executable, and validated states separately |
| Model and workload | Weights/precision, modality, input/output lengths or audio/video shapes, concurrency/arrival rate, cold/warm caches, timing boundary | Different workloads on the same hardware can have different bottlenecks and goals |
Label rated specifications, live probes, component measurements, and inferences separately. Missing values remain unknown; do not substitute zero or values from similar devices. Use logical placeholders for device identifiers in shared evidence; do not publish private mappings. Separate static identity from dynamic state recorded each iteration so normal fluctuations do not all become hardware identity changes.
1.2 Calibrate before freezing
- Clarify user requirements. Select the primary objective: latency, SLO-compliant goodput, capacity, or cost/energy supported by measurement. If preferences are unspecified, produce a reasoned goal recommendation; do not treat that recommendation as an accepted application threshold.
- Check qualification and capacity. Validate the model, precision, execution path, and target workload. Account for weights, KV/state, activations/workspaces, graph buffers, and headroom per device under the actual placement. Do not simply sum memory across GPUs.
- Measure a reference baseline. Fix the scenario, run repeated baselines without a profiler, and collect separate diagnostics. Distinguish warmup/compilation, steady state, queueing, compute, data movement, and communication. This is calibration before search, not evidence of a candidate gain.
- Estimate optimization headroom. Reuse ceiling analysis to distinguish analytical bounds, measured component capability, and end-to-end critical-path analysis. State assumptions, applicable shapes, evidence, and uncertainty. Neither a theoretical peak nor a single empirical best result establishes achievable service performance. Ceiling-analysis levels are also distinct from the repository's L0–L4 validation levels.
- Create and freeze the goal contract. Relate user SLOs to the baseline, noise, and estimated headroom. Specify numeric thresholds, quality/regression constraints, budgets, and stopping conditions. Once established through the existing acceptance process, freeze them before the first candidate search.
For example, if a serial stage accounts for fraction f of baseline latency and its expected stage speedup is s, use 1 / ((1-f) + f/s) for Amdahl screening. With overlap, queueing, or shifting throughput bottlenecks, use the appropriate critical-path/capacity analysis; do not directly apply the serial formula to claim a service gain.
If a target clearly exceeds what current evidence supports, report the gap, limiting factors, and next validation step, along with options to change resources, workload, or goals. Do not automatically lower accepted thresholds or equate “little remaining improvement in this direction” with “the system is globally optimal.”
1.3 Minimum goal contract
The following proposed field semantics belong in the existing campaign plan. They do not introduce a YAML runtime configuration entry point or a second editable source of truth.
| Field | Contract contents |
|---|---|
| Identity | Acceptance issue, goal version, plan/validator digests, hardware-profile reference, model and workload identities |
| Primary objective | Metric, unit, minimize/maximize, timing boundary, statistic, workload rows; declare absolute targets and relative improvement requirements separately |
| Constraints | Hard constraints on quality, success rate/complete outputs, tail latency, memory, device count, precision policy, and their validators |
| Rationale | Source of user requirements, locally measured baseline, noise, bottleneck evidence, ceiling assumptions, and confidence bounds |
| Measurement | Workload/cohort, warmup, repetitions and pairing, statistical decision method, required validation levels, failure/interruption and retry rules |
| Search and stop | Candidate budget, time/device-hour budgets, acceptance reserve, and handling of target attainment, lack of meaningful progress, budget exhaustion, or insufficient evidence |
Numeric thresholds belong only in the specific acceptance issue and frozen evidence. Do not introduce universal percentage, utilization, or memory-headroom defaults across devices. For multiple objectives, default to satisfying hard constraints before comparing the primary objective. If Pareto selection is needed, freeze its selection rule rather than changing weights after seeing results.
Report latency reduction, throughput improvement, and speedup separately. Increasing the device count changes resource conditions and requires separate reporting; it is not a software speedup at equal resources. Cost/energy-efficiency objectives require price/power evidence and a defined measurement scope. Missing inputs cannot be extrapolated into verified metrics.
2. Update the RSI workflow
Add goal preparation before the existing seven-phase search, preserving phase order and the SIAS constraint of modifying one plane per step.
flowchart TD
U["User goals and resource constraints"] --> H["Hardware and workload profile"]
H --> B["Qualification, baseline calibration, and headroom analysis"]
B --> G["Establish and freeze goals, measurement contract, and validators"]
G --> R["Phase 1: Read goals, environment, and prior evidence"]
R --> D["Phase 2: Design a candidate tied to the goals"]
D --> E["Phase 3: Modify one plane"]
E --> Q["Phase 4: Check correctness and quality"]
Q --> P["Phase 5: Measure under the frozen workload"]
P --> V["Phase 6: Evaluate goals and all hard constraints"]
V --> K["Phase 7: Keep or discard; record evidence and remaining gaps"]
K --> S{"Choose the next step using frozen rules"}
S -->|Continue search| R
S -->|Target met or another stop condition, including budget limits| F["Report results, unmet criteria, and reusable lessons"]
R -->|Environment or contract no longer comparable| X["Pause comparison; requalify or create a new contract"]
| Integration point | Added requirement |
|---|---|
| Phase 1 | Read the goal version, current comparable baseline, environment consistency, and remaining budget; do not pool results across device/stack/workload drift |
| Phase 2 | Link each candidate to an explicit target metric, workload row, bottleneck evidence, hardware prerequisites, expected gain range, and kill criterion; filter unavailable methods using hardware conditions |
| Phases 3–5 | Preserve the frozen contract; use component experiments for screening, and complete the required L3 quality and L4 service validation for promotion |
| Phase 6 | Report measurements, target gaps, constraint outcomes, and evidence for each workload row; a gain at one concurrency must not hide a failure at another required concurrency |
| Phase 7 | Update the incumbent and search state, retaining failures and unknown outcomes; record cases with hardware/model/workload applicability |
| Stop/continue | Record the stopping reason and unmet goals for the checkpoint in #137; budget exhaustion, a local ceiling, or stopping search does not establish goal completion |
After freezing, bottleneck assessments, candidate rankings, and headroom estimates may change, but changes must be versioned and cite new evidence. Do not rewrite target thresholds, the baseline, or validators to make existing results pass. Material changes to hardware resources, workloads, or goals require a new comparable contract and, where necessary, a new campaign, retaining the original results. Align with the final freeze semantics of #30.
Keep PASS / FAIL / INCONCLUSIVE: PASS requires complete, comparable evidence satisfying every frozen gate; complete evidence that violates a gate is FAIL; missing critical evidence, incomplete execution, or incomparable environments produce INCONCLUSIVE while retaining observed failures. Passing one RunSpec primary-metric comparison does not automatically establish campaign-wide goal attainment.
3. Example: different goals within a two-GPU speech pipeline
#112 already defines Qwen3-Omni on two L20X GPUs, full streaming speech, and a concurrency matrix. It can serve as a future contract-migration example. This RFC does not change its existing numbers, protocol, or INCONCLUSIVE status.
- At low concurrency, focus on request-to-first-audio, RTF, and the complete request critical path. First measure Thinker/Talker/Code2Wav stage shares and host/launch overhead.
- At high concurrency, focus on goodput under latency/quality constraints, checking queueing, state memory, stage imbalance, and inter-device transfers. Two GPUs alone do not determine TP or stage placement.
- If the model/state does not fit, first establish capacity feasibility. Assess the latency, quality, and resource costs of offloading, precision changes, or additional devices separately, respecting the campaign's frozen precision and device constraints.
- Recalibrate when moving the same workload to devices with different architectures, memory, or interconnects. Reuse methods and hypotheses; do not inherit old-device thresholds as evidence of performance on new hardware.
4. Incremental implementation
- Workflow and templates: Extend goal-preparation guidance in
evidence/_template/plan/, the campaign acceptance form, and quickstart/operations documentation. Add contract-reading/writing guidance within the existing search-policy, ceiling-analysis, and rsi-evidence owners; keep skill content in English. Coordinate a single plan structure with [RFC] Iteration Campaign Plan: one table for model, hardware, goal, search plan and test plan #30. - Minimal runtime support: Extend existing
infra/inspection.py/ platform adapters with capability and resource facts. Bind goal and environment identities in RunSpec/the existing plan. Have the harness evaluator/report consume frozen workloads and constraints. Preserve compatibility and semantics of the existing single-metric interface; do not automatically promote historical results to full goal acceptance. Settle the concrete schema/CLI in implementation PRs. - End-to-end example: Use offline synthetic fixtures to validate goal propagation and failure paths, then validate hardware collection, baselines, and the complete service pipeline in a separately authorized acceptance campaign. Passing software tests establishes workflow implementation, not performance gains.
Do not add a runner, database, or CI lane, or include hardware experiments in default merge CI. Even if a hardware specification table exists, it remains a versioned reference input with provenance; it cannot replace probing and measuring the target devices.
5. Acceptance checklist
- Goals are traceable to hardware, model/workload, user requirements, baseline, and derivation evidence; each iteration references the same frozen goal identity.
- Offline fixtures for the same model under at least two memory/topology conditions explain differences in feasibility or candidate priorities without hard-coding application goals by GPU name.
- Unknown capabilities, unavailable execution paths, insufficient capacity, or missing baselines produce specific gaps and next steps, not fabricated achievable values or PASS.
- Fixtures covering multiple workloads/constraints include improved throughput with failed tail latency, quality failure, a missing workload row, environment drift, and modification of a frozen goal.
- Missing/interrupted evidence and complete-but-failing evidence produce INCONCLUSIVE and FAIL respectively; goal attainment, candidate promotion, search termination, and issue closure are reported separately.
- Target attainment, lack of meaningful progress, budget exhaustion, and insufficient estimated headroom each have a traceable next action. Revising goals preserves previous contracts and results without retroactively relaxing gates.
- Existing RunSpec/CLI behavior is preserved, with migration examples and focused tests; generic templates and CI introduce no new numeric hardware thresholds.
- Any actual service-gain claim includes separate quality and L4 evidence under the campaign's frozen protocol; offline fixtures and microbenchmarks do not substitute for service acceptance.
Ownership and next decision
The user authorized submission of this RFC; the current action is proposal publication only. The implementation owner, goal-acceptance owner, and closure authority will be specified in subsequently accepted scope and remain TBD.
First agree on the workflow/template slice and field ownership with #30, then implement minimal runtime support. #131 consumes goal and bottleneck context; #137 presents remaining gaps and next actions. Publishing this RFC does not mean the workflow has been implemented.
Activity
[-][RFC] Hardware-aware optimization goals: 根据硬件环境设立目标并贯穿 RSI 流程[/-][+][RFC] Hardware-aware optimization goals throughout the RSI workflow[/+]Dong1017 commentedon Oct 26, 2026
Discussion: connect hardware facts, profiling evidence and ceiling analysis
补充这次讨论的 insights 和 proposed changes，供 #143 评估。以下是建议拆分，不代表已接受的实现计划、已交付能力或新的实验授权；本次不创建子 issue。
Insights
从 roadmap 后的 #131、#137、#143、#152 看，已有知识、执行与留证部件之间，还缺少一条一致的路径：环境/工作负载事实 → 条件性 headroom → 冻结目标 → 每轮比较 → 可解释的下一步。优先明确各工件的事实所有者和引用关系，避免继续增加互不连接的入口。
可复用的 profiling 工作
已只读检查 Dong1017/vllm-omni 的 profiling/evidence-framework 分支，固定 SHA
49904e8b7baa8e254e3d8e9454feb5566fc7eda3：profiler/负责运行期采集，profiling/负责既有 artifact 的离线分析，边界适合复用。本次是源码/契约检查，未执行该分支测试或任何 profiler/硬件实验。
复用优先：先从 InferRSI 现有能力提取，而不是新起一套
建议实现时先盘点并 refactor 现有 owner，外部项目用于补充算法、接口与适配经验：
其他可借鉴的开源实现（source-inspected candidates）
按“借鉴哪一层”筛选，而不是把整个列表变成依赖。以下检查固定源码/文档，未安装、运行或完成性能验证；这里只推荐设计参考或可选适配。LLM-Viewer 与 ERT 也已进入现有 ceiling skill 的参考范围，应复用已有来源记录。
1893e4broofline_analyze(bandwidth, max_OPS, OPs, memory_access)的纯计算边界；另有模型 prefill/decode 的分层 work/traffic 估算。适合作为提取现有 skill 确定性计算与测试的参考。0771fa98d16d93c023a08ab8be46ROCm/rocprofiler-compute已归档，当前源码在ROCm/rocm-systems/projects/rocprofiler-compute。AMD 计数器/架构假设不能直接搬给 CUDA/Ascend；采集/benchmark 与离线计算仍需分开。d62cc7d建议阅读/复用优先级： 内部 skill/inspection/contracts → LLM-Viewer 的小计算边界 → 自有 profiling evidence + Perfetto/HTA 的证据语义 → 按实际平台需要参考 ROCm/Ascend；ERT 作为后续实测上限输入的参考。不要把六个项目全部安装成第一版依赖。
直接复制/分发源码前，需核对固定版本的 LICENSE/NOTICE 及逐文件声明、保留来源；表中“可借鉴”不代表许可兼容审查或集成验证已经完成。
采集层另可参考 Kineto，但该独立仓库已归档并说明迁入 PyTorch；不建议把它作为新的独立硬依赖。默认轻量 API 不承担采集职责。
Proposed changes
建议在上述现有能力复用/refactor 基础上讨论三个边界，核心命名采用 Performance Ceiling Analysis API，避免用含义过宽的“Profiling API”同时承载采集、解析、建模和验收。
图中绿色为已有部件/输出，橙色为本次建议边界；虚线表示可选输入或待讨论集成。所有箭头表示数据/契约依赖，不表示执行授权或已实现的自动控制流。
flowchart TD I["Existing inspection + explicit hardware/spec facts"] --> H["Proposed: HardwareProfile"] W["Model/workload + work/traffic estimates"] --> C["Proposed extraction/refactor: Ceiling Analysis API"] H --> C A["Existing profiler artifacts"] --> E["Existing offline profiling framework: OptimizationEvidence JSON"] E -. "optional measured evidence" .-> P["Proposed: Profiling Evidence Import"] P -. "validated, compatible measurements only" .-> C S["Existing ceiling-analysis skill: modeling and interpretation"] -. "guidance" .-> C H --> G["RFC #143: goal preparation and gap assessment"] C --> G U["User requirements + qualified workload + independent validator + baseline"] --> G G -. "review and freeze through existing process" .-> F["Existing Campaign Plan artifacts: frozen contract + versioned search state"] F -. "proposed contract binding" .-> B["Existing Workbench comparison and retained evidence"] B -. "planned integration" .-> K["#137: checkpoint / next-action proposal"] X["#152: shared user journey / thin host adapters"] -. "same operation semantics" .-> G X -. "same operation semantics" .-> B classDef existing fill:#e8f5e9,stroke:#388e3c,color:#111; classDef proposed fill:#fff3e0,stroke:#ef6c00,color:#111; class I,A,E,S,F,B existing; class H,C,P,G,K,X proposed;HardwareProfile 的最小契约是计算 API 的前置；补充探测与计算实现可并行。Profiling Evidence Import 是可选入口，没有 profiling 产物也能进行具备充分输入的 L1 分析。#143 继续拥有目标准备/冻结/消费规则；不另建目标数据库，不复制 #137 或 #152 的责任。
最小验证与讨论点
若当前只有一位实现者，HardwareProfile 与计算 API 可作为同一任务的两阶段，Evidence Import 独立；不按字段或设备 SKU 继续碎片化拆分。
Dong1017 commentedon Oct 27, 2026
Partial task claim: Ceiling / goal-preparation preflight
Claiming a read-only design and scope preflight, coordinated by dot with local Codex task InferRSI-issue#143 (gpt-6.1-sol / xhigh), under the user's request to parallelize non-distillation work.
Deliverables:
This claim does not accept or implement the entire RFC, create a service/hardware registry, launch profilers or model/GPU experiments, or promise achievable acceleration. Existing discussion boundaries remain in force. No other agent needs to duplicate this preflight; implementation ownership and scope will be recorded after its findings are reviewed.
Dong1017 commentedon Oct 27, 2026
Preflight complete: proposed first Ceiling helper slice
Following the partial task claim, the read-only preflight is complete against main
97f77a91373f3e2b61b03afcfd227e21edbffa2f. The existing ceiling skill contains the formulas and uncertainty discipline, but no corresponding Python calculator was found.Recommended bounded implementation, subject to scope/owner agreement:
scripts/, a short input contract, focused CPU tests and a skill reference. Initially support one resource scope and one homogeneous floating-point operation class; callers supply work/traffic counts.F/T, conditional roofmin(P, (F/T)*B), ridgeP/Band conditional runtime floormax(F/P, T/B), plus assumptions, unknowns and evidence references. Missing, unsourced or incompatible facts yieldINSUFFICIENT_EVIDENCEfor dependent outputs; malformed numeric inputs fail explicitly. These results do not establish attainable speedup or an application verdict.conditional_headroom/evidence. A component roof alone cannot establish exposed end-to-end headroom. Keep frozen goals, RunSpec, evaluator gates and historical evidence unchanged in this slice.No implementation, tests, model runs, hardware experiments or profiler launches were performed. This completes only the preflight, not the RFC or its acceptance. Implementation ownership remains to be recorded. HardwareProfile probing, profiling imports and runtime goal propagation are separate follow-ups; no new service, hardware registry or runner is proposed. Coordinate later shared skill/index changes with the active #133/#153 work.
Dong1017 commentedon Oct 27, 2026
Partial implementation claim: deterministic Ceiling helper
Following the completed preflight and the user's instruction to proceed, dot is coordinating the existing InferRSI-issue#143 local Codex task (gpt-6.1-sol / xhigh) for this bounded software slice.
Delivery: a standard-library pure Roofline helper under the existing performance-ceiling-analysis owner, a documented input/output contract and skill reference, focused offline tests, and a worked caller-controlled attachment example for existing campaign-plan row4. Preserve workload/counting boundary, traffic tier, dtype/op class, hardware source facts, assumptions/unknowns and rated-versus-empirical distinctions. No invented specifications or automatic promotion of a conditional estimate to an E2E gain.
Acceptance for this slice: deterministic hand-calculated compute/memory/balanced cases; compatibility and missing-source/unknown handling; invalid/nonfinite numeric rejection; provenance retention; no network/subprocess/hardware probing/file-write side effects in the pure helper; installed-package accessibility and unchanged existing contracts. Shared generated outputs must be regenerated from their canonical owners without overwriting others' contributions.
Not included: HardwareProfile probing, raw profiler import, multi-device aggregation, model work estimation, service/API platform, RunSpec/evaluator/goal authority changes, real model/GPU experiments, or whole-#143 closure. Use ordinary references to this issue, not automatic closing keywords. Independent review and applicable CI precede any merge. Please avoid duplicating this helper implementation; broader RFC follow-ups remain open.
Dong1017 commentedon Oct 27, 2026
Draft delivery: deterministic Ceiling helper
Following the partial implementation claim, draft PR #167 is available for independent review.
ac258e0a1e5b30cb214ea2ac57b047b87d8321c5; head:7af017d8d64fce635e9eeb15a38c31703e1544b7. The material source is03b2ffb4a049c0c067a721ba43a5fee38121287c.Dong1017 commentedon Oct 27, 2026
Integration follow-up claim: PR #167
Continuing the existing #143 task for bounded integration of PR #167, currently at
d71a9a90dc9b548b21c88e524d96b89242f3e563, against current main6b13cf00ccf2e98fcb3144eda69e2442bcb1d652.Scope: inspect the actual main/branch delta and source-binding requirements; preserve the reviewed Ceiling semantics and other contributors' content; make only warranted integration changes and regenerate canonical projections if required; verify focused helper/link/index/package behavior and refresh exact-head evidence/declaration. If no branch change is warranted, report that with current CI rather than creating a no-op trigger commit.
No new features, model/hardware/profiler runs, paid dispatch, permission changes or whole-RFC closure. Parent retains independent delta review and merge responsibility.
feat: add a pure conditional Roofline helper (#167)
Dong1017 commentedon Oct 27, 2026
The bounded pure Roofline-helper slice is now merged via #167 at
f5081da3fab06db8243aea9720ea2bc4fe1b3942(inspected headd71a9a90dc9b548b21c88e524d96b89242f3e563). Final integration review, four passing Ready jobs, and successful latest policy are recorded in the squash commit. Intervening Skill Eval-only main changes were statically checked; no synthetic refresh commit was needed.Delivered: deterministic conditional arithmetic over caller-supplied work/traffic and sourced hardware rates, explicit assumptions/unknowns, packaged helper/example and row-4 attachment guidance. No automatic profiler, hardware discovery, model execution or achievable-speedup claim was introduced.
This closes the implementation/integration work for this software slice only. #143 stays open: broader goal preparation/runtime propagation, hardware/profiling integration and empirical acceptance remain outside #167. Candidate/not_run is preserved; green software CI is not hardware/model efficacy.