From 93d4147770747f55135748166bf94f0fef81d792 Mon Sep 17 00:00:00 2001 From: david6666666 <530634352@qq.com> Date: Thu, 15 Oct 2026 17:31:24 +0000 Subject: [PATCH] Record mixed scheduling diagnostics and codec workspace isolation evidence Signed-off-by: david6666666 <530634352@qq.com> --- .../results/diagnostic-lessons.md | 111 ++++++++++++++++++ .../drama3-b300-scale150/results/status.json | 37 +++++- ...concurrency-scaling-state-and-overlap.json | 52 +++++++- 3 files changed, 190 insertions(+), 10 deletions(-) diff --git a/evidence/drama3-b300-scale150/results/diagnostic-lessons.md b/evidence/drama3-b300-scale150/results/diagnostic-lessons.md index 453b37b4..c1027219 100644 --- a/evidence/drama3-b300-scale150/results/diagnostic-lessons.md +++ b/evidence/drama3-b300-scale150/results/diagnostic-lessons.md @@ -66,3 +66,114 @@ bounds, but throughput still misses the updated baseline and acceptance target. This is one feasibility repetition, not a qualified all-concurrency result. The successful fixed-input fusion and this limited service observation do not promote the rejected asynchronous configuration or resolve its failure cause. + + +## Continuous-admission profile and mixed backbone, 2026-10-16 + +A new diagnostic sends 256 requests at C64 with continuous client replenishment. +The analyzer counts CPU `user_annotation` execute-context scopes, rather than +PIECEWISE GPU graph fragments. It observes 802 scheduler steps: one pure +prefill, 37 mixed, 762 pure decode and two empty steps. Scheduled prompt entries +can include chunks, so their count is not the unique-request count. + +Client timestamps provide an admission/drain proxy. The admission interval has +321 steps (one prefill, 36 mixed and 284 pure decode). Its pure-decode batch +averages 60.718 of 64, with range 52–64; mixed steps average 7.083 scheduled +prompt entries / 1179.33 prompt tokens plus 54.083 decode requests. This interval +includes initial filling; initial large mixed steps should be distinguished +when describing the later refill pattern, without dropping any request from +performance or quality acceptance. Final drain contains 481 steps and its +pure-decode batch averages 8.793. Across the whole trace, the pure-decode average +falls to 28.146. This is direct evidence that whole-run mean batch must not be +substituted for sustained-admission occupancy. + +Within the client-admission interval, correlated GPU annotation unions sum to +999.20 ms for mixed work and 3069.41 ms for pure decode. They are annotations, +not summed kernel duration or SM utilization. CPU wall/monotonic alignment is +an approximation; client in-flight requests include queueing and codec work. +Neither this phase split nor the profiler timings establish a performance gain. +The prior single-wave trace remains retained with its original limitations. + +Local source: `/home/zjy/code/hsliu/artifacts/drama3-mixed-20261016/vllm-continuous-schedule.json`. +Its raw trace SHA-256 is +`29e5cafe329cb85eaba69db7536a28a83fb7e3ac29bc154fc8747f317f6a4b3f`; +request-metadata SHA-256 is +`bd50166976a4753b33c8aeb5a774ebb5711c74ccbba0df1843885f353ecdc996`. + +The native mixed-backbone prototype shares projection/MoE rows while preserving +separate attention/GDN segments and teacher-forced prompt-tower handling. The +initial D58/P6 check passes 730 layer/graph/ownership checks. The expanded +D58/P6, D40/P3 and D12/P1 probe passes 2190 checks, including reused slots in the +latter cases and exact untouched-state checks. These are two overlapping +validation cohorts, not 2920 unique cases. Existing numerical limits remain +unchanged; a passing layer check is not complete generated-audio validation. + +The expanded probe ran on physical GPU0, where external work was observed. +Its numerical results remain recorded; its timing is explicitly unqualified. +The local artifacts are `mixed-layers.json` and `mixed-layers-expanded-v2.json` +in the same artifact directory. The same-source GPU6 complete service comparison +has finished: control 3816.466 and mixed 4094.177 frames/s. All 1088 outputs in +each arm were scored: control WER 1.607636%, SIM 64.48894; mixed WER 1.582517%, +SIM 64.28628. Both meet the single-group quality bounds, but the mixed result +remains below the frozen vLLM C64 baseline and acceptance target. This is one +feasibility repetition on another physical device, not formal acceptance. + +Generated frames differ (102946 versus 101162). Makespan decreases 8.398%, +while frames/s increases 7.277%; shorter final drain contributes about 34% of +the makespan difference. Retain these stochastic workload differences instead +of attributing all observed change to fixed-work kernel acceleration. + +The standalone mixed timeline retains all 384 measured requests. Its 9.998 s +window contains 6.060 s of pure-decode event spans, 1.969 s of mixed backbone +including teacher-forced tower, and 1.454 s of synchronous codec. During ongoing +admission, decode averages 61.071/64 active requests. Event spans and client +phase proxies do not measure SM utilization or establish causal attribution. +The first-frame CPU scope mostly waits for earlier mixed GPU work; it is not +evidence that rewriting the scheduler in another language would remove that cost. + +Six actual-input native FP8 expert GEMMs from three B64 layers show DRAM peak +fractions of 56.36–75.54% and tensor activity of 3.46–4.48%. Measured DRAM reads +are close to one logical active-expert weight footprint. Expert counts come +from a same-logit top-k recomputation with unproven FlashInfer tie equivalence; +Nsight replay, three-layer coverage and a single position limit interpretation. +These samples support inspecting weight traffic, not a many-fold speedup from +unused tensor arithmetic. No expert is removed and no extra quantization is used. + +A separate fixed-state mixed-span probe reduces the static prompt maximum while +keeping token-row capacity unchanged. All 339 output/state checks across three +real cases are exact. This trims diagnostic graph time by about 2.5 ms per case; +the change also affects attention and convolution bounds, so it is not solely +a GDN result. A GPU0 service pair and complete quality checks are required before +selecting it. Frozen plan and validator remain unchanged. + + +A later dynamic replay finally reproduces an asynchronous divergence with fixed +request order. Default capture has a shared ATen cuBLAS workspace and first +changes generated codes at trace entry 207. Dedicated codec capture observes +disjoint workspace ranges and matches serial codes, PCM, completion ordering +and every model/codec state across all 414 trace entries for 80 real requests. +Both configurations also have identical serial trace and PCM hashes. This fixes +the reproduced case; it is not proof that every historical WER failure shares +the cause. Extra diagnostic D2H reads can alter overlap timing. The default-off +candidate is undergoing a fresh full-corpus service/quality comparison. + +For mixed large-M MoE, six counter samples use actual FlashInfer routing and +read only 1.011–1.017 times the unique active-expert weight footprint. The test +preserves original outputs when recording routing. These inputs do not support +a redundant-weight-HBM-read hypothesis, so another blind layout search is not +justified. Three layers and one mixed position remain insufficient for a global +lower bound or a claim that the all-concurrency target is impossible. + + +The fresh GPU0 same-source async service pair is now complete. All 1088 outputs +per arm were scored. Shared capture has WER 6.162606% and SIM 63.124403, failing +the WER gate; private capture has WER 1.590890% and SIM 64.191350, passing the +single-group limits. Private throughput is 4360.917 frames/s, TTFA p50/p95 +143.351/278.698 ms and mean RTF 0.307843. It still misses the frozen C64 baseline +and target. New runtime startup rejects async codec without explicit dedicated +capture, preventing the reproduced unsafe configuration. This is not formal +all-concurrency acceptance or proof about every historical quality failure. + +All new raw groups, diagnostics, source snapshots and limitations are indexed in +[the published mixed investigation](https://github.com/david6666666/drama-3-single-b300-inference/tree/dfdd598a9810cf6e7b8aa6a102f6ab9270ff9cc6/evidence/20261016-mixed). +The runtime PR remains draft and the campaign remains INCONCLUSIVE. diff --git a/evidence/drama3-b300-scale150/results/status.json b/evidence/drama3-b300-scale150/results/status.json index 6038a08e..f632caa3 100644 --- a/evidence/drama3-b300-scale150/results/status.json +++ b/evidence/drama3-b300-scale150/results/status.json @@ -1,10 +1,10 @@ { "verdict": "INCONCLUSIVE", "acceptance_issue": "https://github.com/ThinkFlowLab/OmniRSI/issues/10", - "reason": "No candidate passes the frozen all-concurrency gates. Fresh native-fusion async service controls also fail complete WER; exact replay controls do not resolve the cause.", + "reason": "The dedicated-capture native async candidate passes fixed dynamic replay and complete single-group quality, but C64 throughput still falls below the baseline and target; the required all-concurrency repeated matrix is not complete.", "runtime_pull_request": "https://github.com/david6666666/drama-3-single-b300-inference/pull/1", - "runtime_evidence_commit": "e58c8418005af8972d1616d5368009bc07dbe651", - "runtime_evidence": "https://github.com/david6666666/drama-3-single-b300-inference/tree/e58c8418005af8972d1616d5368009bc07dbe651/evidence/20261015-scale150", + "runtime_evidence_commit": "dfdd598a9810cf6e7b8aa6a102f6ab9270ff9cc6", + "runtime_evidence": "https://github.com/david6666666/drama-3-single-b300-inference/tree/dfdd598a9810cf6e7b8aa6a102f6ab9270ff9cc6/evidence/20261016-mixed", "baseline_quality_groups": 21, "baseline_scored_waveforms": 22848, "packed_layer_graph_ownership_checks": 5076, @@ -16,5 +16,34 @@ "new_lossy_quantization_allowed": false, "asynchronous_codec_configuration_selected": false, "synchronous_native_fusion_feasibility_quality_samples": 1088, - "synchronous_native_fusion_feasibility_quality_passes": true + "synchronous_native_fusion_feasibility_quality_passes": true, + "mixed_backbone_initial_layer_graph_checks": 730, + "mixed_backbone_expanded_layer_graph_checks": 2190, + "mixed_backbone_expanded_layer_graph_checks_pass": true, + "mixed_backbone_expanded_timing_qualified": false, + "mixed_backbone_service_comparison_complete": true, + "mixed_backbone_service_quality_complete": true, + "continuous_admission_profile_requests": 256, + "continuous_admission_profile_is_acceptance_evidence": false, + "mixed_backbone_service_quality_samples_per_arm": 1088, + "mixed_backbone_service_is_formal_acceptance": false, + "mixed_backbone_single_group_quality_passes": true, + "mixed_prompt_span_exact_checks": 339, + "native_moe_counter_launches": 6, + "native_moe_counter_layers": 3, + "mixed_span_gpu0_feasibility_quality_passes": true, + "mixed_span_gpu0_feasibility_quality_samples_per_arm": 1088, + "dynamic_codec_default_capture_diverges": true, + "dynamic_codec_dedicated_capture_exact": true, + "dynamic_codec_dedicated_capture_requests": 80, + "dynamic_codec_dedicated_capture_trace_entries": 414, + "codec_workspace_service_validation_complete": true, + "mixed_moe_counter_launches": 6, + "mixed_moe_actual_routing_recorded": true, + "codec_workspace_shared_service_quality_passes": false, + "codec_workspace_private_service_quality_passes_single_group": true, + "codec_workspace_service_quality_samples_per_arm": 1088, + "latest_c64_gpu0_feasibility_frames_s": 4360.916512808546, + "latest_c64_gpu0_feasibility_wer_percent": 1.5908900611236707, + "latest_c64_gpu0_feasibility_sim": 64.19134978818543 } diff --git a/src/omnirsi/data/repo_packs/vllm_omni/knowledge/cuda/audio/concurrency-scaling-state-and-overlap.json b/src/omnirsi/data/repo_packs/vllm_omni/knowledge/cuda/audio/concurrency-scaling-state-and-overlap.json index daf60fbe..d4cf563b 100644 --- a/src/omnirsi/data/repo_packs/vllm_omni/knowledge/cuda/audio/concurrency-scaling-state-and-overlap.json +++ b/src/omnirsi/data/repo_packs/vllm_omni/knowledge/cuda/audio/concurrency-scaling-state-and-overlap.json @@ -7,11 +7,11 @@ "component": "audio", "scenario": "audio.*", "status": "imported", - "summary": "Diagnose audio service scaling by batch occupancy, mixed prefill and codec overlap; check state ownership, unchanged numerical controls and fixed-request replay, then qualify complete audio quality and service throughput.", + "summary": "Diagnose audio scaling with scheduler-step counts, admission/drain phases, mixed prefill/decode and codec overlap; isolate state, label and RNG ownership before qualifying same-source service throughput and complete audio quality.", "translations": { "zh-CN": { "title": "音频并发扩展：混合预填充、状态归属和 codec 重叠", - "summary": "按实际批大小、混合预填充和 codec 重叠拆解音频并发扩展；检查状态归属、未修改实现的数值对照和固定请求重放，再验收完整音频质量与服务吞吐。" + "summary": "按调度轮而非图碎片计数，分开初始填充、持续接纳与排空阶段，分析混合预填充/解码和 codec 重叠；分别检查状态、标签及随机数归属，再以同源服务对照和完整音频质量验收。" } }, "tags": [ @@ -25,7 +25,15 @@ "state_ownership", "control_run", "layer_parity", - "fixed_request_replay" + "fixed_request_replay", + "scheduler_step", + "piecewise_graph", + "admission_phase", + "mixed_backbone", + "rng_ownership", + "hardware_counters", + "static_sequence_bound", + "output_length_control" ], "sources": [ { @@ -39,6 +47,24 @@ "type": "source", "commit": "d6ed4f4844fbc41573ffc03247e5b2ff0a38d322", "accessed_at": "2026-10-15" + }, + { + "url": "https://github.com/vllm-project/vllm-omni/blob/d6ed4f4844fbc41573ffc03247e5b2ff0a38d322/vllm_omni/model_executor/models/fish_omni/fish_omni_model.py#L1-L28", + "type": "source", + "commit": "d6ed4f4844fbc41573ffc03247e5b2ff0a38d322", + "accessed_at": "2026-10-16" + }, + { + "url": "https://github.com/vllm-project/vllm-omni/blob/d6ed4f4844fbc41573ffc03247e5b2ff0a38d322/vllm_omni/model_executor/models/fish_omni/model_state.py#L264-L379", + "type": "source", + "commit": "d6ed4f4844fbc41573ffc03247e5b2ff0a38d322", + "accessed_at": "2026-10-16" + }, + { + "url": "https://github.com/vllm-project/vllm-omni/blob/d6ed4f4844fbc41573ffc03247e5b2ff0a38d322/vllm_omni/model_executor/models/fish_omni/fish_omni_model.py#L425-L478", + "type": "source", + "commit": "d6ed4f4844fbc41573ffc03247e5b2ff0a38d322", + "accessed_at": "2026-10-16" } ], "applicability": { @@ -55,17 +81,27 @@ }, "recommendations": [ "Measure low and high concurrency independently. Attribute pure decode, prompt work, admission gaps and codec work using actual active batch sizes, not the configured capacity alone.", + "Count scheduler steps from the owning CPU execution annotation or scheduler metadata. A PIECEWISE step can launch many graph fragments; grouping kernel names or graph launches is not a step count. Correlate optional GPU spans explicitly and preserve their scope.", + "Separate initial fill, ongoing request admission and final drain before interpreting batch occupancy. Retain whole-run statistics and phase definitions; client in-flight counts include queueing and codec work and are not direct active-model counts or proof of a steady state.", + "A mixed backbone can share projection and MoE execution across prompt and decode rows while keeping causal attention and recurrent updates segmented. Measure whether joint rows amortize work on the real workload; preserving the expert count alone does not preserve backend dispatch or rounding.", "The source deployment combines chunked prefill, asynchronous CPU scheduling, FULL decode graphs, PIECEWISE mixed graphs, and a stateful codec on overlapping CUDA execution. Test these mechanisms individually when porting the design.", "Preserve decode-first ordering and per-request state indices in a mixed step. Each convolution and recurrent state must advance exactly once; fresh state rows must be reset without affecting other requests.", "Keep acoustic prompt rows in cache identity: the source disables prefix caching because token-ID hashing does not cover their code values.", - "Retain the codec chunk ramp, left context, complete emitted audio and preparation costs while comparing schedules. Do not substitute a kernel timing or text-token rate for playable-audio service throughput." + "Retain the codec chunk ramp, left context, complete emitted audio and preparation costs while comparing schedules. Do not substitute a kernel timing or text-token rate for playable-audio service throughput.", + "Use hardware counters on actual routed inputs to distinguish tensor arithmetic from expert-weight traffic. Report profiler replay policy, measured DRAM bytes and peak fraction, layer/input coverage and routing-count provenance; low tensor activity is not a directly recoverable speedup.", + "Inspect static maximum sequence lengths separately from real packed token counts. Tighter graph bounds can remove inactive recurrent chunks and attention work without changing rows; validate every real and padding sequence fits, preserve a larger-bound fallback, and compare all state and outputs before service.", + "For stochastic service A/B trials, retain generated-frame counts and request-length differences alongside frames/s and drain duration. Capture warmups and scheduling can change random draws, so a faster makespan alone does not prove a fixed-work kernel gain.", + "Separate replay streams and graph memory pools do not by themselves prove BLAS scratch isolation. Inspect the installed runtime capture/workspace ownership; where buffers are keyed by capture stream, test dedicated capture streams and record actual address ranges. Compare identical dynamic admissions, RNG, slot reuse and full mutable state before attributing a failure to workspace sharing." ], "validation_plan": [ "L0: exercise partial batches, variable prompt boundaries, slot reuse, request cancellation and untouched-state checks before any service comparison.", + "For mixed execution, check state slots, teacher-forced labels and sampling/RNG ownership separately. Compare each changed layer on identical inputs, snapshot untouched slots before replay, exercise prompt boundaries and slot reuse, and preserve sampling-row and counter ordering in deterministic controls.", + "Record the model batch, prompt token budget and admission phase for every diagnosed scheduler step. Use a continuously replenished workload when investigating sustained scheduling; distinguish scheduled prompt chunks from unique requests and retain the final drain rather than silently filtering it.", "When batched and single-request outputs differ, preserve the failed comparison and run the unchanged implementation through the same check. Isolate each changed operator on identical layer inputs to separate new errors from accumulated floating-point and routing differences; never silently relax its numeric threshold.", "For stream overlap, hold request ordering, initial model/codec state and sampling counters fixed; compare generated codes, PCM, completion order and reused-slot state. Exact fixed-request replay does not replace quality under dynamic service admission.", "L1/L2: replay heterogeneous real prompts; inspect routing occupancy and graph kernels, and distinguish summed kernel time from elapsed time when streams overlap.", - "L3/L4: score every output in the frozen corpus and repeated concurrency sweep; report acoustic frames/s, first playable audio latency, RTF, errors, WER and speaker similarity. Preserve rejected runs and source/environment provenance." + "L3/L4: score every output in the frozen corpus and repeated concurrency sweep; report acoustic frames/s, first playable audio latency, RTF, errors, WER and speaker similarity. Preserve rejected runs and source/environment provenance.", + "Promote a scheduling or kernel change only after a same-source end-to-end control/candidate comparison and complete repeated quality coverage meet the frozen gates. Operator checks, graph replay and a phase-selected trace cannot substitute for the full service result." ], "limitations": [ "This record describes source-grounded mechanisms, not a reproduced speedup or a portable set of tuned constants.", @@ -73,7 +109,11 @@ "Changing state precision, codec precision or FP8 execution can affect generated samples. Operator parity alone does not establish end-to-end audio quality.", "A faster isolated operation can still regress service throughput. Recheck the unchanged serving configuration and complete audio quality before choosing a fused kernel or a different batch policy.", "CUDA event spans can include host-submission gaps. Do not label their union as SM utilization, or count overlapped codec duration again as an available end-to-end speedup.", - "Qualification of another engine or a different GPU requires its own acceptance issue and complete evidence." + "Qualification of another engine or a different GPU requires its own acceptance issue and complete evidence.", + "A whole-run average batch can be dominated by drain steps; a high batch in an admission subset does not establish an end-to-end speedup. Phase cuts are diagnostic scope, not permission to remove requests from acceptance metrics.", + "A shared-execution prototype may change GEMM shapes, MoE implementation or random-number consumption. Keep those changes explicit, and keep timing from a device with observed external work unqualified even when numerical checks pass.", + "A kernel reading nearly one active-expert weight footprint can remain bandwidth-limited despite low tensor activity; row padding or inactive experts must not be converted directly into a speedup claim.", + "A dedicated capture stream is a workspace-isolation hypothesis, not proof that prior quality failures were races. Require a fixed dynamic schedule and full service quality before attributing a cause or selecting overlap." ], "verification": { "locally_verified": false,