[psv666]

Purpose and summary

Reduce the time Open-Jev-27B-v1.1 users wait for a complete decision by removing repeated candidate work while retaining the trained head, calibration, candidate order and question-level normalization. The frozen cumulative candidate reduced mean full-request latency from 216.31 to 123.02 ms (43.13%) on the historical single-request workload.

Accepted tradeoff: for the six Noul (yes/no) requests with 513–2048 tokens, the 95th-percentile latency increased from 116.32 to 119.77 ms (+2.97%). The contributor explicitly accepted this regression for submission on 2026-10-26. It remains recorded as a regression. Four other historical tail checks remain statistically inconclusive; this PR does not relabel them as passes or claim the original all-group performance gate passed.

complete question
  → bounded tokenization, preserving candidate order
  → compatible candidate groups / within-question shared prefix
  → shared native executor + CUDA operations with independent candidate state
  → original scalar head and whole-question normalization
  → existing typed response

Changes include compatible packed projections and batched Gated DeltaNet (the model's recurrent linear-attention computation), compact prefix continuations, bounded projection-plan caches, vectorized convolution, wider independent work assignment, bounded graph/scratch reuse, and skipping unused rows in the final feed-forward layer when its original matrix-multiplication algorithms support that shape. Unsupported optional paths fall back to existing computation. The 27B worker requires the additive packed-projection/batched-state symbols from this checkout; rebuild its worker and CUDA library together. The existing binary interface remains version 9.

Required human contributor answers

The following scope was explicitly confirmed in the 2026-10-22 planning exchange (Q1–Q18), recorded in PLAN.md with SHA-256 300625b0c6a2c44bf4b20653bc7e0be953e38c81292d02c621e9e68689a93967. The 2026-10-26 submission instruction accepts the specific +2.97% tail regression and requests this PR. These are references to the contributor's private task record; no private transcript is attached.

1. Outcome: lower steady-service Open-Jev-27B-v1.1 decision waiting time without changing decision semantics.
2. Highest-leverage variable: complete-request latency under one-at-a-time requests, including frontend and response handling, with workload-group tail latency checked separately.
3. How this moves the variable: replace measured redundant candidate computation and repeated host/device work; retain changes supported by measured benefit and numerical checks rather than relying on kernel timings alone.
4. Customer and need: System1-Omni Open-Jev deployers who need faster decisions with unchanged probabilities/decisions within the agreed bounds and preserved input capacity; no specific production service-level target was supplied.

Is a PR needed? Yes. The accepted cumulative result requires maintained Rust/CUDA changes, and the contributor explicitly requested submission on 2026-10-26. This is a performance change, so bug-fix failure reproduction is not applicable; the measured baseline is below.

Review scope and map

Integrated base: 05075d0f36e399e3a001ffedea400c666990cda9. Integrated head: efd02a896886c4693f04576faf382fcc2d9ebc0a.

The full diff contains 3,171 changed authored-code lines and 3,363 changed lines overall, across 32 files. The cumulative change is submitted together because the executable, device operations and optional interfaces are coupled, and the quoted overall gain belongs to the combined frozen candidate. No partial component is assigned that cumulative gain. This is larger than a quick precheck: the full source/callers and error/resource paths were inspected, and component plus integration checks are listed below. A split would need validation at each resulting head.

Review area	Main paths	Invariants and validation
Request preparation	src/models/open_jev/native/src/{contract,processing,main}.rs	Tokenize at most 16 candidates per chunk; retain padding, first-error order, question identity and normalization; CPU and pinned-tokenizer tests.
Candidate adapter	src/models/open_jev/native/src/{batching,executor}.rs	Up to 16 packed sequences / 4096 packed tokens; longer prompts retain independent execution. Prefix sharing is within one question and requires compatible 2–16 long candidates; restore original output order. Full-model tests in both graph modes.
Shared executor and storage	src/models/qwen3_5/native/src/model.rs	Per-candidate state, fresh prefix contents, compact suffix bounds, scratch/graph address lifetime; retain upstream fixed-algorithm, shared-prefix, bidirectional, graph-statistics and synchronization behavior.
Projection plans and interface	src/backends/cuda/qwen3_5/{gemm.cu,ops.h}, src/models/qwen3_5/native/src/cuda.rs	Check original algorithm compatibility and workspace; separate 512-entry caches. Replay after descriptor eviction and changed-input tests. Existing entry points retained.
Device operations	src/backends/cuda/qwen3_5/{attention,elementwise,gdn_prefill}.cu	Independent sequence attention/recurrent state, unchanged accumulator/rounding boundaries; numerical references, dirty-workspace and changed-input replay tests. Added explicit future-key coverage for bidirectional attention.
Last layer	model.rs, tests/qwen3_5/last_mlp.rs	Final-row-only feed-forward/norm computation is gated to eligible single-sequence ordinary text units; both original algorithms must support one row. Prefix/suffix, bidirectional and multimodal paths retain full computation.
Maintained documentation	architecture, model/backend READMEs, native recipe	Describes mechanisms, graph limits, memory, historical scope and accepted regression; strict documentation build.

No generated experiment dumps, binary archives or one-off reports are added to the source tree.

Test plan and results

The following checks use integrated head efd02a896886c4693f04576faf382fcc2d9ebc0a; historical timing below uses a different frozen source and is not a fresh performance claim for this head.

Check	Result
cargo fmt --all --check	Pass
cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings	Pass
cargo test --workspace --locked --features omni-laya/serve	368 passed, 0 failed, 82 ignored; ignored tests are not counted as passes
cargo build --workspace --release --locked --features omni-laya/serve	Pass
CUDA shared-library build, native compute capabilities 8.9 and 9.0 plus 9.0 intermediate code	Pass with CUDA 13.0.88
python3 src/backends/cuda/check_contract.py --repo-root .	Zero errors; existing unrelated Laya ABI-macro warning remains
python3 -m unittest discover -s tests/cuda -p 'test_*.py' -v	26 passed
python3 -m unittest discover -s src/backends/cuda/tests -t . -v	68 passed
mkdocs build --strict	Pass
Pinned 27B tokenizer tests	2 passed
CUDA kernel tests	25 passed; one checkpoint-dependent prefix rejection test explicitly excluded
Full-model packed candidates / prefix-buffer reuse	2 passed with graph replay disabled, 2 passed enabled
Final-layer full-hidden-vector comparisons	124 comparisons in each graph mode; both test executions passed
Current upstream fixed/shared-prefix full-model compatibility	2 passed

Current-head HTTP checks completed on 2026-10-26: 508 responses (231 real requests twice, 36 synthetic primary cases and 10 capacity/structure cases), with zero probability/ordinal-score drift, zero decision flips and zero failures against the frozen original responses. All parsed responses were identical after excluding only metadata.inference_seconds and metadata.prefix_cache; the latter changes only where within-question sharing is used. Graph capture failures: zero. The synthetic cases include 16,384-token capacity, 17/33 candidates, 2/10 score levels and mixed question kinds; they are numerical/structural coverage, not new model-quality evidence.

This used the current frontend and worker, graph replay enabled, and one reserved device with measured CUDA capability 9.0, 132 streaming multiprocessors and 150,121,283,584 visible bytes; driver label NVIDIA L20X, UUID GPU-76ec3aee-7fec-b98c-562a-7697a4be04c7, driver 570.133.20 / CUDA 13.0.88. It is a different physical device from the historical campaign. The single measured traversal validates integration and does not establish a new balanced performance comparison. No extra performance acceptance is inferred from it.

Current worker SHA-256: 5ec48fb7d452f131475e5e4669fc31e6b32f4ccd62bd0ed24f9e291a6a9aa96f. Current CUDA library SHA-256: e256ede7524aff3b79d277df428f8b546fd469d637caaf720fe032fa307bd8e2. Current frontend SHA-256: 5c4d458309ca9ed84983a1857c86afaedbf6d37a7124fe2ac059a31ab5b1d265.

Current-head raw validation is retained separately in efd02a8-integration-evidence.tar.gz (13,949,587 bytes, 79 verified archive members), SHA-256 5a5d4f3d53a6636152c5a0608aacb2b005214101c88caa37f2f91c24f94eeeeb. It is host-local; the same public-download limitation described below applies.

Reproduce the device and full-model checks

Prepare the pinned merged checkpoint using recipe/open_jev/native.md, adding --max-length 16384 during export. Build the CUDA library and release tests. All device commands must run inside the host's GPU reservation. Set OPEN_JEV_MODEL to the merged checkpoint, QWEN3_5_MODEL to the same directory, and both OPEN_JEV_CUDA_LIB and CUA_S1_CUDA_LIB to the newly built library.

cargo test --release --locked -p omni-qwen3-5-native -p omni-open-jev-native --no-run
cargo test --release --locked -p omni-open-jev-native --lib -- --ignored --test-threads=1 tokenization_tests::
cargo test --release --locked -p omni-qwen3-5-native --test kernels -- --ignored --test-threads=1 --skip prefix_state_rejects_uncaptured_foreign_and_oversized_prompts
for mode in 0 1; do
  CUA_S1_GRAPH=$mode cargo test --release --locked -p omni-open-jev-native --test prefill_batch -- --ignored --test-threads=1
  CUA_S1_GRAPH=$mode cargo test --release --locked -p omni-qwen3-5-native --lib -- --ignored --test-threads=1 model::last_mlp_tests::last_mlp_matches_complete_forward_at_boundaries_and_replay --exact
done
CUA_S1_GRAPH=0 cargo test --release --locked -p omni-qwen3-5-native --test prefix -- --ignored --test-threads=1

The validation used these release test executables after compilation, with one reserved GPU and serial tests. The shared-prefix rejection test needs a different complete checkpoint and was excluded explicitly. Other complete model checkpoints, multi-request concurrency, other device architectures, and compute-sanitizer remain unverified. The previous candidate's old-library fallback experiment is historical only and was not rerun against the current interface.

Demo / evidence — historical full-request comparison

Before: original source d577bb824a5dbf0f5e385cf829bb6b9025dabf8d, eager execution.
After: frozen cumulative P19 patch on that base, SHA-256 5bb08cafffcf1825c1467de36705ae78baf07e0f7b24662c67ed7237b6504799, graph replay enabled. P18 is the parent experimental candidate. This comparison includes the declared graph-mode treatment; it is not an isolated last-layer gain or a comparison against today's main branch.

Full-request metric	Original	P19	Change
Mean	216.31 ms	123.02 ms	-43.13%
Median	61.04 ms	44.67 ms	-26.82%
95th percentile	1081.56 ms	471.11 ms	-56.44%
Noul 513–2048 tokens, 95th percentile	116.32 ms	119.77 ms	+2.97%, accepted tradeoff
Sampled whole-invocation peak device memory	50,949 MiB	52,035 MiB	+1,086 MiB, approximately 1.06 GiB

231 unique public JevBench requests: 139 Choice, 74 Noul, 18 Score, comprising 775 candidate inputs. Maximum candidate length in the primary workload is 4096 tokens. The persistent client sends the next request only after the previous one finishes. Timing includes request serialization, Rust frontend forwarding, model work, full response receipt, parsing and validation; evidence writes are outside the timer. Loading is outside this request timer. There is no cross-request reuse of prefix contents.

The fixed protocol used all six three-arm orders twice: ABC, BCA, CAB, CBA, ACB, BAC, repeated. A is original, B is P18, C is P19. Each of 36 service starts ran one separate feasibility traversal and eight measured traversals, alternating original order and shuffle seed 20261022. This gives 74,844 responses: 66,528 measured and 8,316 feasibility, or 22,176 measured per arm. No observations were excluded. First traversals mix new-shape captures and existing-shape hits and are retained separately, not called pure cold-start latency.

All 74,844 historical responses had zero observed probability/ordinal-score error, decision flips or request failures against the frozen original responses; the predefined absolute-error tolerance was 0.001. Parsed responses were equal apart from timing and prefix diagnostics. Graph capture failures: zero. These are empirical checks on the stated inputs, not a universal bitwise-equivalence guarantee.

The six-request regressing group has 576 measured responses per arm. Its mean decreased from 75.96 to 75.22 ms, while its 95th percentile increased in all 12 paired blocks. The approximate 95% paired-block interval for the tail change is +2.32% to +3.70%. The original +2% all-group guard therefore failed; the contributor's later acceptance does not rewrite that result. Four additional 95th-percentile intervals remain inconclusive:

Comparison	Group	Approximate 95% interval for change
P19 / original	Score, 129–512 tokens	-12.95% to +8.02%
P19 / P18	Noul, 1–128 tokens	-8.20% to +3.23%
P19 / P18	Score, 1–128 tokens	-7.23% to +4.57%
P19 / P18	Score, 129–512 tokens	-7.60% to +17.54%

The conditional final synthetic timing stage did not run after the real-workload guard failed. Earlier synthetic qualification and current integration checks do not replace that missing timing stage. P19's overall mean was only 0.095% below P18, with an interval spanning zero; the 43.13% cumulative gain must not be attributed to the final increment.

Historical workload-group results and paired-block observations

Groups use the longest candidate token count in a request. Each unique request was measured 96 times per arm.

Task	Tokens	Unique requests	Mean, original → P19 (ms)	95th percentile, original → P19 (ms)	Tail change
Choice	1–128	72	60.51 → 44.07	76.22 → 56.72	-25.58%
Choice	129–512	23	116.38 → 109.27	147.85 → 144.58	-2.22%
Choice	513–2048	17	321.94 → 310.59	579.56 → 561.45	-3.12%
Choice	2049–4096	27	1081.71 → 436.68	1772.95 → 771.10	-56.51%
Noul	1–128	36	21.16 → 19.52	22.39 → 20.43	-8.72%
Noul	129–512	26	33.89 → 32.68	47.66 → 45.55	-4.43%
Noul	513–2048	6	75.96 → 75.22	116.32 → 119.77	+2.97%
Noul	2049–4096	6	253.70 → 245.36	326.69 → 320.41	-1.92%
Score	1–128	12	52.42 → 40.42	56.57 → 43.56	-22.99%
Score	129–512	2	106.93 → 99.65	126.98 → 115.75	-8.84%
Score	513–2048	1	219.74 → 208.15	224.16 → 213.58	-4.72%
Score	2049–4096	3	1190.83 → 430.64	1204.98 → 638.89	-46.98%

Paired-block relative changes (P19 / original minus one; negative is faster). Intervals use the 12 block ratios, Student t with 11 degrees of freedom; they do not treat every repeated request as an independent dataset sample.

Block	Overall mean change	Overall tail change	Noul 513–2048 tail change
1	-43.2193%	-56.4603%	+1.5449%
2	-43.1785%	-56.3364%	+2.9983%
3	-42.9397%	-56.3955%	+3.3214%
4	-43.2351%	-56.5883%	+4.1576%
5	-43.2452%	-56.7117%	+1.3254%
6	-43.2171%	-56.5004%	+2.2788%
7	-43.1473%	-56.3125%	+4.8416%
8	-43.1789%	-56.7328%	+4.1490%
9	-43.2103%	-56.4715%	+2.7791%
10	-43.0929%	-56.5329%	+1.9975%
11	-43.2191%	-56.5996%	+3.4346%
12	-42.6823%	-55.9531%	+3.3395%

Overall mean-change interval: -43.23% to -43.03%. These paired ratios are included for review; they are not the per-request raw response archive.

Historical provenance, controls and evidence availability

• Model: Qwen/Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 plus ZefanCai/Open-Jev-27B-v1.1 revision 28cf73067d5b337860bbef3c85b8b82ba8730956. Merged bfloat16 weights, unchanged 32-bit trained scalar head, temperature 2.5343690298472983, maximum supported length 16,384.
• One reserved device: driver label NVIDIA L20X, but measured CUDA capability 9.0 / 132 streaming multiprocessors / 150,121,283,584 visible bytes. The conflicting label does not establish a commercial GPU model. UUID GPU-46890c79-bf10-5b49-3d4a-a6b78f956122. CUDA 13.0.88, driver 570.133.20, Rust 1.99.1. The library contains native 8.9 and 9.0 code. Other architectures were not executed in the historical campaign.
• Same model, frontend, corpus, runner, device and CPU affinity 56–71 across arms; Rayon threads 4, Tokio worker threads 2. CPU affinity was not exclusive ownership of those CPUs. Graph mode was the intentional treatment difference. Device memory was sampled approximately every 0.5 seconds across loading, warmup and requests; brief peaks may be missed. Throughput under concurrent clients was not measured.
• Corpus source: local JevBench revision 869260649f7dd5b56cac4d91eaae9087f2e985ee, benchmarks/e2e/text.jsonl, pinned upstream corpus f8ce713. Frozen request manifest SHA-256 1f1376b99b12a3eca10f9e0726936585c8875e0eccf54de82ab110fd4e21867f; token-shape manifest 6b6bb344513e9b3024a37d5df0dc4b1442a77d5e62ccc03f00fad8dcf469c7d5.
• Original worker SHA-256 4ef036988633a4e2750d98a7ec916c67bd585e48039bab1028555ce4d8b76b32; P19 worker 2b8ddb4b75ff5a95076c1d6f63c753431e10b3af1f973ed5967e9e9765b93ff3; P19 CUDA library 67e445cfe71f13d5331091878d0a71f47defe6242118621cb08e67ac3cefbe1d.
• Frozen campaign runner SHA-256 6d5eadc2a505e5e93f3d550c23ff70d14084216fa4deafa0a2d795576c6eb0ef; imported benchmark runner 931313c3dc51bca9d3520ac2abdb98ec28a3e45d2c016bf402c8e5bac04d8d56. Each archived invocation records its arguments and binary/model hashes. Reproduction uses campaign.py --worker <arm-worker> --library <arm-library> --passes 8 --graphs <0-for-A-or-1-for-B/C> --reference <frozen-original-responses> --out <new-directory> in the declared 12-block order, followed by the frozen comparison/response-audit scripts.
• All raw historical responses, timing controls, sources, binaries and reports are retained in L4-fixed-confirmation-evidence.tar.gz (42,738,920 bytes, 964 verified members), SHA-256 c2ef5ab2a9a5655386c4f6416514f9df1ae5cd68a74f925d49803c08abde85e2. Earlier failures and qualifications remain archived separately.
• Evidence availability limitation: the complete raw archive and exact campaign driver currently live on the experiment host, not at a public download URL. A repository checkout alone can run the maintained regression tests but cannot replay this exact historical campaign without that archive. The tables above provide the public numeric evidence; independent raw-response audit still needs access to the archive. No weights, credentials, private requests or generated experiment archives are committed here. No video: this is a nonvisual request-latency change.

Merge Danger

Door: two-way. Revert the commit and rebuild the worker/library together; there is no stored-data migration or response-schema migration.

Blast Radius: shared-executor. Open-Jev-27B opts into the optimizations, but its executor and CUDA code are shared with other models. The integration preserves the current bidirectional and fixed/shared-prefix interfaces and tests their affected operators/paths. Complete other-model checkpoint runs are unverified. Larger retained graph/scratch/prefix capacity costs device memory; new shapes still incur capture cost, so graph mode stays opt-in. The measured tail tradeoff and four inconclusive guards remain visible above.

Self-review

Full source review and executed validation are reported above. Keep this PR in draft pending the contributor's own checklist and resolution of the stated review/evidence limitations. Automated source review and local tests do not replace contributor or maintainer review.

• I have reviewed the full diff and addressed the issues I found.
• I have checked that the change follows the project's architecture and stays focused on the stated purpose.
• I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
• I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

[twu3202]

Heads-up: #179 (for #85) also runs Open-Jev's shared candidate prefixes once, so the two overlap. A trial merge conflicts in executor.rs, processing.rs, main.rs, tests/open_jev/processing.rs and recipe/open_jev/native.md, and both add finish_with_prefix with different metadata.prefix_cache fields (saved_tokens with modes fixed/shared/auto in #179, shared_tokens with within_question here).

They cover different cases. #179 is opt-in (OPEN_JEV_PREFIX), works for 9B and 27B, and also shares the page state across questions in one request, using forward_shared from #99. Here it is on by default for 27B, for single-question requests whose candidates are all over 2,048 tokens, and it keeps the default GEMM algorithms so responses don't change. Happy to rebase #179 onto whichever lands first. It would help to settle on one prefix_cache shape.