[New Model]: Add Valen-Preview-0923 reference serving — community help wanted #84

Activity

1. hsliuustc0106

   added

   help wantedExtra attention is neededExtra attention is needed

   new modelRequests to support a new modelRequests to support a new model

   on Oct 5, 2026
2. MrDongsls commented on Oct 5, 2026

   

   MrDongsls

   on Oct 5, 2026

   Collaborator

   More actions

   I’d like to take this issue. Will coordinate with #61 and update #83 during the integration.
3. MrDongsls

   self-assigned this

   on Oct 5, 2026
4. hsliuustc0106 commented on Oct 5, 2026

   

   hsliuustc0106

   on Oct 5, 2026

   ContributorAuthor

   More actions

   Thanks @MrDongsls, please go ahead—you’re already assigned.

   The reference worker with text and single-image choice is a good first slice. Start by pinning the upstream code, base model and preview checkpoint, then bring up the worker and compare its outputs with the reference through the Rust frontend. Please keep all trained visual/LoRA components when loading the preview.

   Coordinate confidence/usage semantics with #61, and update #83 with the implementation and validation status as the work progresses. A small draft PR for the worker and recipe is welcome; native acceleration can follow once this path is reproducible.
5. hsliuustc0106 commented on Oct 5, 2026

   

   hsliuustc0106

   on Oct 5, 2026

   ContributorAuthor

   More actions

   @MrDongsls, one additional requirement: please follow the updated architecture and integration contracts and contributing guide.

   Keep the Valen integration split into clear stages:

   • Model-specific preprocessing prepares inputs and response context; postprocessing handles normalization, confidence, usage and response reconstruction.
   • The model executor owns the forward pass, learned head, weights and device state.
   • Shared admission/scheduling belongs in the runtime; reusable CUDA operations belong under src/backends/. The frontend owns HTTP transport and response delivery.

   The Python reference worker is suitable for the first slice. Preserve these processor/executor boundaries so native follow-ups can use Rust host orchestration and the shared runtime. Reuse the current worker interface while the remaining shared processing and batching layers are being developed.

   Please include the prepared-input/output contract and ownership boundaries in the draft PR, and keep tests under the root tests/ tree.
6. MrDongsls commented on Oct 5, 2026

   

   MrDongsls

   on Oct 5, 2026

   Collaborator

   More actions

   @MrDongsls, one additional requirement: please follow the updated architecture and integration contracts and contributing guide.

   Keep the Valen integration split into clear stages:

   • Model-specific preprocessing prepares inputs and response context; postprocessing handles normalization, confidence, usage and response reconstruction.
   • The model executor owns the forward pass, learned head, weights and device state.
   • Shared admission/scheduling belongs in the runtime; reusable CUDA operations belong under src/backends/. The frontend owns HTTP transport and response delivery.

   The Python reference worker is suitable for the first slice. Preserve these processor/executor boundaries so native follow-ups can use Rust host orchestration and the shared runtime. Reuse the current worker interface while the remaining shared processing and batching layers are being developed.

   Please include the prepared-input/output contract and ownership boundaries in the draft PR, and keep tests under the root tests/ tree.

   Thanks for the guidance. I’ll proceed in line with the project’s conventions and the direction discussed here.
7. MrDongsls commented on Oct 7, 2026

   

   MrDongsls

   on Oct 7, 2026

   Collaborator

   More actions

   @hsliuustc0106 Hi, the first integration slice is up for review: #115 — please take a look.

   Once this lands, if you agree, I'd like to plan and take the follow-up slices as well:
   noul and score question types on the reference worker first (they mostly need protocol/postprocessing work — the pinned compiler already supports them), then video state, and a native Rust/CUDA text-only executor reusing the shared Qwen3.5 executor, with the multimodal native path building on the Cua-S1 vision work (#56 / #59 / #63 / #64).
8. MrDongsls commented on Oct 11, 2026

   

   MrDongsls

   on Oct 11, 2026

   Collaborator

   More actions

   Plan for the remaining slices

   PR #115 gives the reference path: choice over text and one image. This is a proposal for the rest — what we can reuse, what Valen actually needs, in what order, and where I think the traps are.

   What can be reused

   The qwen3_5 CUDA backend can run the whole backbone as-is. I checked the pinned Qwen3.5-2B config (15852e8) against the loader's structural assertions in model.rs:100-178: 24 layers (18 Gated DeltaNet + 6 full attention), head_dim 256/128, mrope [11,11,10]. Everything passes, so the backbone itself needs no backend work.

   The trained vision parts are simple. The checkpoint's vision-top blocks and the merger are full-weight replacements of base tensors — I verified the keys, shapes and values against the base checkpoint — so they load as an ordinary weight update through the existing cua_s1 vision path. The one change that path needs is unpinning its 4B constants (out_hidden_size 2560 → 2048 and the merger width).

   Beyond that:

   • forward_multimodal takes adapted BF16 image embeddings plus three-axis mrope positions, which is exactly the media-state prefix. It expects the vision-language adapter merged into the exported checkpoint.
   • The prefix machinery (alloc_prefix, forward_multimodal_capture/_continue, covering attention KV, the GDN FP32 state and the conv tail) is merged; only jev_vl uses it today.
   • Text and vision CUDA Graph replay are merged behind opt-in switches.
   • The prepare/execute/finish split, serial admission, and the CPU decision heads in open_jev, jev_vl and omnijev give the worker its shape. Valen's head can start on CPU in FP32 like its siblings.
   • Every native worker so far loads exported safetensors, never torch pickles. Valen should do the same.

   Several open issue/PRs matter here:

   • qwen3_5: run prompts that share prefixes once per prefix #99 adds forward_shared (shared prefix only runs once), with qwen3_5: add a GEMM handle with one algorithm per weight shape #98's fixed-GEMM handle underneath.
   • [RFC]: Reuse shared observations across decision questions — community help wanted #85 is the design RFC for reuse; [RFC]: Write down which parts of the /v1/systemone contract all workers share #61 settles the response fields.
   • cua_s1: add native multimodal language CUDA Graph replay #103/docs(jemm): document CUDA Graph controls and validation limits #123 (multimodal language graphs), runtime: optional bounded admission #109/feat(decider): bounded request-local CUDA batching #111/feat: bound pending native execution work #82 (admission, request-local batching) and cuda: add a fused candidate-scoring kernel for CLM and Kev #69 (a fused scoring kernel; Kev's readout is the same form as Valen's head) are relevant later.

   What has to be built

   1. noul and score on the reference worker: The pinned compiler already builds all three task types; PR Valen: serve Valen-Preview-0923 through a Python reference worker #115 only limits the wire. noul is one fixed true/false branch answering P(true). score compiles one branch per level (2–10), takes one decision-head logit from each, softmaxes across them and answers the expected level. Response fields go through [RFC]: Write down which parts of the /v1/systemone contract all workers share #61.
   2. The export recipe: checkpoint.pt holds 428 trainable tensors: a rank-32 FP32 language LoRA, the BF16 vision-top blocks and merger, and two FP32 (256×2048) head projections. The export turns that into safetensors the native path can load. Two details: the base ships 36 FP32 tensors (A_log, linear_attn.norm) that the BF16-only loader rejects, and downcasting them is lossless against Valen's own BF16 runtime; and the LoRA can be merged or served separately (see below). Parity runs against the pinned reference at the existing 1e-6 gate.
   3. The native eager worker: Rust request processing (branch suffixes, candidate and decision token positions), execution on the qwen3_5 backend, an FP32 head on CPU, response construction. The RFC should mirror [RFC]: Native Rust/CUDA inference for Mapika/decider-2b #57 and [RFC]: Native Rust/CUDA support for OmniJev-4B v1.1 and community help #114.
   4. Video on the reference worker first: state.video goes through the pinned processor, which samples at fps-2 (clamped to 4–768 frames) and folds frame pairs into 1536-wide patch rows. The reference defines no media limits, so we set our own: a codec whitelist, duration and size caps, a larger request body. One known blocker, reproduced in the reference environment (transformers 5.19): video compilation fails with ImportError: torchcodec is not installed. Picking that dependency is part of this slice.
   5. The native temporal pipeline: The mrope tables already carry three axes through forward_multimodal; what's missing is bookkeeping, not math: parsing video_token_id, building multi-block sequences, lifting the t==1 assertions (multimodal.rs:36-39 and :139-150, vision/geometry.rs:11-13, vision/model.rs:236, image_preprocess.rs:144), and choosing a decode dependency. An FFmpeg sidecar is closest to what [RFC]: Native Rust/CUDA support for OmniJev-4B v1.1 and community help #114 assumed for OmniJev; its frame-mosaic approach itself does not transfer, since Valen needs real temporal frames.
   6. Multimodal branch continuation: Valen's serving shape is one media/text state prefix with per-question and per-level suffixes inside one admitted unit. That extends qwen3_5: run prompts that share prefixes once per prefix #99's text-only forward_shared to multimodal prefixes and to Score's K branches at once.

   Order

   1. noul/score on the reference worker — independent, the smallest slice, settles the contract fields with [RFC]: Write down which parts of the /v1/systemone contract all workers share #61, and produces the fixtures everything later compares against.
   2. Export recipe + parity — also independent, so it can run alongside 1; it unblocks all native work.
   3. Native eager worker — text+image choice first, then noul/score, with CPU fixtures and GPU parity under a declared tolerance.
   4. Video on the reference worker — independent of 2 and 3; freezes the sampling behavior and limits that native video must later reproduce.
   5. Branch-reuse acceleration — after 3, together with qwen3_5: run prompts that share prefixes once per prefix #99/qwen3_5: add a GEMM handle with one algorithm per weight shape #98/[RFC]: Reuse shared observations across decision questions — community help wanted #85, measured on its own baseline.
   6. Native video — after 3 and 4; graphs and batching only where profiling justifies them.

   Worth attention

   • Merge or serve the LoRA. Merging (W + 2·BA in BF16) differs from the reference's unmerged FP32 application by about one BF16 ulp per layer output. Fine for export, not bit-exact — a decision to make in slice 2 and declare. Strict parity means serving the FP32 LoRA separately; the vision path already loads unmerged LoRA, the text path does not.
   • The native tolerance is declared, not inherited. The 1e-6 gate was Python against Python; cuBLASLt and the GDN kernels round differently, so the native path needs its own gate. This is what qwen3_5: add a GEMM handle with one algorithm per weight shape #98's fixed handle is for.
   • Reuse must not restructure prompts. A Score branch sees only its own level description; concatenating questions or levels into one prompt changes the computation, as [RFC]: Reuse shared observations across decision questions — community help wanted #85 warns. Valen's shared_state training mode is likewise not a serving shortcut — parity is against the per-branch reference. When prefix reuse lands, we should also say how input_tokens (state counted once) and internal_usage.compute_tokens (per-branch recompute) relate to what actually runs.
   • The vision-constant unpin touches shared code. Per CONTRIBUTING, coordinate before changing the CUDA ABI and keep cua_s1's regressions green.
   • One acceleration, one measurement. Each optimization is a separate hypothesis with its own baseline; no combined claims.
   • Video is bounded by the 8192 max_length: ceil(frames/2)·(H'/32)·(W'/32) vision tokens plus roughly 4–8 text tokens per frame. The request limits in slice 4 are what actually protects this.