Skip to content

vela: serve the 4B and 9B decoder members - #195

Draft
MrDongsls wants to merge 42 commits into
InferMatrixLab:mainfrom
MrDongsls:model/vela2-4b-9b
Draft

MrDongsls wants to merge 42 commits into
InferMatrixLab:mainfrom
MrDongsls:model/vela2-4b-9b

Conversation

@MrDongsls

@MrDongsls MrDongsls commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Required human contributor answers

  1. What outcome are you trying to achieve?

    Answer: Serve the Vela 2.0 4B and 9B decoder members on system1-omni the same way the 0.8B is served.

  2. What is the highest-leverage variable for that outcome?

    Answer: Exact-match parity with each member's pinned reference runtime.

  3. How will this work move that variable rather than create busywork?

    Answer: The three members ship one byte-identical runtime, so one registry and the shape-derived heads extend the validated 0.8B path to both with no new executor code.

  4. Who is the customer or user, and what they actually need?

    Answer: Anyone who wants Vela 2.0 decisions from system1-omni on the larger members, and they need correct answers on all five question kinds.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision: Yes, the new members are served only after this registry, tests and docs merge.

For a bug fix: link actual observed failure evidence.

N/A, not a bug fix.

Purpose

Fourth slice of the Vela 2.0 native chain. It serves the 4B and 9B decoder members on the Python reference worker and the native library that already serve the 0.8B, one pinned member per worker process.

  • The worker keeps one member registry keyed by the package's config.json model_name. Each entry pins the repo, revision, the ten hashed files and the shard layout, and a package naming an unpinned member is refused by its own name.
  • The native bundle gate mirrors that registry as a member table with the exported tensor inventories. The F32 heads derive every width from the tensors' own shapes and the executor cross-checks the backbone hidden size, so one implementation serves hidden 1024, 2560 and 4096.
  • Fixtures are relabeled per member and JSON fixtures are committed. The head-input safetensors of the larger members are not committed, about 103 MB of binaries, and are regenerated with recipe/vela/export_head_inputs.py on the pinned package.

Builds on #151, #174, #175, #176 and #177.

Stacked PRs and merge order

This is the fourth slice of one chain and stacks on the open 0.8B native PRs; the branch history contains the #177 head 064244d, so most of this diff's native-crate lines are those PRs' content plus the member-parameterization on top:

Order PR Layer
1 #151 (merged) 0.3B Python reference worker
2 #174 (merged) 0.8B hybrid through the reference worker
3 #175 0.8B native bundle export, wire contract, token-level processing
4 #176 0.8B native F32 heads, scoring and span decoding
5 #177 0.8B native response finishing and CUDA executor
6 this PR the 4B/9B member layer

Recommended merge order is #175 → #176 → #177 → this PR. Once the 0.8B stack lands, this PR's diff collapses to its own layer (the hybrid.py member registry, the per-member fixtures, the member-stamping exporters and the docs); until then, reviewers can treat the files under src/models/vela/native/ as the #175–#177 surface under review there. The branch merges current main in 1b8d025, so it stays conflict-free while the stack lands.

Test Plan

System1-Omni Version / Commit: 73af078 on model/vela2-4b-9b, based on the #177 head 064244d.

  • CPU on the branch head: cargo fmt --all --check, cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings, cargo test --workspace --locked --features omni-laya/serve, PYTHONPATH=src pytest tests/vela, and mkdocs build --strict. The release build was not run locally and is left to CI.
  • Author-run GPU evidence recorded in the recipes, one rented RTX 4090 48 GB, sm_89, driver CUDA 13.3, toolkit 13.2, the bf16 autocast setting: package gate, load and all-kind warmup for both members, nine-case exact-match reference parity against the pinned upstream server, frozen oracle, token-plan and head-input fixtures exported under each member's label, and the native replays with the member-calibrated gates in the tests.

Test Result

  • CPU: all checks above pass on 73af078. The workspace tests report 100 test targets ok with no failure, the omni-vela-native crate 30 passed and 4 ignored that need an exported bundle or a GPU, and pytest 160 passed including the member-parametrized oracle cases.
  • Reference parity: exact on all nine cases for both members, four 200s byte-equal including key order and agreed 422 and 413. A warm 4B worker answered the example request in about 0.2 s, a single observation, not a benchmark.
  • Native heads and response: 4B head logit drift at most 8.97e-5, answer drift at most 5.56e-3, response drift at most 5.56e-3. 9B: 4.20e-5, 6.80e-4 and 5.92e-3. All inside the member gates. The larger ceilings against the 0.8B baseline come from the GPU-recorded fixtures and are documented in the tests and native.md. One divergence is kept deliberately: on borderline span cases the replay flips the discrete span set while its probabilities still match, so the tests assert probabilities there and keep full structural assertion on the 0.8B CPU-recorded fixtures.
  • Head updated to b2559fe (independent-review fixes plus the merge of current main): the same CPU checks pass there — fmt, workspace clippy and workspace tests 421 passed / 0 failed / 92 ignored after the merge, omni-vela-native 32 passed / 4 ignored (two new bundle-walk tests), pytest tests/vela 160 passed also re-verified torch-free on system Python, and mkdocs build --strict clean. main merge conflicts (Cargo.toml, README.md, docs/supported-models.md, recipe/README.md) are resolved in 1b8d025; Cargo.lock keeps only the omni-vela-native addition against main.
  • Executor on CUDA: 4B replays all four small cases with worst logit drift 0.2271 against the 0.35 bf16 gate. 9B recorded two of the four, worst 0.1899, before the rented session ended. The 9B cjk and score cases are unverified and remain to be run.

Demo / evidence

The runs above are author-run and recorded with hardware, revisions and gates in recipe/vela/reference.md and recipe/vela/native.md. No separate raw archive or video. The change is a serving integration and the parity numbers are the evidence. The 9B executor cjk and score replays are labeled unverified above and will be fully verified when this PR is ready for review.

Self-review

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked that the change follows the project's architecture and stays focused on the stated purpose.
  • I have run the checks appropriate to this change and reported commands, results, and anything that I could not verify above.
  • I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

Mr.Dong added 30 commits October 25, 2026 02:57
- recipe/vela/export_native.py: gate-verified export into a native
  bundle — 320 backbone tensors re-keyed backbone. -> model. with
  per-tensor dtypes preserved (204 BF16 Linear/conv1d + 116 F32), the
  37 F32 heads incl. the side-loaded broad span head in one file,
  backbone_config at the config root (the family Config::load shape),
  and a manifest pinning the source revision, the gate table, the
  tensor inventory and every bundle file's SHA-256.
- src/models/vela/native (omni-vela-native): the bundle manifest gate
  and the wire admission mirroring protocol.py and the pinned runtime's
  preamble with byte-equal 422/400 bodies.
- recipe/vela/export_token_plan.py and five frozen token plans under
  tests/vela/fixtures/: the predict mapping, the row decomposition and
  the rendered records (p_ids, blocks, readout positions, truncation,
  a windowed long-span case); example-request totals 1,041 tokens,
  matching the oracle's usage.

Checks: cargo test (15 tests, no weights, no GPU), fmt/clippy clean,
mkdocs --strict.
The Rust processing of the native path, mirrored from the pinned
runtime and checked field-for-field against the five frozen token
plans (opt-in render parity with the exported bundle's tokenizer):

- state.rs: the state -> typed-parts mapping (role grouping, field
  ranges, canonical DEV2.0 JSON) with the pinned rejection triples;
- questions.rs: the SystemOne -> predict mapping — the validation walk
  and order, Noul folded into a two-option choice (choice renders
  [ABS], the reference's abstain default), over resolution,
  threshold/head carried, the set-label collision check;
- words.rs: the GLiNER2/CJK _WORD splitter and the first-token word
  mapping;
- render.rs: the encoder (char-offset tokens, memoized), the six-rule
  span-head dispatch with the hallucination-alias label replacement,
  the truncation budget, the DEV2.0 rendering (parts pass, question
  blocks with option ends and query positions, span layout a/c with
  word positions) and the window rows (head only, as the reference);
- processing.rs: the Processor over the exported bundle.

Also: the envelope error locs are flat arrays, matching the reference
triples exactly.

Checks: cargo test (26 tests) + fmt/clippy clean + opt-in render
parity (5/5 fixtures).
The Rust processing of the native path, mirrored from the pinned
runtime and checked field-for-field against the five frozen token
plans (opt-in render parity with the exported bundle's tokenizer):

- state.rs: the state -> typed-parts mapping (role grouping, field
  ranges, canonical DEV2.0 JSON) with the pinned rejection triples;
- questions.rs: the SystemOne -> predict mapping — the validation walk
  and order, Noul folded into a two-option choice (choice renders
  [ABS], the reference's abstain default), over resolution,
  threshold/head carried, the set-label collision check;
- words.rs: the GLiNER2/CJK _WORD splitter and the first-token word
  mapping;
- render.rs: the encoder (char-offset tokens, memoized), the six-rule
  span-head dispatch with the hallucination-alias label replacement,
  the truncation budget, the DEV2.0 rendering (parts pass, question
  blocks with option ends and query positions, span layout a/c with
  word positions) and the window rows (head only, as the reference);
- processing.rs: the Processor over the exported bundle.

Also: the envelope error locs are flat arrays, matching the reference
triples exactly.

Checks: cargo test (26 tests) + fmt/clippy clean + opt-in render
parity (5/5 fixtures).
- heads.rs: CandidateHead (option-end + query hiddens; bilinear +
  scalar-MLP, [ABS] last), both SpanHeadV2 instances (router and the
  side-loaded broad head) and set_bias, loaded from the exported
  bundle's heads.safetensors — plain float32, the reference's own
  autocast-off setting (erf-based GELU via libm).
- scoring.rs: the calibration mirror — per-type temperatures, set
  thresholds, the PII length rule (log-space interp) and sparse gate,
  span thresholds by id/label set, softmax/sigmoid, span_decode_r4
  (word units, unit vote, fill, edge/URL trim), label aliasing, score
  value normalization (i/(n-1)), and the window aggregation (mean over
  covering windows, -30 uncovered).
- recipe/vela/export_head_inputs.py and the recorded head-input
  fixtures: the reference's consumed hidden states, its head logits
  and its calibrated predict answers for the five frozen cases.

Opt-in parity (VELA_NATIVE_BUNDLE): five cases, head-logit drift
<= 3.8e-5, predict-answer drift <= 7.3e-7 (the windowed long-span case
exact), discrete picks, span structures and threshold rules equal.
- heads.rs: CandidateHead (option-end + query hiddens; bilinear +
  scalar-MLP, [ABS] last), both SpanHeadV2 instances (router and the
  side-loaded broad head) and set_bias, loaded from the exported
  bundle's heads.safetensors — plain float32, the reference's own
  autocast-off setting (erf-based GELU via libm).
- scoring.rs: the calibration mirror — per-type temperatures, set
  thresholds, the PII length rule (log-space interp) and sparse gate,
  span thresholds by id/label set, softmax/sigmoid, span_decode_r4
  (word units, unit vote, fill, edge/URL trim), label aliasing, score
  value normalization (i/(n-1)), and the window aggregation (mean over
  covering windows, -30 uncovered).
- recipe/vela/export_head_inputs.py and the recorded head-input
  fixtures: the reference's consumed hidden states, its head logits
  and its calibrated predict answers for the five frozen cases.

Opt-in parity (VELA_NATIVE_BUNDLE): five cases, head-logit drift
<= 3.8e-5, predict-answer drift <= 7.3e-7 (the windowed long-span case
exact), discrete picks, span structures and threshold rules equal.
scoring.rs system_one_response: the SystemOne envelope from the
predict answers and the response plan — Noul views (span = the
range-filtered max word probability, set fan-out <qid>.<label>), span
clipping to the plan's field ranges with re-trim, span_heads,
thresholds, the choice top-two margin and the score variance
confidence, and usage. Replaying the recorded reference logits into
the frozen oracle responses matches within 1.5e-6 on the four oracle
cases (opt-in with VELA_NATIVE_BUNDLE).
executor.rs: the backbone through the shared qwen3_5 crate. The
family Model exposes only last-position hidden states, so every
readout position runs as its own full sequence [parts .. position] —
causality makes that hidden identical to the reference's in-tree
readout — packed into forward_batch. The backbone runs from the
uniform-BF16 export (export_native.py --bf16-out, the shared loader's
format); the heads stay F32 on the faithful tensors. Opt-in GPU
parity (VELA_NATIVE_BUNDLE + VELA_CUDA_LIB): four cases replay the
recorded reference logits, worst drift 0.285 (the bf16-vs-FP32 scale;
gate 0.35). First recorded execution of these kernels on sm_120
(RTX 5060, CUDA 13.1).
…processing

# Conflicts:
#	Cargo.toml
#	README.md
#	docs/supported-models.md
#	recipe/README.md
…heads

# Conflicts:
#	Cargo.toml
#	README.md
#	docs/supported-models.md
#	recipe/README.md
…executor

# Conflicts:
#	Cargo.toml
#	README.md
#	docs/supported-models.md
#	recipe/README.md
…ows alongside the Clef/pplx-decider additions
…native-processing

# Conflicts:
#	recipe/README.md
…heads

# Conflicts:
#	recipe/vela/native.md
# Conflicts:
#	recipe/vela/native.md
#	src/models/vela/native/Cargo.toml
#	src/models/vela/native/src/bundle.rs
#	tests/vela/native_heads.rs
#	tests/vela/native_processing.rs
…egistry

One vela2-decoder program for all three members: hybrid.py keys the
registry by config.json model_name (repo, revision, pin table, shard
layout), the native bundle gate mirrors it as a member table, and the
F32 heads derive every width from the tensors' own shapes with the
executor cross-checking the backbone hidden. Fixtures are relabeled
per member; the larger members ship their JSON fixtures only — the
head-input safetensors are regenerated with recipe/vela/export_head_inputs.py
on the pinned package instead of committing ~103 MB of binaries.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant