[RFC]: Model support and feature roadmap for Ouro, Nanbeige4.2, and Huginn #69

Motivation.

Maintain one model and feature progress tracker for vllm-rlt, initially limited to Ouro, Nanbeige4.2, and Huginn. Contributors should be able to see which implementations exist, which work is in progress, and which combinations still need integration.

Track architecture implementations rather than duplicating the same backend for every checkpoint size. Keep PD as a core priority alongside loop-level batching, depth-aware KV, asynchronous execution, adaptive depth, and self-speculative decoding.

This issue complements the architecture/refactoring roadmap in #32. Recipes and training/rollout integration will have separate RFCs; this issue links to their results rather than collecting unrelated benchmark scripts or training tasks.

Proposed Change.

Scope and model reuse

• Ouro: one model implementation for 1.4B and 2.6B, configured by checkpoint metadata.
• Ouro-Thinking: reuse the Ouro core and runtime features. Track Thinking-specific basic integration separately: checkpoint configuration, BOS/EOS, chat-template/enable_thinking behavior, and generation/reference checks. Do not create another recurrent backend solely for the Thinking weights.
• Nanbeige4.2: one implementation covering 3B and 3B-Base, starting with the official fixed two-loop semantics. Native fixed two-loop support merged in [MODEL] : support Nanbeige/Nanbeige4.2-3B recurrent causal LM #63; checkpoint-specific and cross-feature qualification remains separate.
• Huginn: a separate architecture integration for Huginn-0125, including its prelude/recurrent/coda structure, state initialization/input injection, and cache semantics.

Official references: Ouro, Ouro-Thinking, Nanbeige4.2, Huginn.

Status convention

• ✅ Implemented and merged for the tracked architecture/path.
• 🚧 In progress: a partial integration, open implementation PR, or explicitly active work item in a linked RFC.
• ⬜ Not implemented/integrated for this model in the tracked scope.
• ➖ No separate implementation task: reuse the Ouro implementation and track its progress in the Ouro row/sub-tables.

These are engineering progress indicators, not claims of validation on every checkpoint, hardware platform, workload, or feature combination. A parent feature is 🚧 while some listed subfeatures remain incomplete. An existing shared utility does not by itself complete a new model's integration.

Progress refreshed on 2026-10-22 against main at 4a350344027960a48384626aa646f1765fda6d4f. #63 Nanbeige, #77 Nanbeige PD, #82 Huginn and #106 Thinking basic integration are merged. #118 stage/loop execution and #107 scheduler result application are also merged. These statuses identify implementation scope, not qualification of every checkpoint, backend or feature combination.

Overall progress

Each feature has an Assignee(s) column listing contributors to its claimed work. Multiple contributors may share a feature or claim different subfeatures. — means no assignee is recorded; Ouro-Thinking runtime cells marked ➖ inherit the corresponding Ouro task.

Names prefilled from existing PRs/RFC claims identify contributors only to the linked scope, as shown in the sub-tables. They do not assign all remaining work in the parent feature to those contributors. Record new subfeature claims as status · issue/PR · @username in the corresponding model cell and reflect the contributors in the overall table.

Basic implementation includes configuration and weight loading, tokenizer/chat template, model computation, fixed recurrence, greedy and temperature/top-k/top-p sampling, EOS/length handling, and text generation. These define acceptance for basic model integration; contributors may split the implementation work as needed.

For Ouro-Thinking, only basic integration requires a separate task. All other cells are marked ➖ because they reuse Ouro and require no separate implementation; their progress is maintained in the Ouro row and sub-tables. This does not mean the features are unsupported or that checkpoint validation can be skipped.

@YouHengfei's shared Thinking-specific basic integration merged in #106. The recipe covers bounded BF16 fixed-four-loop synchronous checks for both 1.4B-Thinking and 2.6B-Thinking. Other runtime combinations and base Ouro-2.6B quality/performance are not inherited from those checks. See the current checkpoint matrix.

Scheduling and batching

Request arrival, completion and cancellation are part of continuous batching. #118 merged split-prefill and compatible prompt/decode RECURRENT batching under LAST_EXITED with non-atomic prefill; PRELUDE/CODA stay separate and speculative rounds are excluded. It does not implement same-request multi-chunk wavefront scheduling (#65). #60 and #50 closed without merge; #68/#113 scheduling fixes and #107 scheduler-owned result application are merged. Further #81 validation should build on this implementation.

KV cache

Incremental allocation draws pages from the engine's KV pool; it does not allocate a new CUDA tensor per token. Snapshot/restore preserves request state rather than recomputing from scratch. #57 public KV views, #109 async device-token restoration and #118 shared metadata/resource contracts are merged; speculative preemption (#52) is still open. Opt-in FP8 KV (#129) is open, including fused writes and scale-aware restoration. FP8 PD transfer is not implemented. Combining #129 with #128 packed/split attention requires scale reads in every consuming path; the second merged change must include that integration.

GPU execution

This table covers ordinary inference. Huginn's #82 implementation and synchronous LAST_EXITED core/CODA Graph path are merged; async Huginn is not claimed. Nanbeige validation PR #76 closed unmerged, so broader graph qualification remains separate from basic model/PD support. Unchanged empty cells require model-specific integration or qualification rather than inheriting Ouro results. #46 attention contracts are merged; #112 FlashInfer and #128/#130 Triton kernel changes remain open.

Hardware and attention backend adaptation

Adapt suitable attention backends for the hardware available to contributors, and track progress per model and hardware family. These are recommended adaptation targets, not claims that every listed backend is already integrated or fastest on every workload. The existing GPU execution table above describes the implemented CUDA paths; broader hardware/backend coverage is tracked here.

For this table: ✅ completed integration for the explicitly identified device/path; 🚧 partial integration or ongoing work/validation; ⬜ adaptation or target-device validation pending. Record the claimed contributor and implementation PR in the relevant cell when work starts. Performance and accuracy evidence belongs in the corresponding recipes.

• Ouro-Thinking inherits Ouro. Existing Ouro hardware evidence primarily covers 1.4B; it does not qualify every checkpoint or device in the same family.
• FA2/FA3 interfaces already exist, but target-device validation remains pending. H800 FA4 results are reported in the still-open [DOCS] Report synchronous self-speculation evaluation on H800 #47. Nanbeige basic integration merged in [MODEL] : support Nanbeige/Nanbeige4.2-3B recurrent causal LM #63; that alone does not establish support for the recommended backends above.
• MindIE-SD is a candidate operator integration. Confirm causal attention, paged KV and incremental decode applicability before claiming a complete language-model attention backend. Retain the existing Triton path as a comparison/fallback where supported.

Selection references: FlashAttention, FlashInfer hardware support, AMD backend guidance, and MindIE-SD attention APIs.

Adaptive depth

Fixed recurrence is part of basic support. Adaptive exit is an optional research extension for Nanbeige4.2, not a requirement for its fixed two-loop integration. Huginn's stopping rules require their own implementation/validation rather than reuse of the Ouro gate. #56 also tracks accuracy, mean-depth, and throughput evidence. Its adaptive-exit evaluation harness and depth statistics merged in #58; this does not complete the exact-policy asynchronous exit implementation.

Self-speculative decoding

#43 tracks algorithms and #47 the H800 evaluation report (still open; retarget/rebase reported complete). #48/#51/#66/#115 are merged. #118 adds shared runtime integration, async speculative core/CODA graphs and PD D-side speculation, with scoped B300 Triton/FA4 device coverage. Async stochastic sampling and speculative preemption remain unsupported; #52 is open for synchronous boundary preemption. #127 remains open for graph buckets, derived limits, vectorization and adaptive K; #121/#133 are open pipelining experiments. #59 closed unmerged after a negative result; #120 documents that result and remains open.

Prefill/decode disaggregation (PD)

PD speculation combines PD deployment with local speculation on D; it does not put draft and target on separate P/D workers. #118 provides shared stage connectors and D-side speculative execution; #122 ready-handoff prioritization and #77 Nanbeige basic PD are merged. Current deployment scope is same-host multi-GPU, not cross-node RDMA qualification. #64 closed unmerged; #111 retains contracts, #131 protocol typing and dependent #132 reservation-credit naming are draft/open. #119 is a draft real-checkpoint agreement test; equal-prefill-geometry checks do not imply bitwise equality under arbitrary concurrent BF16 batching.

Independent tasks, dependencies, and inherited support

These relationships describe implementation scope, not contributor ownership. A dependent task can have a different assignee from its prerequisite and can be developed in parallel against an agreed interface.

For each implementation issue, list concrete prerequisites and reused components. A dependency is not a new implementation task if the needed support already exists. Inherited support must be distinguished from integration that still needs model-specific work or validation; do not mark another architecture complete merely because Ouro supports it.

Task assignment and claiming

Progress and linked claims refreshed on 2026-10-22. Existing claims are preserved; this refresh assigns no new work. Implementation and qualification remain separate.

• @yanbao1217 owns Ouro FlashInfer adaptation for the SM120 path, with RTX 5090 results reported in open feat: add FlashInfer attention backend for SM120 decode #112. Accuracy differences remain under review; this backend is not marked complete. Coordinate with the attention-interface work in [M8 1/3] Extract attention contracts and add automatic backend selection #46; retain explicit unsupported cases and matched Triton correctness/performance comparisons. Ouro GPU execution is marked 🚧 for this additional backend work; existing completed CUDA paths remain ✅ in their subfeature rows.
• @kunkunblueberry's Nanbeige fixed-two-loop PD integration merged in [MODEL][PD] Integrate Nanbeige4.2 with PD serving #77. Broader model/backend/worker-count qualification remains separate; coordinate follow-ups with [RFC]: PD coordination and KV transfer contracts #111 and the current stage connectors.

@Carlos779988's mixed-batching claim remains recorded in #81; #118 has now delivered its implementation foundation and mixed recurrent batching. Coordinate remaining validation with @LyxWxj, keeping multi-chunk WCPB in #65 separate. @0z5a's Nanbeige/Huginn graph claim is retained: Huginn #82 merged, Nanbeige #76 closed unmerged. @CXJorz has no implementation task assigned.

Contributors may freely claim and split work at any practical scope, individually or collaboratively. Comment with the intended scope and link the implementation issue/PR; record the contributor(s) under Assignee(s) and note relevant dependencies. Ownership applies only to the claimed scope.

Track subfeature progress independently and mark a parent feature complete once all applicable subfeatures are integrated and validated. Technical dependencies and inherited support describe how implementations relate; they do not restrict task assignment.

Open qualification and evidence work (2026-10-22)

• feat: add FlashInfer attention backend for SM120 decode #112 FlashInfer SM120 remains blocked on accuracy investigation (reported Triton 61/87 versus FlashInfer 58/87); a BF16 accumulation-order hypothesis is not a resolved quality gate.
• [Kernel] Split-KV decode: one launch per call in eager mode (stacked on #128) #130 latest matching-base GSM8K-87 comparison is 60/87 versus 59/87, failing its declared 1 pp regression budget despite passing scoped GPU/PD tests. [Kernel] Triton packed paged prefill and split-KV decode attention #128/[KV] Opt-in FP8 e4m3 KV cache with per-vector scales #129 historical performance results need validation against their actual integration heads.
• [Architecture] Unify stage execution with loop-level scheduling and mixed batching #118's recorded full GSM8K ordinary/mixed FA4 runs pass its quality gate; Triton runs do not. Those runs precede later graph changes and are not blanket final-head or feature-combination qualification.
• [RFC]: Adaptive early exit on Ouro-1.4B: accuracy–throughput evaluation and exact-policy async exit #56's adaptive-exit harness merged in bench: add adaptive-exit options and depth statistics to the GSM8K ha… #58. The author plans Phase 1 measurements by Oct 25; exact-policy async exit remains data-gated, and BF16 multi-stream divergence needs separate reproduction.
• [RFC] Add support for LoopCD and validate its correctness #85/Add initial LoopCD support for Ouro and correctness checks #86 LoopCD and Ouro LoopCD preemption #98/Support eager preemption for Ouro LoopCD #101 LoopCD preemption remain open, with BF16/quality gaps retained. [Feature] Add experimental deferred Ouro deep batching #125 deferred deep batching is an independent draft experiment with real-weight GPU qualification outstanding.
• [RFC]: Training support and verl/vime integration for looped-transformer inference optimization #70 training interfaces remain open in Expose rollout logprobs and full policy publication on shared engines #78/[FEAT] Rollout outputs: selected-token logprobs, effective seeds and stop token IDs (part of #70) #83; their shared API/integration order is unresolved. [Doc]: Validation and benchmark artifacts for Ouro and Nanbeige #124 requests a public reproducibility index for Ouro/Nanbeige evidence and remains open.
• ci: add CPU lint and unit-test gate for pull requests #136 CPU lint/unit-test CI is merged; it is not GPU qualification. No new experiments were run for this tracker refresh.

Completion and maintenance

• Update the relevant cell, assignee(s), and linked implementation issue/PRs when work starts or merges. Ownership applies only to the explicitly claimed scope; parent status aggregates the subfeature results.
• For basic model integration, validate the intended checkpoint against its reference computation and generation behavior. Reusing an architecture does not automatically qualify every checkpoint.
• For a feature, exercise its real engine path, relevant request/KV/RNG lifecycle, and the combinations being claimed. Unsupported combinations should be rejected explicitly and documented.
• For performance work, report matched, repeated before/after results, including high-concurrency/large-batch workloads and low-concurrency controls where relevant. Fewer synchronizations or kernels alone is not a throughput claim.
• Every new model or feature must update the README support matrix, relevant documentation, and its recipe. Keep commands, checkpoint/configuration pins, correctness results, and performance measurements in the recipe; avoid adding a separate benchmark script for every tracker cell.

Alternatives and impact

We considered separate implementations/progress matrices for every checkpoint size and Thinking variant. The shared-architecture approach keeps runtime work in one place while preserving a small, explicit Thinking integration task and separate checkpoint validation. A single flat feature list would hide important PD, asynchronous, and speculative integration gaps, so those features have sub-tables.

This RFC itself changes no runtime behavior, public API, or memory usage. Implementation PRs must describe any API/configuration changes and their measured performance and memory effects. It does not expand model scope beyond the three families above, introduce training integration, or declare performance gains without evidence.

Feedback Period.

At least one week after publication. Keep this issue open as a living implementation tracker after the initial scope discussion.

CC List.

@bjf-frz @hsliuustc0106

Anything else.

Related architecture and implementation work is linked in the tables. The progress snapshot should be maintained as those PRs evolve; a refactoring PR and a feature-support PR are different kinds of work.

Before submitting a new issue...

• Make sure you already searched for relevant issues in the issue tracker, and looked for the answer in the documentation.