[quic-mkshirsa]

Summary

• Change Qwen3.6 convolution-state layout to channel-last
• Add decode convolution-state updates
• Support independent FP16 recurrent-state retention
• Adapt continuous-batching convolution-state gather/scatter to channel-last layout

Testing

• Ran the relevant Qwen3.6 unit tests

Note

This changes the retained conv_state layout to channel-last [B, K, conv_dim]. Previously exported ONNX models, compiled QPCs, and external state allocations—including disaggregated/vLLM and _kv_dma_handoff integrations—may need to be regenerated. The export hash now includes a layout version to prevent reuse of old cached artifacts.

[Vinayak Baddi (vbaddi)]

test_recurrent_state_dtype_override_preserves_mxint8_for_other_retained_states
only calls the two helpers directly. It never checks:

• that _validate_retained_state_dtype rejects an unsupported value (e.g. "int4"), the
  only reason the validator exists;
• that compile() actually emits the overridden dtype in the custom-IO entries for
  recurrent_state.*_RetainedState (the wiring at modeling_auto.py:2347-2356).
  Also the GDN fix for the recurrent reset mask
  (modeling_qwen3_5_moe.py:1019, reshape to recurrent_state.ndim) has no test.

Repro:  in modeling_auto.py:2354 stop passing recurrent_state_dtype= to
_resolve_retained_state_dtype; the new test still passes.

Ask: add a rejection case, a compile-level custom-IO assertion, and a test that
decode with batch_index and a position-0 reset keeps the recurrent state's shape/values.

[Vinayak Baddi (vbaddi)]

The retained conv_state layout changes from [B, conv_dim, K] (grouped [B, H, C/H, K])
to channel-last [B, K, conv_dim]. That changes the shape of the retained-state ONNX
inputs/outputs and QPC buffers,. But the export
hash is built from config, transform names and kwargs (modeling_qeff.py:412-431, 1192-1207);
it has no layout or code version. A user with an existing QEFF_HOME will get a cache hit and
reuse the old-layout ONNX/QPC, while the runtime now allocates [B, K, conv_dim] zeros
(modeling_qwen3_5.py:1378, 2188). The result is a shape mismatch at
inference time, or silently wrong state if K == conv_dim.
Repro:

1. On main, export and compile a tiny qwen3_5 (or tiny-random/qwen3.5-moe) with CB.
2. Checkout pr-1444 with the same QEFF_HOME and re-run export. The hash is identical, so the
   old ONNX is reused.
3. Compare the conv_state.* input shapes in the cached ONNX with the dummy inputs
   from get_dummy_inputs.

Ask: include the state-layout version in hash_params (or bump an export version) and
state in the PR description that previously exported/compiled QPCs and any external
state allocators (disagg/vLLM, _kv_dma_handoff users) must be regenerated.