Repository navigation
Conversation
| # cannot be rolled back by positions. Only an accepted-prefix pass commits them. | ||
| for i, kind in enumerate(self.config.layer_types): | ||
| if kind == "linear_attention": | ||
| past_key_values.conv_states[i] = torch.where( |
There was a problem hiding this comment.
Is it intentional that KV-attention cache is not rolled back on cache_commit=0 -> verification permanently writes KV slots?
There was a problem hiding this comment.
Yes, this is intentional. cache_commit controls persistence of the convolution and recurrent states, not attention KV writes. Verification writes KV at absolute positions but rejected suffix entries are not logically committed: positional masking excludes them during accepted-prefix replay, and subsequent verification overwrites those positions before reading them. Recurrent states cannot use this mechanism because they summarize the entire processed prefix, so verification restores them and replay advances only the accepted prefix. I can clarify this distinction in the comment.
| if target_layer_ids: | ||
| # A new prompt may reuse an already-loaded QPC. Unlike attention KV, | ||
| # recurrent state has no positional mask to hide the previous request. | ||
| reset = (text_position_ids[:, :1] == 0).reshape(-1, 1, 1, 1) |
There was a problem hiding this comment.
Is position 0 the only sentinel for "new prompt" ? could a continuation legitimately start at 0 and silently zero out state from the previous request?
There was a problem hiding this comment.
Yes, in the current DFlash flow, the first text position being 0 means we are starting a sequence. We use it to clear the convolution and recurrent states when reusing the same QPC for another prompt.
A continuation keeps its absolute positions. For example, after a 100-token prefix, the next block starts at position 100, not 0. Prefill chunks and accepted-prefix replay follow the same rule.
If a caller restarted positions at 0 while expecting to keep the previous state, this code would indeed clear it. Our current runtime does not do that. Supporting that behavior would need a separate reset signal rather than relying on position 0.
| if dflash_chunk_size is not None: | ||
| # DFlash uses a fixed multi-token shape, including prefix replay. | ||
| # Select at export time so no one-token recurrent branch is traced. | ||
| if self.gdn_chunk_size < dflash_chunk_size: |
There was a problem hiding this comment.
(for my understanding) what's the valid use case where gdn_chunk_size > dflash_chunk_size?
There was a problem hiding this comment.
One quick explanation about the idea we took to improve the performance of DFlash:
DFlash verifies a block of 16 (block_size) tokens at a time, but the existing GatedDeltaNet uses a chunk size of 64, which adds unnecessary padded computation. Our approach uses a 16-token chunk for DFlash and skips the single-token recurrent branch. This reduces unnecessary work and simplifies the exported graph, while keeping the original behavior unchanged for non-DFlash runs.
Now related to your question:
This happens with the default settings: gdn_chunk_size is 64, while dflash_chunk_size is 16 in this PR.
gdn_chunk_size is the existing GatedDeltaNet setting and is also used to create the masks and identity matrix. For DFlash, we reuse only their first 16×16 entries and pass chunk_size=16 to the calculation. We are not doing a 64-token calculation.
So this check only makes sure those precomputed tensors are large enough for the DFlash chunk. The sizes can be equal, but they do not have to be.
|
vjanfaza Can you please resolve conflicts and update branch? |
|
vjanfaza get_dflash_block_size() has some ValueError paths where test case is missing, can you please add a test case for that? for this, missing ValueError cases worth testing: missing block_size, empty dflash_config, non-int values like "16", and values < 2. |
Enable DFlash for dense Qwen3.5 vision-language targets with fixed 16-token verification and accepted-prefix cache replay. DFlash2 and Qwen3.5-MoE support are intentionally excluded. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Cover missing block_size, empty dflash_config, non-integer values, and values below two in both supported config locations. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
aaa2a9b to
d394ef8
Compare
Extend the existing Qwen3.5 DFlash approach to Qwen3.5-MoE, reusing the same hybrid-cache handling and accepted-prefix replay. Keep the existing MoE routing and expert-parallel implementation unchanged. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Done! |
Done! |
Select only the last valid position before the LM head during DFlash prefill, while retaining all verification positions and DLM target features. Keep non-DFlash behavior unchanged. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Co-authored-by: Mohammed Faizanulla Azeez <mazeez@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
9070cf5 to
5028468
Compare
|
CI-Ready |
Reuse the scripted gated-delta chunk scan for larger prefill lengths while keeping verification and accepted-prefix replay block-sized. Update compile/runtime validation and existing regression tests. Keep non-DFlash behavior unchanged. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Return vocabulary logits for Qwen3-VL, Qwen3.5, and Qwen3.5-MoE DFlash targets, and reserve Gemma4 sliding-cache headroom for block verification. Load draft embeddings from the VLM target's nested language model and distinguish these exports in the artifact cache. These changes are limited to DFlash paths; existing non-DFlash outputs and cache behavior are unchanged. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
| ones_lower=self._ones_lower[:dflash_chunk_size, :dflash_chunk_size], | ||
| eye=self._eye[:dflash_chunk_size, :dflash_chunk_size], | ||
| use_matrix_solve=getattr(self, "dflash_matrix_solve", True), | ||
| use_dynamic_chunk_loop=True, |
There was a problem hiding this comment.
What is dflash_chunk_gated_delta_scan? is it TorchScript-compiled, and why does it generalize over variable chunk counts when the Python for i in range(total_sequence_length // chunk_size) loop doesn't?
There was a problem hiding this comment.
Yes, it is a @torch.jit.script helper that performs the same recurrent calculation across chunks. Its purpose is to preserve the loop during ONNX export, not to change the attention calculation.
The regular Python loop works in eager execution, but our tracing-based exporter unrolls it using the export example’s chunk count. Marking the sequence dimension dynamic does not make that unrolled loop dynamic.
The scripted helper preserves an ONNX Loop, with the iteration count taken from query.shape[2]. For example, with chunk size 16, the same exported graph can support one chunk for verification and eight chunks for a 128-token prefill. The QAIC compiler still specializes the graph for the requested sequence lengths.
| raise ValueError("Qwen3.5 DFlash requires prefill_seq_len >= 2 for the exported gated-delta graph.") | ||
| # Keep tracing small; the scripted scan preserves the dynamic chunk count. | ||
| chunk_size = getattr(self.model.language_model, "dflash_chunk_size", QWEN3_5_DFLASH_CHUNK_SIZE) | ||
| dummy_seq_len = min(dummy_seq_len, chunk_size) |
There was a problem hiding this comment.
Export always traces a single chunk regardless of prefill_seq_len. How does the ONNX graph handle a prefill_seq_len=256 at runtime if it was traced with seq_len=16 ? is the dynamic iteration count baked into the scripted scan's loop, and does the compiler see a fixed or symbolic trip count?
There was a problem hiding this comment.
The 16-token input is only the export example; it does not fix the ONNX sequence length. seq_len is marked dynamic, and reshaping into fixed-size chunks preserves a variable chunk dimension.
The scripted scan exports an ONNX Loop whose trip count comes from query.shape[2]. The graph contains that shape-derived calculation, not a constant iteration count of 1 from the export example.
The compiler initially receives a symbolic trip count. Once the compile specialization sets seq_len=256, the count is statically determined as 256 / 16 = 16 iterations. The verification specialization with seq_len=16 needs one iteration. Whether the compiler unrolls those iterations is a backend optimization decision.
At runtime, we use the matching compiled specialization. This does not mean a QPC compiled only for 16 tokens can accept 256 tokens without recompilation.
| ) | ||
| target_seq_len = prompt_chunk_size if is_qwen35 else block_size | ||
| if tlm_config.model_type == "qwen3_5_moe": | ||
| target_seq_len = block_size |
There was a problem hiding this comment.
Previously Qwen3.5 dense set target_seq_len = prompt_chunk_size (so TLM was called with the full prefill length per verification step). Now target_seq_len = block_size unconditionally and get_qwen35_target_seq_len is used for both dense and MoE. Just confirming: for dense, TLM verification is now always block-sized regardless of prefill length -> is that correct, and is this the intended behavioral change?
There was a problem hiding this comment.
Yes, that is correct and intentional for both dense and MoE Qwen3.5. Prompt prefill uses prompt_chunk_size, while verification uses block_size. For example, with prefill length 256 and block size 16, prefill processes 256 tokens per call, but verification processes only 16.
Previously, dense verification was tied to the prefill length, which caused unnecessary padded computation when prefill was larger than the draft block. This change separates the two. Accepted-prefix replay also uses the block-sized shape, with unused positions masked.
get_qwen35_target_seq_len() checks that the QPC contains the required block-sized specialization and raises an error requesting recompilation if it does not. The acceptance and cache-commit logic remain unchanged.
| logger.info(f"[compile_qwen3vl_tlm] fc/hidden_norm injected (fc scale s={s:.6f})") | ||
| if is_qwen35: | ||
| # The fixed chunk and matrix solve change the graph, not its I/O names. | ||
| tlm_qeff.lang_model.hash_params["dflash_cache_protocol"] = "qwen35_chunk16_matrix_replay_v3" |
There was a problem hiding this comment.
This removes the qwen35_chunk16_matrix_replay_v3 fingerprint for dense Qwen3.5; the protocol is now set to qwen35_dynamic_replay_v1 in modeling_auto.py. Any QPCs compiled under the old protocol string will be treated as cache misses and need recompilation. Is that intentional (old graph is incompatible anyway), and should there be a user-facing note about this artifact invalidation?
There was a problem hiding this comment.
Yes, the cache invalidation is intentional. The new graph supports variable-length prefill with block-sized verification, so we should not automatically reuse the earlier fixed-chunk export. Moving the fingerprint into modeling_auto.py also applies it consistently outside the example script.
I agree that a short README note would help. This fingerprint change affects the Qwen3.5 DFlash TLM artifacts, not the model weights or non-DFlash runs.
Suggested note:
Recompilation: Qwen3.5 DFlash TLM artifacts from before the variable-length prefill update need re-export and recompilation. The updated cache key triggers this automatically through the normal compilation flow. When supplying QPC paths explicitly, replace the old TLM QPC with a newly compiled one.
An old QPC may still work with its original settings; the important point is that it should not be reused automatically for the updated graph.
| logits = self.model.lm_head(hidden_states).float() | ||
| predicted_token_ids = logits.argmax(dim=-1).to(torch.int32) | ||
| output_logits = (getattr(self, "qaic_config", None) or {}).get("speculative_model_type") == "target" | ||
| predictions = logits if output_logits else logits.argmax(dim=-1).to(torch.int32) |
There was a problem hiding this comment.
modeling_qwen3_vl.py: line 790 / modeling_qwen3_5.py: line 1981 / modeling_qwen3_5_moe.py: line 2033: When speculative_model_type == "target", predictions is float32 [batch, vocab_size] instead of int32 [batch, 1]. Does the vLLM adapter detect this via output binding name or index, and is the changed dtype and shape reflected in the ONNX output spec / compile options so the compiler doesn't reject it?
There was a problem hiding this comment.
The vLLM adapter looks up the output by the name "logits" through binding_index_map, rather than assuming a fixed output index. It reads the dtype from the compiled QPC binding and allocates matching buffers.
One clarification: the exported logits shape is [batch, num_logits_to_keep, vocab_size], while the greedy-token output is [batch, num_logits_to_keep]. Prefill specializes num_logits_to_keep=1; verification uses block_size.
This choice is made at export time, not switched at runtime within one QPC. ONNX records the selected output tensor’s dtype and rank, and modeling_auto.py marks its batch and num_logits_to_keep dimensions dynamic. Compilation then specializes those dimensions; there is no custom-I/O override forcing logits to the previous
int32 type.The adapter uses the final QPC dtype, so it also handles precision conversion during compilation. The dflash_output_logits hash flag separates this export from the greedy-token version, and the VLM adapter rejects QPCs returning token IDs instead of vocabulary logits.
| self.lang_model.hash_params["dflash_cache_protocol"] = "qwen35_dynamic_replay_v1" | ||
| self.lang_model.hash_params["dflash_chunk_size"] = 16 | ||
| if self.dflash_tlm and self.model.config.model_type == "gemma4": | ||
| self.lang_model.hash_params["dflash_cache_protocol"] = "gemma4_block_window_v1" |
There was a problem hiding this comment.
Is dflash_cache_protocol validated by the QAIC compiler or runtime, or is it purely an opaque artifact-cache fingerprint key? If validated, where is the set of valid strings defined?
There was a problem hiding this comment.
It is purely an internal artifact-cache fingerprint, not a QAIC compiler or runtime parameter. The string is included in hash_params to prevent reuse of an older exported graph when the DFlash implementation changes.
There is no set of valid strings or validation against them. The name describes the graph version; it does not select or validate runtime behavior. Changing it changes the artifact cache key and triggers a fresh export/compilation through the normal cache flow.
| config = AutoConfig.from_pretrained(tlm_repo, trust_remote_code=True, token=hf_token) | ||
| is_qwen35 = config.model_type == "qwen3_5" | ||
| is_qwen35_moe = config.model_type == "qwen3_5_moe" | ||
| qaic_config = {"target_layer_ids": tlm_target_ids} |
There was a problem hiding this comment.
The TLM qaic_config only includes target_layer_ids, but the example later injects fc/hidden_norm from dlm_repo, so the language export hash can collide across different DLM checkpoints and reuse a stale ONNX/QPC. Pass/hash the DLM projection source, or add a weight checksum to lang_model.hash_params before export.
There was a problem hiding this comment.
Thanks for pointing this out. I’ve added the DLM repository/path to the language export hash for the standalone VLM examples. This follows the existing vLLM target behavior and prevents different
DLM sources from sharing the same cached language export. Non-DFlash behavior is unchanged.
| "mxfp6_matmul": True, | ||
| "mxint8_kv_cache": True, | ||
| # Keep hybrid verification/replay state in FP16 until MXINT8 is validated. | ||
| "mxint8_kv_cache": not is_qwen35, |
There was a problem hiding this comment.
MXINT8 KV cache is disabled only for dense Qwen3.5, while Qwen3.5-MoE takes the same hybrid verification/replay path and the adjacent comment says this state is not validated. Use not (is_qwen35 or is_qwen35_moe) or document why MoE is safe.
There was a problem hiding this comment.
We have tested MXINT8 with both dense and MoE models. I removed the exclusion and the outdated comment, so MXINT8 is now enabled by default for all supported DFlash models.
There was a problem hiding this comment.
The documented defaults do not match the CLI: basic_inference_vision.py:80 defaults --ctx_len to 4096 for all modes and basic_inference_vision.py:82 defaults --generation_len to 1800. Update the table.
There was a problem hiding this comment.
Fixed. The table now shows ctx_len=4096 for both scripts and generation_len=256 for the text script / 1800 for the vision script.
| grid_h, grid_w = resized_height // patch_size, resized_width // patch_size | ||
| grid_height = grid_h * grid_w | ||
| grid_width = patch_size * patch_size * temporal_patch_size * channel | ||
| vision_size = (grid_height // 4) * time |
There was a problem hiding this comment.
I only see MoE regression coverage for get_specializations(..., kv_offload=True), please add the same coverage for dense Qwen3.5 so this VLM compile path stays protected.
There was a problem hiding this comment.
Added dense Qwen3.5 coverage alongside the existing MoE tests for get_specializations(..., kv_offload=True), covering multiple resolutions, logger-adapter compatibility, and separate vision batch sizes. All six cases pass.
Hash the DLM projection source for standalone VLM targets, enable MXINT8 KV cache by default, and correct the documented CLI defaults. Extend existing VLM specialization tests to cover dense Qwen3.5 alongside MoE. Merge upstream/main at 118ee70 and initialize the non-DFlash flag in the upstream compile-test stubs. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
3de69ea to
bc44fde
Compare
Use baseline expert parallelism for GPT-OSS DFlash, with last-token prefill logits and block-sized verification. Fix target hidden-state normalization and keep sliding-window cache changes scoped to DFlash. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
4863280 to
f2c9447
Compare
| else None | ||
| ) | ||
| prompt_chunk_size = get_spd_prompt_chunk_size(tlm_prefill_session or tlm_session) | ||
| validate_spd_decode_specialization(tlm_session, config.block_size) |
There was a problem hiding this comment.
Here basic_inference_text.py still reads config.block_size directly and again at line 241. This bypasses the new get_dflash_block_size() helper, so DFlash configs that only define dflash_config.block_size will fail in the text path even though the vision path supports them. Please compute block_size = get_dflash_block_size(config) once and use it for validation and run_spd_inference_single().
There was a problem hiding this comment.
You are right, I missed these two accesses in the text example. Updated it to compute block_size = get_dflash_block_size(config) once and reuse it for validation and run_spd_inference_single(). This supports both top-level and nested dflash_config.block_size, consistent with the vision path
| ``` | ||
|
|
||
| Use `--prompt "..."` instead of `--image --image_prompt ...` for text-only input. | ||
| Prefill length can be any positive multiple of the draft block size; verification and replay remain block-sized. |
There was a problem hiding this comment.
The previous thread agreed to document that older Qwen3.5 DFlash TLM artifacts/QPCs need re-export/recompile after the variable-length prefill + block-sized verification change. I don’t see that note in the final README, so users supplying old --tlm_qpc paths can still accidentally reuse incompatible artifacts. Please add the recompilation note near the Qwen3.5 section or “Skipping compile” section.
Use the shared block-size helper once in the text example for both validation and inference. Document required re-export and recompilation of legacy Qwen3.5 DFlash TLM artifacts. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
There was a problem hiding this comment.
run_vision_encoder_qwen3_vl() is now unused and still assumes every vision QPC returns deepstack_features, while the new Qwen3.5 path handles that conditionally. Either remove the helper or update it to mirror the new has_deepstack behavior so future callers don’t hit a KeyError.
|
vjanfaza Please address the comment I posted and resolve the conflicts to update the branch. I will trigger the CI once it is ready. |
860942f to
aac8ea6
Compare
Prefill optimization: the DLM consumes a full TLM prefill chunk in one call instead of several block-sized calls, reducing prefill overhead and TTFT. Keep one specialization with prefill-width target inputs and block-sized noise inputs and draft outputs. During decoding, unused target positions are padded with -1. Existing DLM QPCs retain their compiled target width. Disaggregated serving: provide the QEfficient-side support used by the vLLM DFlash integration for separate TLM prefill/decode QPCs and pipeline prefilling across Qwen3, Llama, GPT-OSS, Gemma4, Qwen3-VL, Qwen3.5, and Qwen3.5-MoE, including Qwen3.6 checkpoints. Keep recurrent-state and compiler-unrolled scan placement DFlash-gated. The vLLM and launcher integration is maintained separately. Remove the unused Qwen vision helper, merge upstream/main at 3736940, resolve the Qwen3.5 dense/MoE conflicts while retaining upstream export support, and align existing DFlash test fixtures. Local benchmarks and unfinished VLM batching changes are excluded. Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com> Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Combine accepted-prefix commitment and verification in one target call for Qwen3.5 dense and MoE models. Use twice the draft block width for target inputs while keeping block-sized logits and draft features. Update the export cache protocol, validate fixed-shape QPCs with folded logits selectors, and align the DFlash examples, timing labels, and regression coverage with the concatenated path. Preserve the existing BS=1 VLM restrictions. Validation: 175 CPU tests passed; hardware compilation and execution were not run for this push. Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
25e2629 to
560c02a
Compare
Summary
Add DFlash support for dense Qwen3.5 hybrid vision-language models through the
existing DFlash VLM workflow.
Qwen3.5 uses Gated Delta Net linear-attention layers. The target is therefore
exported with a fixed 16-token verification graph. Verification leaves target
cache state unchanged, and a separate accepted-prefix replay commits only the
accepted tokens. This keeps convolution and recurrent state aligned with the
speculative-decoding result.
Scope
Qwen/Qwen3.6-27B,Qwen/Qwen3.5-27B,Qwen/Qwen3.5-9B, andQwen/Qwen3.5-4B.Equally Contributors