Skip to content

Add DFlash support for Qwen3.5 hybrid VLMs - #1389

Open
vjanfaza wants to merge 12 commits into
qualcomm:mainfrom
vjanfaza:dflash-qwen3_5
Open

vjanfaza wants to merge 12 commits into
qualcomm:mainfrom
vjanfaza:dflash-qwen3_5

Conversation

@vjanfaza

@vjanfaza vjanfaza commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add DFlash support for dense Qwen3.5 hybrid vision-language models through the
existing DFlash VLM workflow.

Qwen3.5 uses Gated Delta Net linear-attention layers. The target is therefore
exported with a fixed 16-token verification graph. Verification leaves target
cache state unchanged, and a separate accepted-prefix replay commits only the
accepted tokens. This keeps convolution and recurrent state aligned with the
speculative-decoding result.

Scope

  • Add the Qwen3.5 and Qwen3.5-MoE target-side DFlash execution path.
  • Add per-layer sliding/full-attention masks for the DFlash draft model.
  • Add Qwen3.5 and Qwen3.5-MoE VLM compile and inference support to the existing DFlash example.
  • Support the validated Qwen/Qwen3.6-27B, Qwen/Qwen3.5-27B, Qwen/Qwen3.5-9B, and Qwen/Qwen3.5-4B.

Equally Contributors

# cannot be rolled back by positions. Only an accepted-prefix pass commits them.
for i, kind in enumerate(self.config.layer_types):
if kind == "linear_attention":
past_key_values.conv_states[i] = torch.where(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional that KV-attention cache is not rolled back on cache_commit=0 -> verification permanently writes KV slots?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is intentional. cache_commit controls persistence of the convolution and recurrent states, not attention KV writes. Verification writes KV at absolute positions but rejected suffix entries are not logically committed: positional masking excludes them during accepted-prefix replay, and subsequent verification overwrites those positions before reading them. Recurrent states cannot use this mechanism because they summarize the entire processed prefix, so verification restores them and replay advances only the accepted prefix. I can clarify this distinction in the comment.

if target_layer_ids:
# A new prompt may reuse an already-loaded QPC. Unlike attention KV,
# recurrent state has no positional mask to hide the previous request.
reset = (text_position_ids[:, :1] == 0).reshape(-1, 1, 1, 1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is position 0 the only sentinel for "new prompt" ? could a continuation legitimately start at 0 and silently zero out state from the previous request?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, in the current DFlash flow, the first text position being 0 means we are starting a sequence. We use it to clear the convolution and recurrent states when reusing the same QPC for another prompt.

A continuation keeps its absolute positions. For example, after a 100-token prefix, the next block starts at position 100, not 0. Prefill chunks and accepted-prefix replay follow the same rule.

If a caller restarted positions at 0 while expecting to keep the previous state, this code would indeed clear it. Our current runtime does not do that. Supporting that behavior would need a separate reset signal rather than relying on position 0.

if dflash_chunk_size is not None:
# DFlash uses a fixed multi-token shape, including prefix replay.
# Select at export time so no one-token recurrent branch is traced.
if self.gdn_chunk_size < dflash_chunk_size:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(for my understanding) what's the valid use case where gdn_chunk_size > dflash_chunk_size?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One quick explanation about the idea we took to improve the performance of DFlash:
DFlash verifies a block of 16 (block_size) tokens at a time, but the existing GatedDeltaNet uses a chunk size of 64, which adds unnecessary padded computation. Our approach uses a 16-token chunk for DFlash and skips the single-token recurrent branch. This reduces unnecessary work and simplifies the exported graph, while keeping the original behavior unchanged for non-DFlash runs.

Now related to your question:
This happens with the default settings: gdn_chunk_size is 64, while dflash_chunk_size is 16 in this PR.
gdn_chunk_size is the existing GatedDeltaNet setting and is also used to create the masks and identity matrix. For DFlash, we reuse only their first 16×16 entries and pass chunk_size=16 to the calculation. We are not doing a 64-token calculation.
So this check only makes sure those precomputed tensors are large enough for the DFlash chunk. The sizes can be equal, but they do not have to be.

@quic-mamta

Copy link
Copy Markdown
Contributor

vjanfaza Can you please resolve conflicts and update branch?

@Shrubabati7

Shrubabati7 commented Oct 12, 2026 •

Copy link
Copy Markdown

vjanfaza get_dflash_block_size() has some ValueError paths where test case is missing, can you please add a test case for that? for this, missing ValueError cases worth testing: missing block_size, empty dflash_config, non-int values like "16", and values < 2.

vjanfaza and others added 2 commits October 12, 2026 11:56
Enable DFlash for dense Qwen3.5 vision-language targets with fixed 16-token verification and accepted-prefix cache replay.

DFlash2 and Qwen3.5-MoE support are intentionally excluded.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Cover missing block_size, empty dflash_config, non-integer values, and values below two in both supported config locations.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Extend the existing Qwen3.5 DFlash approach to Qwen3.5-MoE, reusing the same hybrid-cache handling and accepted-prefix replay. Keep the existing MoE routing and expert-parallel implementation unchanged.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
@vjanfaza

Copy link
Copy Markdown
Contributor Author

vjanfaza Can you please resolve conflicts and update branch?

Done!

@vjanfaza

Copy link
Copy Markdown
Contributor Author

vjanfaza get_dflash_block_size() has some ValueError paths where test case is missing, can you please add a test case for that? for this, missing ValueError cases worth testing: missing block_size, empty dflash_config, non-int values like "16", and values < 2.

Done!

Select only the last valid position before the LM head during DFlash
prefill, while retaining all verification positions and DLM target
features. Keep non-DFlash behavior unchanged.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Co-authored-by: Mohammed Faizanulla Azeez <mazeez@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
@quic-mamta

Copy link
Copy Markdown
Contributor

CI-Ready

vjanfaza and others added 2 commits October 14, 2026 11:59
Reuse the scripted gated-delta chunk scan for larger prefill lengths
while keeping verification and accepted-prefix replay block-sized.
Update compile/runtime validation and existing regression tests.
Keep non-DFlash behavior unchanged.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Return vocabulary logits for Qwen3-VL, Qwen3.5, and Qwen3.5-MoE
DFlash targets, and reserve Gemma4 sliding-cache headroom for block
verification.

Load draft embeddings from the VLM target's nested language model and
distinguish these exports in the artifact cache.

These changes are limited to DFlash paths; existing non-DFlash outputs
and cache behavior are unchanged.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
ones_lower=self._ones_lower[:dflash_chunk_size, :dflash_chunk_size],
eye=self._eye[:dflash_chunk_size, :dflash_chunk_size],
use_matrix_solve=getattr(self, "dflash_matrix_solve", True),
use_dynamic_chunk_loop=True,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is dflash_chunk_gated_delta_scan? is it TorchScript-compiled, and why does it generalize over variable chunk counts when the Python for i in range(total_sequence_length // chunk_size) loop doesn't?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is a @torch.jit.script helper that performs the same recurrent calculation across chunks. Its purpose is to preserve the loop during ONNX export, not to change the attention calculation.

The regular Python loop works in eager execution, but our tracing-based exporter unrolls it using the export example’s chunk count. Marking the sequence dimension dynamic does not make that unrolled loop dynamic.

The scripted helper preserves an ONNX Loop, with the iteration count taken from query.shape[2]. For example, with chunk size 16, the same exported graph can support one chunk for verification and eight chunks for a 128-token prefill. The QAIC compiler still specializes the graph for the requested sequence lengths.

raise ValueError("Qwen3.5 DFlash requires prefill_seq_len >= 2 for the exported gated-delta graph.")
# Keep tracing small; the scripted scan preserves the dynamic chunk count.
chunk_size = getattr(self.model.language_model, "dflash_chunk_size", QWEN3_5_DFLASH_CHUNK_SIZE)
dummy_seq_len = min(dummy_seq_len, chunk_size)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Export always traces a single chunk regardless of prefill_seq_len. How does the ONNX graph handle a prefill_seq_len=256 at runtime if it was traced with seq_len=16 ? is the dynamic iteration count baked into the scripted scan's loop, and does the compiler see a fixed or symbolic trip count?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 16-token input is only the export example; it does not fix the ONNX sequence length. seq_len is marked dynamic, and reshaping into fixed-size chunks preserves a variable chunk dimension.

The scripted scan exports an ONNX Loop whose trip count comes from query.shape[2]. The graph contains that shape-derived calculation, not a constant iteration count of 1 from the export example.

The compiler initially receives a symbolic trip count. Once the compile specialization sets seq_len=256, the count is statically determined as 256 / 16 = 16 iterations. The verification specialization with seq_len=16 needs one iteration. Whether the compiler unrolls those iterations is a backend optimization decision.

At runtime, we use the matching compiled specialization. This does not mean a QPC compiled only for 16 tokens can accept 256 tokens without recompilation.

)
target_seq_len = prompt_chunk_size if is_qwen35 else block_size
if tlm_config.model_type == "qwen3_5_moe":
target_seq_len = block_size

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously Qwen3.5 dense set target_seq_len = prompt_chunk_size (so TLM was called with the full prefill length per verification step). Now target_seq_len = block_size unconditionally and get_qwen35_target_seq_len is used for both dense and MoE. Just confirming: for dense, TLM verification is now always block-sized regardless of prefill length -> is that correct, and is this the intended behavioral change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is correct and intentional for both dense and MoE Qwen3.5. Prompt prefill uses prompt_chunk_size, while verification uses block_size. For example, with prefill length 256 and block size 16, prefill processes 256 tokens per call, but verification processes only 16.

Previously, dense verification was tied to the prefill length, which caused unnecessary padded computation when prefill was larger than the draft block. This change separates the two. Accepted-prefix replay also uses the block-sized shape, with unused positions masked.

get_qwen35_target_seq_len() checks that the QPC contains the required block-sized specialization and raises an error requesting recompilation if it does not. The acceptance and cache-commit logic remain unchanged.

Comment thread examples/performance/dflash/utils.py Outdated
logger.info(f"[compile_qwen3vl_tlm] fc/hidden_norm injected (fc scale s={s:.6f})")
if is_qwen35:
# The fixed chunk and matrix solve change the graph, not its I/O names.
tlm_qeff.lang_model.hash_params["dflash_cache_protocol"] = "qwen35_chunk16_matrix_replay_v3"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes the qwen35_chunk16_matrix_replay_v3 fingerprint for dense Qwen3.5; the protocol is now set to qwen35_dynamic_replay_v1 in modeling_auto.py. Any QPCs compiled under the old protocol string will be treated as cache misses and need recompilation. Is that intentional (old graph is incompatible anyway), and should there be a user-facing note about this artifact invalidation?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the cache invalidation is intentional. The new graph supports variable-length prefill with block-sized verification, so we should not automatically reuse the earlier fixed-chunk export. Moving the fingerprint into modeling_auto.py also applies it consistently outside the example script.

I agree that a short README note would help. This fingerprint change affects the Qwen3.5 DFlash TLM artifacts, not the model weights or non-DFlash runs.

Suggested note:

Recompilation: Qwen3.5 DFlash TLM artifacts from before the variable-length prefill update need re-export and recompilation. The updated cache key triggers this automatically through the normal compilation flow. When supplying QPC paths explicitly, replace the old TLM QPC with a newly compiled one.

An old QPC may still work with its original settings; the important point is that it should not be reused automatically for the updated graph.

logits = self.model.lm_head(hidden_states).float()
predicted_token_ids = logits.argmax(dim=-1).to(torch.int32)
output_logits = (getattr(self, "qaic_config", None) or {}).get("speculative_model_type") == "target"
predictions = logits if output_logits else logits.argmax(dim=-1).to(torch.int32)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

modeling_qwen3_vl.py: line 790 / modeling_qwen3_5.py: line 1981 / modeling_qwen3_5_moe.py: line 2033: When speculative_model_type == "target", predictions is float32 [batch, vocab_size] instead of int32 [batch, 1]. Does the vLLM adapter detect this via output binding name or index, and is the changed dtype and shape reflected in the ONNX output spec / compile options so the compiler doesn't reject it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The vLLM adapter looks up the output by the name "logits" through binding_index_map, rather than assuming a fixed output index. It reads the dtype from the compiled QPC binding and allocates matching buffers.

One clarification: the exported logits shape is [batch, num_logits_to_keep, vocab_size], while the greedy-token output is [batch, num_logits_to_keep]. Prefill specializes num_logits_to_keep=1; verification uses block_size.

This choice is made at export time, not switched at runtime within one QPC. ONNX records the selected output tensor’s dtype and rank, and modeling_auto.py marks its batch and num_logits_to_keep dimensions dynamic. Compilation then specializes those dimensions; there is no custom-I/O override forcing logits to the previous
int32 type.

The adapter uses the final QPC dtype, so it also handles precision conversion during compilation. The dflash_output_logits hash flag separates this export from the greedy-token version, and the VLM adapter rejects QPCs returning token IDs instead of vocabulary logits.

self.lang_model.hash_params["dflash_cache_protocol"] = "qwen35_dynamic_replay_v1"
self.lang_model.hash_params["dflash_chunk_size"] = 16
if self.dflash_tlm and self.model.config.model_type == "gemma4":
self.lang_model.hash_params["dflash_cache_protocol"] = "gemma4_block_window_v1"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is dflash_cache_protocol validated by the QAIC compiler or runtime, or is it purely an opaque artifact-cache fingerprint key? If validated, where is the set of valid strings defined?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is purely an internal artifact-cache fingerprint, not a QAIC compiler or runtime parameter. The string is included in hash_params to prevent reuse of an older exported graph when the DFlash implementation changes.

There is no set of valid strings or validation against them. The name describes the graph version; it does not select or validate runtime behavior. Changing it changes the artifact cache key and triggers a fresh export/compilation through the normal cache flow.

Comment thread examples/performance/dflash/utils.py Outdated
config = AutoConfig.from_pretrained(tlm_repo, trust_remote_code=True, token=hf_token)
is_qwen35 = config.model_type == "qwen3_5"
is_qwen35_moe = config.model_type == "qwen3_5_moe"
qaic_config = {"target_layer_ids": tlm_target_ids}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TLM qaic_config only includes target_layer_ids, but the example later injects fc/hidden_norm from dlm_repo, so the language export hash can collide across different DLM checkpoints and reuse a stale ONNX/QPC. Pass/hash the DLM projection source, or add a weight checksum to lang_model.hash_params before export.

@vjanfaza vjanfaza Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. I’ve added the DLM repository/path to the language export hash for the standalone VLM examples. This follows the existing vLLM target behavior and prevents different
DLM sources from sharing the same cached language export. Non-DFlash behavior is unchanged.

Comment thread examples/performance/dflash/utils.py Outdated
"mxfp6_matmul": True,
"mxint8_kv_cache": True,
# Keep hybrid verification/replay state in FP16 until MXINT8 is validated.
"mxint8_kv_cache": not is_qwen35,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MXINT8 KV cache is disabled only for dense Qwen3.5, while Qwen3.5-MoE takes the same hybrid verification/replay path and the adjacent comment says this state is not validated. Use not (is_qwen35 or is_qwen35_moe) or document why MoE is safe.

@vjanfaza vjanfaza Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have tested MXINT8 with both dense and MoE models. I removed the exclusion and the outdated comment, so MXINT8 is now enabled by default for all supported DFlash models.

Comment thread examples/performance/dflash/README.md Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documented defaults do not match the CLI: basic_inference_vision.py:80 defaults --ctx_len to 4096 for all modes and basic_inference_vision.py:82 defaults --generation_len to 1800. Update the table.

@vjanfaza vjanfaza Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The table now shows ctx_len=4096 for both scripts and generation_len=256 for the text script / 1800 for the vision script.

grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
grid_height = grid_h * grid_w
grid_width = patch_size * patch_size * temporal_patch_size * channel
vision_size = (grid_height // 4) * time

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only see MoE regression coverage for get_specializations(..., kv_offload=True), please add the same coverage for dense Qwen3.5 so this VLM compile path stays protected.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added dense Qwen3.5 coverage alongside the existing MoE tests for get_specializations(..., kv_offload=True), covering multiple resolutions, logger-adapter compatibility, and separate vision batch sizes. All six cases pass.

Hash the DLM projection source for standalone VLM targets, enable MXINT8 KV cache by default, and correct the documented CLI defaults.

Extend existing VLM specialization tests to cover dense Qwen3.5 alongside MoE.

Merge upstream/main at 118ee70 and initialize the non-DFlash flag in the upstream compile-test stubs.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Use baseline expert parallelism for GPT-OSS DFlash, with last-token prefill logits and block-sized verification. Fix target hidden-state normalization and keep sliding-window cache changes scoped to DFlash.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
else None
)
prompt_chunk_size = get_spd_prompt_chunk_size(tlm_prefill_session or tlm_session)
validate_spd_decode_specialization(tlm_session, config.block_size)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here basic_inference_text.py still reads config.block_size directly and again at line 241. This bypasses the new get_dflash_block_size() helper, so DFlash configs that only define dflash_config.block_size will fail in the text path even though the vision path supports them. Please compute block_size = get_dflash_block_size(config) once and use it for validation and run_spd_inference_single().

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I missed these two accesses in the text example. Updated it to compute block_size = get_dflash_block_size(config) once and reuse it for validation and run_spd_inference_single(). This supports both top-level and nested dflash_config.block_size, consistent with the vision path

Comment thread examples/performance/dflash/README.md Outdated
```

Use `--prompt "..."` instead of `--image --image_prompt ...` for text-only input.
Prefill length can be any positive multiple of the draft block size; verification and replay remain block-sized.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous thread agreed to document that older Qwen3.5 DFlash TLM artifacts/QPCs need re-export/recompile after the variable-length prefill + block-sized verification change. I don’t see that note in the final README, so users supplying old --tlm_qpc paths can still accidentally reuse incompatible artifacts. Please add the recompilation note near the Qwen3.5 section or “Skipping compile” section.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed!

Use the shared block-size helper once in the text example for both validation and inference. Document required re-export and recompilation of legacy Qwen3.5 DFlash TLM artifacts.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_vision_encoder_qwen3_vl() is now unused and still assumes every vision QPC returns deepstack_features, while the new Qwen3.5 path handles that conditionally. Either remove the helper or update it to mirror the new has_deepstack behavior so future callers don’t hit a KeyError.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed!

@quic-mamta

Copy link
Copy Markdown
Contributor

vjanfaza Please address the comment I posted and resolve the conflicts to update the branch. I will trigger the CI once it is ready.

@vjanfaza
vjanfaza force-pushed the dflash-qwen3_5 branch 3 times, most recently from 860942f to aac8ea6 Compare October 27, 2026 21:06
vjanfaza and others added 2 commits October 28, 2026 16:39
Prefill optimization: the DLM consumes a full TLM prefill chunk in one call instead of several block-sized calls, reducing prefill overhead and TTFT. Keep one specialization with prefill-width target inputs and block-sized noise inputs and draft outputs. During decoding, unused target positions are padded with -1. Existing DLM QPCs retain their compiled target width.

Disaggregated serving: provide the QEfficient-side support used by the vLLM DFlash integration for separate TLM prefill/decode QPCs and pipeline prefilling across Qwen3, Llama, GPT-OSS, Gemma4, Qwen3-VL, Qwen3.5, and Qwen3.5-MoE, including Qwen3.6 checkpoints. Keep recurrent-state and compiler-unrolled scan placement DFlash-gated. The vLLM and launcher integration is maintained separately.

Remove the unused Qwen vision helper, merge upstream/main at 3736940, resolve the Qwen3.5 dense/MoE conflicts while retaining upstream export support, and align existing DFlash test fixtures. Local benchmarks and unfinished VLM batching changes are excluded.

Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Combine accepted-prefix commitment and verification in one target call for Qwen3.5 dense and MoE models. Use twice the draft block width for target inputs while keeping block-sized logits and draft features.

Update the export cache protocol, validate fixed-shape QPCs with folded logits selectors, and align the DFlash examples, timing labels, and regression coverage with the concatenated path. Preserve the existing BS=1 VLM restrictions.

Validation: 175 CPU tests passed; hardware compilation and execution were not run for this push.
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants