feat(mtmd): add Qwen2VL SMT patch preprocessing

[LFF28]

feat(mtmd): add Qwen2VL SMT patch preprocessing

Overview

This PR adds a Qwen2VL-specific SMT vision preprocessing path for ONNX vision encoders exported from model.visual.

Qwen2VL model.visual does not consume a normal CHW image tensor. It expects the processor output tensor shaped as:

[grid_h * grid_w, 3 * temporal_patch_size * 14 * 14]

The SMT preprocessor now detects qwen2vl / qwen2_vl architectures and converts resized RGB input into the flattened patch tensor expected by the exported ONNX model. The path applies the configured rescale and normalization values, then lays out patches in Qwen2VL merge order instead of returning a CHW tensor.

This enables the MinerU2.5 split deployment path:

text side:   GGUF model loaded by llama.cpp
vision side: Qwen2VL visual ONNX loaded through the SMT vision backend

Additional information

Companion deployment/export project:

https://gitlab.dc.com:8443/liangjunzhao/mineru2.5-split

Model artifacts have been uploaded externally and are not included in this PR:

https://archive.spacemit.com/spacemit-ai/model_zoo/vlm/

Validation context:

• Model: MinerU2.5-Pro-2605-1.2B
• Text model: mineru2.5-text-Q4_1.gguf
• Vision models:
  • mineru2.5-vision-224.f16.onnx
  • mineru2.5-vision-504.f16.onnx
  • shared external data: mineru2.5-vision-shared.f16.data
• Current deployment config uses the 504 static vision graph.
• 224 and 504 ONNX graphs share the same external initializer data; the graph files differ by static input/output shapes.

The 1008 static vision export was attempted but could not complete on the available local machine configuration. The 504 static graph was exported and validated on the target device.

Requirements

• git diff --check
• cmake --build build --target llama-server -j 8
• Verified that tools/mtmd/smt-vision-preprocess.cpp compiles as part of llama-server
• Verified the MinerU2.5 504 deployment starts with SMT media backend and /health returns {"status":"ok"}
• Verified input.pdf inference through the split GGUF + ONNX deployment on the target device
• Model binaries and exported ONNX/GGUF/data files are archived externally, not committed to source

[Copilot]

Pull request overview

Adds a Qwen2VL-specific vision preprocessing path to the SMT (ONNX) vision pipeline so that Qwen2VL model.visual exports can consume the flattened, merged patch tensor layout they expect (instead of a CHW image tensor). This supports split deployments where llama.cpp runs the text GGUF while SMT runs the Qwen2VL vision ONNX.

Changes:

• Extend SMT vision preprocessing spec resolution to detect qwen2vl / qwen2_vl architectures and enable a new patch-flatten mode.
• Implement RGB->normalized CHW conversion + Qwen2VL patch flattening in Qwen2VL merge order, producing [grid_h * grid_w, 3 * temporal_patch_size * 14 * 14]-style packed float data.
• Route preprocessing output selection through the new qwen2vl_patch_flatten flag and reuse a single resolved preprocess_config.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.