vLLM support broken on Windows with WSL2

[VIEWVIEWVIEW]

Hi,

I tried to follow the tutorial from https://www.docker.com/blog/docker-model-runner-vllm-windows/ for vLLM. Unfortunately it did not work, even after trying different models etc. It seems like vLLM support is completely broken on Windows, while models with the llama.cpp backend work fine.

Every vLLM model fails with this error:

ImportError: libtorch_cuda.so: cannot open shared object file: No such file or directory

Below are more verbose logs from two different models I tried. Since it's exactly the same error and it's occuring before the model loads, I assume it's a docker model runner-runtime issue.

gemma3-vllm

>docker model run ai/gemma3-vllm "Say hello in one short sentence."
Unable to find model 'ai/gemma3-vllm' locally. Pulling from the server.
~snip~
Model pulled successfully
Failed to generate a response: error response: status=500 body=unable to load runner: error waiting for runner to be ready: vLLM terminated unexpectedly: vLLM failed:     module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm-env/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 22, in <module>
    import vllm._C_stable_libtorch  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: libtorch_cuda.so: cannot open shared object file: No such file or directory
y", line 44, in is_pin_memory_available
    from vllm.platforms import current_platform
  File "/opt/vllm-env/lib/python3.12/site-packages/vllm/platforms/__init__.py", line 277, in __getattr__
    _current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm-env/lib/python3.12/site-packages/vllm/utils/import_utils.py", line 109, in resolve_obj_by_qualname

smollm2-vllm

> docker model run ai/smollm2-vllm "Tell me about Docker."
Failed to generate a response: error response: status=500 body=unable to load runner: error waiting for runner to be ready: vLLM terminated unexpectedly: vLLM failed:     module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm-env/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 22, in <module>
    import vllm._C_stable_libtorch  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: libtorch_cuda.so: cannot open shared object file: No such file or directory
y", line 44, in is_pin_memory_available
    from vllm.platforms import current_platform
  File "/opt/vllm-env/lib/python3.12/site-packages/vllm/platforms/__init__.py", line 277, in __getattr__
    _current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm-env/lib/python3.12/site-packages/vllm/utils/import_utils.py", line 109, in resolve_obj_by_qualname

Meanwhile, models with llama work fine:

>docker model run ai/smollm2 "whatsupp?"
Hi, I'm here to help with your programming tasks, such as debugging, troubleshooting, and providing guidance on various programming languages and concepts. What's your question or issue?

Environment

• Host OS: Windows 10 Pro, build 19045.7417
• Docker context: desktop-linux
• Docker Desktop: 4.82.0 (233772)
• Docker Engine: 29.6.1, Linux amd64
• Docker Model Runner client: v1.2.5
• Docker Model Runner server: v1.2.6
• Docker Desktop CLI plugin: v0.4.1
• WSL: 2.7.10.0
• WSL kernel: 6.18.33.2-2
• Default WSL distribution: Ubuntu, version 2
• GPU: NVIDIA GeForce RTX 4070 series, 12 GB VRAM
• NVIDIA driver: 610.47
• CUDA UMD version: 13.3
• Model Runner vLLM version: 0.24.0
• Runner image: docker/model-runner:latest-vllm-cuda
• Runner image digest: sha256:105c9bc23f9546de54bb898298bce94aeb75b4ee9e558212ae6260de03c8957b

docker model status

> docker model status
Docker Model Runner is running

BACKEND    STATUS         DETAILS
llama.cpp  Running        llama.cpp 72874f559
vllm       Running        vllm 0.24.0
diffusers  Not Installed
mlx        Not Installed  only supported on Apple Silicon
sglang     Not Installed  package not installed
PS C:\Users\w>

GPU access works from Docker/WSL2:

docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

This successfully reports the RTX 4070 GPU inside the Linux container.

Reproduction

docker model install-runner --backend vllm --gpu cuda
docker model status
docker model run ai/smollm2-vllm "Say hello in one short sentence."

Also reproduced with:

docker model run ai/gemma3-vllm "Say hello in one short sentence."

---

Btw, the documentation on this page has a hallucinated/invalid parameter: https://docs.docker.com/ai/model-runner/inference-engines/#running-models-with-vllm

--backend does not exist on docker model run.

> docker model run ai/gemma3-vllm --backend vllm
unknown flag: --backend

Usage:  docker model run MODEL [PROMPT]

Run 'docker model run --help' for more information

Output of help command does not list --backend either. I think it only exists for installing the runner backend, but not for explicitly choosing the backend when running a model.

> docker model run --help
Usage:  docker model run MODEL [PROMPT]

Run a model and interact with it using a submitted prompt or chat mode

Options:
      --color string       Use colored output (auto|yes|no) (default "no")
      --debug              Enable debug logging
  -d, --detach             Load the model in the background without
                           interaction
      --openaiurl string   OpenAI-compatible API endpoint URL to chat with
      --websearch          Enable web search tool during chat

[VIEWVIEWVIEW]

Seems like #952 is related, and the runner image is simply shipping with the wrong torch version.

> docker exec docker-model-runner /opt/vllm-env/bin/python -c "import torch; print('torch:', torch.__version__); print('CUDA build:', torch.version.cuda); print('CUDA available:', torch.cuda.is_available())"
torch: 2.11.0+cpu
CUDA build: None
CUDA available: False

[michaelmanly]

@VIEWVIEWVIEW
This looks like the runner image is shipping CPU-only Torch rather than a usable CUDA build.

As a workaround, you can run the same model remotely in a known-good GPU environment instead of debugging the WSL2 image:

badgr serve ai/smollm2-vllm \
  --gpu RTX-4090 \
  --max-cost 5

Badgr provisions the CUDA environment, starts the vLLM endpoint, health-checks it, and returns the URL and logs. That should help confirm whether the model itself is fine and the failure is isolated to Docker Model Runner’s Windows image.