Feature Request: vLLM Backend

[dxvsh]

Are there any plans for supporting vLLM as an additional inference engine apart from llama.cpp?

Llama.cpp is a great option for tinkering and playing around with language models but vLLM is a much better alternative for production workloads and it performs much better on multi-gpu setups thanks to vLLM's support for Tensor Parallelism. It would be great to see it included as an additional backend in model-runner.

[francesco-corti]

Hi,
Yes, we have it in our radar and we already did a spike.
Cannot commit on dates (yet) becuase of the fast changing priorities.

[ericcurtin]

If it helps... I did put together a build script for vLLM, is capable of building vLLM for CUDA and CPU inferencing for aarch64/x86_64

https://github.com/containers/ramalama/blob/main/container-images/scripts/build-vllm.sh

would be neat to run safetensors files and for those who have the beefier GPUs and want maximum performance:

https://developers.redhat.com/articles/2025/08/08/ollama-vs-vllm-deep-dive-performance-benchmarking

Ollama can be treated as llama.cpp performance I guess...

[ericcurtin]

Just saying also:

docker offload start
docker model run hf.co/openai/gpt-oss-20b # forks a vLLM server since this is a safetensors model

would be a killer feature that the vLLM/enterprise inferencing community is missing.

There are so many people that can't run vLLM locally because they don't have Nvidia hardware (I've heard all the complaints):

docker offload+docker model runner+vLLM+safetensors

is a great match for this.

vLLM is a little different to llama.cpp in that users cannot run it accelerated on macOS. docker offload would be very useful with vLLM, more useful than it is with llama.cpp because of hardware constraints.

[ericcurtin]

@dxvsh any interest in working on this? I'd take a crack at opening some PRs to get started but I'm off for the next month...

[ericcurtin]

Another complaint about vLLM usage, this time usability, I think docker model runner can solve this:

ollama/ollama#11714 (comment)