vLLM

“probably one of the reasons why vllm, sglang, the llamas etc. so often produce buggy output”

“vLLMs tool parsing is much more precise than for example llama cpp or Ollama”

“I really appreciate this. I hope LLM inference providers and Chinese LLM labs start using this latest vLLM version ASAP.”

“I'm just now working on switching to vllm since it is better for concurrent inference”

“I'm running vllm on k3s on the dgx and do not know at quick glance which model is which process”

“most have decent support for AMD, it's not that different from CUDA once you get it working”

“there is series of bugs in the vllm middleware that prevent GLM models from getting all of their reasoning content”

“vLLM seems to have a lot more gotchas and kludged together stuff once you get outside of anything straightforward.”

“things have improved since the days when even chat templates/tool parsers were problematic”

“It finally stabilized for me on 2 x H200 using a commit a few days before 0.28”

“there was an issue of abysmal performance of models on Strix Halo, that is open for half a year”

“I mostly end up using vLLM these days and I'm a bit more performance sensitive”

“vLLM which required me to reinstall CUDA drivers because by default it installed the latest one”

“patching up vllm to run one or two models you care about on AMD hardware is a much easier proposition”

“And that's before you get hooked on vLLM's PagedAttention and can run multiple sequences without a ton of extra overhead.”

“separating the web server and GPU process, continuous batching, kv caching / chunking, and a huge model library including low precision mattered more”

“mature great open source tools that don't require a anti-consumer company's hands in the jar”

“they've ditched the Jinja templates for an in-engine parser and I've seen no issues”

“I have also seen a 2-3x improvement in then speeds.from improvements in vllm (mainly MTP).”

“vLLM will just fill whatever VRAM is available, no matter the context size, or the model size”

“The parallelism and MTP on top of better tuned kernels gave an extremely nice boost!”

“vllm is nicer (ray, handles queueing, doesn't have the cache invalidation bug for qwen/gemma models)”

“it's slow to start, it's Linux only, it doesn't like WSL much, ended up with either old or nightly container builds, I more or less have given up”

“It took a good 4 minutes plus to load up on the 2x 3090 rig, and served a single request 3 tokens/second slower.”

“I've tried every compatible vLLM version + currently using it from git (#main) yet the issue persists.”

“On vllm with a 5090 I get 120-180TPS with the awq 4 bit quant + MTP speculative decoding”

“I've heard that vLLM performs much better, scaling particularly better in the multi GPU case.”

“I also got vLLM to run in a rocm container, but I didn't have time to to diagnose perf problems”

“If you want to use multiple GPUs you might want to switch to vLLM or something else.”

“vLLM's continuous batching helps, but there's still a physical limit with shared VRAM/compute.”

“The big advantage with vLLM for OCR models is its continuous batching capability”

“do tensor parallelism and concurrent batch requests with vllm and it becomes extremely cost competitive”

“I tried FP8 in vLLM and it used 110GB and then my machine started to swap when I hit it with a query.”

“You can get concurrency gains as local/single user (multi-agent) use case with vLLM”

“vLLM can squeeze performance out of your GB10 or GPU that you wouldn't get otherwise”

“large performance difference between llama.cpp and vLLM for the models I use”

“It took me quite some time to figure the magic combination of versions and commits”

“vLLM ususally only plays out its strength when serving multiple users in parallel”

“it's drastically slower than Ollama (tested over a few weeks, always using the official AMD vLLM nightly releases)”

“I've actually found the highlight -> explain tool to be useful when sent to my local vLLM instance”