TL;DR: vLLM is an open-source engine for serving language models to many users at once. Its core trick, PagedAttention, manages GPU memory the way an operating system manages RAM, which lets far more requests share one GPU. If you are self-hosting an open-weight model for a team or a product - rather than for yourself - vLLM is the default place to start.
How it works
When a model generates text, it keeps a running memory of the conversation so far - the KV cache - and that cache is what actually fills up a GPU. Naive servers reserve one big contiguous block per request, sized for the worst case, so most of the memory sits reserved but empty. vLLM's PagedAttention (introduced by researchers at UC Berkeley in 2023) splits the cache into small pages and allocates them on demand, exactly like virtual memory in an operating system. Wasted space drops from the majority of the cache to a few percent, and the reclaimed memory holds more concurrent requests.
The second pillar is continuous batching. Instead of waiting for a full batch of requests and running them in lockstep, vLLM admits new requests and retires finished ones at every generation step, so the GPU never idles while there is work queued. The two techniques compound: paging frees the memory, and continuous batching fills it. Together they are why one GPU running vLLM can serve many times the traffic of a naive server on identical hardware.
In practice, you point vLLM at an open-weight model - most anything on Hugging Face works - and it exposes an OpenAI-compatible HTTP API. That compatibility matters more than it sounds: any client library written for a hosted provider can talk to your vLLM server by changing one base URL, which makes moving a workload from a provider to your own GPUs (or back) a configuration change rather than a rewrite. It also supports quantized models, multi-GPU tensor parallelism for models too big for one card, prefix caching, and speculative decoding.
Knowing when vLLM is the wrong tool is just as useful. For one person chatting with a model on a laptop, Ollama is simpler and fits better on consumer hardware. vLLM assumes server-class GPUs and concurrent traffic; its payoff is throughput per dollar of hardware, which only matters once there is real load. It has serious competition in the same niche - SGLang and TensorRT-LLM trade wins with it on benchmarks - but vLLM's ecosystem and model coverage keep it the common default.
Where it sits in the AI stack
vLLM is the serving engine: the layer that turns a downloaded model and a GPU into an API your applications can call:
Key tools and implementations
-
SGLang
The closest rival - a serving engine with RadixAttention prefix caching, strong on agentic and structured workloads.
-
TensorRT-LLM
NVIDIA's own serving stack - peak performance on NVIDIA GPUs at the cost of more setup work.
-
Text Generation Inference
Hugging Face's production server, tightly integrated with its model hub and inference endpoints.
-
llama.cpp
The other end of the spectrum - single-user local inference on consumer hardware rather than server throughput.
Related entries
- Batching Grouping multiple requests into one GPU pass so serving hardware stays fully utilized.
- Ollama A tool that downloads open-weight models and runs them locally with a single command and a local API.
- GPUs for AI The parallel processors that run neural networks, where memory size matters as much as raw speed.
- Quantization Storing a model's weights in lower-precision numbers so it needs less memory and runs faster.