TL;DR: A single request leaves most of a GPU idle, so serving systems group many requests into one pass through the model - that grouping is batching. Modern engines do it continuously, adding and removing requests mid-flight, which is the single biggest reason one GPU can serve hundreds of users. The same word also names providers' discounted bulk-job APIs.
How it works
Generating one token means streaming the model's entire weights through the GPU's compute units - billions of parameters read from memory to produce a single word for a single user. Here is the key asymmetry: reading the weights is the expensive part, and the arithmetic per request is comparatively cheap. So if 64 requests ride along on the same pass, the weights are read once and 64 tokens come out, one per conversation. Throughput multiplies while the hardware does barely more work. That is batching, and it is why per-token prices from inference providers can be as low as they are - you are splitting the cost of each memory pass with everyone else on the GPU.
Static batching, the naive version, collects requests until a batch fills, runs them in lockstep, and returns everything when the longest one finishes. Its flaws are obvious in the tail: early arrivals wait for the batch to fill, and short answers wait for long ones - a five-word reply stuck behind an essay.
Continuous batching (also called in-flight batching) fixes both. Because generation happens one token step at a time anyway, the scheduler recomposes the batch at every step: a request that just arrived joins immediately, a request that just finished leaves immediately, and its slot is refilled from the queue. The GPU stays saturated with no lockstep penalty. Engines like vLLM pair this with paged memory management so the freed capacity actually holds more requests - and since responses are streamed token by token, each user still sees their answer flow the moment it starts.
The trade-off is latency versus throughput: deeper batches mean more tokens per second from the hardware but slightly slower steps for each individual request, and serving teams tune where to sit on that curve. Separately, providers sell batch APIs - submit thousands of requests, get results back within hours at a steep discount (commonly around half price). Same word, different mechanism: the discount exists because deferred jobs let the provider fill idle capacity on its own schedule. For eval suites, bulk classification, and any pipeline nobody is watching live, it is the easiest cost lever there is.
Where it sits in the AI stack
Batching lives in the serving engine's scheduler, between the incoming request queue and the GPU:
Key tools and implementations
-
vLLM
The engine that made continuous batching plus paged memory the standard recipe for open-weight serving.
-
SGLang
A rival engine with the same continuous-batching core and aggressive prefix reuse across requests.
-
TensorRT-LLM
NVIDIA's serving stack - in-flight batching tuned for maximum throughput on its own GPUs.
-
Provider batch APIs
Deferred bulk-job endpoints from the major providers, trading turnaround time for a large discount.
Related entries
- vLLM An open-source serving engine that squeezes high throughput out of GPUs with paged attention and continuous batching.
- Streaming (token streaming) Sending each generated token to the user as it is produced instead of waiting for the full reply.
- GPUs for AI The parallel processors that run neural networks, where memory size matters as much as raw speed.
- Token cost tracking Counting tokens per request and pricing them per model so AI spend is a metric, not a surprise bill.