TL;DR: Inference is using a trained model: your prompt goes in, the model's frozen weights process it on GPUs, tokens come out. Training happens once, at enormous cost; inference happens on every single request, forever - so nearly all of the latency, throughput, and cost engineering in production AI lives here.
How it works
A model's life has two phases. Training adjusts billions of weights against data until the model is good; it is done rarely and costs a fortune. Inference runs the finished model with its weights frozen: nothing is learned, the network simply computes outputs from inputs. When a chatbot answers you, that is inference. The distinction matters commercially because training cost is paid once while inference cost scales with every user - which is why so much of the stack, from quantization to specialized serving software, exists to make this phase cheaper and faster.
For a language model, one request has two distinct stages with different personalities. Prefill processes your entire prompt in parallel - compute-heavy but quick, and it determines the delay before the first token appears. Decode then generates the response one token at a time, each new token requiring another pass through the model; this stage is limited by how fast weights can be streamed through GPU memory, and it sets the pace at which text appears. That is why answers arrive as a stream rather than all at once, and why streaming UIs are the norm - showing tokens as they decode hides the total latency.
Serving efficiently at scale is a genuine systems problem. Because decode leaves GPU compute underused, servers batch many users' requests together to keep the hardware busy - the core trick behind engines like vLLM. Caches avoid recomputing repeated prompt prefixes. The metrics that matter to an application are time to first token, tokens per second, and cost per million tokens; these vary widely across providers and configurations for the same model.
Practically, a developer chooses where inference happens: a managed API from an inference provider (simplest, pay per token), rented GPUs running an open-weights model (more control, more ops), or fully local hardware (privacy, no per-call cost, capped capability). Most products start with an API and revisit the choice when volume, latency, or data constraints force the question.
Where it sits in the AI stack
Inference is the serving layer - the bridge between a finished model and every application request:
Key tools and implementations
-
Provider APIs
Hosted inference from Anthropic, OpenAI, Google, and aggregators - pay per token, zero ops.
-
vLLM
The leading open-source serving engine for running open-weights models at high throughput.
-
Ollama
The easy path to local inference - pull a model and run it on a laptop or workstation.
-
GPU clouds
Rentable accelerator infrastructure for teams serving their own models at scale.
Related entries
- Inference provider A company that runs AI models on its own hardware and sells access through an API.
- vLLM An open-source serving engine that squeezes high throughput out of GPUs with paged attention and continuous batching.
- Quantization Storing a model's weights in lower-precision numbers so it needs less memory and runs faster.
- GPUs for AI The parallel processors that run neural networks, where memory size matters as much as raw speed.