Category
Where the models actually run
Training gets the headlines, but inference is where AI meets users: every answer is a model running on real hardware somewhere. These entries map the options - hosted provider APIs, self-hosted serving engines like vLLM, and local runners like Ollama - along with the techniques that make serving fast and affordable: batching, prompt caching, streaming, and structured output.
Entries in this category
- Batching Grouping multiple requests into one GPU pass so serving hardware stays fully utilized. Read the entry
- GPUs for AI The parallel processors that run neural networks, where memory size matters as much as raw speed. Read the entry
- Inference provider A company that runs AI models on its own hardware and sells access through an API. Read the entry
- Local LLM Running a language model entirely on hardware you control instead of calling a hosted API. Read the entry
- Ollama A tool that downloads open-weight models and runs them locally with a single command and a local API. Read the entry
- Prompt caching Reusing the computed state of a repeated prompt prefix so later requests skip that work. Read the entry
- Streaming (token streaming) Sending each generated token to the user as it is produced instead of waiting for the full reply. Read the entry
- Structured output Forcing a model's reply to match a schema, such as valid JSON, so software can parse it reliably. Read the entry
- vLLM An open-source serving engine that squeezes high throughput out of GPUs with paged attention and continuous batching. Read the entry
Where this layer fits
Inference is the delivery layer of the stack. Upstream, models define what is being served and core concepts like tokens and quantization explain the raw mechanics. Downstream, evals and observability tell you whether all that serving machinery is producing good answers at a sane cost.
Not sure where to start? Inference providers frames the hosted-versus-self-hosted decision that everything else here builds on. Or browse everything at once in the A-Z index.