TL;DR: GPUs run AI because neural networks are mostly enormous matrix multiplications, and a GPU performs thousands of those small calculations at once where a CPU does a handful. For language models the surprising bottleneck is memory, not math: how much VRAM a card has, and how fast it can move data, matters more than its raw compute.
How it works
A CPU is a few very fast cores built for sequential work; a GPU is thousands of simple cores built to do the same operation on huge arrays of numbers simultaneously. Graphics needed that shape - shade a million pixels at once - and neural networks turned out to need exactly the same thing, since running a model is layer after layer of matrix multiplication. That accident of geometry is why the graphics-card industry became the AI hardware industry.
For inference, two memory numbers dominate. Capacity (VRAM) sets what fits: the model's weights plus the KV cache - the growing per-conversation state that expands with every token of context. A 70B-parameter model needs roughly 140 GB at 16-bit precision, which is why quantization to 4-bit - cutting that to about 40 GB - decides whether a model fits on one card, needs several, or cannot run at all. Bandwidth sets speed: generating each token requires streaming essentially all the weights through the compute units, so tokens per second tracks memory bandwidth far more than it tracks raw FLOPS. Serving tricks like batching exist precisely to squeeze more useful work out of each pass through memory.
The hardware tiers follow from those numbers. Datacenter GPUs (NVIDIA's H100 generation and successors, AMD's Instinct line) offer the most memory and bandwidth plus fast interconnects for spreading one model across many cards - at prices only businesses pay. Consumer GPUs - RTX-class cards with 12 to 24 GB - run quantized mid-size models well and power most hobbyist local LLM setups. Apple silicon is the wildcard: unified memory means a Mac can hold a far larger model than any consumer GPU, though with lower bandwidth, so big models fit but generate more slowly.
GPUs are not the only answer. Google's TPUs and purpose-built inference chips from Groq, Cerebras, and others drop graphics heritage entirely and design for tensor math or token generation specifically. For most people the practical decision is simpler than the silicon: rent GPU time from a provider until utilization is high and steady enough that owning hardware wins, and let VRAM - not FLOPS - drive any card you do buy.
Where it sits in the AI stack
Hardware is the floor of the stack - everything above it ultimately resolves to weights moving through a GPU:
Key tools and implementations
-
Datacenter GPUs
NVIDIA's H-series and AMD's Instinct line - maximum memory, bandwidth, and interconnect for serving at scale.
-
Consumer GPUs
RTX-class cards with 12-24 GB of VRAM - the workhorses of local inference with quantized models.
-
Apple silicon
Unified memory lets Macs load very large models; bandwidth, not capacity, is the constraint.
-
Custom accelerators
Google TPUs and inference-first chips from Groq and Cerebras, designed around tensor math rather than graphics.
Related entries
- Local LLM Running a language model entirely on hardware you control instead of calling a hosted API.
- vLLM An open-source serving engine that squeezes high throughput out of GPUs with paged attention and continuous batching.
- Quantization Storing a model's weights in lower-precision numbers so it needs less memory and runs faster.
- Batching Grouping multiple requests into one GPU pass so serving hardware stays fully utilized.