TL;DR: A local LLM is a language model running entirely on hardware you control - a laptop, a workstation, or a server in your rack - instead of behind a provider's API. Your data never leaves the machine, it works offline, and there is no per-token bill. The trade: local models are smaller and slower than frontier hosted ones, and the hardware and upkeep are on you.
How it works
Running a model locally starts with an open-weight model - one whose trained parameters are published for download, like Llama, Qwen, Mistral, or Gemma. A runtime such as Ollama or llama.cpp loads those weights into memory and runs the same token-by-token generation loop a hosted provider would, just on your silicon. A concrete example: a lawyer can point a local model at privileged documents and ask questions about them, knowing nothing ever crosses the network.
The binding constraint is memory. A model's weights must fit in your GPU's VRAM (or in unified memory on Apple silicon) with room left over for the working state of the conversation. This is where quantization earns its keep: storing weights at 4-bit precision instead of 16-bit shrinks a model to roughly a quarter of its size with a modest quality cost, which is the difference between a model fitting on a consumer GPU and not running at all. As a rough rule, a 7-8B parameter model runs comfortably on 8 GB of memory once quantized; bigger models scale up from there.
Why bother? Four reasons come up again and again: privacy (medical, legal, and proprietary data that cannot leave the building), offline use (planes, ships, air-gapped networks), cost (a model you run has no meter on it, which matters for high-volume batch work), and control (the model never changes underneath you, gets deprecated, or rate-limits you).
The honest downside is capability. The models that fit on personal hardware are far smaller than frontier hosted models, and it shows on hard reasoning and long documents. Local LLMs shine for drafting, summarization, classification, code completion, and private question-answering - and many teams run a hybrid: local models for routine private work, a hosted API for the hard problems. Scaling up is a spectrum, too; the far end of "local" is self-hosting on rented GPUs with a serving engine like vLLM, which trades the simplicity of a laptop for real multi-user throughput.
Where it sits in the AI stack
A local LLM collapses the provider layer into your own machine - the whole serving path runs on hardware you own:
Key tools and implementations
-
Ollama
The easiest on-ramp - pull and run a model with one command, then talk to it through a local API.
-
llama.cpp
The C/C++ engine underneath much of the local ecosystem, built for quantized models on modest hardware.
-
LM Studio
A desktop app for browsing, downloading, and chatting with local models - no terminal required.
-
vLLM
The step up for self-hosting on server GPUs when several users or services share one model.
Related entries
- Ollama A tool that downloads open-weight models and runs them locally with a single command and a local API.
- GPUs for AI The parallel processors that run neural networks, where memory size matters as much as raw speed.
- Quantization Storing a model's weights in lower-precision numbers so it needs less memory and runs faster.
- Open-weights models Models whose trained weights are published for anyone to download, run locally, and fine-tune.