TL;DR: Quantization stores a model's weights in lower-precision numbers - fewer bits each - so the model takes less memory and runs faster, at a small cost in quality. Halving precision halves the memory footprint. It is the main reason capable models now run on laptops and phones instead of only datacenter GPUs.
How it works
A model is billions of numbers, each typically trained in 16-bit floating point. Memory is just arithmetic: a 70-billion-parameter model at 16 bits needs about 140 GB - beyond any consumer GPU. Quantization maps each weight to a smaller representation: 8 bits halves the footprint, 4 bits quarters it, bringing that same model to roughly 35 GB, inside reach of a high-end workstation. Conceptually it is rounding on a grid: within each block of weights you record a scale, then snap every value to the nearest of a limited set of levels. Each weight moves slightly; the model gets smaller and slightly blurrier.
The surprise is how little quality this costs. Networks are redundant, and errors partially cancel across billions of weights: 8-bit quantization is generally indistinguishable from the original, and well-executed 4-bit typically costs only a few percent on benchmarks. Push further - 3-bit, 2-bit - and degradation gets obvious, with small models suffering more than large ones. Modern schemes protect the handful of outlier weights that matter most, keeping them at higher precision while compressing the rest.
Speed improves too, for a structural reason: token generation during inference is bottlenecked on streaming weights through memory, not on arithmetic - so halving the bytes moves you meaningfully toward doubling generation speed on the same hardware. This is the engine of the local LLM ecosystem: the models you pull with Ollama are quantized builds, usually in the GGUF format, offered at several precision levels so you can trade quality against your available RAM.
Two practical notes. Most quantization is post-training: a finished checkpoint is converted in minutes to hours, no retraining required, which is why community quantizations of new open-weights models appear almost immediately. And quality loss is task-dependent - a lightly degraded model may chat flawlessly yet fumble edge-case code or arithmetic, so evaluate the quantized build on your workload rather than trusting a benchmark table.
Where it sits in the AI stack
Quantization is a compression step between a trained checkpoint and the hardware that serves it:
Key tools and implementations
-
GGUF / llama.cpp
The de facto format and runtime for quantized models on consumer hardware.
-
bitsandbytes
The library that made 8-bit and 4-bit loading a one-line change in Python stacks.
-
GPTQ and AWQ
Calibration-based methods that squeeze models to 4 bits with minimal quality loss.
-
QLoRA
Fine-tunes on top of a quantized base model, putting training within reach of one GPU.
Related entries
- Local LLM Running a language model entirely on hardware you control instead of calling a hosted API.
- GPUs for AI The parallel processors that run neural networks, where memory size matters as much as raw speed.
- Distillation Training a smaller model to imitate a larger one, keeping much of the capability at far lower cost.
- Ollama A tool that downloads open-weight models and runs them locally with a single command and a local API.