TL;DR: Mixture of experts (MoE) replaces parts of a transformer with many parallel "expert" subnetworks plus a router that picks a few of them per token. The model can hold a huge amount of knowledge while spending only a small model's compute on each token - which is how several frontier-class models keep serving costs down.
How it works
In a standard "dense" transformer, every token passes through every weight - all of the model works on all of the input, all the time. MoE observes that this is wasteful: the parameters useful for parsing Python are mostly not the ones useful for French poetry. So the architecture swaps each feed-forward block for a set of parallel expert subnetworks plus a small learned router. For every token at every MoE layer, the router scores all experts and sends the token through only the top few - commonly two of eight or more. The distinction that follows is between total parameters (all experts, what the model knows) and active parameters (the slice used per token, what you pay in compute).
That split is the entire commercial appeal: capacity scales with total parameters while per-token cost scales with active ones, so an MoE can know like a giant and run like a mid-size. The "experts" are not human-legible specialists - no clean "math expert" emerges - but routing is far from random: experts develop consistent affinities for kinds of tokens, and different experts fire on different layers of the same sentence. Routing decisions happen per token and per layer, thousands of times across one generation.
The catch is memory and orchestration. Every expert must sit in GPU memory even though most are idle for any given token, so an MoE needs the RAM of its total size while delivering the speed of its active size - a trade that favors well-provisioned inference servers and penalizes memory-poor local setups. Training brings its own headache: the router must be pushed (via auxiliary losses) to spread load across experts, or it collapses onto a favored few while the rest stay undertrained. Serving efficiently at scale means balancing tokens across experts that may live on different GPUs.
MoE went from research curiosity to default choice for large models because the economics won: several prominent open-weights families - from Mistral to DeepSeek - are MoE designs, and it is widely used in frontier lab models. For a developer the takeaway is mostly interpretive: when a model card quotes "active" versus "total" parameters, that is MoE talking, and small active counts explain how big models got cheap.
Where it sits in the AI stack
MoE lives inside the model's layers - a routing decision made for every token on its way through the network:
Key tools and implementations
-
Mixtral
The open-weights release that put sparse MoE on every practitioner's radar.
-
DeepSeek's MoE models
Fine-grained expert designs with very low active-to-total ratios, driving down serving cost.
-
Switch Transformer
Google's early large-scale demonstration that sparse routing could work at trillion scale.
-
Expert-parallel serving
Inference-engine support for spreading experts across GPUs and balancing token traffic.
Related entries
- Transformer The neural network architecture, built on attention, that underpins nearly every modern language model.
- LLM (large language model) A neural network trained on huge amounts of text to predict the next token, which lets it generate and understand language.
- DeepSeek A Chinese AI lab known for open-weight models with strong reasoning and unusually low training costs.
- Mistral A French AI lab whose model family spans open-weight downloads and a commercial API, with a focus on efficiency.
- Inference Running a trained model to produce output - the phase where every user request is actually served.