TL;DR: The transformer is the neural network architecture behind essentially every modern AI model. Its core trick, attention, lets every token in a sequence look at every other token and decide which ones matter. That made models parallelizable enough to train at enormous scale, and scale is what made them good.
How it works
Before transformers, language models read text the way you read a ticker tape: one word at a time, carrying a running summary forward. Long sentences degraded that summary, and the sequential design meant training could not be parallelized well. The 2017 paper "Attention Is All You Need" replaced the running summary with attention: every token computes a relevance score against every other token in the input, then builds its representation as a weighted blend of the ones that matter. In "The dog chased its tail because it was excited", attention is what lets "it" bind strongly to "dog" rather than "tail" - no grammar rules required, just learned scores.
A transformer stacks this move dozens of times. Input text is split into tokens, each token becomes a vector (an embedding), and the vectors flow through repeated layers of attention plus small feed-forward networks. Each layer refines what every position "knows" about the whole sequence - early layers pick up syntax, later layers something closer to meaning. At the top, the model converts the final vector into a probability for every possible next token.
The reason this architecture won is economic as much as scientific. Because attention scores for all positions can be computed simultaneously, training parallelizes across thousands of GPUs - which made it practical to scale models and data by orders of magnitude. The same blueprint also generalized beyond text: vision, audio, and protein-structure models are all transformers with different inputs. Variants like mixture of experts change how much of the network runs per token, but the attention core remains.
The architecture's main cost is that attention compares every token with every other token, so work grows quickly as input length grows. That pressure is why context windows are finite and why serving long inputs is expensive - constraints the rest of the stack spends a lot of effort working around.
Where it sits in the AI stack
The transformer is the machinery inside the model box - the part that turns embedded tokens into predictions:
Key tools and implementations
-
Decoder-only transformers
The variant behind chat models - generates left to right, one token at a time.
-
Encoder models
Transformers that read whole inputs at once, powering embeddings, search, and classification.
-
Vision transformers
The same architecture applied to image patches instead of word tokens.
-
Hugging Face Transformers
The open-source library most teams use to load, run, and fine-tune transformer models.
Related entries
- LLM (large language model) A neural network trained on huge amounts of text to predict the next token, which lets it generate and understand language.
- Embedding A list of numbers capturing a piece of content's meaning, so similar things sit close together in vector space.
- Mixture of experts An architecture that activates only a few specialist subnetworks per token, so a huge model runs at a smaller model's cost.
- Context window The maximum amount of text, measured in tokens, that a model can consider in a single request.