TL;DR: A token is the unit a language model actually reads and writes - a chunk of characters that is usually most of a word, sometimes a whole one, sometimes just punctuation. Everything about working with models is priced and limited in tokens: API bills, context windows, and generation speed all count in them, not in words or characters.
How it works
Models cannot process raw text; they need a fixed vocabulary of pieces they can map to numbers. A tokenizer builds that vocabulary before training by finding the character sequences that appear most often in the training data - an approach called byte-pair encoding. Common words like "the" earn their own token. Rarer words get split: "tokenization" might become "token" + "ization". Nothing is ever unrepresentable, because in the worst case a string falls back to single characters. A useful rule of thumb for English is that one token is about four characters, or roughly three-quarters of a word.
Here is why a developer should care. First, money: API pricing is per token, in and out, so a verbose system prompt repeated on every request costs real dollars at scale - which is what makes prompt caching and cost tracking worth setting up. Second, limits: a model's context window is a token budget, so "how many documents fit" is a token math question. Third, speed: models generate one token at a time, so long outputs take proportionally longer.
Tokenization also explains some famous quirks. A model that seems unable to count the letters in "strawberry" is not being lazy - it never sees letters, only token IDs, so character-level questions are genuinely hard for it. Arithmetic suffers because numbers split into arbitrary chunks ("12345" might be "123" + "45"). And non-English text often uses two or three times more tokens per sentence than English, because the vocabulary was tuned on mostly English data - which quietly raises both cost and effective context usage for other languages.
The same idea extends beyond text: multimodal models chop images and audio into token-like patches so one architecture can consume them all. Whatever the medium, the token is the atom.
Where it sits in the AI stack
Tokens are the boundary between human text and model machinery - everything crossing into or out of a model passes through them:
Key tools and implementations
-
Byte-pair encoding (BPE)
The dominant algorithm for building token vocabularies from frequent character sequences.
-
tiktoken
OpenAI's open-source tokenizer library, handy for counting tokens before you send a request.
-
SentencePiece
A language-agnostic tokenizer used by many open-weights model families.
-
Provider token counters
Count-tokens API endpoints that let you measure a prompt's cost before running it.
Related entries
- Context window The maximum amount of text, measured in tokens, that a model can consider in a single request.
- LLM (large language model) A neural network trained on huge amounts of text to predict the next token, which lets it generate and understand language.
- Prompt caching Reusing the computed state of a repeated prompt prefix so later requests skip that work.
- Token cost tracking Counting tokens per request and pricing them per model so AI spend is a metric, not a surprise bill.