TL;DR: Before a model writes anything, it must process every token of your prompt - and most applications resend the same instructions, tools, and history on every call. Prompt caching stores the computed state of that repeated prefix so later requests skip straight to the new part. Providers typically discount cached input heavily, so it cuts both latency and the bill.
How it works
Processing a prompt is real work: for every token, the model computes attention state - the KV cache - that generation then reads from. Crucially, this state is deterministic: the same model reading the same tokens in the same order always produces the same cache. So when thousands of requests share an identical opening - a long system prompt, tool definitions, a policy document - recomputing that state every time is pure waste. Prompt caching computes it once, keeps it, and on the next matching request restores it and starts from the first new token.
The catch is in the word prefix. Matching is exact and positional: the cache covers the unbroken run of identical tokens from the very start of the prompt, and the first difference ends it. That single rule dictates prompt architecture - stable content first (system prompt, tools, reference documents), volatile content last (the user's question). Even a timestamp or a random ID placed early in the prompt silently destroys every cache hit after it.
A concrete example: a support bot with 20,000 tokens of instructions and product documentation ahead of each question. Uncached, every one of those tokens is reprocessed on every request, and the user waits through it before the first word appears. Cached, the model starts at the question - time to first token drops from several seconds toward instant, and since providers bill cached input at a fraction of the normal rate (often around a tenth), the per-request cost falls with it. The same mechanics power multi-turn chat: each turn extends the previous prompt, so everything before the newest message is one long cache hit.
The heaviest beneficiaries are agents, which loop - resending a growing transcript plus a large toolset dozens of times per task, exactly the shape caching rewards. Major providers all offer it (some automatically, some via explicit cache markers, with entries living minutes to about an hour), and self-hosted engines have their own versions, like vLLM's prefix caching. One boundary worth knowing: this caches prompt processing, not answers - for identical repeated questions, response-level caching is the separate, complementary trick.
Where it sits in the AI stack
Prompt caching sits inside the serving layer, between the incoming prompt and the model's forward pass:
Key tools and implementations
-
Provider prompt caching
Anthropic, OpenAI, and Google all cache repeated prefixes, billing cached input at a steep discount.
-
vLLM prefix caching
Automatic KV-cache reuse across requests that share a prefix on self-hosted deployments.
-
SGLang RadixAttention
A radix-tree cache that shares prefixes across many concurrent requests, built for agentic workloads.
-
Response caches
Gateway-level caching of full answers to repeated questions - a different layer that pairs well with prefix caching.
Related entries
- Context window The maximum amount of text, measured in tokens, that a model can consider in a single request.
- System prompt Hidden instructions sent before user messages that set a model's role, rules, and tone for the conversation.
- AI agent A system where a language model plans, calls tools, and loops on results to finish a task with minimal supervision.
- Token cost tracking Counting tokens per request and pricing them per model so AI spend is a metric, not a surprise bill.