TL;DR: Language models produce their answer one token at a time, and streaming sends each token to the user the moment it exists instead of holding the reply until it is finished. The total wait is the same; the experience is not. Text that starts appearing in half a second feels instant, while the identical answer delivered whole after twenty seconds feels broken.
How it works
Generation is inherently sequential: the model predicts one token, appends it to the context, and predicts the next. A 500-word answer might mean 700 of those steps. A non-streaming API buffers all of them and returns one complete response; a streaming API opens a long-lived HTTP connection and pushes each token down it as the loop runs - which is exactly the typewriter effect every AI chat interface shows. The standard transport is server-sent events (SSE): plain one-way chunks over HTTP, no WebSocket machinery needed, and the format every major provider's streaming API uses.
The metric streaming transforms is time to first token (TTFT) - how long before the user sees anything. Once tokens are flowing, people read along at reading speed and the remaining generation time hides behind their own eyes; a response can take fifteen seconds in total and never once feel slow. This is also why prompt caching pairs so naturally with streaming: caching attacks TTFT (skipping the prompt-processing wait), and streaming hides everything after it.
Streaming does put obligations on your code. The client receives arbitrary fragments - half a word, a broken Markdown link, an unclosed code fence - so interfaces re-render progressively and tolerate incomplete markup. Anything that consumes model output downstream has to either buffer to completion or parse incrementally, which is where structured output gets interesting: partial-JSON parsers exist precisely so applications can act on a half-arrived object. Error handling changes shape too - a connection can drop mid-answer, after the user has already read half of it.
One useful side effect: most providers let you cancel a stream, and generation stops - you pay only for tokens produced so far. And the rule of thumb writes itself: stream whenever a human is watching; skip it for batch jobs, pipelines, and anywhere only the complete result matters, where the buffered response is simply easier to work with.
Where it sits in the AI stack
Streaming is the delivery leg of inference - the pipe between the generation loop and the user's screen:
Key tools and implementations
-
Server-sent events
The one-way HTTP streaming format every major provider's API uses to deliver tokens.
-
Provider SDKs
Official client libraries wrap the event stream in iterators, so consuming tokens is a for-loop.
-
UI streaming toolkits
Frameworks like the Vercel AI SDK handle progressive rendering, backpressure, and reconnection in the browser.
-
Partial-JSON parsers
Incremental parsers that make sense of half-arrived structured output before the stream finishes.
Related entries
- Token The chunk of text - roughly three-quarters of a word - that a language model reads and writes one at a time.
- Batching Grouping multiple requests into one GPU pass so serving hardware stays fully utilized.
- Prompt caching Reusing the computed state of a repeated prompt prefix so later requests skip that work.
- Structured output Forcing a model's reply to match a schema, such as valid JSON, so software can parse it reliably.