TL;DR: Models naturally produce prose, but software needs data it can parse. Structured output makes a model's reply conform to a schema - usually JSON with specific fields - either by constraining generation itself so invalid output is impossible, or by validating and retrying. It is the plumbing that lets model output flow into ordinary code.
How it works
Suppose an app extracts invoices from emails: it needs vendor, amount, and due_date as fields it can insert into a database, not a paragraph that starts "This invoice appears to be from...". Asking nicely - "respond only with JSON" - works most of the time, and most of the time is exactly the problem. At thousands of requests a day, the occasional markdown fence, trailing comma, or chatty preamble becomes a steady stream of parse failures in production.
The robust fix is constrained decoding. A model produces its next token by scoring every candidate in its vocabulary - and the serving layer can zero out any candidate that would violate your schema before one is picked. Compiled from a JSON Schema or grammar, the constraint walks along with generation: right after the opening of a vendor field, only tokens that continue a valid string are even considered. Malformed output is not detected and rejected; it is unrepresentable. Provider "structured output" or "JSON mode" features, and open-source libraries like Outlines behind self-hosted engines such as vLLM, all work this way.
The other route is validate and retry: let the model answer freely, check the result against the schema, and re-prompt with the errors on failure. Libraries like Instructor wrap this loop around Pydantic or Zod definitions. It burns occasional extra calls but works with any model, and a schema violation becomes a handled error instead of a crash. Constraint has its own subtle cost, worth knowing: forcing format can shave answer quality, since the model cannot "think out loud" before committing to fields - one reason schemas often include a reasoning field first, giving the model room to work before the answer fields.
Structured output is also the mechanism underneath tool use: a tool call is just a schema-conforming object naming a function and its arguments, which is what makes agents dependable enough to act. It interacts awkwardly with streaming - half-arrived JSON is not yet valid - which is exactly the gap partial-JSON parsers fill. The rule of thumb: whenever code, not a person, consumes the output, use enforced structure; hope is not a parser.
Where it sits in the AI stack
Structured output lives at the boundary between generation and your code, shaping tokens as they are chosen:
Key tools and implementations
-
Provider structured modes
Schema-enforced output built into the major APIs - pass a JSON Schema, get conforming JSON back.
-
Outlines
An open-source constrained-decoding library that compiles schemas and grammars for self-hosted engines.
-
Instructor
The validate-and-retry pattern packaged around Pydantic models, working across many providers.
-
Pydantic and Zod
The schema-definition layers most structured-output tooling builds on in Python and TypeScript.
Related entries
- Tool use (function calling) Letting a language model request actions by emitting structured calls that your code executes and whose results feed back in.
- Token The chunk of text - roughly three-quarters of a word - that a language model reads and writes one at a time.
- Streaming (token streaming) Sending each generated token to the user as it is produced instead of waiting for the full reply.
- Guardrails Programmatic checks on an AI system's inputs, outputs, and actions that block or correct behavior outside defined limits.