TL;DR: Guardrails are programmatic checks wrapped around a model - validating what goes in, what comes out, and what an agent is allowed to do. The model stays probabilistic; the guardrails are deterministic code. Together they turn "the model usually behaves" into "the system cannot do these specific things."
How it works
A language model follows instructions statistically, not contractually. Prompts saying "never reveal internal data" or "always answer in JSON" are usually honored - and "usually" is not an engineering guarantee. Guardrails restore the guarantee by surrounding the model with ordinary code that checks inputs before they reach it, outputs before they reach the user, and actions before they execute. The model proposes; the guardrails dispose.
Input guardrails screen what arrives: filters for off-topic or abusive requests, scans for prompt injection patterns in user text and retrieved documents, and redaction of secrets or personal data before they enter the context. Output guardrails screen what leaves: schema validation against structured output contracts, content classifiers for policy violations, groundedness checks that compare claims against retrieved sources, and pattern scans that catch leaked credentials. A failed check can block the response, strip the offending part, or send the model back for a retry with the error attached.
For agents the critical third layer is action guardrails, because an agent's tool calls have real-world effects. Permission systems define what each tool may touch: allowlists for commands and domains, read-only modes, spending caps, rate limits, sandboxed working directories. Risk-tiered policies let routine actions pass silently while destructive ones - deleting data, sending money, contacting customers - are blocked outright or escalated to a human in the loop. The enforcement point is the tool boundary: since every action flows through the harness's code, the harness is where "cannot" gets implemented.
Two design realities keep the field honest. First, checks sit on the request path, so every one adds latency and sometimes a second model call - teams tier them, running cheap regex and schema checks on everything and expensive classifier checks only where stakes are high. Second, guardrails fail in both directions: too loose and incidents slip through, too tight and legitimate work gets blocked until users route around the system. Treat the rule set as living software - tested against real traffic, tuned with feedback, and probed by red-teaming before attackers do it for you.
Where it sits in the AI stack
Guardrails sit between the model's proposal and anything that acts on it:
Guardrails are the runtime enforcement arm of a broader safety posture: policy decides what should never happen, guardrails make it mechanically difficult, and monitoring proves whether the two agree.
Key tools and implementations
-
Schema validators
JSON Schema and typed-parsing libraries that reject malformed model output deterministically.
-
Content classifiers
Moderation models and APIs that score text for policy violations on the way in and out.
-
Permission systems
Allowlists, sandboxes, and approval rules that bound what an agent's tools can touch.
-
Guardrails frameworks
Open-source libraries for declaring validation rules and wiring them around model calls.
Related entries
- Human in the loop A design pattern where a person reviews, approves, or corrects an AI system's actions at defined checkpoints.
- Prompt injection An attack where malicious instructions hidden in content an LLM reads override what the developer told it to do.
- Tool use (function calling) Letting a language model request actions by emitting structured calls that your code executes and whose results feed back in.
- Red teaming Deliberately attacking your own AI system to find jailbreaks and harmful failures before attackers do.