TL;DR: Prompt injection hides instructions inside content an LLM processes - a web page, an email, a document - so the model follows the attacker's directions instead of the developer's. It is the defining security risk of LLM applications, because models cannot reliably tell trusted instructions apart from untrusted data. There is no complete fix; teams defend in layers.
How the attack works and how teams defend
Picture an email assistant that summarizes your inbox. One incoming message contains a paragraph addressed not to you but to the assistant, telling it to forward your recent messages elsewhere. The assistant was only supposed to read that email as data - but a language model processes everything in its context as one stream of text, and text that looks like an instruction can get treated as one. That is prompt injection. It comes in two forms: direct, where the user typing into the app supplies the malicious instruction themselves, and indirect, where an attacker plants it in content the app will later retrieve - a web page, a shared document, a review, a calendar invite.
The root cause is architectural. Traditional software separates code from data; an LLM does not. The system prompt, the user's message, and every retrieved document all arrive as tokens in the same context window, with no privilege boundary between them. Models are trained to prioritize developer instructions, but that is a learned tendency, not an enforced rule - and a sufficiently persuasive piece of injected text can override it. The stakes rise sharply with tool use: an injected instruction in a chatbot produces a bad answer, but the same instruction in an agent that can send email, write files, or call APIs produces a bad action.
Defense starts from an assumption: some injection will get through. So teams limit what a successful one can do. Give the model the narrowest tool permissions its task allows, and never more access to data than the current user legitimately has. Require human confirmation before consequential actions - the pattern covered under human-in-the-loop. Wrap the model in guardrails that screen inputs for injection patterns and outputs for data leaving that should not. Mark untrusted content clearly in the prompt, and treat the model's own output as untrusted downstream. Then test it all adversarially, repeatedly, because attackers will.
Where it sits in the AI stack
Prompt injection enters wherever untrusted content meets the model's context - retrieval pipelines, browsing, email, and file uploads are all doorways:
Key defenses and mitigations
-
Least-privilege tools
Scope every tool and data source to the minimum the task needs, so a hijacked model can do little.
-
Guardrail filters
Classifier layers that screen incoming content for injection patterns and outgoing responses for leaks.
-
Human approval gates
A person confirms irreversible or sensitive actions before the system executes them.
-
Adversarial testing
Red-team exercises and injection test suites that probe the app the way an attacker would.
Related entries
- Guardrails Programmatic checks on an AI system's inputs, outputs, and actions that block or correct behavior outside defined limits.
- Tool use (function calling) Letting a language model request actions by emitting structured calls that your code executes and whose results feed back in.
- Jailbreaking Crafting prompts that talk an AI model into ignoring its safety training and producing restricted output.
- Red teaming Deliberately attacking your own AI system to find jailbreaks and harmful failures before attackers do.