TL;DR: Red teaming is deliberately attacking your own AI system to find its failure modes before strangers do. Testers probe for jailbreaks, data leaks, and harmful outputs using the tricks real adversaries use, then every successful attack becomes a documented finding, a fix, and a regression test.
How it works
It starts with a threat model: what could go wrong with this system, for these users? A customer-facing bank bot worries about leaking other customers' data and giving unauthorized financial advice; an internal coding assistant worries about exfiltrating secrets from its context. The threat model turns "try to break it" into a concrete checklist of behaviors to hunt for, ranked by how much damage each would do. Scope matters as much as ambition: a team that goes deep on the two failure modes that would make headlines finds more than one that tests everything shallowly.
Then comes the attacking. Human red teamers improvise the way real adversaries do - role-play framings, multi-turn setups that walk the model somewhere it would refuse to go directly, encoding tricks, and jailbreaking patterns collected from the wild. If the system reads external content - web pages, documents, email - testers also plant prompt injection payloads there, since instructions hidden in retrieved data are now a primary attack path, not an exotic one.
Manual creativity is then scaled with automation. Attack libraries replay hundreds of known jailbreak and injection patterns against every build, and attacker LLMs generate fresh variants of whatever succeeded last time. A common cadence: a deep human-led exercise before launch and after major changes, with the automated suite running continuously in between as a regression net. Coverage claims deserve skepticism either way - passing a fixed attack library proves the system beats yesterday's attacks, not tomorrow's, which is why the human exercises never fully retire.
What separates red teaming from mischief is what happens after. Every successful attack is written up - the exact prompts, the harmful output, reproduction steps, severity - and routed to a fix: a tightened system prompt, new guardrails, restricted tool permissions. The attack then joins the automated suite so the same hole can never quietly reopen, exactly as production bugs get folded into an eval dataset. The goal is defensive throughout: find it first, fix it, and prove it stays fixed.
Where it sits in the AI stack
Red teaming probes the deployed system from the outside, and its findings flow back into the layers that defend it:
Key tools and implementations
-
garak
Open-source LLM vulnerability scanner that fires a broad library of attack probes at a model.
-
PyRIT
Microsoft's open-source toolkit for orchestrating automated multi-turn attack campaigns.
-
promptfoo red teaming
Adversarial test generation built into the same runner teams already use for evals.
-
Giskard
Open-source scanner covering injection, leakage, and harmful-output checks for LLM apps.
Related entries
- Jailbreaking Crafting prompts that talk an AI model into ignoring its safety training and producing restricted output.
- Prompt injection An attack where malicious instructions hidden in content an LLM reads override what the developer told it to do.
- Guardrails Programmatic checks on an AI system's inputs, outputs, and actions that block or correct behavior outside defined limits.
- Evals (AI evaluations) Structured tests that score an AI system's outputs so teams can measure quality and catch regressions.