AI Red Teaming
Definition
AI red teaming is adversarial testing of AI systems, such as models, LLM applications and agents, to find ways they can be manipulated or abused.
What Is AI Red Teaming?
AI red teaming attacks an AI system the way an adversary would. It covers prompt injection, jailbreaks, data leakage, unsafe outputs and agents taking actions they should not. The most useful tests follow an attack from the prompt into the tools, APIs and data the AI can reach.
Types of AI Red Teaming
- Model-level: probing the model for unsafe or leaking outputs
- Application-level: attacking the LLM app, its prompts and integrations
- Agent-level: manipulating agents into misusing tools or data
AI Red Teaming Use Cases
- Testing a customer-facing chatbot before launch
- Checking that an internal agent cannot exfiltrate data
- Regression testing after a model or prompt change
Limitations of AI Red Teaming
Model-only testing misses risks in the surrounding application. Results also vary between runs because model behavior is not deterministic, so continuous or repeated testing matters.
Related ranking
Related terms
Proven ExploitabilityProven exploitability means a vulnerability has been confirmed by actually exploiting it, typically documented with a working exploit and reproducible steps.Business Logic VulnerabilityA business logic vulnerability is a flaw in how an application is designed to work, such as broken authorization or a workflow that can be bypassed, rather than a flaw in input handling.Exploit ChainAn exploit chain combines several weaknesses, often individually low severity, to reach an impact that none of them reaches alone.Continuous Penetration TestingContinuous penetration testing runs pentests on an ongoing basis, or on every significant change, instead of once or twice a year.