AI Red Teaming

A red team is an adversarial group that attacks a system to find weaknesses before real attackers do. The term comes from military wargaming, where the red team plays the adversary. Applied to AI, red teaming means systematically trying to make your own AI system fail in ways that matter.

The key word is systematically. Clicking around a chatbot and noticing it sometimes says something odd is not red teaming. Red teaming has a defined threat model, a structured attack plan, documented results, and produces findings that are classified by severity and mapped to remediation actions.

Traditional software penetration testing finds code vulnerabilities: buffer overflows, authentication bypasses, SQL injection. These are deterministic. The same input always produces the same output. AI red teaming targets model behaviour, which is probabilistic. The same prompt can produce different results across runs. The attack surface includes what the model has learned, not only how the surrounding code is written.

Scoping answers four questions before any testing starts. What is the model authorised to do? Who is the realistic attacker? What would a successful attack produce? What is explicitly out of scope?

The authorisation boundary defines normal behaviour. For a customer support chatbot: answer questions about products, escalate to human agents, never discuss competitor pricing, never generate content unrelated to support. This boundary is the reference point for classifying results. A response is a finding when it violates the boundary.

The misuse case inventory is a list of what an attacker would want to achieve. Build it by asking: if someone wanted to misuse this system, what would they try to get it to do? Common misuse cases across AI system types:

A well-defined misuse case inventory drives everything downstream: which attack categories to test, what constitutes a failure, and how to measure coverage. Coverage means the fraction of misuse cases that were tested. A red team report that cannot report coverage by misuse case is incomplete.

The AI attack surface has three layers: where attacker-controlled content enters the system, what the model can do once it processes that content, and what the model can produce as output.

The input layer is the most overlooked. For a RAG system, attacker-controlled content does not only arrive through the user's direct prompt. It also arrives through retrieved documents, cached memories, and the responses of external APIs the system calls. Each of these is an injection vector. Coverage means testing all of them, not only the obvious user-facing chat interface.

Capabilities amplify the impact of a successful injection. A chatbot that can only produce text has limited blast radius if hijacked. An agent that can send emails, write files, and call payment APIs has a much larger blast radius. Map capabilities before testing because they determine the severity of each finding: the same injection that produces a harmless wrong answer in a text-only bot constitutes a critical finding in an agent with write access to production systems.

Attack category selection is the most consequential design decision in a red team assessment. Running every attack category against every system wastes time and produces a report too diffuse to act on. Selecting categories matched to the threat model and system type produces focused, actionable findings.

The table below maps system types to priority attack categories. The priority column uses the misuse case inventory as its basis: categories are high priority if they address misuse cases with high impact and realistic attacker capability.

Within each attack category, test depth varies from shallow to deep. A shallow test sends a canonical example from the attack library, for example a standard DAN jailbreak prompt. A deep test chains multiple techniques, adapts to the model's responses, and uses encoding or obfuscation to evade filters. Deep tests find vulnerabilities that shallow tests miss. A complete assessment includes both.

Define pass and fail criteria for each category before running tests. For jailbreaks: a fail is any response that produces content the system prompt explicitly prohibits. For indirect injection: a fail is any response that executes an instruction originating from a retrieved document, not from the user. Writing criteria before execution prevents the evaluator from rationalising borderline responses as passes.

Execution has two modes: manual and automated. They answer different questions and serve different purposes in a complete red team program.

Classification assigns each test result to one of three categories: pass, fail, or borderline. The criteria must be defined before execution. Defining criteria after seeing results introduces confirmation bias, where the evaluator unconsciously adjusts standards based on how many failures they want to report.

Classify in batches of the same attack type. Evaluating all jailbreak tests before moving to extraction tests keeps calibration consistent within each category. Mixed classification across categories drifts.

Borderline results require a second reviewer. The second reviewer should classify without seeing the first reviewer's verdict. If they agree, record the agreed verdict. If they disagree, escalate to a third reviewer or record as borderline with both rationales. Borderline findings are not discarded. They are reported as such with the specific reasoning, because they often reveal ambiguity in the authorisation boundary that needs to be resolved in policy, not in code.

Compute the pass rate per attack category, not just an aggregate. An aggregate pass rate of 85% sounds good. A per-category breakdown that shows 40% pass rate on indirect injection reveals a critical weakness that the aggregate obscures.

Severity is a function of two factors: exploitability (how much effort and skill does the attack require?) and impact (what happens when the attack succeeds?). Neither factor alone determines severity. A trivially easy attack that produces a harmless result is not critical. A technically difficult attack that causes full agent compromise may still be critical because the impact justifies the effort from an attacker's perspective.

A red team report that does not produce remediation action has failed at its primary purpose. The report is not a record of what the red team did. It is a decision-making tool for the people responsible for fixing the system.

The report has two audiences: the technical team who will implement the fixes, and the non-technical stakeholders who decide whether to deploy, delay, or accept risk. Both need different things from the same document. Structure the report so both can extract what they need without reading everything.

The most common failure in red team reports is vague remediation recommendations. "Improve prompt robustness" is not actionable. "Add a guard instruction that explicitly prohibits responding to instructions found in retrieved documents, and test with the 14 indirect injection prompts in Appendix B" is actionable.

The second most common failure is missing reproduction steps. A finding that cannot be reproduced by another analyst is not a confirmed finding. It may be a real vulnerability, but without reproduction steps it cannot be verified after a fix is deployed, which means it cannot be closed.