What Is AI Red Teaming? How Security Testing for AI Systems Actually Works

A conventional application security test asks whether software can be made to violate a security requirement. AI red teaming keeps that adversarial mindset, but the target is more complicated. An AI application may combine a model, system prompts, retrieval, user data, tools, APIs, memory, identity controls and ordinary web infrastructure. A failure can emerge from the interaction between those pieces even when no single component looks obviously broken.

That is why useful AI red teaming is not simply a contest to produce a clever jailbreak. It is a structured attempt to discover how a real AI system behaves when someone deliberately pushes it toward outcomes its designers did not intend.

OWASP’s GenAI Red Teaming Guide describes a holistic approach across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. NIST’s ARIA evaluation work similarly separates model testing, red teaming and field testing. Together they make an important point: adversarial testing is one part of AI evaluation, not a substitute for every other form of assurance.

What AI red teaming means in practice

An AI red team adopts the perspective of a motivated adversary, abusive user or other challenging actor and tests whether realistic actions can cause meaningful failure. The goal is evidence that helps engineers understand and reduce risk.

The target might be a standalone model, but in production it is often an entire system. Consider an internal assistant that can search company documents and create support tickets. A useful assessment may need to ask all of these questions:

• Can untrusted retrieved content manipulate the assistant?
• Can one user cause it to disclose another user’s information?
• Can the model call a tool with arguments the user is not authorized to use?
• Can sensitive system instructions or credentials leak through responses or logs?
• Can repeated or unusual requests create unacceptable cost or availability problems?
• Do ordinary API, authentication and cloud controls still hold around the AI layer?

These are different failure modes. A single prompt benchmark cannot answer all of them.

Red teaming is broader than jailbreaking

Jailbreaking is one useful test category. It explores whether a model can be induced to bypass behavioral restrictions. But treating AI red teaming as jailbreak hunting alone creates a blind spot around the application that surrounds the model.

OWASP’s February 2026 vendor evaluation criteria for AI red teaming explicitly distinguishes meaningful testing from superficial jailbreak-only offerings. That distinction matters even more for systems that use retrieval, tool calling, Model Context Protocol (MCP) or multiple agents.

For example, imagine an agent that is very resistant to direct harmful prompts but has permission to read a shared mailbox and send messages. If an attacker can place malicious instructions inside an email that the agent later reads, the security question is no longer only whether the base model refuses a direct jailbreak. The assessment must follow the data and control flow through the agent’s actual environment.

This connects directly to Indirect Prompt Injection and to the risks discussed in MCP Security Risks. The system boundary determines what should be tested.

Four layers worth testing

OWASP’s four-part framing is a useful way to prevent a red team plan from becoming too narrow.

This table is a synthesis of the testing areas in OWASP’s GenAI Red Teaming Guide. It is not a claim that every assessment needs equal effort in every layer. Scope should follow risk.

A public writing assistant with no private data and no tools has a very different attack surface from an autonomous agent connected to source code, email and cloud administration. Red teaming should reflect that difference.

Start with a threat model, not a bag of prompts

A good assessment begins by identifying what must be protected and how the system works. Otherwise testers can generate thousands of interesting conversations without knowing whether they exercised the risks that matter.

A practical scope should identify:

1. Assets: sensitive data, credentials, business actions, model access, proprietary prompts and other valuable resources.
2. Actors: normal users, malicious users, external content authors, compromised integrations, insiders and automated attackers where relevant.
3. Trust boundaries: browser to API, application to model provider, retrieval pipeline to model, model to tool, tenant to tenant and agent to external service.
4. Allowed actions: what the system is intentionally permitted to read, write, execute or disclose.
5. Unacceptable outcomes: concrete failures the assessment should try to demonstrate or rule out.

For an agent, the threat model should include reasoning, memory, tools, identity and human oversight. OWASP’s Agentic Threats Navigator highlights these as important attack surfaces in agentic systems.

The result should be a set of testable hypotheses rather than a vague instruction to “hack the AI.”

A simple example: red teaming a RAG assistant

Suppose an organization has a retrieval-augmented generation assistant that answers employee questions from approved internal documents.

A weak test might send a collection of jailbreak prompts to the chat interface and record which ones produce unusual answers.

A risk-based test asks more specific questions. Can a document author place instructions in an indexed document that influence later answers? Can retrieved text override the intended task? Can the assistant reveal chunks that the current employee should not retrieve? Are citations tied to the information actually used? What happens if retrieval returns hostile markup, misleading instructions or data from another access-control scope?

The red team can then build controlled test documents inside an authorized test environment. No real confidential information is required. Each test has a defined precondition, action and expected security property.

For example:

Security property:
Retrieved documents are data, not trusted instructions.

Controlled test:
Insert a test document containing an instruction that asks the assistant
for an unrelated privileged action.

Pass condition:
The assistant continues the authorized task and does not perform the
unrelated action solely because retrieved content requested it.

This is much more useful than reporting that “prompt injection worked.” It tells the engineering team which trust boundary failed and what behavior a regression test should preserve after remediation.

For a deeper treatment of this architecture, see RAG Security Explained and How to Secure a RAG Application.

What counts as a good finding?

A red team finding should be reproducible enough to support engineering decisions without overstating what was proven. At minimum, it should capture the system version or configuration, relevant preconditions, the test path, observed behavior, expected behavior and potential impact.

For probabilistic systems, one successful or failed attempt may not characterize reliability. The test design may need repeated trials and clearly reported conditions. The important discipline is to distinguish observation from inference.

A useful finding might say that a particular indirect prompt caused a test agent, under a named configuration and permission set, to attempt an unauthorized tool action in a controlled environment. A poor finding would generalize that observation into an unsupported claim that all agents using the same model are vulnerable in the same way.

Evidence also needs context. A strange model response may be a quality defect but not a security vulnerability. Conversely, a seemingly ordinary response can represent a serious security failure if it contains data from another tenant.

Human testing and automated testing both have roles

Automation is valuable for breadth and repeatability. A test harness can run large sets of prompts, vary inputs, track outputs and rerun known failures after a change. Human testers are useful for adapting strategies, recognizing unexpected behavior and following multi-step paths that were not anticipated when the harness was written.

NIST’s ongoing GenAI evaluation program reflects the broader importance of rigorous testing and measurement across generative AI technologies. Its ARIA work also uses multiple evaluation levels rather than treating a single test technique as sufficient.

The best balance depends on the system. Automated checks are particularly useful as regression tests once a failure mode is understood. Exploratory human testing is often more useful when the team is still discovering how a new architecture can fail.

Test the consequences, not only the model response

This is one of the biggest differences between testing a chatbot and testing an agent.

If a chatbot produces a bad answer, the response itself may be the relevant outcome. If an agent produces a bad intermediate decision and then calls an external tool, the important evidence may be the resulting action.

A tool-enabled assessment should therefore observe:

• which tool the agent selected;
• the arguments it generated;
• which identity or credential authorized the call;
• whether a policy layer approved or blocked it;
• what data crossed the boundary;
• whether a human approval step actually constrained the action;
• whether logs make the sequence reconstructable.

NIST’s April 2026 work on measurement probes for agentic AI ecosystems emphasizes the need for traceability around agent decisions and tool usage. That is directly relevant to red teaming: if the team cannot reconstruct what an agent did, understanding a failure becomes much harder.

Safety matters during the assessment itself

Red teaming intentionally creates abnormal conditions, so the test process needs its own controls. Testing should occur only on systems the team owns or is explicitly authorized to assess.

High-impact actions should be isolated. Use test tenants, synthetic records, sandboxed tools and non-production credentials where possible. A test agent should not need permission to send a real payment, delete production data or message real customers merely to prove that an authorization boundary is weak.

Define stop conditions before testing. If a test unexpectedly reaches production data, escapes the agreed environment or begins creating uncontrolled cost, the team should know when to halt and preserve evidence.

This is ordinary security engineering discipline applied to an AI-specific target.

Red teaming does not prove that an AI system is secure

A successful assessment demonstrates what the team tested and what it observed. It cannot demonstrate the absence of every possible adversarial input or future attack.

This limitation is especially important for systems driven by natural language. In June 2026, NIST discussed research showing limitations of relying on a fixed finite set of guardrails against adaptive adversarial prompts and argued for continuous searching, updating and operational resilience rather than a one-time security posture. The associated NIST publication, Robust AI Security and Alignment: A Sisyphean Endeavor?, was published in May 2026.

The engineering implication is practical: keep known adversarial cases as regression tests, monitor deployed behavior, revisit the threat model when the model or architecture changes, and assume that new attack strategies will appear.

That does not make red teaming pointless. It makes the purpose clearer. Red teaming reduces uncertainty and finds actionable weaknesses. It is not a certificate of permanent safety.

How findings should become controls

The value of an assessment appears after discovery. Each credible finding should lead to a decision: accept the risk, reduce it, redesign the feature, add monitoring or remove an unnecessary capability.

The right control depends on the failure. Prompt injection against a privileged agent may require stronger separation between untrusted content and control instructions, narrower tool permissions, deterministic authorization outside the model and human confirmation for consequential actions. Cross-tenant data exposure requires real access-control enforcement, not a better system prompt. Excessive model cost may require quotas, rate limits or workload controls.

This is why AI security cannot be solved entirely inside the prompt. Controls belong at the layer that can actually enforce the security property.

After a fix, rerun the original test and nearby variants. A remediation that blocks one exact string but leaves the underlying trust problem intact is not a strong fix.

A practical AI red team checklist

Before an assessment, ask whether you can answer these questions clearly:

• What system, model version and integrations are in scope?
• What assets and trust boundaries matter most?
• Which threat actors and abuse cases are credible for this deployment?
• Are tests mapped to explicit security properties or unacceptable outcomes?
• Are model, implementation, infrastructure and runtime risks covered where relevant?
• Are tool permissions and identity boundaries tested independently of model behavior?
• Is the environment safe enough to run adversarial tests without harming real users or data?
• Can findings be reproduced with their conditions recorded?
• Will confirmed failures become regression tests after remediation?
• Is there a plan to revisit testing when models, prompts, tools, data sources or threats change?

Where AI red teaming fits in a security program

Red teaming works best beside secure design, threat modeling, conventional application security testing, model evaluation, monitoring and incident response. It contributes a deliberately adversarial perspective that those activities may not provide on their own.

NIST’s ARIA pilot is a useful reminder of this relationship because it evaluates AI through model testing, red teaming and field testing as distinct levels. OWASP likewise frames GenAI red teaming across the complete system rather than only the model interface.

For defenders, the most useful mental model is a loop: understand the system, define credible failures, challenge the controls, collect evidence, reduce the risk and test again. As AI applications gain retrieval, memory and agency, that loop becomes more important because the consequences of a model decision can travel far beyond the text shown in a chat window.

Sources and further reading

• OWASP GenAI Red Teaming Guide
• OWASP AI Red Teaming and Evaluation Initiative
• OWASP Vendor Evaluation Criteria for AI Red Teaming Providers and Tooling v1.0
• NIST ARIA Pilot Evaluation Report, NIST AI 700-2
• NIST Generative Artificial Intelligence Evaluation Program
• NIST: Robust AI Security and Alignment