What Is AI Red Teaming? Methods, Frameworks, and Tools

AI red teaming is structured adversarial testing of an AI system to find the flaws and failure modes an attacker could use, before the attacker does. It targets the parts classic security testing skips: the model's behavior, the prompts that reach it, the documents it retrieves, the tools and agents it can call, and the permissions behind them, as well as the usual infrastructure. NIST defines it as "a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment, in collaboration with AI developers." The work combines automated probing at scale with human testers who find the creative abuse paths tools miss. This guide covers how AI red teaming differs from classic red teaming, the attack classes and the frameworks that name them, the tools, how an engagement runs, and what it costs.

AI red teaming, defined

AI red teaming is adversarial testing aimed at how an AI system behaves, reaching past the servers to the model, the prompts, the retrieval, and the agents. The NIST glossary definition, drawn from NIST SP 800-218A, frames it as a structured effort to find flaws in an AI system, often in a controlled environment and alongside the developers. The important word is behavior. A model can be fully patched at the infrastructure level and still hand an attacker another user's data because someone hid an instruction in a document it read.

AI red teaming differs from safety benchmarking, and the distinction matters for what you buy. A benchmark scores a model against a fixed test set and produces a number. Red teaming hunts for the specific inputs and conditions that make the system fail, and produces proven abuse paths with evidence. Microsoft's team, writing up lessons from red teaming over 100 generative AI products, put two findings plainly: you do not need to compute gradients to break an AI system, and the human element of AI red teaming is central. Most real failures come from simple abuse of the system as built, found by a person who thinks like an attacker.

How AI red teaming differs from classic red teaming

Classic red teaming works toward a crown jewel through networks, identities, and endpoints, and AI red teaming adds a set of targets that behave differently from any of those. The model follows instructions in its input, which means text anywhere the system reads, a user message, a retrieved document, a tool result, can carry an attack. The retrieval layer can be poisoned. An agent holds tools, credentials, and memory, so a single successful injection can turn into real actions: a sent email, a database write, a payment. None of these targets has a patch in the traditional sense, because the failure is in behavior rather than a version number. If you are new to the discipline, start with our guide to what red teaming is, then add the layers below.

A scope matrix by layer

Scoping an AI red team means naming the layers in play, because each one has its own attacks and its own evidence. A chatbot wrapper is a narrow scope. An agent holding tools, credentials, and retrieval is a wide one. The matrix sets out what gets tested at each layer.

The agent layer is where the most damage concentrates, because an agent can act. Model Context Protocol servers, which give agents tools, are a new piece of attack surface worth testing on their own, covered in our guide to MCP security.

Attack classes: OWASP LLM Top 10 and MITRE ATLAS

Two public references name the attack classes, and using them keeps an engagement complete and legible to auditors. The OWASP Top 10 for LLM Applications, 2025 edition, lists the application risks. MITRE ATLAS catalogs adversary techniques against AI systems the way ATT&CK does for networks. The table maps the common attack classes to both.

We go deeper on the application risks in the OWASP Top 10 for LLM applications, and on the single most common one in prompt injection. The OWASP list and MITRE ATLAS are both free and maintained.

What the frameworks require

Adversarial testing of AI is moving from good practice to a named obligation, and three documents set the current bar. The NIST AI Risk Management Framework, NIST AI 100-1, published January 2023, organizes the work under four functions: Govern, Map, Measure, and Manage. Its companion for generative AI, NIST AI 600-1, the Generative AI Profile from July 2024, names 12 generative AI risks and calls, in suggested action MS-2.7-007, for AI red teaming to assess resilience against prompt injection, adversarial examples, data poisoning, membership inference, and model extraction.

In the European Union, the AI Act makes adversarial testing a legal duty for the largest models. Article 55(1)(a) requires providers of general-purpose AI models with systemic risk to "perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks." Those obligations have applied since August 2, 2025, and the AI Office's enforcement powers, including fines, apply from August 2, 2026. For high-risk systems, Article 15(5) of Regulation (EU) 2024/1689 requires resilience against data poisoning, model poisoning, adversarial examples, and confidentiality attacks. The 2026 Digital Omnibus on AI adjusted some timelines but left the general-purpose model obligations in Article 55 in place.

The tools

A capable AI red team runs a small set of open-source tools for the automated work, then adds human testers for the rest. Naming them accurately matters, because the category is young and the marketing is loud.

These tools cover breadth: they run thousands of known attack prompts and catch regressions. They do not replace a tester who reasons about your specific agent, its tools, and the business actions it can take. The garak scanner and Microsoft PyRIT are the common starting points.

Automated versus human-led testing, and why you need both

Automated and human-led testing cover different failures, so a serious engagement uses both. Automated tools run large libraries of known attacks against the system on every build, which catches regressions and the well-documented jailbreaks cheaply and repeatably. Human testers find the abuse paths that depend on your system: the agent that will send an email if a retrieved document tells it to, the tool whose output the model trusts without checking, the memory that carries a planted instruction into a later session. Microsoft's review of its own practice reached the same split, crediting automation for coverage and human judgment for the failures that matter most. An engagement that runs only tools gives a false sense of safety, and one that runs only humans does not scale to every release.

How an AI red team engagement runs

An AI red team engagement runs as a sequence that ends in a regression suite, so the testing keeps paying off after the report. It starts with a threat model: what the system can do, what it can reach, and who would want to abuse it. Next comes a harm and abuse taxonomy, the list of failures that would matter for this system, from data leakage to an unauthorized action. From that, the team writes a test plan covering each layer in the scope matrix. Execution combines automated probing with human-led attacks, and every finding is reproduced and evidenced by a person before it counts, with a severity assigned on review rather than by a tool. Remediation names the specific fixes: tighter tool scopes, input and output handling, retrieval controls, approval gates on destructive actions. The last step is the one that separates a mature engagement: a regression suite of the successful attacks, wired to run on every model swap and prompt change, because an AI system's behavior shifts when the model or the prompt does. Our agentic AI security work is built around that permanent test loop.

How much does AI red teaming cost?

A focused test of a single LLM application anchors to our verified AI penetration testing range of $10,000 to $40,000, where a chatbot wrapper is days of work and an agent holding tools, credentials, and retrieval is weeks. A broader AI red team, covering multiple models or agents, the retrieval and data pipelines behind them, human-led depth, and a standing regression suite, runs higher. As a BD Emerson planning range, that broader engagement is roughly $30,000 to $90,000. The drivers are the number of models and applications in scope, whether agents and tools are in play, how many retrieval and data sources feed the system, the depth of human-led testing, and whether a regression suite is built and handed over. These are planning figures for budgeting; a scoped proposal replaces them once we see the architecture.

What you get

The deliverable is a set of proven abuse paths with evidence, mapped to the frameworks, plus the means to keep testing. Each finding carries a reproduction, the impact demonstrated rather than asserted, and a severity assigned by a human on review. Findings map to the OWASP Top 10 for LLM Applications and to NIST AI RMF, so the report drops into an existing governance program, an ISO 42001 management system, or a SOC 2 scope without translation. Remediation is specific and ordered by the defense it buys. The regression suite ships with the engagement, so the attacks that worked become tests that run on every change.

Frequently asked questions

What is AI red teaming? AI red teaming is structured adversarial testing of an AI system to find the flaws and failure modes an attacker could use. It targets the model's behavior, the prompts that reach it, the documents it retrieves, and the tools and agents it can call, as well as the infrastructure. NIST describes it as a structured effort to find flaws in an AI system, often in a controlled environment and alongside the developers.

How is AI red teaming different from a normal penetration test? A normal penetration test targets networks, applications, and identities, and the failures have patches. AI red teaming targets behavior: a model follows instructions hidden in the text it reads, retrieval can be poisoned, and an agent can be driven to take real actions. Those failures live in how the system behaves, not in a software version, so the methods and the fixes are different.

What is automated red teaming? Automated red teaming runs large libraries of known attacks, such as jailbreaks and injection payloads, against an AI system on every build, using tools like PyRIT, garak, and promptfoo. It catches regressions and documented attacks cheaply and repeatably. It does not replace human testers, who find the abuse paths specific to your agent, tools, and business actions, so strong programs use both.

What tools are used for AI red teaming? The common open-source tools are Microsoft PyRIT, which automates multi-turn attacks; NVIDIA garak, an LLM vulnerability scanner built from probes and detectors; promptfoo, for red teaming and evaluations of LLM apps and agents; and Inspect from the UK AI Security Institute for AI evaluations including agentic tasks. Teams run these for breadth and add human testers for the system-specific failures.

Does the EU AI Act require AI red teaming? For the largest models, yes. Article 55 of the EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing of the model to identify and mitigate systemic risks. Those obligations have applied since August 2, 2025, and the AI Office's enforcement powers apply from August 2, 2026. High-risk systems must also be resilient against data poisoning and adversarial examples under Article 15.

How much does AI red teaming cost? A focused test of one LLM application anchors to the AI penetration testing range of $10,000 to $40,000. A broader AI red team covering multiple models or agents, their retrieval and data pipelines, human-led depth, and a regression suite runs higher, at a BD Emerson planning range of roughly $30,000 to $90,000. Scope, the number of agents and data sources, and whether a regression suite is built set the number.

If you are putting an AI system or an agent into production, our agentic AI security team can scope a red team and leave behind the regression suite that keeps it tested on every change.