How to Do AI Penetration Testing

To do AI penetration testing, you probe the whole AI system, not just the model, for the security flaws that make AI dangerous: prompt injection, indirect injection, data leakage, excessive agency, insecure tool calls, and RAG poisoning. You scope every part of the system the agent touches, run the attack classes from the OWASP Top 10 for LLM Applications, map what you find to MITRE ATLAS so it can be prioritized, and then, critically, enforce the fixes at runtime so the results do not expire the next time someone changes a prompt. A pentest report finds the holes; it does not keep them closed.

Here is how to run one that produces useful, actionable findings.

Step 1: Scope the whole system, not the model

The most common mistake is treating an AI pentest as prompting a chatbot with jailbreaks. The model is one component. The real attack surface is the system around it: the prompts and their assembly, the retrieval corpus and every document in it, each tool and MCP server the agent can call, the application logic that turns model output into action, and the APIs behind all of it. Start by mapping that surface completely. Enumerate every tool and every data source the agent can reach, because an agent connects to more systems than the diagram usually shows, and each integration that wires the agent to your apps and databases is a place an attacker can reach through it.

Step 2: Test the core attack classes

A thorough AI pentest covers a defined set of classes rather than improvising. These map onto the OWASP LLM Top 10 and are the ones that cause real incidents:

1. Direct prompt injection. Try to override the system prompt or refusal rules with crafted user input.
2. Indirect prompt injection. Plant instructions in a document, web page, email, or tool output the agent will read, and see if it obeys them. This is the vector that turns RAG and browsing into attack surface.
3. Sensitive data disclosure. Attempt to extract secrets, other users' data, or the system's own instructions.
4. Excessive agency. Probe what actions the agent can take beyond its job, and measure the blast radius when it does.
5. Insecure tool and MCP calls. Poison a tool description or output and see whether it redirects the agent; check that tool access is scoped to least privilege.
6. Insecure output handling. Test whether model output is rendered, executed, or passed downstream without validation.

Step 3: Account for the probabilistic nature of the target

AI defects are not deterministic. An injection that fails on the first attempt can succeed on the fiftieth, because the model samples differently each run. So a single passing test proves little. Run each attack multiple times, vary the phrasing, and treat a low success rate as a finding, not a pass. This is also why AI testing has to be continuous: a result that held during a Tuesday engagement can break on a Wednesday prompt edit.

Step 4: Map findings to a shared taxonomy

Raw findings are hard to act on. Mapping each one to the OWASP LLM Top 10 and to MITRE ATLAS techniques gives every stakeholder a shared language, makes prioritization defensible, and connects your results to the wider body of published attack research. It also makes the report reusable: next quarter's test can measure against the same taxonomy and show whether you improved.

Step 5: Enforce the fixes at runtime

This is the step that separates a pentest that improves security from one that just documents it. A report that says an agent is injectable does not stop the next injection. The fix belongs in a runtime control plane that blocks the same attack classes on every live request: inspect untrusted input, enforce least-privilege tool and data permissions, gate irreversible actions for human approval, and log every decision. Then the gap between engagements is covered, and real attempts in production feed back into your next round of testing.

How often to run it

Run a full AI penetration test at each major release and on any significant change to the model, system prompt, tools, or retrieval sources, since any of those can reopen a closed hole. We go deeper on how often you should pentest an AI agent if you need to defend a cadence to an auditor. Between engagements, use continuous automated red-teaming and a runtime firewall so coverage does not lapse. The reliable pattern is test plus enforce, which we lay out on the AI penetration testing page. For the adversarial side specifically, see how to red team an AI agent, and to harden an agent from your findings, AI agent hardening maps each class to a control. Try Agentshield against a live prompt injection in the console above.