AI pentesting, what the research says about offensive agents

For a long time, "AI pentesting" was more of a marketing promise than a real capability. That has changed. A recent review by Fluid Attacks, cross-referencing several studies and offensive-agent frameworks, shows these systems already do meaningful security work, but it also makes clear where they still fail and why human oversight remains indispensable. For anyone who hires or runs pentests, the message is direct: the right question isn't "does AI replace the pentester?", but "how do we combine AI and specialists to find more flaws, with evidence that they are exploitable?".

Architecture matters more than the model

The most consistent finding is counterintuitive: what separates a useful offensive agent from a useless one is not the size of the language model, but the engineering around it, planning, validation, memory and course correction.

One example illustrates it well. In tests with a relatively small model, systems with a structured attack tree (based on MITRE ATT&CK) reached 71.8% task completion, versus 13.5% without that guidance. Architectures that separate roles (one agent plans, another executes, another perceives the environment) and correction modules raised the success rate significantly, in one case from 48.6% to 84.3% on the same target. The lesson applies to corporate security too: an AI tool without process architecture behind it doesn't deliver reliable results.

Where offensive AI already shines

The strengths are concrete and measurable:

• Scale and parallelism. One research system scanned a live university network with about 8,000 hosts and found 9 valid vulnerabilities, outperforming 9 of 10 human pentesters in that exercise.
• Command-line exploitation. Agents explore CLI paths that humans sometimes skip due to interface limitations, finding old flaws that slipped through.
• Falling cost. CTF competitions showed agents at the top of the ranking with inference cost dropping from thousands to a few dozen dollars per billion tokens.

In other words: for broad, repeatable and cheap coverage, AI is unbeatable on speed.

Where AI still fails (a lot)

And here is the part vendors rarely highlight:

• Chaining exploits is hard. The research pointed to failure in about 83% of attempts to chain multiple vulnerabilities into a full compromise.
• Proving real exploitation is another story. Building an effective payload from a CVE succeeded around 43% of the time. Recognizing a vulnerability is very different from proving it's exploitable.
• Stochasticity. Running the same model 100 times against the same target gave results ranging from 56 to 85 successes out of 100, depending on the model. A single successful run doesn't prove capability.
• Judgment and context. On visual interfaces and strategic pivots, humans still dominate: in one study, 80% of human participants found flaws the AI missed.

One methodological detail is telling: systems "captured the flag" via unintended vulnerabilities, masking real capability gaps. Without rigorous measurement, the number looks better than it is.

What this means when hiring (or running) pentests

The research converges on a model of layered workflows: planners, specialized models, validators and human triage working together, not a magical autonomous agent replacing the security team. Before trusting any AI pentesting solution, demand:

• Reproducible findings across multiple runs, not one lucky round.
• Clear separation between discovery, exploitation and validation.
• Documented false-positive rate and explicit handling of hallucinations.
• Audit trail with human approval points.
• Scope and credential controls that prevent unauthorized actions.

How Intelliway applies this in practice

This is exactly the principle behind ISA Horizon, our continuous AI-powered pentesting platform: AI agents explore the attack surface recurrently, covering scale and the environment's changes between manual tests, while Intelliway's offensive security specialists run the scenarios that require human creativity and, above all, validate whether a finding is actually exploitable. Every result arrives with evidence and reproduction steps, not a loose alert.

That combination, AI for breadth and frequency, humans for depth and judgment, is what the research shows to be the most effective path today. AI didn't retire the pentester. It gave them a team of tireless agents, as long as someone knows how to orchestrate them and demand proof of what they find.

Want to assess your attack surface with continuous AI-assisted pentesting validated by specialists? Talk to our team and discover ISA Horizon.