AI pentesting has gone from experimental to crowded in about a year. It feels like new solutions are launching on a weekly basis, and nearly all AI pentesting vendors describe what they sell as autonomous, agentic or continuous.
Behind those labels are some significant differences. When a Stanford-led study set different AI agent frameworks loose to run live penetration tests on an 8,000-host network, between 55% and 82% of their findings were confirmed as real vulnerabilities.
That spread matters for anyone comparing AI pentesting vendors. The capability is real, but the products differ far more than their marketing suggests. The differences show up mostly in how these solutions validate findings, how they prove their performance, how they behave on live systems, how they handle data and what they mean by “continuous.” Knowing the red flags in each area is the most effective way to tell vendors apart.
Why AI Pentesting Vendors Are So Hard to Compare
Labels like autonomous and agentic now cover products that work in very different ways. One system may run familiar scanners and use an LLM to summarize the results. Another may use AI to decide what to probe next, adapt after a failed attempt and keep working down an attack path.
The Stanford research shows how much those differences matter. When researchers changed only the underlying model, one agent framework’s valid finding rate moved from 57% to 80%, and another’s went from 55% to 82%. The model isn’t the whole story, though. Setups running the same model also produced very different results, because the surrounding system matters too. That includes how it plans, what tools it can call, how it reads responses and when it decides something has actually been exploited. A product page won’t show any of that.
When researching solutions, it’s therefore a good idea to ask vendors which models they use and how the system decides what to test next. How it confirms what it finds is a separate question, and the most important one. Once you look past the labels, the differences between vendors will show up as a handful of recurring red flags.
Red Flag: Findings Without Proof of Exploitation
A finding without evidence is a hypothesis. A pentesting system can flag an issue that looks vulnerable without proving that it is. A credible finding should show what happened, including the request and response, reproduction steps, the sequence of actions that led to exploitation and evidence of the access or impact that resulted.
The Stanford study shows what happens when that validation is missing. One agent reported successful authentication with default credentials because the server returned “200 OK” responses. In reality, those responses were redirects back to the login page after failed login attempts. An unvalidated report would have sent a team chasing a vulnerability that didn’t exist.
Even the strongest setup in the study had roughly one in five findings turn out invalid, and the weakest came close to half. Without proof of exploitation, buyers have no way to tell which kind of system they’re looking at.
Ask what happens between a system flagging a potential issue and a finding reaching your team. Does it try to reproduce the issue, and what evidence does it collect? Who validates each finding before it reaches the customer, a human or the system itself? And how does the vendor measure its false-positive rate on real engagements rather than in controlled tests?
Validity alone isn’t enough, either. The framework whose valid finding rate jumped from 57% to 80% after a model swap posted the lowest overall score of any participant in the study, human or AI. Its findings were mostly scanner-level issues, so the model swap made them more accurate but not deeper. Ask vendors about the severity and depth of validated findings, not just how many are correct.
Red Flag: Benchmark Scores Instead of Real-World Results
Capture-the-flag challenges and vulnerability labs are designed around known, planted weaknesses, so an agent being tested on them knows there is something to find. A production network offers no such guarantee. It may contain legacy systems, unusual authentication flows, partly decommissioned services and applications that behave differently from their documentation. That is why the Stanford researchers tested on a live network instead of relying on lab benchmarks.
Their own results show the problem. On a standard benchmark built from capture-the-flag challenges, the researchers’ agent framework performed about the same as a simpler framework running the same model, at 48.6% versus 45.9%. On the live network, the researchers’ framework scored nearly three times higher on the study’s measure of the severity and complexity of what each agent found. The benchmark couldn’t tell them apart, but performance on a real network could.
Treat benchmark scores as one piece of evidence, not proof. Ask for results from environments that resemble yours in size and complexity. Better still, run a scoped pilot on your own assets, because what the system finds, proves and misses there tells you far more than a leaderboard position.
Red Flag: Vague Answers About Scope and Safety Controls
A system that actively probes and exploits live services needs hard limits. A vendor should be able to explain how the agent stays inside the authorized scope, how request rates are throttled and what prevents actions that could damage a production system.
There should also be a kill switch that stops testing immediately and a log of every action the agent took.
The Stanford researchers treated live testing as risky enough to require strict safeguards. The work ran under the university’s vulnerability disclosure policy, which provided safe-harbor protections and prohibited destructive or excessively disruptive activity. A member of the research team watched the agents at all times and could terminate any session, while university IT staff monitored the network.
A commercial system should be able to describe its safeguards with the same specificity. If the answer is that the agent is simply “designed to be safe,” keep asking for more details on what that means.
Red Flag: Unclear Answers About Where Your Data Goes
Pentest data can include vulnerable endpoints, internal architecture, credentials, captured traffic and detailed evidence of how systems can be compromised. It is some of the most sensitive data a company holds, so data handling belongs in the evaluation.
Ask which model provider processes the data, whether prompts or findings are used for model training, where information is stored, how long it is retained and who can access it.
These should be straightforward questions. Vague answers add another unknown to an already sensitive process.
Red Flag: Point-in-Time Testing Sold as Continuous
“Continuous” means very different things depending on which vendor is using it.
One product may rerun the same test every week against a fixed asset list the customer supplied. Another may discover new external assets, bring them into scope and test them as they appear. Both can claim to be continuous, but they work extremely differently in practice.
To sort this out, ask how assets enter the testing scope, how often testing actually runs and what triggers a test. Then ask what happens after a fix. Does the system retest automatically, and how soon? Does it confirm the original attack path no longer works, or simply mark the finding resolved when the customer says it’s fixed?
The goal is to make sure the service being sold matches the service being delivered.
Let the Evidence Decide
AI pentesting has enough real capability to deserve serious evaluation, and enough loosely used vocabulary that buyers should be skeptical of labels on their own. Every red flag above comes down to the same test. Can the vendor show it, on your assets, with evidence?
The right vendor will welcome a pilot and hard questions about validation, scope, safety and data handling. A vendor that resists them has already answered your most important question.