Much AI security testing checks whether a model can be tricked into saying something harmful. But agents act. They send email, run commands and change databases, and the damage depends on what they're allowed to reach. So I wanted to know whether the testing is aimed at the wrong target.
The main industry guidance, from OWASP, already treats permission limits as essential and tells builders to assume the model will be fooled and to limit what it can do. Despite that, none of the twelve agent security benchmarks I read changes what the agent is allowed to access; they compare models, not deployments. Where researchers did restrict what an agent could use, attacks dropped sharply, sometimes at a cost in usefulness. Of the security products I sampled, most check the text going in and out of the model and fewer control what the agent is permitted to do.
There's a decent case the other way. Better-trained models are much harder to hijack in agent tests, so model testing clearly isn't useless, and permission controls in real coding tools have been bypassed, so they're not a complete answer either. Government bodies and model makers do test agents realistically. Several real harms also came from what a model said or agreed to do rather than from its permissions, the Air Canada chatbot case among them.
So I can't actually answer this, which isn't what I expected to write. No public source measures how much testing is output-only, and none classifies agent incidents by cause, so the claim can't be shown or refuted either way. What I'd say is narrower: the benchmarks everyone cites test the model and not the deployment, and if the damage depends on permissions then that's a gap worth noticing, even if I can't size it.