Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Weekly Paper Notes — one of the top picks from the 2026-07-25 CS paper digest. Area: NLP / LLM Inference. Author: Alagappan Valliappan arXiv: 2607.21535 · PDF TL;DR Frontier LLMs increasingly ship a built-in Multi-Token-Prediction (MTP / NEXTN) draft head for speculative decoding, based on the assumption that the draft is negligibly cheap. Windowed-MTP shows that assumption breaks catastrophically at million-token context: the native MTP head does full attention over the entire KV cache at every draft step, so its cost grows linearly with context and comes to dominate — precisely where speculation is supposed to matter most....