Weekly Paper Notes — one of the top picks from the 2026-07-25 CS paper digest. Area: NLP / LLM Inference.

Author: Alagappan Valliappan arXiv: 2607.21535 · PDF

TL;DR

Frontier LLMs increasingly ship a built-in Multi-Token-Prediction (MTP / NEXTN) draft head for speculative decoding, based on the assumption that the draft is negligibly cheap. Windowed-MTP shows that assumption breaks catastrophically at million-token context: the native MTP head does full attention over the entire KV cache at every draft step, so its cost grows linearly with context and comes to dominate — precisely where speculation is supposed to matter most. Worse, the effect compounds with draft length (deep native drafts can go net-negative — slower than no speculation at all) and sharpens under hybrid / linear-attention targets, where cheap verification leaves the draft’s full-attention read exposed. The fix: apply a StreamingLLM-style sliding window plus attention sink to the draft’s attention only, leaving verification untouched. It is training-free, drop-in, and lossless by construction — the full-attention target still decides every accepted token, so windowing changes what is proposed, never what is accepted. Across three architecture families (Qwen GDN-MoE 35B/122B and a Mamba2-hybrid NoPE 120B) at 1M context on a single GPU in SGLang, windowing cuts per-decode-step cost by +28% to +44%, an input-invariant margin that widens with context. Unread draft KV (~7.7–11% of total at 1M) is reclaimed via a compact ring buffer at no acceptance cost.

Why this matters

Speculative decoding is the most reliable inference speedup of the last two years — but it has a nasty scaling failure mode that vendors have politely under-reported: at long context the draft stops being free. This paper isolates the exact cause (the draft head reads the whole KV every step) and applies the smallest possible fix (window the draft, not the verifier). The insight that “lossless” only needs to hold at the verifier, not the drafter is the kind of asymmetric constraint that unlocks a whole design space — you can now use much cheaper, much sloppier draft heads as long as the target is still doing full attention on the accepted tokens.

The finding that a deep native MTP head can go net-negative — slower than no speculation — is the sort of empirical detail that will change how frontier labs ship their next model. Expect to see either shallower default MTP heads or windowed drafts baked into the reference inference implementations within a quarter. The paper’s second contribution — reclaiming the unused draft KV via a ring buffer — is a smaller but very practical win: 7–11% of total KV at 1M is measurable memory, and getting it back for free at long context makes single-GPU million-token serving materially cheaper.

Especially notable is the interaction with hybrid / linear-attention targets (Mamba2-hybrid NoPE at 120B): when verification is cheap, the draft’s full-attention read is a larger fraction of total cost, and the windowing win is proportionally bigger. This is a rare paper where the fix scales better on the newest architectures rather than being a legacy patch.

Read alongside

  • StreamingLLM (Xiao et al., 2023) — the attention-sink + sliding-window primitive Windowed-MTP borrows.
  • Medusa / EAGLE / MTP heads — the draft-head lineage the paper is critiquing.
  • SGLang / vLLM speculative-decoding docs — where this will land first.
  • DeepSeek-V3 MTP paper — the reference for how native MTP heads are trained today.
  • H2O / SnapKV — sibling KV-eviction approaches that operate on the verifier instead.

📄 arXiv abstract · 📄 PDF

Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint abstract.