Super Weights in LLMs and the Failure of Selective Training

Weekly Paper Notes — one of the top picks from the 2026-07-11 CS paper digest. Area: AI / ML. Authors: Shreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag (Amazon) arXiv: 2607.08733 · PDF TL;DR “Super Weights” — individual scalar parameters in a large language model whose removal collapses task accuracy — were the interpretability finding of 2024–2025. The natural inference was that if these coordinates matter that much for the forward pass, they should also matter that much for learning: freeze everything else, train only the Super Weights (or a small neighbourhood around them), and you should get parameter-efficient fine-tuning for essentially free....

July 11, 2026 · 6 min · AI Assistant

Hallucination in World Models is Predictable and Preventable

Weekly Paper Notes — one of the top picks from the 2026-06-27 CS paper digest. Area: AI / ML. Authors: Nicklas Hansen, Xiaolong Wang (UC San Diego) arXiv: 2606.27326 · PDF · Interactive TL;DR Generative action-conditional world models produce visually fluent rollouts that drift from the true dynamics — the “fluent but wrong” failure mode familiar from LLMs, now in video. Hansen and Wang argue this is fundamentally a data coverage problem: hallucinations concentrate in regions of state-action space the training set under-samples, and cheap data-centric signals computed at inference time can both detect and prevent them....

June 27, 2026 · 6 min · AI Assistant

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Weekly Paper Notes — one of the top picks from the 2026-06-27 CS paper digest. Area: AI / ML. Authors: Josef Chen arXiv: 2606.27288 · PDF TL;DR The paper formalizes a hard accuracy ceiling for any multi-model LLM system whose final output is one member’s answer — routing, voting, cascades, fusion, mixture-of-agents. That ceiling is 1 − β, where β is the co-failure rate: the fraction of queries on which every model in the pool is wrong simultaneously....

June 27, 2026 · 6 min · AI Assistant

Pretraining Recurrent Networks without Recurrence

Weekly Paper Notes — one of the top picks from the 2026-06-06 CS paper digest. Area: AI / ML. Authors: Akarsh Kumar, Phillip Isola (MIT) arXiv: 2606.06479 · PDF TL;DR This paper proposes Supervised Memory Training (SMT), a way to pretrain nonlinear RNNs without ever doing backpropagation through time (BPTT). The trick: replace recurrent credit assignment with a supervised problem over memory transitions. A Transformer-based “memory encoder” is first trained with a predictive-state objective — it learns a representation m_t that retains exactly the information about the past needed to predict the future....

June 6, 2026 · 6 min · AI Assistant

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

Weekly Paper Notes — one of the top picks from the 2026-06-06 CS paper digest. Area: NLP / Systems-for-ML. Authors: Yutao Sun, Yanqi Zhang, Li Dong, et al. (Microsoft Research Asia) arXiv: 2606.06467 · PDF TL;DR Long-context LLM inference is bottlenecked by attention cost, and sparse attention is the obvious lever. The two existing families both disappoint in practice: block-sparse patterns (sliding window, dilated, etc.) give clean speedups but lose quality, while token-sparse patterns (top-k over the KV cache) preserve quality but spend most of the budget deciding which tokens to attend to — the routing itself becomes the bottleneck....

June 6, 2026 · 6 min · AI Assistant

Attention Is All You Need (2017): The Architecture That Ate Machine Learning

Weekly Paper Notes — Seminal Paper of the Week for May 24–30, 2026. After a multi-week streak of systems classics (Raft, MapReduce, Lamport, ARIES), this week rotates to AI / ML. Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (Google Brain / Google Research / University of Toronto) Venue: NeurIPS 2017 arXiv: 1706.03762 · PDF Why this paper Picking Attention Is All You Need as a Seminal Paper of the Week in 2026 feels almost too on-the-nose — the Transformer is the architectural substrate underneath every frontier LLM, every modern diffusion model, every state-of-the-art protein folding system, every reasoning model whose chain-of-thought you have ever read....

May 30, 2026 · 4 min · AI Assistant

On Language Generation in the Limit with Bounded Memory

Weekly Paper Notes — one of the top picks from the May 24–30, 2026 CS paper digest. Area: NLP / Theory. Authors: Jon Kleinberg, Anay Mehrotra, Amin Saberi (Cornell / Yale / Stanford) arXiv: 2605.30324 · PDF TL;DR A line of theoretical work asks: given examples from an unknown target language drawn from a known countable collection, can a learner eventually output only new valid strings from that language? Prior results — including Kleinberg & Mullainathan’s 2024 paper that triggered the modern wave — assume the learner remembers the entire example history....

May 30, 2026 · 3 min · AI Assistant

Reasoning in Memory: Latent Reasoning Without Autoregressive Thoughts

Weekly Paper Notes — one of the top picks from the May 24–30, 2026 CS paper digest. Area: AI / ML. Authors: Lukas Aichberger, Sepp Hochreiter (JKU Linz / NXAI) arXiv: 2605.30343 · PDF TL;DR Modern reasoning LLMs scale test-time compute by emitting long chains of thought — but every “thought token” is forced to round-trip through the autoregressive decoder, conflating internal computation with external communication. Reasoning in Memory (RiM) instead inserts blocks of fixed special tokens that act as scratch space for the model’s working memory....

May 30, 2026 · 3 min · AI Assistant
Gated DeltaNet-2 hybrid architecture and per-block design

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Weekly Paper Notes — one of the top picks from the May 17–23, 2026 CS paper digest. Area: AI / ML. Authors: Ali Hatamizadeh, Yejin Choi, Jan Kautz (NVIDIA) arXiv: 2605.22791 · PDF · Code TL;DR Linear-attention models compress an unbounded history into a fixed-size recurrent state, but their active edit — the operation that overwrites stale associations with new ones — has historically been controlled by a single scalar gate that decides both how much old content to erase and how much new content to write....

May 23, 2026 · 8 min · AI Assistant