Weekly Paper Notes — one of the top picks from the 2026-07-11 CS paper digest. Area: AI / ML.

Authors: Shreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag (Amazon) arXiv: 2607.08733 · PDF

TL;DR

“Super Weights” — individual scalar parameters in a large language model whose removal collapses task accuracy — were the interpretability finding of 2024–2025. The natural inference was that if these coordinates matter that much for the forward pass, they should also matter that much for learning: freeze everything else, train only the Super Weights (or a small neighbourhood around them), and you should get parameter-efficient fine-tuning for essentially free. Subramanian et al. show, in a careful 10-seed ablation on OLMo-1B and OLMo-7B, that this is false in the strongest possible way. Training 100 to 8,192 Super Weight coordinates in isolation drives accuracy to random-guessing levels. Expanding the neighbourhood to 36,000 parameters doesn’t fix it. Training the same number of random coordinates in the same down_proj layers actually improves over baseline. Vanilla LoRA on attention matrices, using only 0.16% of parameters, works. The takeaway is deceptively simple and important: parameter importance in the forward pass does not imply parameter trainability in isolation. Super Weights are load-bearing for inference but not levers for adaptation.

What problem is the paper actually attacking?

The lineage here is the “individual weights matter” strand of LLM interpretability. Dettmers et al.’s outlier features work established that a small number of activation channels carry most of the signal that survives quantisation. Yu et al.’s Super Weights paper (2024) sharpened this from features to individual scalar parameters: it identified a handful of specific down_proj positions whose ablation would collapse a 7B model’s zero-shot benchmarks to near random. Subsequent work extended the finding to more architectures and used it to design better quantisation schemes (preserve the Super Weights at full precision, quantise the rest aggressively).

The natural next step — implicit in a lot of Twitter takes and at least one workshop paper — is that Super Weights should be the ideal target for parameter-efficient fine-tuning. If 100 scalars carry order-of-magnitude importance for the forward pass, then updating those 100 scalars should be an incredibly cheap way to adapt the model. It’s a beautiful hypothesis. This paper is the negative result that kills it.

The observation that motivates the study is that “important for what’s already there” and “important for changing what’s there” are different questions. The gradient of the loss with respect to a Super Weight is not necessarily large, and even if it is, updating a Super Weight in isolation moves the model along a direction the rest of the network can’t compensate for. Ablation and training are asymmetric operations.

The mechanism: a clean negative-result design

The experimental setup is the paper’s main contribution and it is worth stating precisely. Four training regimes are compared, all with the same total optimisation budget, on OLMo-1B and OLMo-7B fine-tuned for standard downstream benchmarks:

  1. Train Super Weights only. Freeze every parameter in the model except the identified Super Weight coordinates (100, 1024, 4096, 8192 parameters in successive experiments).
  2. Train Super Weights + local neighbourhood. Expand the trainable set to include up to ~36,000 coordinates in the same down_proj rows/columns.
  3. Train the same number of randomly chosen coordinates in the same down_proj layers. This is the critical control: same layer, same count, different positions.
  4. Vanilla LoRA on attention weight matrices (rank-adapter updates), matched to a small parameter budget.

A separate constrained-LoRA experiment (10 seeds) further pins down the failure: run vanilla LoRA, then mask out the low-rank update wherever it would touch a Super Weight coordinate. This tests whether avoiding Super Weights during LoRA hurts — the answer is that the constrained and unconstrained runs are statistically indistinguishable.

Why the accounting is airtight

The design closes off almost every escape hatch a reviewer might raise. If Super Weight training failed simply because you were training too few parameters, the random-coordinate control would fail too — but it improves over baseline. If it failed because the layer is wrong, LoRA on the same down_proj would fail too — but it succeeds. If it failed because LoRA specifically helps by touching Super Weights, the masked-LoRA experiment would show a gap — but it doesn’t. The failure is specific to the targeting of Super Weight coordinates in isolation.

There is a subtler point about universality that the paper flags in passing. The abstract notes that “degradation due to pruning Super Weights does not universally apply to all LLMs.” Some architectures survive Super Weight ablation better than others; the phenomenon is real but not uniform. This is worth pairing with the training result: not every model has a clear Super Weight population, and even when it does, the population isn’t the right lever for adaptation.

Results

The headline results, summarised from the abstract:

  • Super-Weight-only training on OLMo-1B and OLMo-7B drops accuracy to random-guessing levels across the tested benchmarks, at every parameter budget from 100 up to 8,192.
  • Expanding to a 36K local neighbourhood provides no improvement — the collapse is not about parameter count.
  • Random coordinate training in the same layers improves over baseline. This is the smoking gun that the failure is Super-Weight-specific, not sparsity-specific.
  • Vanilla LoRA on attention matrices succeeds with only ~0.16% of parameters trainable.
  • LoRA on down_proj also succeeds.
  • Masked LoRA (LoRA with updates zeroed at Super Weight positions) is statistically indistinguishable from unconstrained LoRA over 10 seeds.

The overall picture: the model can adapt through low-rank updates distributed across many positions, including positions in the same layer as Super Weights, but it cannot adapt through concentrated updates at exactly those positions. The Super Weights are, in a real sense, load-bearing scaffolding — you can neither remove them nor swing them around without the rest of the network coming down.

Why this matters

Two things. First, this is a clean and much-needed corrective to the “individual weights matter” narrative. Interpretability findings that identify important parameters have been quietly conflated with prescriptions for how to train those parameters. The paper draws a sharp line: importance for inference ≠ trainability in isolation, and any PEFT method that assumes otherwise is optimising against a false intuition.

Second, it is a small but pointed piece of evidence for the “LoRA works because it distributes updates, not because it targets” school of PEFT theory. If concentrated updates at the most important positions collapse the model but distributed low-rank updates ignoring those positions succeed, then the geometry of LoRA — updating everything through a rank constraint — is doing something the targeted approaches structurally can’t. This connects the empirical result to a growing set of theoretical papers on why low-rank fine-tuning generalises so well.

The obvious follow-ups: does the failure hold at 70B+ scale? Does it hold for models with less pronounced Super Weight populations? What’s the minimal distribution over positions that works — how localised can updates get before they collapse? The paper is a starting point for those questions, not the end of them.

Read alongside

  • Yu, Bandarkar, et al., The Super Weight in Large Language Models (2024) — the finding this paper stress-tests.
  • Dettmers et al., LLM.int8() / SpQR — the earlier outlier-features line that Super Weights sharpened.
  • Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021) — the PEFT baseline that keeps winning.
  • Recent work on the geometry of low-rank fine-tuning (Malladi et al., Zhu et al., 2024–2025) — the theoretical companion to this negative result.

📄 arXiv abstract · 📄 PDF

Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint’s abstract and framing; readers should consult the PDF for the full ablation tables and seed statistics.