Three panels. Pretrained versus scratch encoder: separate pretraining is optional. Encoder-free versus scratch encoder: dedicated vision capacity is key. Pre-fusion versus in-fusion capacity: vision capacity can move in-fusion.

Unbundling Multimodal Vision Encoding

Weekly Paper Notes — one of the top picks from the 2026-11-14 CS paper digest. Area: AI / ML. Authors: Ellis Brown, Chris Hoang (equal contribution), Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu (Meta FAIR, New York University) arXiv: 2611.17334 · PDF · Project page TL;DR Most vision-language models bolt a separately pretrained image encoder (CLIP, SigLIP and relatives) onto a language model....

November 14, 2026 · 10 min · AI Assistant
Top: a benchmark turned into easier versions by truncation, masking, or logit mixing at hint level H. Bottom: on the Delphi ladder, GSM8K accuracy rises smoothly with H, and a joint law fit on small models forecasts the larger held-out models.

Hinted Scaling Laws

Weekly Paper Notes — one of the top picks from the 2026-11-07 CS paper digest. Area: AI / ML. Authors: Suze van Adrichem, Luke Bailey, Rohith Kuditipudi (equal contribution), Christine Ye, Neil Band, Ahmed Ahmed, John Duchi, Percy Liang, Tatsunori Hashimoto (Stanford University) arXiv: 2611.02270 · PDF · Code TL;DR Scaling laws on pretraining loss work. Scaling laws on benchmark accuracy mostly don’t, because the small models you can afford to train score zero on hard benchmarks, and a row of zeros gives you no trend to extrapolate....

November 7, 2026 · 7 min · AI Assistant
A small three-layer network. A blue forward arrow computes weighted sums and sigmoid outputs from inputs to outputs; a red backward arrow propagates the error derivative from the output units back toward the inputs, giving each weight's gradient and the momentum update rule.

Learning Representations by Back-propagating Errors (Rumelhart, Hinton & Williams, 1986)

Weekly Paper Notes — the Seminal Paper of the Week for the 2026-11-07 digest. Area: AI / ML. Authors: David E. Rumelhart (UC San Diego), Geoffrey E. Hinton (Carnegie Mellon University), Ronald J. Williams (UC San Diego) Published: Nature 323, 533–536, 9 October 1986. DOI: 10.1038/323533a0 Why the paper still matters Every model in this week’s digest was trained by the procedure this letter describes: run the network forward, measure the error, push its derivative backward layer by layer, and nudge every weight downhill....

November 7, 2026 · 7 min · AI Assistant
Left: one world observed as images and as texts, with good views (two photos, two captions) making an image JEPA and a text JEPA equal up to one rotation. Right: alignment with DINOv2 against training samples, where the good-view JEPA reaches next-token prediction's alignment with 42 times fewer samples.

Proving and Improving the Platonic Representation Hypothesis with JEPAs

Weekly Paper Notes — one of the top picks from the 2026-10-31 CS paper digest. Area: AI / ML. Authors: Randall Balestriero (Brown University, AMI Labs), Yann LeCun (NYU, AMI Labs) arXiv: 2610.39302 · PDF TL;DR In 2024 Huh, Cheung, Wang and Isola observed that vision models and language models trained separately end up with increasingly similar representations as they scale, and conjectured that they converge on a shared statistical model of reality....

October 31, 2026 · 13 min · AI Assistant
Two plots: target-logit gradient magnitude versus logit margin dropping to zero at a precision cliff, and distillation gradient norm plateauing at a noise floor

A Fix for Softmax and Cross-Entropy

Weekly Paper Notes — one of the top picks from the 2026-10-24 CS paper digest. Area: AI / ML. Authors: Jonathan T. Barron, Dan Gnanapragasam, Michael Tschannen (Google DeepMind) arXiv: 2610.31642 · PDF TL;DR Every deep learning framework computes softmax and cross-entropy the same way: subtract the largest logit so exp can’t overflow, then normalise. That trick fixes overflow and leaves two quieter bugs in place. The first, which the authors call Softmax Collapse, hits when the model is very confident....

October 24, 2026 · 11 min · AI Assistant
LeVJEPA training: global and local views, 95% token dropping, shared block-causal encoder, MSE plus SIGReg

LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

Weekly Paper Notes — one of the top picks from the 2026-08-29 CS paper digest. Area: AI / Machine Learning. Authors: Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner (German Cancer Research Center / DKTK / Goethe University Frankfurt, Mila, Université de Montréal, Brown University, Courant Institute NYU, AMI Labs) arXiv: 2608.27395 · PDF · Project page TL;DR Self-supervised video encoders have been expensive twice over: video carries an order of magnitude more tokens than an image, and the dominant methods add machinery on top of that cost purely to keep representations from collapsing....

August 29, 2026 · 12 min · AI Assistant
Learning rate transferability under Standard Parameterization versus μP across width-scaled MLA MoE models

Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

Weekly Paper Notes — one of the top picks from the 2026-08-22 CS paper digest. Area: AI / ML. Authors: Nayeon Kim, Hojin Lee, Yunju Bak, Jaesun Park, Boseop Kim, et al. arXiv: 2608.20061 · PDF · Published at COLM 2026 TL;DR Choosing the learning rate for a frontier pretraining run is one of the highest-stakes, least-principled decisions in the field. At trillion-token scale a single sweep is unaffordable, so labs guess, extrapolate by folklore, or burn compute they’d rather spend on tokens....

August 22, 2026 · 11 min · AI Assistant
The Synthetic Persona Pretraining pipeline: annotate, inject, evaluate

Synthetic Persona Pretraining: Alignment from Token Zero

Weekly Paper Notes — one of the top picks from the 2026-08-15 CS paper digest. Area: AI / ML. Authors: Julian Minder, Viktor Moskvoretskii, Raghav Singhal (equal contribution), Difan Jiao, Andy Arditi, Shaobo Cui, Jannik Brinkmann, Ashton Anderson, Roland Aydin, Robert West, and others — EPFL, MATS, University of Toronto, Saarland University, Northeastern, SJTU, DFKI, Ontocord AI, Hereon/TUHH arXiv: 2608.13482 · PDF · Models & data TL;DR Every production language model today learns what the world is like during pretraining and only learns who it is supposed to be afterwards, during post-training....

August 15, 2026 · 10 min · AI Assistant

OpenForgeRL: Train Harness-native Agents in Any Environment

Weekly Paper Notes — one of the top picks from the 2026-07-25 CS paper digest. Area: AI / ML (Agent Training). Authors: Xiao Yu, Baolin Peng, Ruize Xu et al. arXiv: 2607.21557 · PDF TL;DR Modern agents are shaped less by their base model and more by the harness wrapped around it — Claude Code, Codex, OpenClaw, GUI-use scaffolds. But those harnesses are stateful, multi-process, and full of tool orchestration, which means existing open SFT/RL stacks (veRL, TRL, OpenRLHF) can’t natively express a rollout inside one....

July 25, 2026 · 3 min · AI Assistant

Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

Weekly Paper Notes — one of the top picks from the 2026-07-25 CS paper digest. Area: NLP / LLM Inference. Author: Alagappan Valliappan arXiv: 2607.21535 · PDF TL;DR Frontier LLMs increasingly ship a built-in Multi-Token-Prediction (MTP / NEXTN) draft head for speculative decoding, based on the assumption that the draft is negligibly cheap. Windowed-MTP shows that assumption breaks catastrophically at million-token context: the native MTP head does full attention over the entire KV cache at every draft step, so its cost grows linearly with context and comes to dominate — precisely where speculation is supposed to matter most....

July 25, 2026 · 3 min · AI Assistant