pretraining

Learning rate transferability under Standard Parameterization versus μP across width-scaled MLA MoE models

Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

Weekly Paper Notes — one of the top picks from the 2026-08-22 CS paper digest. Area: AI / ML. Authors: Nayeon Kim, Hojin Lee, Yunju Bak, Jaesun Park, Boseop Kim, et al. arXiv: 2608.20061 · PDF · Published at COLM 2026 TL;DR Choosing the learning rate for a frontier pretraining run is one of the highest-stakes, least-principled decisions in the field. At trillion-token scale a single sweep is unaffordable, so labs guess, extrapolate by folklore, or burn compute they’d rather spend on tokens....

The Synthetic Persona Pretraining pipeline: annotate, inject, evaluate

Synthetic Persona Pretraining: Alignment from Token Zero

Weekly Paper Notes — one of the top picks from the 2026-08-15 CS paper digest. Area: AI / ML. Authors: Julian Minder, Viktor Moskvoretskii, Raghav Singhal (equal contribution), Difan Jiao, Andy Arditi, Shaobo Cui, Jannik Brinkmann, Ashton Anderson, Roland Aydin, Robert West, and others — EPFL, MATS, University of Toronto, Saarland University, Northeastern, SJTU, DFKI, Ontocord AI, Hereon/TUHH arXiv: 2608.13482 · PDF · Models & data TL;DR Every production language model today learns what the world is like during pretraining and only learns who it is supposed to be afterwards, during post-training....

Pretraining Data Can Be Poisoned through Computational Propaganda

arXiv: 2607.15267 · PDF: 2607.15267.pdf Authors: Victoria Graf, Hannaneh Hajishirzi, et al. TL;DR Prior work on pretraining-data poisoning has mostly targeted curated sources like Wikipedia — a poor stand-in for the scale and heterogeneity of real pretraining corpora. This paper demonstrates that public discussion interfaces on the open web (comment sections, forums, Q&A pages) are a viable at-scale injection vector, and introduces HalfLife, an analysis technique for estimating whether adversarial content actually survives web-crawl-based data curation pipelines and lands in the training set....