Weekly Paper Notes — one of the top picks from the 2026-11-07 CS paper digest. Area: AI / ML.
Authors: Suze van Adrichem, Luke Bailey, Rohith Kuditipudi (equal contribution), Christine Ye, Neil Band, Ahmed Ahmed, John Duchi, Percy Liang, Tatsunori Hashimoto (Stanford University) arXiv: 2611.02270 · PDF · Code
TL;DR
Scaling laws on pretraining loss work. Scaling laws on benchmark accuracy mostly don’t, because the small models you can afford to train score zero on hard benchmarks, and a row of zeros gives you no trend to extrapolate.
This paper adds a second axis. Give the model a hint drawn from a reference solution, with the hint level H running from 0 (the original benchmark) up to the full solution. Even tiny models score above zero once the hint is big enough, and accuracy turns out to rise smoothly with H and with model scale together. Fit one six-parameter sigmoid over both, then read off its H = 0 slice at the target scale.
On the Delphi pretraining ladder, a law fit only on models of at most 10²⁰ FLOPs predicts the GSM8K accuracy of a 10²³ FLOP model (25.9%) within 3 points. A standard scaling law misses by 13 points on the same data, and still by 12 when given models up to 10²² FLOPs. Across Delphi and four post-trained families (Qwen2.5, Qwen3, Qwen3.5, Gemma-3) on GSM8K, AIME and Humanity’s Last Exam, median forecast error drops from 21 points to 4.3. The same trick ranks near-identical weak models with one hinted rollout per problem about as well as five unhinted ones.
Three ways to make a benchmark easier
A benchmark here is a problem set plus a sampling protocol. A hinting method keeps the problems and changes the protocol so they get easier as H grows. It has to equal the original benchmark at H = 0, and accuracy should rise predictably in both H and model scale.
The paper uses three:
- Truncation puts the first H fraction of the reference solution in the prompt, cut at a word boundary and reworded so it never states the final answer.
- Masking puts the whole solution in the prompt but replaces a random 1 − H fraction of its words with
[MASK]. - Logit mixing decodes the evaluated model together with a stronger teacher that shares its tokenizer, sampling from
z_model + H · z_teacher. At H = 0 it’s the model alone. As H grows the teacher takes over.
Truncation and masking need only reference solutions. GSM8K ships with them, and for AIME and HLE the authors generated them. Logit mixing needs a teacher. For Delphi they used Llama-3.1-8B-Instruct with the reference solution in its context.
Hint level trades for scale
The empirical result that makes the method work is simple. For every model, accuracy rises with H along a sigmoid. At every fixed H, accuracy rises with log-compute along a sigmoid. And the curves rarely cross, so bigger models stay ahead at every hint level.
On Delphi, the unhinted GSM8K score sits on the noise floor until 10²² FLOPs. With masking at H = 0.5, every model from 9×10¹⁸ FLOPs upward scores above 7%. Plot iso-accuracy contours over the (scale, hint) plane and they come out as roughly parallel diagonals. Some fixed amount of hint is worth some fixed amount of compute, at every accuracy level.
That shape motivates the fitted law, which the authors call the floored sigmoid:
L(C, H) = max( F, L + (A − L) · σ(β₁·C + β₂·H + β₃) )
C is log₁₀ FLOPs (or parameters for post-trained families). A lets accuracy saturate below 1, and the floor F lets it sit flat until an emergence point. Six parameters per family, against 42 to 102 if you fit every slice separately, and the joint fit costs at most 1.7 extra points of mean absolute error.
To forecast, fit L(C, H) to all hint levels of the n smallest models, then evaluate L(C_large, 0). The baseline sees exactly the same models and the same unhinted scores. The only difference is whether the hinted scores are available.
Results
Figure: Logit mixing on Delphi/GSM8K. Iso-accuracy contours are close to parallel straight lines, which is what the floored sigmoid assumes. Source: arXiv:2611.02270, Figure 7.
Post-trained families. With masking and truncation averaged into a “hinting ensemble”, the hinted forecast beats the baseline on the largest model in 17 of 19 panels. The baseline’s failures are often extreme. Fit on the two smallest Qwen2.5 models, which score near zero on AIME, it predicts close to 100% at 32B against a true 12%, an 88-point miss. The hinted forecast from the same two models is off by 3.0 points.
Pretraining ladder. Masking and truncation alone help only as much as a weak model can use the solution text. Delphi models below 10²² FLOPs score at most 12% even when shown the whole solution, and the two methods under- and over-forecast respectively when extrapolating past 10× compute. Logit mixing with a teacher fixes that, since the teacher pulls every model up toward a shared ceiling. Mean forecast error across nine training sets is 3.3 points, worst case 6.6 when extrapolating from 0.03% of target compute. Seeds matter: with only one seed per budget, mean error rises to 10.2 points, so the authors trained extra seeds for the small budgets.
Ranking. Hinting spreads near-zero models apart while mostly keeping their order. On a set of seven models between 0.06% and 0.81% on AIME, a ranking that spends one hinted rollout per problem beats plain avg@k by 0.33 in Kendall’s τ at a budget of one rollout. Avg@k needs about five rollouts on AIME (six on HLE) to catch up. The combined score shifts weight to unhinted accuracy as the budget grows, so it never falls more than 0.02 τ behind.
What doesn’t work, and why that matters
Two negative results make the method easier to trust.
First, not every hint is informative. A trivial method that hands the model the correct answer with probability H looks like hinting but carries no information about the model. The appendix proves that any method whose hinted accuracy depends only on the model’s unhinted accuracy can’t beat a forecaster that sees unhinted scores alone. When every small model scores zero, such a method yields the same forecast for every ladder. A useful hint has to separate models that score the same at H = 0 according to their real capability.
Second, logit mixing with a teacher drawn from the same post-trained family produced poor forecasts. Those teachers wrote unnatural solutions that mentioned “the hint” explicitly. Logit mixing also has a quirk: at high H, a model with near-uniform outputs can beat a stronger one because it lets the teacher through unopposed, so capability slices aren’t guaranteed monotone.
Why this matters
Downstream scaling laws are what you’d want for choosing data mixes and recipes by the benchmark you actually care about, and for safety teams asking at what scale a dangerous capability shows up. Until now the options were to fit only models already above chance, switch to a continuous proxy like answer likelihood, or pay for huge pass@k sampling. Hinting keeps the real metric and gets signal from models that score zero on it.
The ranking result may matter sooner in practice. The discussion section prices a single run of UK AISI’s cyber-range evaluation at $3,000–$5,000 per rollout per frontier model. If a hinted rollout, such as replaying part of a successful agent trajectory and letting the model finish, can stand in for several unhinted ones, expensive agentic evaluations get cheaper. The paper doesn’t test agentic benchmarks, so that’s a hypothesis for now.
The limits are clear. Delphi is the only true single-recipe scaling ladder in the study, so it’s hard to separate method error from quirks of the released model families. And every method needs verified, readable reference solutions, which get harder to write as benchmarks move toward the frontier.
Read alongside
- Kaplan et al., Scaling Laws for Neural Language Models (2020), and Hoffmann et al., Training Compute-Optimal Large Language Models (2022): the loss-based laws this work extends to accuracy.
- Wei et al., Emergent Abilities of Large Language Models (2022), and Schaeffer et al., Are Emergent Abilities of Large Language Models a Mirage? (2023): the debate over why accuracy curves look discontinuous.
- Lourie et al., Scaling Laws Are Unreliable for Downstream Tasks (2025): the failure mode this paper sets out to fix.
- Jones, Scaling Scaling Laws with Board Games (2021): the closest precedent, where game size plays the role hint level plays here.
Links
📄 arXiv abstract · 📄 PDF · 💾 tatsu-lab/hinted-scaling-laws
Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint; figures from the arXiv HTML version (CC BY 4.0).