Scaling to Long Horizons: What Galactica Taught Us About RL

Most retellings of the modern AI wave start with ChatGPT arriving out of nowhere in November 2022. Ross Taylor has a different vantage point: he shipped a competing language model two weeks earlier, watched it get torn apart in public, and spent the following four years working out exactly why. This talk is the compressed version of that education — half war story, half technical agenda for what comes after the current generation of agents.

Taylor was the reasoning lead at Meta AI (Papers With Code → Galactica → Llama 2/3 post-training) and now runs General Reasoning, a London RL lab. He’s joined by co-founder Chengxi Taylor for the forward-looking half.

The natural experiment nobody asked for

Galactica and ChatGPT launched two weeks apart on comparable base models. Galactica shipped as a raw base-model demo — it hallucinated, it happily wrote a research paper on Dyson spheres or a report on the benefits of eating crushed glass, and the internet reacted accordingly. ChatGPT shipped with RLHF on top of GPT-3.5.

“A good base model is not enough.”

Taylor frames this, with visible dark humour, as the first clean natural experiment demonstrating that RL creates product value — and one he happened to be on the wrong side of. The signal was already public: InstructGPT had shown a 1B-parameter RLHF’d model beating a 175B model, two orders of magnitude fewer parameters for better results.

What Galactica actually got right

The uncomfortable part of the story is that Galactica was a genuinely strong model. It beat PaLM, Chinchilla, and GPT-3.5 in scientific domains at a fraction of the compute — 68% vs 49% on LaTeX/science tasks against GPT-3.5, and 36% vs 19% chain-of-thought against a 540B PaLM with a 30B model.

Three ideas from that paper aged extremely well:

1. Data efficiency. A 105B-token curated corpus against Chinchilla’s trillion. Deeply contrarian in 2022, when the consensus was simply “more tokens.”
2. Multi-epoch training. The “don’t train more than one epoch” rule was orthodoxy; Galactica produced the first real empirical result against it, later formalised as the ~4-epoch repeated-data rule.
3. Thinking tokens. Buried in the paper: the argument that intermediate reasoning is an internal working memory process that belongs inside dedicated tags where you spend inference compute before answering. Not chain-of-thought prompting, not scratchpads — the thing the industry now calls reasoning traces.

The reasoning recipe that almost worked

Taylor’s personal obsession was applying RL pressure directly to what happens inside those thinking tags. Two years before R1 and o1, his team at Meta ran an unpublished recipe:

1. Continue pre-training Llama 2 on mathematics and science data (Llama 2’s math corpus was, in his words, terrible).
2. PPO with verifiable rewards — notably not GRPO — using a strong outcome reward model to initialise the value model.

Internally this produced state-of-the-art math and reasoning results. What it did not produce was inference-time scaling: no backtracking, no reflection, none of the emergent behaviour that became the hallmark of o1 and R1.

The answer arrived two years later, and it was the bitter lesson in its purest form — better base models, more RL compute, bigger context windows. Nothing clever required. Taylor draws a sharper sociological point out of it:

“The fact that OpenAI had a GPT-4 level model before anyone else allowed them to see further.”

In an age of scaling, prerequisites compound into perception. If you don’t have the base model, you can’t see the next idea.

Long horizon is a mindset, not an engineering ticket

Chengxi Taylor takes the second half. The framing: if we want AI to cure cancer, solve Millennium Prize problems, or get to Mars, the unit of work has to stretch from minutes to months. Fermat’s Last Theorem took a mathematician over a decade of reading, scratchpad thinking, and walks — tens or hundreds of billions of tokens if you convert it. We have a 1M-token context window.

The near-term workaround is compaction: generate to the end of the context window, summarise, continue. The elegant move is applying RL to the compaction step and the task simultaneously — one stone, two birds.

Three problems that break RL at length

1. Gradient variance scales with trajectory length.
2. Sparse reward creates a brutal credit assignment problem.
3. Variable-length trajectories further destabilise optimisation.

Their answer is to bring back the critic — a value model. It reduces variance, operates at trajectory level (which fits compaction naturally), encourages batch diversity, and enables bootstrapping: getting a learning signal before the episode ends. The cost is honest: it’s more complicated than GRPO, and you’re now training a second model alongside the policy.

Tooling helps too — filesystem tools as external scratchpad, self-search over prior trajectory, archive tools to build on previous results. With a caveat: in some settings an archive lets the agent cheat by retrieving the old answer instead of thinking.

Frontier models lose all the money

General Reasoning built Kelly Bench: agents build ML models to bet on Premier League football matches over a one-year horizon, starting with $100K. Real money, real-world implications, an actual industry — not a Kaggle leaderboard. It landed on the front page of the Financial Times.

Every frontier model lost the money.

The diagnosis is a critique of the field’s current taste: the industry is over-indexed on coding and procedural tasks (“do this, fix that”) where the solution space is one or two options and creativity has nowhere to go. Not enough focus on open-ended tasks. Not enough simulation of a world containing other players with different objectives and different games in their heads.

The compute trade-off nobody escapes

Long horizons wreck the training/inference balance. Classically you finish inference, then train. Pipeline RL starts training while sequences are still generating — buying GPU utilisation with off-policy staleness. In General Reasoning’s experience, up to about eight steps off-policy is fine.

But when a single inference run takes weeks, you blow straight past that eight-step budget and your GPUs idle. The escape hatch is again the value model: bootstrap an expectation before the episode ends — “like dopamine in the human brain” — and train on that. You get full GPU utilisation and you pay in value-model bias. Every solution here is a trade, not a fix.

Infrastructure matters as much as algorithms: General Reasoning runs Open Review (openreview.ai), hosting 350+ RL environments behind a single API endpoint, used internally and by frontier and new labs.

Key takeaways

1. A state-of-the-art base model is not a product. Galactica beat PaLM and GPT-3.5 in its domain and still failed publicly; ChatGPT’s RLHF layer is what crossed the Rubicon.
2. Galactica’s real legacy is three ideas — data curation over raw token count, multi-epoch training, and thinking tokens as internal working memory — all now industry defaults.
3. The o1/R1 unlock was the bitter lesson, not a trick. Better base models plus more RL compute plus longer context produced reflective behaviour that a 2023 recipe with the same objective could not.
4. Capability compounds into foresight. Whoever holds the strongest base model sees the next research idea first.
5. Long-horizon RL breaks on three axes — gradient variance with length, sparse reward and credit assignment, and variable trajectory length.
6. Value models are back. They cut variance, fit compaction at trajectory level, and enable mid-episode bootstrapping — at the cost of a second trained model and value bias.
7. Frontier models are bad at genuinely open-ended tasks. Every model tested on Kelly Bench lost real money over a one-year horizon.
8. Pipeline RL is a GPU-utilisation trade, not a free lunch — and it stops working entirely when inference takes weeks.

Source

• Talk: Scaling to Long Horizons
• Speakers: Ross Taylor (CEO) & Chengxi Taylor (co-founder/president), General Reasoning
• Origin: AI Engineer conference, published 31 July 2026
• Duration: 18m 07s
• URL: https://www.youtube.com/watch?v=2bvtay8wGYI