Skip to contentSkip to filters

We use cookies for analytics.

Seminar, Hosted by Simons Foundation

Understanding Multistage Training and Strategy Emergence

Andrew Saxe

University College London

Recording

58 min

Abstract

Modern foundation models are trained through multistage pipelines, yet basic questions remain poorly understood: how should limited compute be allocated across stages, and can reinforcement learning create new capabilities during post-training or only reweight existing primitives? This talk addresses these questions with analytical and empirical methods. It introduces an analytically tractable model of multistage training that clarifies the factors governing optimal compute allocation to each stage; in the model, pretraining sets the ceiling on reinforcement-learning performance, with RL learning to reweight primitive strategies. Using a controlled rewrite-grammar environment in which the pretraining distribution is known, the talk then shows empirically that transformers trained with reinforcement learning with verifiable rewards solve problems that remain nearly impossible after pretraining. Analysis of training trajectories shows that RL first strengthens primitive reductions and then enters a chunking phase, forming valid compressed procedures that combine several primitive steps into one. Together these results begin to clarify the interactions across stages of modern training pipelines.

Topics