58 min
Seminar, Hosted by Simons Foundation
Understanding Multistage Training and Strategy Emergence
University College London
Recording
Abstract
Modern foundation models are trained through multistage pipelines, yet basic questions remain poorly understood: how should limited compute be allocated across stages, and can reinforcement learning create new capabilities during post-training or only reweight existing primitives? This talk addresses these questions with analytical and empirical methods. It introduces an analytically tractable model of multistage training that clarifies the factors governing optimal compute allocation to each stage; in the model, pretraining sets the ceiling on reinforcement-learning performance, with RL learning to reweight primitive strategies. Using a controlled rewrite-grammar environment in which the pretraining distribution is known, the talk then shows empirically that transformers trained with reinforcement learning with verifiable rewards solve problems that remain nearly impossible after pretraining. Analysis of training trajectories shows that RL first strengthens primitive reductions and then enters a chunking phase, forming valid compressed procedures that combine several primitive steps into one. Together these results begin to clarify the interactions across stages of modern training pipelines.