1 h 6 min
Seminar, Hosted by Simons Institute for the Theory of Computing
Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression
Stanford University
Recording
Abstract
Surya Ganguli varies the diversity of pretraining tasks to study when transformers learn to solve new regression problems from prompt examples. With limited diversity, predictions resemble a Bayesian estimator tied to the training tasks. Above an empirical threshold they resemble ridge regression and generalize beyond those tasks, revealing a role for task diversity alongside data and model scale.