Skip to contentSkip to filters

We use cookies for analytics.

Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression

Surya Ganguli

Stanford University

Recording

1 h 6 min

Abstract

Surya Ganguli varies the diversity of pretraining tasks to study when transformers learn to solve new regression problems from prompt examples. With limited diversity, predictions resemble a Bayesian estimator tied to the training tasks. Above an empirical threshold they resemble ridge regression and generalize beyond those tasks, revealing a role for task diversity alongside data and model scale.

Topics