Skip to contentSkip to filters

We use cookies for analytics.

Learning Theory of Transformers: Generalization and Optimization of In-Context Learning

Taiji Suzuki

University of Tokyo

Recording

46 min

Abstract

Taiji Suzuki analyzes the statistical and optimization properties of transformer in-context learning. He discusses when pretrained transformers achieve minimax rates and improve on methods without pretraining, then gives conditions for optimizing nonlinear features in a mean-field model. For single-index targets, the analysis links computational efficiency to the information exponent.

Topics