46 min
Seminar, Hosted by Simons Institute for the Theory of Computing
Learning Theory of Transformers: Generalization and Optimization of In-Context Learning
University of Tokyo
Recording
Abstract
Taiji Suzuki analyzes the statistical and optimization properties of transformer in-context learning. He discusses when pretrained transformers achieve minimax rates and improve on methods without pretraining, then gives conditions for optimizing nonlinear features in a mean-field model. For single-index targets, the analysis links computational efficiency to the information exponent.