1 h 17 min
Seminar, Hosted by Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University
Generalization in Attention-Based Models: Insights from Solvable High-Dimensional Models
Physics and Computer Science, École Polytechnique Fédérale de Lausanne
Recording
Abstract
Statistical physics has long provided a framework for understanding learning in high dimensions, revealing phase transitions, fundamental limits on generalization, and gaps between optimal inference and practical algorithms. This talk discusses recent efforts to extend that perspective from classical perceptron-type models to attention-based architectures that operate on sequences, as in transformers. Solvable high-dimensional models yield analytical insight into generalization, scaling laws and the spectral properties of learned representations. Although necessarily simplified, these models provide a controlled setting in which theoretical predictions can be derived and compared with empirical phenomena observed in modern AI systems.