Skip to contentSkip to filters

We use cookies for analytics.

Generalization in Attention-Based Models: Insights from Solvable High-Dimensional Models

Lenka Zdeborová

Physics and Computer Science, École Polytechnique Fédérale de Lausanne

Recording

1 h 17 min

Abstract

Statistical physics has long provided a framework for understanding learning in high dimensions, revealing phase transitions, fundamental limits on generalization, and gaps between optimal inference and practical algorithms. This talk discusses recent efforts to extend that perspective from classical perceptron-type models to attention-based architectures that operate on sequences, as in transformers. Solvable high-dimensional models yield analytical insight into generalization, scaling laws and the spectral properties of learned representations. Although necessarily simplified, these models provide a controlled setting in which theoretical predictions can be derived and compared with empirical phenomena observed in modern AI systems.

Topics