The Expressive Power of Large Language Models

Large language models alternate multilayer perceptrons and self-attention to process token sequences. Gabriel Peyré compares the proof methods and approximation capabilities of these components. Whereas perceptron approximation is well understood, attention presents less explored mathematical questions. The talk connects their expressive power to approximation theory in infinite-dimensional spaces representing distributions of input tokens and identifies open problems at that interface.