Topic: self-attention

Large language models alternate multilayer perceptrons and self-attention to process token sequences. Gabriel Peyré compares the proof methods and approximation capabilities of these components. Whereas perceptron approximation is well understood, attention presents less explored mathematical questions. The talk connects their expressive power to approximation theory in infinite-dimensional spaces representing distributions of input tokens and identifies open problems at that interface.

Computations in neural circuits are often construed as being implemented through the coordinated dynamics of neurons. In this picture, the role of synaptic connectivity is to sculpt neuronal dynamics to implement computations of interest. Of course, synapses are not static but change on a variety of timescales, including fast timescales comparable to those of neurons. Thus, a more accurate view of computation in neural circuits may involve the coupled dynamics of neurons and synapses. This form of computation is closer to what is implemented by Transformers via an equivalence between ongoing s