Seminar, Hosted by Fields Institute for Research in Mathematical Sciences
Geometry and Dynamics of Attention: From Gradient Flows to Inertial Transformers
Technical University of Munich
Abstract
Interpreting tokens as interacting particles leads to equations for the distribution of their representations. Stein develops geometries for softmax and linear attention, then uses them to introduce inertia and analyze when token norms grow with depth. In decoder-only experiments on TinyStories and OpenWebText, the resulting token and momentum corrections improve validation loss against momentum-only controls at matched training budgets.