Skip to contentSkip to filters

We use cookies for analytics.

Geometry and Dynamics of Attention: From Gradient Flows to Inertial Transformers

Viktor Stein

Technical University of Munich

Abstract

Interpreting tokens as interacting particles leads to equations for the distribution of their representations. Stein develops geometries for softmax and linear attention, then uses them to introduce inertia and analyze when token norms grow with depth. In decoder-only experiments on TinyStories and OpenWebText, the resulting token and momentum corrections improve validation loss against momentum-only controls at matched training budgets.

Topics