57 min
Seminar, Hosted by Simons Institute for the Theory of Computing
Transformers, parallel computation, and logarithmic depth
Columbia University
Recording
Abstract
The speaker connects self-attention layers to communication rounds in a massively parallel model of computation. Simulations in both directions explain how logarithmic depth lets transformers solve tasks that several other sequence architectures and subquadratic approximations struggle with, and identify parallelism as a source of the difference.