51 min
Seminar, Hosted by Simons Institute for the Theory of Computing
Computational Benefits and Limitations of Transformers and State-Space Models
Kempner Institute, Harvard University
Recording
Abstract
Why do language models differ at copying or retrieving a long sequence? The talk compares transformers with recurrent and state-space architectures using theoretical and empirical results, then shows how a transformer’s ability to copy can support length generalization in algorithmic and arithmetic tasks.