Skip to contentSkip to filters

We use cookies for analytics.

Computational Benefits and Limitations of Transformers and State-Space Models

Eran Malach

Kempner Institute, Harvard University

Recording

51 min

Abstract

Why do language models differ at copying or retrieving a long sequence? The talk compares transformers with recurrent and state-space architectures using theoretical and empirical results, then shows how a transformer’s ability to copy can support length generalization in algorithmic and arithmetic tasks.

Topics