Skip to contentSkip to filters

We use cookies for analytics.

Transformers, parallel computation, and logarithmic depth

Daniel Hsu

Columbia University

Recording

57 min

Abstract

The speaker connects self-attention layers to communication rounds in a massively parallel model of computation. Simulations in both directions explain how logarithmic depth lets transformers solve tasks that several other sequence architectures and subquadratic approximations struggle with, and identify parallelism as a source of the difference.

Topics