37 min
Seminar, Hosted by Simons Institute for the Theory of Computing
Towards Understanding Modern Alchemy
MIT
Recording
Abstract
Ekin Akyurek studies whether a model can infer the rules of a probabilistic formal language from example strings. In experiments with randomly generated finite automata, transformers outperform recurrent and convolutional models. The analysis points to higher-order induction mechanisms that estimate next-token distributions in context. Building these mechanisms into the architecture also improves language modeling on SlimPajama.