Skip to contentSkip to filters

We use cookies for analytics.

Towards Understanding Modern Alchemy

Recording

37 min

Abstract

Ekin Akyurek studies whether a model can infer the rules of a probabilistic formal language from example strings. In experiments with randomly generated finite automata, transformers outperform recurrent and convolutional models. The analysis points to higher-order induction mechanisms that estimate next-token distributions in context. Building these mechanisms into the architecture also improves language modeling on SlimPajama.

Topics