Transformers

…networks and LSTMs to attention and Transformers. Exercises use TensorFlow and Keras to connect model training and performance evaluation with the biological interpretation of genomic data. 2027-01-25–2027-01-27, 14:00–20:00 daily, Europe/Berlin (CET, UTC+01:00). Online course; fee €480. Participants should be familiar with Pytho

…models such as diffusion networks, transformers and DNA-sequence models in JAX or PyTorch, improves distributed training and inference, and develops research libraries and evaluation tools.

Maissam Barkeshli uses synthetic data of adjustable complexity to study how transformers learn. Sequences from graph, hypergraph and hierarchical random walks make it possible to investigate scaling laws, sample complexity and unexpected temperature effects in controlled settings. The presentation also considers how parameterisation changes hyperparameter scaling, including the importance of scaling the embedding-layer learning rate.

With the recent success of Transformers in the field of automatic natural language processing [1], it has become clear that we are able to create complex, high-performance neural network architectures that can extract, combine and transform information to meet a specific objective, such as predicting the next word in a text. However, to function, Transformers need to receive a context of all N elements of the past sequence (N = 2k to 32k tokens) at each time step, causing the number of computations to grow quadratically (N^2) with the length of the sequence [1]. This "perfect" memory of a co

…an obvious example, but in addition transformers need it for matrix-matrix multiplications. Without weight sharing, deep networks perform badly on hard tasks. This is potentially problematic for the brain, since weight sharing is biologically implausible - a fact that is ignored in deep learning models of brain activity, especially of visual processing, which are becoming widely used. Recently it was shown that partial weight sharing can be implemented with a "sleep phase", in which plasticity is anti-Hebb