Topic: Transformers

Apply neural networks to biological sequence analysis, including sequence classification, functional-feature discovery, source tracking and sequence generation. Hands-on notebooks introduce sequence encoding and language-model concepts, progressing from convolutional networks and LSTMs to attention and Transformers. Exercises use TensorFlow and Keras to connect model training and performance evaluation with the biological interpretation of genomic data. 2027-01-25–2027-01-27, 14:00–20:00 daily, Europe/Berlin (CET, UTC+01:00). Online course; fee €480. Participants should be familiar with Pytho

Translate frontier machine-learning prototypes into scalable systems for drug discovery. The role optimises models such as diffusion networks, transformers and DNA-sequence models in JAX or PyTorch, improves distributed training and inference, and develops research libraries and evaluation tools.

Maissam Barkeshli uses synthetic data of adjustable complexity to study how transformers learn. Sequences from graph, hypergraph and hierarchical random walks make it possible to investigate scaling laws, sample complexity and unexpected temperature effects in controlled settings. The presentation also considers how parameterisation changes hyperparameter scaling, including the importance of scaling the embedding-layer learning rate.

With the recent success of Transformers in the field of automatic natural language processing [1], it has become clear that we are able to create complex, high-performance neural network architectures that can extract, combine and transform information to meet a specific objective, such as predicting the next word in a text. However, to function, Transformers need to receive a context of all N elements of the past sequence (N = 2k to 32k tokens) at each time step, causing the number of computations to grow quadratically (N^2) with the length of the sequence [1]. This "perfect" memory of a co