Towards a Science of AI: Scaling laws and synthetic data

Maissam Barkeshli uses synthetic data of adjustable complexity to study how transformers learn. Sequences from graph, hypergraph and hierarchical random walks make it possible to investigate scaling laws, sample complexity and unexpected temperature effects in controlled settings. The presentation also considers how parameterisation changes hyperparameter scaling, including the importance of scaling the embedding-layer learning rate.