Topic: Neural scaling laws

Deep networks are often thought of as black boxes. Their ability to encompass vast swathes of knowledge indeed makes them hard to explain. Yet many of their behaviours — generalization under overparametrization, grokking, OOD failures, neural scaling laws — recur across architectures and scales, and each, however surprising, can be reproduced and explained in controlled settings. I will review these efforts to identify and explain the universal phenomena of deep learning, and suggest that an LLM may amount to a sum of such tractable sub-phenomena, interpolative in nature. Finally, leaving scie

Stanford researcher Surya Ganguli connects statistical physics, theoretical neuroscience and machine learning. He discusses scaling laws, data selection and the origins of diffusion models, then turns to neural experiments on perception, self-related processing and describing neuronal responses. The conversation closes with questions about human versus machine data efficiency and the different ways brains and artificial networks acquire useful algorithms. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.