1 h 4 min
Seminar, Hosted by Simons Institute for the Theory of Computing
Theory of Scaling: A General Framework for Scale-aware Training
UC Berkeley
Recording
Abstract
Neural networks often need different training rules as their width or depth grows. The speaker presents a framework for deriving learning rules that adjust with model scale, discusses how it accounts for stability and efficient learning, and relates the theory to empirical rules used to train large models.