Skip to contentSkip to filters

We use cookies for analytics.

Theory of Scaling: A General Framework for Scale-aware Training

Soufiane Hayou

UC Berkeley

Recording

1 h 4 min

Abstract

Neural networks often need different training rules as their width or depth grows. The speaker presents a framework for deriving learning rules that adjust with model scale, discusses how it accounts for stability and efficient learning, and relates the theory to empirical rules used to train large models.

Topics