52 min
Seminar, Hosted by Simons Foundation
How Much Data Is Required to Learn the Structure of Language?
Johns Hopkins University
Recording
Abstract
Experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, yet no existing theory can quantitatively predict their exponents for any modern large language model trained on natural language, and it is unclear whether they can be beaten by training with other objectives. This talk reviews the Physics of Learning collaboration's work on this phenomenon, including the first such theory for data-limited scaling laws. It then introduces synthetic languages with a hidden hierarchy, a central notion in both linguistics and computer vision, and discusses how these languages are learned by deep but not shallow networks. When trained at the token level, scaling laws in these synthetic languages follow from the same principle as in natural languages, but the languages can be learned far more efficiently by machines that predict in an abstract space, which helps explain the success of joint embedding predictive architectures (JEPA) and other systems that learn from their own latents.