Statistical Physics

Salvatore Torquato gives a public lecture in the Simons Foundation's 2026 Randomness series, connecting the geometry of complex systems with random structures and applications across mathematics, computer science, physics and chemistry.

Deep networks are often thought of as black boxes. Their ability to encompass vast swathes of knowledge indeed makes them hard to explain. Yet many of their behaviours — generalization under overparametrization, grokking, OOD failures, neural scaling laws — recur across architectures and scales, and each, however surprising, can be reproduced and explained in controlled settings. I will review these efforts to identify and explain the universal phenomena of deep learning, and suggest that an LLM may amount to a sum of such tractable sub-phenomena, interpolative in nature. Finally, leaving scientific rigor aside, I'll argue that what separates this prosaic picture from the apparent magic of LLMs may well be the industrial scale of compute and human labour behind it, and that AGI in its deeper extrapolative sense may be much further away than claimed.

We introduce a statistical physics framework for learning in neural architectures composed of single or interconnected asymmetric attractor networks. These systems can exhibit a manifold of global fixed points capable of implementing sophisticated input-output mappings, which we characterize analytically. Learning from extensive datasets is achieved through the stabilization of fixed points via a fully distributed and local learning process, implemented at the single-neuron level. This simple mechanism yields performance comparable to that of conventional feedforward deep neural networks trained using gradient-based methods. The effectiveness of the model stems from the dense and accessible manifolds of stable fixed points, which encode the internal representations of data. Unlike other approaches to deep learning without backpropagation, our method does not attempt to estimate gradients. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2025-06-18. Recording duration: 00:45:35.

The factorization of a large matrix into the product of two matrices is an important mathematical problem encountered in many tasks, ranging from dictionary learning to machine learning. Statistical physics can provide on the one hand theoretical limits on the possibility of factorizing matrices in the limit of infinite size, and also practical algorithms. While this program has been successful in the case of finite rank matrices, the regime of extensive rank (scaling linearly with the dimension of the matrix) turns out to be much harder. This talk will describe a new approach to matrix factorization that maps it to neural network models of associative memory: each pattern found in the associative memory corresponds to one factor of the matrix decomposition. A detailed theoretical analysis of this new approach shows that matrix factorization in the extensive rank regime is possible when the rank is below a certain threshold. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2024-01-03. Recording duration: 00:44:31.

Deep networks are often thought of as black boxes. Their ability to encompass vast swathes of knowledge indeed makes them hard to explain. Yet many of their behaviours — generalization under overparametrization, grokking, OOD failures, neural scaling laws — recur across architectures and scales, and each, however surprising, can be reproduced and explained in controlled settings. I will review these efforts to identify and explain the universal phenomena of deep learning, and suggest that an LLM may amount to a sum of such tractable sub-phenomena, interpolative in nature. Finally, leaving scientific rigor aside, I'll argue that what separates this prosaic picture from the apparent magic of LLMs may well be the industrial scale of compute and human labour behind it, and that AGI in its deeper extrapolative sense may be much further away than claimed.

The factorization of a large matrix into the product of two matrices is an important mathematical problem encountered in many tasks, ranging from dictionary learning to machine learning. Statistical physics can provide on the one hand theoretical limits on the possibility of factorizing matrices in the limit of infinite size, and also practical algorithms. While this program has been successful in the case of finite rank matrices, the regime of extensive rank (scaling linearly with the dimension of the matrix) turns out to be much harder. This talk will describe a new approach to matrix factorization that maps it to neural network models of associative memory: each pattern found in the associative memory corresponds to one factor of the matrix decomposition. A detailed theoretical analysis of this new approach shows that matrix factorization in the extensive rank regime is possible when the rank is below a certain threshold. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2024-01-03. Recording duration: 00:44:31.

We introduce a statistical physics framework for learning in neural architectures composed of single or interconnected asymmetric attractor networks. These systems can exhibit a manifold of global fixed points capable of implementing sophisticated input-output mappings, which we characterize analytically. Learning from extensive datasets is achieved through the stabilization of fixed points via a fully distributed and local learning process, implemented at the single-neuron level. This simple mechanism yields performance comparable to that of conventional feedforward deep neural networks trained using gradient-based methods. The effectiveness of the model stems from the dense and accessible manifolds of stable fixed points, which encode the internal representations of data. Unlike other approaches to deep learning without backpropagation, our method does not attempt to estimate gradients. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2025-06-18. Recording duration: 00:45:35.