Learning and prediction in artificial deep neural networks: scaling, data manifolds, and universality
Developing scientifically-grounded theories for representation learning and generalization in artificial deep neural networks remains a grand challenge of fundamental interest to theoretical neuroscience and machine learning. I will discuss our work on one facet of this challenge — namely understanding generalization or “scaling laws” in learned neural networks as a function of basic control variables. I’ll discuss a taxonomy we develop that classifies different regimes of scaling behavior. We identify regimes where generalization exhibits universal scaling behavior and others where it can be traced back to properties of the data and neural architecture. The theoretical analysis is enabled by leveraging exactly solvable models of deep neural networks that arise naturally in the limit of large hidden layers. Along the way, I’ll also discuss our work on these theoretical models, which have been a useful starting point for theoretical descriptions of neural network dynamics. Finally, I’ll discuss our findings connecting generalization in neural networks to properties of the learned data manifold. I’ll close by discussing future directions and new hypotheses that emerge from our findings Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2024-06-19. Recording duration: 00:46:53.
Is there universality in biology?
It is sometimes said that there are two reasons why physics is so successful as a science. One is that it deals with very simple problems. The other is that it attempts to account only for universal aspects of systems at a desired level of description, with lower level phenomena subsumed into a small number of adjustable parameters. It is a widespread belief that this approach seems unlikely to be useful in biology, which is intimidatingly complex, where “everything has an exception”, and where there are a huge number of undetermined parameters. I will try to argue, nonetheless, that there are important, experimentally-testable aspects of biology that exhibit universality, and should be amenable to being tackled from a physics perspective. My suggestion is that this can lead to useful new insights into the existence and universal characteristics of living systems. I will try to justify this point of view by contrasting the goals and practices of the field of condensed matter physics with materials science, and then by extension, the goals and practices of the newly emerging field of “Physics of Living Systems” with biology. Specific biological examples that I will discuss include the following: Universal patterns of gene expression in cell biology Universal scaling laws in ecosystems, including the species-area law, Kleiber’s law, Paradox of the Plankton Universality of the genetic code Universality of thermodynamic utilization in microbial communities Universal scaling laws in the tree of life The question of what can be learned from studying universal phenomena in biology will also be discussed. Universal phenomena, by their very nature, shed little light on detailed microscopic levels of description. Yet there is no point in seeking idiosyncratic mechanistic explanations for phenomena whose explanation is found in rather general principles, such as the central limit theorem, that every microscopic mechanism is constrained to obey. Thus, physical perspectives may be better suited to answering certain questions such as universality than traditional biological perspectives. Concomitantly, it must be recognized that the identification and understanding of universal phenomena may not be a good answer to questions that have traditionally occupied biological scientists. Lastly, I plan to talk about what is perhaps the central question of universality in biology: why does the phenomenon of life occur at all? Is it an inevitable consequence of the laws of physics or some special geochemical accident? What methodology could even begin to answer this question? I will try to explain why traditional approaches to biology do not aim to answer this question, by comparing with our understanding of superconductivity as a physical phenomenon, and with the theory of universal computation. References Nigel Goldenfeld, Tommaso Biancalani, Farshid Jafarpour. Universal biology and the statistical mechanics of early life. Phil. Trans. R. Soc. A 375, 20160341 (14 pages) (2017). Nigel Goldenfeld and Carl R. Woese. Life is Physics: evolution as a collective phenomenon far from equilibrium. Ann. Rev. Cond. Matt. Phys. 2, 375-399 (2011).