ePoster, COSYNE 2023
Influence of Learning Rules on Representation Dynamics in Wide Neural Networks
- Montreal, Canada
- Machine Learning, Computational Neuroscience and
Poster
Abstract
The learning rules implemented by neural networks in the brain are currently unknown. Understanding how candidate learning rules influence learned representations and prediction dynamics in neural networks could provide a link between experimentally accessible neural response measurements and possible underlying learning rules. To gain insight into how learning rules alter representation dynamics, we provide theoretical analysis of a broad class of learning rules which include exact gradient descent (GD), Feedback-Alignment (FA), Direct-Feedback-Alignment (DFA), Hebbian learning (Hebb) and gated linear networks (GLN). By developing a dynamical mean field theory (DMFT) for wide neural networks, we derive predictions for how neural representations evolve during learning with each of these rules in the large population limit. We show that, for each of these learning rules, the evolution of the output function at infinite width is governed by a time varying effective neural tangent kernel (eNTK). In the lazy training limit, this eNTK is static and does not evolve, while in the rich mean-field regime this kernel’s evolution can be determined self-consistently with DMFT. We show that richness accelerates learning by aligning the true feature gradients to the pseudo-gradient fields and aligning the feature kernels to the structure of the task.
Topics
- learning rules, search World Wide
- neural networks, search World Wide
- learned representations, search World Wide
- prediction dynamics, search World Wide
- gradient descent, search World Wide
- Feedback-Alignment, search World Wide
- Direct-Feedback-Alignment, search World Wide
- Hebbian learning, search World Wide
- dynamical mean field theory, search World Wide
- neural tangent kernel, search World Wide
Details
- Session
- Poster Session II