Influence of Learning Rules on Representation Dynamics in Wide Neural Networks

The learning rules implemented by neural networks in the brain are currently unknown. Understanding how candidate learning rules influence learned representations and prediction dynamics in neural networks could provide a link between experimentally accessible neural response measurements and possible underlying learning rules. To gain insight into how learning rules alter representation dynamics, we provide theoretical analysis of a broad class of learning rules which include exact gradient descent (GD), Feedback-Alignment (FA), Direct-Feedback-Alignment (DFA), Hebbian learning (Hebb) and gated linear networks (GLN). By developing a dynamical mean field theory (DMFT) for wide neural networks, we derive predictions for how neural representations evolve during learning with each of these rules in the large population limit. We show that, for each of these learning rules, the evolution of the output function at infinite width is governed by a time varying effective neural tangent kernel (eNTK). In the lazy training limit, this eNTK is static and does not evolve, while in the rich mean-field regime this kernel’s evolution can be determined self-consistently with DMFT. We show that richness accelerates learning by aligning the true feature gradients to the pseudo-gradient fields and aligning the feature kernels to the structure of the task.