Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons

Fast cortical processing is required in many scenarios where both sensory input and the corresponding cognitive responses change rapidly. Therefore, efficient learning in such networks needs to tackle the issue of credit assignment continuously, in real time. Recent years have witnessed a surge of cortical learning models which address this question by approximating the error backpropagation algorithm. However, all of these either require long relaxation phases following a change in sensory stimuli, which renders them unable to cope with the fast time scales imposed by, e.g., saccades, or impose some form of rapidly phased learning, which is difficult to reconcile with experimental observations. We introduce Latent Equilibrium, a framework for inference and learning in networks of slow components which avoids these issues by harnessing the ability of biological neurons to phase-advance their output with respect to their membrane potential. This mechanism enables quasi-instantaneous inference independent of network depth and avoids the need for computationally expensive relaxation phases, allowing networks to learn from stimulus-target pairs with dynamics on near-arbitrarily short time scales. We derive neuron morphology, network structure, and in particular disentangled neuronal and synaptic weight dynamics from a single prospective energy function. The resulting model can be interpreted as a real-time, biologically plausible approximation of error backpropagation in deep cortical networks with continuous-time, leaky neuronal dynamics and continuously active, local synaptic plasticity. We demonstrate successful learning from continuous input streams, achieving competitive performance with both fully-connected and convolutional architectures on standard benchmark datasets. We further show how our mathematical framework can be embedded within cortical microcircuits. Finally, we study the robustness of our model to spatio-temporal substrate imperfections to demonstrate its feasibility for physical realization, both in vivo and in silico.