Only two types of attractors support representation of continuous variables, and learning over long time-spans

Attractor dynamics are essential components of neural computation— they have given rise to elegant, interpretable models of population coding and working memory. Frequently, attractors are classified based on whether their invariant manifolds can represent discrete/continuous variables, with point attractors being an example of the former and ‘continuous attractors’ of the latter. Here we highlight that ‘learnability’ is another quality that may serve to differentiate neural dynamics while offering novel insight. Counterintuitively, good working memory does not necessarily imply that a well-defined, directional learning signal (i.e., gradient) exists. We develop a differential geometric theory that shows that only two kinds of neural dynamics–continuous attractors and quasi-periodic attractors–are fundamentally capable of learning arbitrarily distant temporal relationships. Surprisingly, we find that requiring that an invariant manifold carries persistent learning signals is equivalent to requiring that it faithfully represent continuous variables. The intuitive explanation for this equivalence is that the learning signal (gradient) is an infinitesimal perturbation of the neural state. For such perturbations to be faithfully representable in a biological neural network, the neural states: (1) need to be bounded and, (2) must not asymptotically approach a point attractor. Our analysis further distinguishes continuous and quasi-periodic attractors. Continuous attractors infamously suffer from the fine-tuning problem but also have trivial gradients; both problems hinder their ability to learn and produce temporally structured behavior. By comparison, quasi-periodic attractors do not require fine-tuning and generate richly structured gradients, which facilitates learning and generation of temporally-complex behavior with long-range dependencies. Our theory proposes that oscillatory activity is an observable signature of biological neural dynamics that can support temporal dependence learning Dually, these results have broad implications for the design of artificial learning systems. We propose a novel initialization scheme for RNNs that outperforms specialized architectures in tasks with long-range temporal dependencies. Lastly, the framework we introduce significantly expands upon the dynamical systems tools in use, making it of independent interest.