Topic: credit assignment

Learning in neural networks requires assigning the right values to thousands to trillions or more of individual connections, so that the network as a whole produces the desired behavior. Neuroscientists have gained insights into this “credit assignment” problem through decades of experimental, modeling, and theoretical studies. This has suggested key roles for synaptic eligibility traces and top-down feedback signals, among other factors. Here we study the potential contribution of another type of signaling that is being revealed in greater and greater fidelity by ongoing molecular and genomic

Humans have the ability to quickly adapt to new tasks and generalise what they learned to improve the learning process itself. At the core of this capacity for meta-learning lies a difficult credit assignment problem. After a learning episode, the system needs to determine how to change the components of plasticity such that the next time a similar task is encountered, a better learning outcome can be arrived at more quickly. In machine learning, this problem is often approached by storing the entire learning trajectory and revisiting it in reverse-time order, a process that places large deman

A central problem in biological learning is how information about the outcome of a decision or behavior can be used to reliably guide learning across distributed neural circuits while obeying biological constraints. This “credit assignment” problem is commonly solved in artificial neural networks through supervised gradient descent and the backpropagation algorithm. In contrast, biological learning is typically modelled using unsupervised Hebbian learning rules. While these rules only use local information to update synaptic weights, and are sometimes combined with weight constraints to reflec

Current studies suggest that individuals estimate the value of their choices based on observed feedback. Here, we ask whether individuals update the value of their unchosen actions, even when the associated feedback remains unknown. Two hundred and three individuals completed a multi-armed bandit task, making choices to gain rewards. We found robust evidence suggesting inverse value updating for unchosen actions based on the chosen action’s outcome. Computational modeling results suggested that this effect is mainly explained by a value updating mechanism whereby individuals integrate the outc

In reinforcement learning (RL), we try to learn a function f that maps a state representation s to actions a: a=f(s). Learning f is hard, and is typically performed in each environment. What if, instead, we learn how to rapidly construct s out of general building blocks, for which the same f always works? We demonstrate (1) that this is possible in spatial worlds using object- and goal-vector representations that exist in the brain and (2) that this ‘behavioural construction’ dramatically outperforms traditional RL in simple spatial problems. In a companion paper, we demonstrate that the “buil

The error-backpropagation (backprop) algorithm has stood at the forefront as a solution to the credit assignment problem in artificial neural networks. Whether the brain adopts a similar strategy to ensure that the correct synapses are modified remains unclear. Recent work has attempted to bridge this gap with backprop-like learning mechanisms that are consistent with several cortical experimental observations. However, these models are either unable to effectively backpropagate error signals across several brain areas or require a multi-phase learning process, neither of which are reminiscent

The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output. However, the majority of current attempts at biologically-plausible learning methods are either non-local in time, require highly specific connectivity motives, or have no clear link to any known mathematical optimization method. Here, we introduce Deep Feedback Control (DFC), a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose con

The neurotransmitter dopamine is essential for normal reward learning and motivational arousal processes. Indeed these core functions are implicated in the major neurological and psychiatric dopamine disorders such as schizophrenia, substance abuse disorders/addiction and Parkinson's disease. Over the years, we have made significant strides in understanding the dopamine system across multiple levels of description, and I will focus on our recent advances in the computational description, and brain circuit mechanisms that facilitate the dual role of dopamine in learning and performance. I will

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens (NAc), which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex (PL) and midline regions of the thalamus (mTH). However, little is known about what is represented in PL or mTH neurons that project to NAc (PL-NAc and mTH-NAc). By comparing these inputs during a reinforcement learning task in mice, we discovered that i) PL-NAc preferentially represents actions and choices, ii) mTH-NAc preferentially represents cues, iii) choice-selective activity in PL-NAc is organized in s

Learning is fundamental for interacting with a changing environment. Critical to this process is the ability to rapidly adapt previously acquired skills to external perturbations while avoiding catastrophic forgetting of previous tasks. For the learning system, this corresponds to a trade-off between being able to maintain stable memories of learned tasks, and the ability to swiftly adapt such memories to changes. We hypothesize that such a problem can be addressed by using a two-timescale [1,2] strategy, where memorization and adaptation are performed by two separate modules. This structure a

The success of Artificial Neural Networks (ANNs) trained with error-backpropagation algorithms relies on the combination of offline training, vast amounts of data, and energy-intensive simulations on dedicated hardware accelerators. In contrast, biological neurons can learn quickly in a self-supervised manner by utilizing (a) recurrent connections within their local network and from other areas, (b) segregated dendrites receiving inputs from feedforward and feedback pathways, and (c) learning rules that combine locally available information from pre- and -post-synaptic sources. Inspired by loc

By causally mapping neural activity to behavior [1], Brain-Computer Interfaces (BCI) offer a means to study the dynamics of sensorimotor learning. Here, we combine computational modeling and data analysis to study the neural learning algorithm [2] monkeys use to adapt to a changed output mapping in a center-out reaching task. We exploit that the mapping from neural space (ca. 100 dimensions) to the 2D cursor position is a credit assignment problem [3] that is underconstrained, because changes along a large number of output-null dimensions do not influence the behavioral output. We hypothesized

Learning in neuronal circuits requires coordinated modifications of synaptic strengths across hierarchical networks to refine computation and improve behavioral outcomes. This coordination requires solving the credit assignment problem. Yet, how neuronal circuits achieve credit assignment remains a central unsolved question in systems neuroscience [1]. Several modeling studies have suggested putative biological mechanisms for back-propagating error signals through multi-layer networks. These functionally motivated models assume distinct neuronal compartments representing local error signals t