Credit assignment

…such networks can perform efficient credit assignment, remains, to a large extent, an open question. In machine learning (ML), the answer is almost universally given by the error backpropagation algorithm, through b

…In contrast, the brain's credit assignment mechanism remains elusive, and implementing effective credit assignment algorithms for multilayer networks in a biologically plausible manner has been challenging. Due to the locality constraint of synaptic plasticity, synaptic weights should only be updated using signals available at presynaptic and postsynaptic neurons, without accessing information from unconnected neurons. How a biological neuron encodes and transmits feedforward predictions and er

Behavioral Timescale Synaptic Plasticity (BTSP) is a type of synaptic plasticity in which large, Ca2+ plateau potentials in hippocampal pyramidal cells drive the formation of hippocampal fields such as place fields. BTSP differentiates itself from typical forms of plasticity by a) learning pre-post correlations on the timescales of seconds and b) learning in a one-to-few-shot manner. Though BTSP has been posited to be the primary driver of field formation in hippocampus, theoretical models that describe the population-level computations of hippocampus (such as the formation of "cognitive" or "

Learning is fundamental for interacting with a changing environment. Critical to this process is the ability to rapidly adapt previously acquired skills to external perturbations while avoiding catastrophic forgetting of previous tasks. For the learning system, this corresponds to a trade-off between being able to maintain stable memories of learned tasks, and the ability to swiftly adapt such memories to changes. We hypothesize that such a problem can be addressed by using a two-timescale [1,2] strategy, where memorization and adaptation are performed by two separate modules. This structure a

The success of Artificial Neural Networks (ANNs) trained with error-backpropagation algorithms relies on the combination of offline training, vast amounts of data, and energy-intensive simulations on dedicated hardware accelerators. In contrast, biological neurons can learn quickly in a self-supervised manner by utilizing (a) recurrent connections within their local network and from other areas, (b) segregated dendrites receiving inputs from feedforward and feedback pathways, and (c) learning rules that combine locally available information from pre- and -post-synaptic sources. Inspired by loc

…to the 2D cursor position is a credit assignment problem [3] that is underconstrained, because changes along a large number of output-null dimensions do not influence the behavioral output. We hypothesized

…This coordination requires solving the credit assignment problem. Yet, how neuronal circuits achieve credit assignment remains a central unsolved question in systems neuroscience [1]. Several modeling studies have suggested putative biological mechanisms for back-propagating error signals through multi-layer networks. These functionally motivated models assume distinct neuronal compartments representing local error signals t

…have gained insights into this “credit assignment” problem through decades of experimental, modeling, and theoretical studies. This has suggested key roles for synaptic eligibility traces and top-down feedback signals, among other factors. Here we study the potential contribution of another type of signaling that is being revealed in greater and greater fidelity by ongoing molecular and genomic

…for meta-learning lies a difficult credit assignment problem. After a learning episode, the system needs to determine how to change the components of plasticity such that the next time a similar task is encountered, a better learning outcome can be arrived at more quickly. In machine learning, this problem is often approached by storing the entire learning trajectory and revisiting it in reverse-time order, a process that places large deman

Humans and animals construct internal models linking actions with their likely outcomes, and use these models to guide behavior. Such model-based cognition is often referred to as planning, and its neural mechanisms remain poorly understood (Miller & Venditto, 2021). The hippocampus, known for its role in spatial learning and memory, is necessary for planning (Miller et al., 2017; Vikbladh et al., 2019). Specifically, hippocampal sequences, which “sweep out” nonlocal trajectories both during active behavior (“theta sequences”) and at rest (“sharp-wave ripple sequences”; SWR), have been pro

Reinforcement Learning studies how animals and humans adapt to their environment by learning from experience which actions maximize reward acquisition. Extant research has focused on situations where learners receive veridical reward-feedback. However, as social beings, much of the feedback we receive is provided by others. Critically, such feedback may be misleading because others might manipulate us based on their own –even if well-intended– interests. For example, a colleague might spare your feelings, expressing enthusiasm regarding your imminent Cosyne submission when, they actually find

…This “credit assignment” problem is commonly solved in artificial neural networks through supervised gradient descent and the backpropagation algorithm. In contrast, biological learning is typically modelled using unsupervised Hebbian learning rules. While these rules only use local information to update synaptic weights, and are sometimes combined with weight constraints to reflec

Current studies suggest that individuals estimate the value of their choices based on observed feedback. Here, we ask whether individuals update the value of their unchosen actions, even when the associated feedback remains unknown. Two hundred and three individuals completed a multi-armed bandit task, making choices to gain rewards. We found robust evidence suggesting inverse value updating for unchosen actions based on the chosen action’s outcome. Computational modeling results suggested that this effect is mainly explained by a value updating mechanism whereby individuals integrate the outc

Identifying situations in which exploration could be beneficial, and learning from such choices, is critical for survival in complex environments. Nonetheless, it is still unclear how animals adapt their choice strategies depending on the potential benefits of exploration and what the neural underpinning of this ability is. We thus adapted a 2-option forced-choice task in which 3 rhesus macaques made choices while the information and the feedback they received was being manipulated: (i) feedback was given about either just the chosen option or both the chosen and unchosen counterfactual option

In reinforcement learning (RL), we try to learn a function f that maps a state representation s to actions a: a=f(s). Learning f is hard, and is typically performed in each environment. What if, instead, we learn how to rapidly construct s out of general building blocks, for which the same f always works? We demonstrate (1) that this is possible in spatial worlds using object- and goal-vector representations that exist in the brain and (2) that this ‘behavioural construction’ dramatically outperforms traditional RL in simple spatial problems. In a companion paper, we demonstrate that the “buil

…networks needs to tackle the issue of credit assignment continuously, in real time. Recent years have witnessed a surge of cortical learning models which address this question by approximating the error backpropagation algorithm. However, all of these either require long relaxation phases following a change in sensory stimuli, which renders them unable to cope with the fast time scales imposed by, e.g., saccades, or impo

…biologically plausible model of credit assignment with a cholinergic adaptive module based on adaptive deep learning rules. Using a multi-class perceptual discrimination task we show that cholinergic adaptive learning leads to rapid learning when compared with non-adaptive mode

…at the forefront as a solution to the credit assignment problem in artificial neural networks. Whether the brain adopts a similar strategy to ensure that the correct synapses are modified remains unclear. Recent work has attempted to bridge this gap with backprop-like learning mechanisms that are consistent with several cortical experimental observations. However, these models are either unable to effectively backpropagate error signals across several brain areas or require a multi-phase learning process, neither of which are reminiscent

…using similar techniques for assigning credit to each synaptic weight for its contribution to the network output. However, the majority of current attempts at biologically-plausible learning methods are either non-local in time, require highly specific connectivity motives, or have no clear link to any known mathematical optimization method. Here, we introduce Deep Feedback Control (DFC), a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose con

The neurotransmitter dopamine is essential for normal reward learning and motivational arousal processes. Indeed these core functions are implicated in the major neurological and psychiatric dopamine disorders such as schizophrenia, substance abuse disorders/addiction and Parkinson's disease. Over the years, we have made significant strides in understanding the dopamine system across multiple levels of description, and I will focus on our recent advances in the computational description, and brain circuit mechanisms that facilitate the dual role of dopamine in learning and performance. I will

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens (NAc), which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex (PL) and midline regions of the thalamus (mTH). However, little is known about what is represented in PL or mTH neurons that project to NAc (PL-NAc and mTH-NAc). By comparing these inputs during a reinforcement learning task in mice, we discovered that i) PL-NAc preferentially represents actions and choices, ii) mTH-NAc preferentially represents cues, iii) choice-selective activity in PL-NAc is organized in s