Reward prediction error

Understanding the neural mechanisms of reward-guided learning is a long-standing goal of computational neuroscience. Recent methodological innovations enable us to collect ever larger neural and behavioral datasets. This presents opportunities to achieve greater understanding of learning in the brain at scale, as well as methodological challenges. In the first part of the talk, I will discuss our recent insights into the mechanisms by which zebra finch songbirds learn to sing. Dopamine has been long thought to guide reward-based trial-and-error learning by encoding reward prediction errors. Ho

Understanding the neural mechanisms of reward-guided learning is a long-standing goal of computational neuroscience. Recent methodological innovations enable us to collect ever larger neural and behavioral datasets. This presents opportunities to achieve greater understanding of learning in the brain at scale, as well as methodological challenges. In the first part of the talk, I will discuss our recent insights into the mechanisms by which zebra finch songbirds learn to sing. Dopamine has been long thought to guide reward-based trial-and-error learning by encoding reward prediction errors. Ho

…which dopamine (DA) neurons transmit reward prediction error (RPE) signals---the difference between actual and expected rewards---to enable value learning via cortico-striatal plasticity. Recent studies have highlighted two novel aspects: first, that RPE signals from midbrain DA neurons can encode entire reward distributions through a distributional RL algorithm that mirrors cutting-edge machine

The dopaminergic system facilitates associative learning and motivation by signaling reward expectation and positive or negative valence in selected brain regions. Previous studies measuring somatic and axonal activity suggest a heterogeneous representation of reinforcement signaling by dopaminergic neurons, but it remains unclear how dopamine signals for expectation and valence interact in various brain regions. Here, we used an improved fluorescent dopamine sensor to record dopamine signals associated with reward predictions and valence in eight distinct striatal and non-striatal regions. We

…and that DA acts analogously to the reward prediction error (RPE) term within the TD algorithm. TD assigns reward predictions (values) to states by learning not only from rewards but also when its internal value estimates change, requiring an approximate temporal derivative calculation. Although many studies have demonstrated similarities between DA

Artificial recurrent neural networks (RNNs) are difficult to train due to their tendency towards instability, and common training algorithms that tame such networks are not biologically plausible, i.e., back-propagation through time (BPTT). Node perturbation learning in combination with local Hebbian weight updates has been shown to approximate BPTT [1] and, thus, could achieve similar performance while being biologically plausible. However, where these perturbations could arise from and how they are utilized within neurons remains obscure. For years, the neuroscientific community has known ab

Neuromodulatory systems, more specifically dopamine, noradrenaline, serotonin, and acetylcholine can dynamically change brain states and rapidly alter information processing of large number of neurons. Therefore, they can play a crucial role in cognitive functions, including learning and memory, and are accordingly severely implicated in neurodegenerative diseases. Interestingly, neuromodulatory systems often regulate seemingly overlapping cognitive processes and show similar modes of action. To reveal possible synergistic or antagonistic relations and to better understand the exact role of th

Humans and other animals require flexibility to make decisions appropriate for an ever-changing environment. The way mammalian brains achieve flexible decision making is not fully understood. The probabilistic reversal learning (PRL) task is a powerful tool used to assess behavioral flexibility and how positive and negative feedbacks influence decision making, both in healthy subjects and in patients with a wide range of neurological conditions like Autistic Spectrum Disorder and Schizophrenia, as well as animal models. We trained C57BL/6 mice on PRL and conducted high-density single unit reco

…the future reward e

…correlation of their release with the reward prediction error (RPE). To address this, we trained mice on a sound detection Pavlovian conditioning task with a 50% reward schedule that allowed examining clean representations of postivite (rewarded trials) and negative RPE (reward omission trials), while we measured DA, ACh, NE and 5-HT release by fiber photometry. Behavioral updating of value representations based on the outcome of the previous trial wa

Dopamine (DA) is classically associated with learning and reward prediction error, though it is clear from deficits in the striatal dopaminergic system that DA plays a major, yet unclear, role in controlling motor behaviour. Several recent works have begun to uncover the complex roles DA plays in movement control. We have developed an experimental framework based on a dual fibre-photometry system, EMG recording and a freewheeling linear treadmill to explore the dynamics of dLight readouts during spontaneous locomotion and brief muscle twitches in head-fixed mice. In this work, we find high cor

…neuronal elements should represent the reward prediction error (RPE), which means they signal the difference between the expected future rewards and the actual rewards. The prominence of the TD theory arises from the observation that firing properties of dopaminergic neurons in the ventral tegmental area appear similar to those of RPE model-neurons in TD learning. Previous implementations of TD learning assume a fixed temp

…activity in dopamine neurons as a reward prediction error signal that drives learning in basal ganglia and prefrontal circuits. However, recent findings suggest that dopaminergic prediction error signals have access to complex, structured reward predictions and are sensitive to more properties of outcomes than learning theories with simple scalar value predictions might suggest.

…call "action surprise," in addition to reward prediction error. In the same model, an implementation of learning from a motor efference copy also predicts a novel

…act adaptively to maximize fitness and rewards. Reinforcement learning (RL) algorithms have been successful at training artificial agents to superhuman performance in an array of complex tasks and the activity of dopaminergic neurons has been described by one of their key components, the reward prediction error (RPE). However, classical RL occurs at a single timescale, set by the discount factor, while agents have to make decisions at multiple timescales and behaviorally, humans and animals are known to

…in the ventral striatum acts as a reward prediction error (RPE) to support cue-outcome associations while DA manipulations in the dorsolateral striatum produce gross effects on movement. The dorsomedial striatum (DMS), located between the ventral and dorsolateral striatum, may represent an intermediate position al

…DA activity is observed to signal reward prediction error (RPE), a key quantity in many RL algorithms used to modulate updates to state-reward and state-action associations. How the state representations underlying these associations are themselves learned is typically outside the scope of RL models of the basal ganglia. However, modern RL algorithms in machine learning typically employ deep RL, in which state representations are learned as part of the RL algorithm, to attain state-of-the-art performance. In this wor

Many proposed neural circuits for learning are theoretically well-grounded but have limited biological evidence to support them. Emerging anatomical and functional data from neurons in the fruit fly brain allows us to identify interconnections in the mushroom body neuropil that form an 'incentive circuit', supporting rich dynamics of memory acquisition, consolidation and forgetting. In addition, new insights into dopamine function in the fruit fly brain motivate a novel 'dopaminergic plasticity rule' that alters the weights between input sensory patterns and output neurons of the mushroom body

…neuron function is that they signal reward prediction error (RPE) in their phasic activity. Some recent studies have observed ramping dopamine neuron activity, but its relationship to RPE signalling is currently under debate. We set out to explain the function of this ramp and its relationship to phasic RPEs, using both experimental and theoretical methods. We chose to examine goal-directed navigation, as it requires learning to accurately estimate location and select optimal actions in each location. Given that VTA dopamine neuro

…This signal may come in the form of a reward prediction error, which can either be signed, reflecting the value of an unexpected outcome, or unsigned, reflecting the capture of attention. The role of dorsolateral prefrontal cortex (dlPFC), known to carry both value and attention related signals, in communicating when expectations are violated is not well understood [1,2]. How are unexpected outcomes signaled in dlPFC? To address this question, we trained macaque monkeys to view displays consisting o

Understanding the activity of single neurons in relation to features in the environment is the first step in many neuroscience studies. We propose a method using algorithm unrolling, an emerging technique in interpretable deep learning, to deconvolve single-trial neuronal activity into interpretable components. Specifically, we model the firing rates of single neurons using a set of kernels characterizing neurons' responses to time-sensitive sparse events/stimuli. The kernels can be either unique or shared across the population and are weighted by codes whose amplitude and timing are trial-spe

…We show that reusing the traditional reward prediction error machinery at multiple hierarchical levels allows complex tasks to be automatically decomposed in a compositional manner, leading to fast and flexible reinforcement learning. In this compositional reinforcement learning (CRL) framework, the agent computes a set of predictions for each state in the form of hierarchically organized general value functions (GVFs). Level 0 GVFs predict whether continuing straig

…(DA) neurons in the brain can exhibit reward prediction error (RPE), usually through direct analogy to temporal difference (TD) learning. However, there are two key hurdles in imagining how TD could plausibly be implemented in the brain. First, TD models of DA learning frequently require arbitrarily constructed and unrealistic components, such as a temporal chain of feature-specific neurons that uniformly tile the time from stimulus onset to reward arrival. Secondly, various p

…in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g. no/no-go, choose left/right). However, there is mounting evidence that the basal ganglia functions not as a selector between predefined actions, but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a nee