Topic: reward prediction error

Understanding the neural mechanisms of reward-guided learning is a long-standing goal of computational neuroscience. Recent methodological innovations enable us to collect ever larger neural and behavioral datasets. This presents opportunities to achieve greater understanding of learning in the brain at scale, as well as methodological challenges. In the first part of the talk, I will discuss our recent insights into the mechanisms by which zebra finch songbirds learn to sing. Dopamine has been long thought to guide reward-based trial-and-error learning by encoding reward prediction errors. Ho

Understanding the neural mechanisms of reward-guided learning is a long-standing goal of computational neuroscience. Recent methodological innovations enable us to collect ever larger neural and behavioral datasets. This presents opportunities to achieve greater understanding of learning in the brain at scale, as well as methodological challenges. In the first part of the talk, I will discuss our recent insights into the mechanisms by which zebra finch songbirds learn to sing. Dopamine has been long thought to guide reward-based trial-and-error learning by encoding reward prediction errors. Ho

The dominant theoretical framework to account for reinforcement learning in the brain is temporal difference (TD) reinforcement learning. The TD framework predicts that some neuronal elements should represent the reward prediction error (RPE), which means they signal the difference between the expected future rewards and the actual rewards. The prominence of the TD theory arises from the observation that firing properties of dopaminergic neurons in the ventral tegmental area appear similar to those of RPE model-neurons in TD learning. Previous implementations of TD learning assume a fixed temp

Theories from reinforcement learning have been highly influential for interpreting neural activity in the biological circuits critical for animal and human learning. Central among these is the identification of phasic activity in dopamine neurons as a reward prediction error signal that drives learning in basal ganglia and prefrontal circuits. However, recent findings suggest that dopaminergic prediction error signals have access to complex, structured reward predictions and are sensitive to more properties of outcomes than learning theories with simple scalar value predictions might suggest.

I will discuss work with Jack Lindsey modeling reinforcement learning for action selection in the basal ganglia. I will argue that the presence of multiple brain regions, in addition to the basal ganglia, that contribute to motor control motivates the need for an off-policy basal ganglia learning algorithm. I will then describe a biological implementation of such an algorithm that predicts tuning of dopamine neurons to a quantity we call "action surprise," in addition to reward prediction error. In the same model, an implementation of learning from a motor efference copy also predicts a novel

Many proposed neural circuits for learning are theoretically well-grounded but have limited biological evidence to support them. Emerging anatomical and functional data from neurons in the fruit fly brain allows us to identify interconnections in the mushroom body neuropil that form an 'incentive circuit', supporting rich dynamics of memory acquisition, consolidation and forgetting. In addition, new insights into dopamine function in the fruit fly brain motivate a novel 'dopaminergic plasticity rule' that alters the weights between input sensory patterns and output neurons of the mushroom body

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g. no/no-go, choose left/right). However, there is mounting evidence that the basal ganglia functions not as a selector between predefined actions, but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a nee

We must constantly adapt the rules we use to guide our attention. To understand how the brain learns these rules, we designed a novel task that required monkeys to learn which color is the most rewarded at a given time (the current rule). However, just as in real life, the monkey was never explicitly told the rule. Instead, they had to learn it through trial and error by choosing a color, receiving feedback (amount of reward), and then updating their internal rule. After the monkeys reached a behavioral criterion, the rule changed. This change was not cued but could be inferred based on reward

Artificial recurrent neural networks (RNNs) are difficult to train due to their tendency towards instability, and common training algorithms that tame such networks are not biologically plausible, i.e., back-propagation through time (BPTT). Node perturbation learning in combination with local Hebbian weight updates has been shown to approximate BPTT [1] and, thus, could achieve similar performance while being biologically plausible. However, where these perturbations could arise from and how they are utilized within neurons remains obscure. For years, the neuroscientific community has known ab