Behavioral and brainwide correlates of dynamic reward prediction

Adaptive behavior is guided by dynamic evaluations of the reward environment, which can be influenced by factors like reward frequency and internal motivational state. We implemented a behavioral task in water deprived headfixed mice that included fluctuating probabilities of reward following auditory stimuli. Mice adjusted their anticipatory licking, a proxy for reward expectation, according to both the recent reward rate and their diminishing motivation for reward. We developed a reinforcement learning model that revealed the synergistic impacts of recent reward history, overall reward rate, and satiety on individual subjects’ behavior. This model captured session and subject variability and allowed us to calculate a trial-by-trial estimate of expected value. To interrogate the neural implementation of expected value, we recorded the spiking activity of ~10,000 individual neurons across multiple brain systems, including prefrontal cortex, striatum, amygdala, hypothalamus, thalamus, midbrain, and pons. We then performed a kernel regression to predict the activity of individual neurons. By incorporating the model-generated value estimates into our kernel model, we were able to identify neurons across all regions with activity that fluctuated along with value. Interestingly, these value correlates were present in both the tonic activity and phasic cue-evoked responses of individual neurons, with regional differences in the prominence of each. For instance, substantia innominata value neurons had strong value modulation of cue-evoked activity, while prefrontal cortex value neurons modulated their tonic firing according to trial value. Overall, our findings demonstrate a role for proximal and global reward rate, as well as satiety, in subjects’ reward predictions, and indicate the presence of diverse value coding schemes spread across many brain systems with potential to dynamically contribute to adaptive behavior.