Richly structured reward predictions in dopaminergic learning circuits
Theories from reinforcement learning have been highly influential for interpreting neural activity in the biological circuits critical for animal and human learning. Central among these is the identification of phasic activity in dopamine neurons as a reward prediction error signal that drives learning in basal ganglia and prefrontal circuits. However, recent findings suggest that dopaminergic prediction error signals have access to complex, structured reward predictions and are sensitive to more properties of outcomes than learning theories with simple scalar value predictions might suggest. Here, I will present recent work in which we probed the identity-specific structure of reward prediction errors in an odor-guided choice task and found evidence for multiple predictive “threads” that segregate reward predictions, and reward prediction errors, according to the specific sensory features of anticipated outcomes. Our results point to an expanded class of neural reinforcement learning algorithms in which biological agents learn rich associative structure from their environment and leverage it to build reward predictions that include information about the specific, and perhaps idiosyncratic, features of available outcomes, using these to guide behavior in even quite simple reward learning tasks.
Dynamical population coding during defensive behaviours in prefrontal circuits
Coping with threatening situations requires both identifying stimuli predicting danger and selecting adaptive behavioral responses in order to survive. The dorso medial prefrontal cortex (dmPFC) is a critical structure involved in the regulation of threat-related behaviour, yet it is still largely unclear how threat-predicting stimuli and defensive behaviours are associated within prefrontal networks in order to successfully drive adaptive responses. To address these questions, we used a combination of extracellular recordings, neuronal decoding approaches, and optogenetic manipulations to show that threat representations and the initiation of avoidance behaviour are dynamically encoded in the overall population activity of dmPFC neurons. These data indicate that although dmPFC population activity at stimulus onset encodes sustained threat representations and discriminates threat- from non-threat cues, it does not predict action outcome. In contrast, transient dmPFC population activity prior to action initiation reliably predicts avoided from non-avoided trials. Accordingly, optogenetic inhibition of prefrontal activity critically constrained the selection of adaptive defensive responses in a time-dependent manner. These results reveal that the adaptive selection of active fear responses relies on a dynamic process of information linking threats with defensive actions unfolding within prefrontal networks.