Cortical dopamine enables deep reinforcement learning and leverages dopaminergic heterogeneity

Midbrain dopamine (DA) activity is implicated in reinforcement learning (RL). DA activity is observed to signal reward prediction error (RPE), a key quantity in many RL algorithms used to modulate updates to state-reward and state-action associations. How the state representations underlying these associations are themselves learned is typically outside the scope of RL models of the basal ganglia. However, modern RL algorithms in machine learning typically employ deep RL, in which state representations are learned as part of the RL algorithm, to attain state-of-the-art performance. In this work, we investigate the requirements for biologically plausible deep RL by studying a model in which DA projections to cortical regions upstream of the striatum drive state representation learning. We show that coarse, global DA projections are sufficient to enable effective representation learning, provided there exist two classes of cortical neurons that undergo DA-modulated plasticity according to learning rules with opposite signs (as experimental evidence suggests is the case for D1 and D2 receptor-expressing neurons). We next apply our model to an open question in the field: recent experimental evidence suggests DA population activity does not uniformly signal RPE, but rather heterogeneously encodes higher-dimensional information about an animal’s state. While prior work has proposed interpretations of this phenomenon, it remains unclear whether such heterogeneity serves a useful function in the context of RL. We show that although DA heterogeneity provides no benefit in classic RL models without state representation learning, it improves learning performance in our model. We develop a theoretical framework that explains this performance improvement and enables predictions about the optimal set of DA signals for a given task. We describe a key prediction of our model, that cortical representations of state should change over the course of learning to carry more of the same information as DA activity.