Improved striatal learning with vector-valued errors mediated by diffusely transmitted dopamine

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g. no/no-go, choose left/right). However, there is mounting evidence that the basal ganglia functions not as a selector between predefined actions, but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of dynamic outputs, rather than discrete action values.

Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which striatal dopamine carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). Using a recurrent network model of the basal ganglia, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. The corticostriatal plasticity rule (based on the RFLO algorithm) we employed is a fully local, ”three-factor” product of the presynaptic firing rate, a post-synaptic factor and the unique dopamine concentration perceived by each striatal neuron. Crucially, we demonstrate that under this plasticity rule, the improvement in learning does not require precise nigrostriatal synapses, but is compatible with random placement of varicosities and diffuse volume transmission of dopamine.