MULTI-TIMESCALE REWARD PREDICTION LEARNING DYNAMICS WITH ENTANGLED CONNECTIVITY BETWEEN THE STRIATUM AND THE DOPAMINE SYSTEM

Adaptive behaviour requires animals to learn to predict future rewards. The dopamine neurons of the ventral tegmental area (VTA) are believed to support this process by broadcasting a teaching signal called a reward prediction error (RPE) into the ventral striatum. Classical computational modelling suggests that dopamine neurons homogeneously encode a temporally-discounted RPE whose predictions follow a characteristic time horizon. More recently, with experimental work showing highly heterogeneous discounting in dopamine neurons, a suggestion has been that multiple time horizons are tracked in parallel. However, this suggestion depends on biologically-unrealistic one-to-one connectivity between neural ensembles in VTA and striatum. We use simulations to explore the effect of relaxing this assumption. First, we present a minimal biologically-plausible network that produces diverging reward predictions. Second, we exhibit a Gaussian fan-out connectivity scheme that eventually produces a valid, albeit linearly entangled, multi-timescale reward prediction in the striatum. Third, we show that non-specific connectivity induces complex learning dynamics, including damped fluctuations, even in simple trace conditioning tasks. This is partially explained by the differing learning rates that non-specific connectivity implies for coarse and fine distinctions across reward prediction time horizons. Overall, while our results broadly support the biological plausibility of dopamine-driven multi-timescale reward learning, they also highlight major differences from classical models. Whether the unusual dynamics we observe manifest as behavioural idiosyncrasies in animals learning to predict reward remains an open question.