Signatures of belief representations in recurrent neural networks and prefrontal cortex

A popular model of the basal ganglia uses the theory of reinforcement learning (RL), where phasic dopamine activity encodes reward prediction errors to drive the learning of reward associations. In contrast to classical RL, however, where relevant state information is always provided to the agent, animals must find optimal behaviors using only incomplete state information. One theoretical observation is that animals (or agents) can solve these partially observable tasks by forming "beliefs," or optimal Bayesian posterior estimates of the state of the task. However, it is unknown whether animals could truly learn these theoretical constructs directly from experience, nor is it clear what signatures we might look for in neural data to know whether the brain truly uses a belief representation. To tackle this, we developed a two-pronged approach for linking the theory of beliefs with analyses. Using recurrent neural networks (RNNs), we can build brain-like model systems that must solve the task of estimating value given the same observations as animals. We then use these model systems to develop statistical analyses that determine whether or not the neural representations are consistent with beliefs. Using these analyses, we find that RNNs trained to estimate value do indeed discover belief-like representations. Tentative analysis of neural population activity from mouse cortex suggests cortex also exhibits similar dynamical structure as beliefs and RNNs.