Harnessing the flexibility of neural networks to predict meaningful theoretical parameters in a multi-armed bandit task

Classical studies in reinforcement learning (RL) heavily rely on normative models of behavior, models that often stress interpretability over predictive capabilities. More recently, neural network models emerged as a descriptive modeling paradigm capable of high predictive power, however, with limited interpretation. To address these issues, we sought to augment the expressiveness of theoretically interpretable RL models with the high flexibility and predictive power of neural networks. Specifically, we introduce a novel framework whereby a neural network is trained to predict theoretically meaningful RL parameters from behavioral observations. We first simulated behavior from a large number of artificial RL agents performing a two-armed bandit task. We then trained a recurrent neural network, using the trial-by-trial artificial data to predict the agent’s hidden theoretical RL parameters from observed action-reward history. At test time, and without further optimization, we illustrate the network's successful recovery of unseen agents' RL parameters, even when their true RL parameters dynamically changed across task performance. We then applied our network to estimate non-stationary theoretical RL parameters of empirical data, where healthy and psychiatric individuals completed a similar two-armed bandit task. We found better performance in the action prediction of our network compared to two alternative methods of RL parameters estimation (non-stationary Bayesian particle filtering and stationary maximum-likelihood). We demonstrate the advantage of our modeling approach by describing high volatility in the parameters’ estimation of psychiatric individuals, a group known for unstable internal mental states. Finally, we discuss how our framework may facilitate future research where temporal changes in latent parameters can be correlated with meaningful high-temporal neural measures.