Novelty drives human exploration even when it is suboptimal

How do humans explore environments with sparse rewards? Recent studies in computational neuroscience have proposed an answer by integrating intrinsic reward signals into traditional reinforcement learning models of human behavior. However, different choices of intrinsic reward result in fundamentally different exploration strategies. Here, we focus on three representative intrinsic reward signals (novelty, surprise, and information-gain) and ask which one explains human exploration best.

We design an experimental paradigm for multi-step decision-making where the three representative intrinsic rewards make different predictions: Participants search for a goal state in a complex environment with 59 states and 3 actions per state that, importantly, includes a large sub-region (50 states) with highly stochastic transitions. This sub-region mimics the main features of a noisy TV as in the infamous 'noisy TV problem'. We show via simulations that exploration driven by different reward signals exhibits different levels of attraction to this stochastic sub-region. Simulated agents driven by information-gain eventually lose their interest in stochasticity when they realize that there is no information to gain, agents driven by novelty exhibit a persistent attraction to stochasticity and the ones driven by surprise a detrimentally increasing attraction. Performing the same experiment on human participants shows that human participants who are optimistic about the availability of goal states of higher value than those already known exhibit a persistent attraction to stochasticity. We show that this behavior is both qualitatively and quantitatively consistent with that of novelty-driven agents and not with those driven by surprise or information-gain – despite the evident efficiency of information-gain in dealing with stochasticity.

Our work provides evidence for novelty-driven reinforcement learning algorithms as models of human exploration and suggests that humans use suboptimal but computationally cheap policies for exploration in complex environments.