A normative framework for balancing reward- and information-seeking behaviors in dynamic environments

Our understanding of how the brain makes decisions has benefited greatly from normative theories that have, for example, established benchmarks for how specific forms of evidence accumulation and decision rules can optimally balance quantities like decision speed and accuracy. These theories have recently been expanded to include adaptive components that are necessary for effective decision making in dynamic environments. However, these extensions remain incomplete, because they have focused primarily on how an unconstrained agent manages single, independent decisions [1]. In contrast, real-world decision-makers are often constrained by limitations on resources like energy and time. These constraints can have complicated effects on not just single decisions, but also on sequences of decisions in volatile environments. Here we present a new theoretical framework for understanding quantitatively, and optimizing with respect to, the costs and benefits of different sequential decision-making behaviors in dynamic environments.
Our framework is based on a Bayesian model of decision making in environments with temporally correlated structure. Specifically, we assumed that after each trial, the correct choice changes according to a discrete-state Markov process, and that the agent can take a limited number of actions per block of trials. The resulting rich, optimal choice behaviors depend strongly on task conditions, but generally exhibit the explore-exploit trade-off characteristic of many foraging tasks. Our major contribution is a predicted, sharp transition from exploratory to exploitative strategies when choice feedback provides more information about the next trial than the expected environmental evidence, and thus a comparison of reward-optimal and information-optimal strategies [2]. More generally, our work provides a broad, quantitative account, including experimentally testable predictions, of the effectiveness of different decision strategies under natural constraints.