Controlling human cortical and striatal reinforcement learning with meta prediction error

Value-based decision-making in a context-changing environment is known to be guided by the two distinctive reinforcement learning (RL) strategies: goal-directed and habitual learning. Despite decades-long efforts to reveal that different environmental variables influence these RL processes, it remains unknown exactly how we create a situation by manipulating a set of environmental variables to guide these processes separately. This is mostly because of the complex interactions between environmental variables and RL, often beyond the experimenter’s comprehension. Here we show that an RL algorithm can be trained to control the task variables to guide human RL in a goal-directed manner. We individually fitted a computational model1 of goal-directed and habitual learning (called cognitive models) to the 82 subjects’ behavior data collected with a two-stage Markov decision-making task. Then we trained another RL algorithm (called a task controller) on another task to separately manipulate the key variables of the cognitive models: reward and state prediction error. Specifically, the task controller’s state and action are respectively defined as the cognitive model’s prediction errors and the task parameters: transition probability and reward profiles. The task controller was set to maximize or minimize the cognitive model’s prediction errors, while the cognitive model acts as a conventional human reward-maximizing agent: obtaining the reward from the task as much as possible. This asymmetric game setting enables the task controller to predict the prediction error of human RL (called meta prediction error). First, we showed in a subject-wise permutation test that the task controller learned a subject-independent policy. Second, we ran two fMRI experiments (n=50 in total, 25 for each controller condition) with the task whose parameters are determined by this task controller. We found the effect of task control on the behavior (choice optimality), latent process (prediction errors), and neural levels (cortex and striatum).