Modeling the trial-by-trial dynamics of a reversal learning task with reinforcement learning

Analysis of behavioral data usually averages responses across subjects and/or across trials. However, this approach obscures the actual dynamics of learning, which can occur within few trials and differ between subjects. For instance, reexamination of previous data (Uengoer and Lachnit, 2006) reveals that during learning subjects do not change their responses gradually, but abruptly (Fig. A). These “jumps” occur at different trials for each subject and averaging therefore paints a misleading picture of gradual learning curves. These data are not easily accounted for with simple associative learning models such as the Rescorla-Wagner model. Hence, we used Deep Reinforcement Learning (Deep RL) to model the choice behavior of subjects in reversal learning. In the task, subjects were shown stimuli (food items) in either of two contexts (restaurants) in each trial and had to predict whether the presented item will result in stomach troubles. The Deep RL agent takes simplified observations representing the item-restaurant pairings as input (Fig. B) and outputs the Q-values, i.e., expected future reward, for two actions which reflect the possible choices of the subjects. Deep RL agents were trained on the experiences of each subject. A simple grid search was used to determine the hyper-parameters, e.g., learning rate, which yielded the best fit. Finally, we compared our results to the Rescorla-Wagner model. In summary, we show the importance of analyzing subject-level learning dynamics and test the ability of RL and associative models to account for them.