Although reinforcement learning has been highly successful in training artificial agents to perform a wide variety of complex tasks, the principles governing how animals actually learn to perform new tasks remain poorly understood. While normative approaches have provided profound insights into the design of optimal agents, they often make assumptions that may not be aligned with the sub-optimal learning strategies employed by animals in experimental settings. Even in simplified two-alternate force choice task, mice have shown significant variability in learning dynamics despite highly standardized training protocols. On the other end, descriptive approaches have explored the dynamics and variability extensively, but typically without a learning model. To overcome these limitations, we derive a framework to infer empirical reinforcement learning objective functions itself driving learning dynamics from single-animal behavioral learning trajectories. We introduce a semi-parametric form for the gradient of the objective that encompasses and generalizes known optimal RL policy gradient rules, and use probabilistic inference methods to infer its parameters. We show that mouse learning can be better described by accounting for forgetting terms and as a mixture of optimal RL updates, and that these update rules can be leveraged to disentangle the learning from the noise, paving the path towards using these methods for active experimental training.