I will discuss work with Jack Lindsey modeling reinforcement learning for action selection in the basal ganglia. I will argue that the presence of multiple brain regions, in addition to the basal ganglia, that contribute to motor control motivates the need for an off-policy basal ganglia learning algorithm. I will then describe a biological implementation of such an algorithm that predicts tuning of dopamine neurons to a quantity we call "action surprise," in addition to reward prediction error. In the same model, an implementation of learning from a motor efference copy also predicts a novel solution to the problem of multiplexing feedforward and efference-related striatal activity. The solution exploits the difference between D1 and D2-expressing medium spiny neurons and leads to predictions about striatal dynamics.

During the course of an animal’s interaction with its environments, activity within central neural circuits is orchestrated exquisitely to structure goal-oriented movement. During walking, for example, the head, body and limbs are coordinated in distinctive ways that are guided by the task at play, and also by posture and balance requirements. Hence, the overall performance of goal-oriented walking depends on the interplay between task-specific motor plans and stability reflexes. Copies of motor plans, typically described by the term efference copy, modulate stability reflexes in a predictive manner. However, the highly uncertain nature of natural environments indicates that the effect of efferent copy on movement control is insufficient; additional mechanisms must exist to regulate stability reflexes and coordinate motor programs flexibly under non-predictable conditions. In this talk, I will discuss our recent work examining how self-generated visual signals orchestrate the interplay between task-specific motor plans and stability reflexes during a self-paced, goal-oriented walking behavior.