Musculoskeletal skill learning with curriculum-based Static to Dynamic Stabilization

Building realistic models for complex and skilful motor tasks such as dexterous object manipulation is hampered by two key challenges: the need for physiologically-detailed musculoskeletal models and powerful algorithms that are able to tackle such high-dimensional problems. The newly released MyoSuite ecosystem provides such musculoskeletal models and is 4000x faster than previous simulators, thereby enabling us to develop and test algorithms for agile object manipulation. Combining ideas from stochastic optimal control and reinforcement learning (RL), we develop a training curriculum called Static to Dynamic Stabilization (SDS). The SDS curriculum first learns stable static solutions at several intermediate points along the desired object trajectory and gradually relaxes them to yield dynamically stable movement motifs. Akin to coaching techniques for skill-learning in humans, SDS allows the agent to experience intermediate configurations before learning a policy that reaches those configurations from the default initial state of the task at hand. We trained a policy gradient method using the SDS curriculum on the Baoding Balls task which involves rotating two balls in the hand. The task is extremely challenging since the environment is noisy, unstable, partially observable, and involves a time-varying decision-making component. Our model beat all SOTA baselines by a wide margin and won the MyoChallenge, a NeurIPS 2022 competition hosted by Meta AI. To the best of our knowledge, our model is the first successful example of fully-learned musculoskeletal control in a highly skilled object manipulation task. This highlights the usefulness of the SDS curriculum to develop realistic, high-performance sensorimotor models, thereby paving the way to compare and contrast artificial and biological control systems performing arbitrarily complex motor skills.