Efficient, physiologically-detailed musculoskeletal simulators and powerful learning algorithms provide new computational tools to tackle the grand challenge of understanding biological motor control. Our winning solution for the first NeurIPS MyoChallenge leverages an approach mirroring human learning and showcases reinforcement and curriculum learning as mechanisms to find motor control policies in complex object manipulation tasks. The training procedure, which we call static-to-dynamic stability (SDS), defines a curriculum of progressively more complex state-space neighborhoods, which guide a policy towards dexterous object manipulation. In this way we can train a recurrent neural network to control a biologically-realistic human arm and rotate two Baoding balls in variable conditions. Analyzing the policy against data from human subjects reveals insights into efficient control of complex biological systems. Indeed, the SDS policy exhibits a number of properties that have been observed in primates. Firstly, we found a low-dimensional posture and control space, reminiscent in humans performing the Boading balls task. Secondly, we found that the controller is robust to activity perturbations and that low-variance principal components still contain task-relevant signals. Thirdly, we found lower tangling of the dynamics in the learned controller than in the action space, akin to what Russo et al. found for motor cortex vs. muscle dynamics (Neuron 2018). Overall, our work highlights the new possibilities emerging at the interface of musculoskeletal physics engines, reinforcement learning and neuroscience.