Artificial neural networks (ANNs) for simulated motor control and robotics often adopt generic architectures like fully connected multi-layer perceptrons (MLPs) and randomly connected recurrent neural networks (RNNs). While general, these tabula rasa architectures rely on large amounts of experience to learn, and their internal dynamics are difficult to interpret. In nature, animals are born with highly structured connectivity in their nervous systems shaped by evolution; this innate circuitry acts synergistically with learning mechanisms to provide inductive biases that enable most animals to function well soon after birth and improve abilities efficiently. Convolutional networks inspired by visual circuitry encode the inductive biases of spatial locality and weight sharing to improve data and parameter efficiency for vision. It is unknown the extent to which ANN architectures inspired by neural circuitry can yield useful inductive biases for other domains. We asked what advantages biologically inspired network architecture can provide in the context of motor control. Specifically, we translate C. elegans circuits for locomotion into an ANN model applied to a simulated Swimmer agent. In contrast to previous work on neuromechanical models of movement, our model is designed within the abstract discrete-time ANN framework and is fully differentiable, enabling parameters to be trained with reinforcement learning (RL) and evolution strategies (ES) just like standard MLPs. Further, our model controls a body significantly different from the original. On a locomotion task, our architecture achieves good initial performance and asymptotic performance comparable with MLPs, while dramatically improving data efficiency and requiring orders of magnitude fewer parameters. Our architecture is more interpretable and generalizable to new body designs. An ablation analysis shows that principled excitation/inhibition significantly contributes to learning. Our work demonstrates several advantages of and design principles for ANN architectures inspired by systems neuroscience and suggests a path towards modeling more complex animals.