Deep reinforcement learning (RL) has successfully trained machines to excel in video games, board games, and robotics. It has been argued that together, deep RL and robotics can be used to study biological learning by virtue of being embodied intelligence in the real world (Tan et al., 2021). In fact, RL itself was originally formulated to model animal behavior (Sutton & Barto, 1998). Here we go one step further and directly interface deep RL agents with a living neural network: that of the nematode C. elegans. We present a hybrid deep RL - C. elegans closed-loop computational system wherein an agent reads animal states through a camera and uses optogenetics to control neuronal activity. We trained the system to move animals to target locations. Agents successfully learned to control the movement of three genetic lines, each with different neurons that responded to optogenetic control. Building and training our system led to insights about which algorithmic choices mattered in learning the task; notably, appropriate agent regularization and data augmentation were important for success. In parallel, analyzing the policies of trained agents shed light on how neurons involved in each line could guide target-finding. Finally, we found that neither agent nor animal was always in complete control. Instead, existing sensory and motor systems in C. elegans integrated with RL agents to avoid obstacles during target-finding, or to override optogenetic input when animals reached a desirable state (like a patch of food). Remarkably, these behaviors emerged without any retraining of the system. Thus, we demonstrate that biologically-integrated deep RL can be used to control C. elegans behavior, to learn about neuron capabilities, to find key algorithmic principles for coordinating animal behaviors, and to study the interaction of biological and artificial neural networks.