The conversion of visual input into behavior requires whole-brain circuits of diverse neurons. Due to the inaccessibility of mammalian model systems, there exists a gap in understanding how these neurons are connected and influence neural response dynamics that underlie behavior. Recent studies into the visually guided optomotor response (OMR) in larval zebrafish have attempted to bridge this gap by proposing circuit models that predict connectivity among functionally identified response classes of motion-processing neurons driving the OMR [1,2]. However, these models only consider responses of overrepresented classes averaged across many fish and therefore ignore large swaths of the motion-responsive population, fail to capture inherent circuit dynamics, and cannot describe idiosyncrasies in neural responses and behavior. Here, we train recurrent neural networks (RNNs) with calcium imaging data to model all neurons in the pretectum (Pt), the central sensory processing region underlying the OMR, responding to eye- and direction-specific visual motion. These models generate predictions of local and inter-hemispheric connectivity, including the signs, strengths, and numbers of synapses. Our model estimates connections between individual neurons, providing insight into the potential role of underrepresented response classes, previously from models [2]. To drive the RNNs with visual stimuli, we implemented a novel, biologically plausible set of directional-selective retinal ganglion cells providing external input to the population. We show the RNNs to be exceptional at reproducing activity from neurons with diverse responses to motion. We find the RNNs predict excitatory connections among neurons with shared motion preference and of inhibitory connections among neurons responsive to conflicting stimuli, supporting the hypothesis of shared functional roles of previously identified neural response classes. Our RNNs provide a realistic, dynamic circuit model of a complete sensory population and generates hypotheses about the nature of vertebrate information processing that will inform future photostimulation experiments to illuminate neural circuitry underlying the OMR.