Being able to identify key patterns of neural network connectivity that are specifically required to generate functional response patterns is a challenging and aspirational goal in neuroscience. The difficulty, in part, is because we are typically only able to probe a network with a limited number of stimulus conditions, and there exists huge degeneracies in the ways we can connect the neurons to reproduce the observed responses. Here we present a new geometric ensemble modeling approach to this problem. Inspired by whole-brain imaging approaches, we assume that we have access to a finite number of steady state response patterns of all the relevant neurons in a given network, and we then ask if we can identify key excitatory or inhibitory connections that must exist to generate the responses. We assume that the network can be modeled by a recurrent neural network of rectified linear units that receive feedforward inputs. We develop a geometric framework to analytically tackle the problem. We then apply our theory to predict biological connectivity required to generate binocular responses of zebrafish pretectal neurons to various optomotor stimuli, which were previously recorded using whole-brain imaging.