Deep feedforward and recurrent networks have become popular choices for models of neural computation, but they do not provide a clear link to the details of the underlying biophysical processes. Here we argue that the reason this link has been difficult to establish is because biological circuits operate in a fundamentally different way. To illustrate this, we present a new theory of neural computation in excitatory-inhibitory spiking networks, which captures the precise computational role of each neuron and every spike. By assuming low-rank recurrent connectivity, we show that spiking population activity is confined to a well-defined nonlinear manifold in a low-dimensional latent space, with the spikes of individual neurons pushing the latent dynamics along this manifold. We then show that the network’s recurrent connectivity can be factorized into a part that determines the manifold geometry and another part that determines the manifold dynamics. The stability of the on-manifold dynamics can be enforced through sign constraints --- either through an all-inhibitory network with a constant background input, or through an (inhibition-stabilized) excitatory-inhibitory network --- thereby suggesting a functional role for Dale’s law. We show that such networks can approximate arbitrary continuous dynamical systems, and demonstrate several examples including a limit cycle, ring attractor, and a set of Hopfield-like fixed-point attractors. Overall, our work proposes a new way of understanding the dynamics and computations of spiking neural networks, and suggests the possibility of an intriguing distinction between biological and artificial computation.