The current dogma in neuroscience is that neurons primarily convey stimulus information through their firing rate. However, recent studies suggest a remarkable speed of sensory processing which may be incompatible with traditional rate-coding schemes, and it has been hypothesized that sensory encoding may rely on the spike timing relationships among neurons (Thorpe et al., 2001 and Resulaj et al., 2018). To study multi-neuron spiking patterns, we developed SpikeShip, an unsupervised, linear, geometry-based dissimilarity measure that aligns spikes across pairs of epochs based on optimal transport cost. This method has linear computation cost and is sensitive to higher-order correlations in spike trains. We used both rate and timing codes to find clusters across N > 8000 neurons from six visual areas during natural video presentations in 32 mice. We split the video into 30 sub-videos of one second each as in previous studies (Deitch et al., 2021), and compared information content in firing rate population codes and multi-neuron temporal spiking patterns. Using SpikeShip, we show that (1) multi-neuron temporal patterns convey substantially more information about natural movies than population firing rates; (2) multi-neuron temporal patterns show high reliability across presentations, in contrast to firing rate codes; (3) firing rate codes exhibit memory across frames, whereas temporal patterns form a discrete and discontinuous manifold separating different movie frames; (4) the advantage of temporal information becomes larger as the number of neurons grows. These findings reveal the importance of temporal spiking patterns in the encoding of natural visual inputs.