Information in the nervous system is encoded by the spiking patterns of large populations of neurons. The analysis of such high-dimensional data is typically restricted to simple, arbitrarily defined features like spike rates, which discards information in the temporal structure of spike trains. Here, we took a fundamentally different approach to neural coding by using a novel method, SpikeShip, that is based on optimal transport theory. SpikeShip captures the information in all the relative spike-timing relations among neurons, without defining arbitrary features and counting spikes within windows. We compared spike-rate and spike-timing codes in neural ensembles from six visual areas in mice during natural video presentations. We show that the temporal structure of spiking patterns among thousands of neurons encodes natural video content substantially more reliably than population rate vectors. Furthermore, we show this advantage only becomes apparent when considering neural ensembles of hundreds of neurons and larger. Additionally, population rate vectors exhibited substantial drift across repetitions and between blocks. Conversely, encoding through temporal sequences was stable over time, and did not show representational drift both within and between blocks. These findings unveil a neural code based solely on relative spike timing in ensembles, challenging the conventional method of counting spikes within windows for cortical coding mechanisms. The findings will be of interest to a wide audience of neuroscientists interested in temporal coding, representational drift, and the encoding of natural movies by visual cortex.