Efficient coding theory, which stipulates that the nervous system attempts to minimize redundancy in stimulus coding, has proven widely successful in predicting observed features in early sensory systems, including center-surround receptive fields (RFs), ON and OFF parallel pathways, and the relative arrangement of ON and OFF RF mosaics. However, relatively limited work has addressed the encoding of spatiotemporal stimuli. Ocko et al. (2018) examined spatiotemporal encoding in the retina using an approach based on a convolutional autoencoder, replicating known features of retinal ganglion cell (RGC) receptive fields like midget and parasol cells. However, this work assumed a configuration consisting of exactly two cell types with a global stride factor to represent the size and density of the RFs, and the input data were synthetic Gaussian noise. This leaves a number of questions unanswered, including the optimal number of RF types, their size/mosaic arrangements, and the impact of natural movies’ higher-order statistics. Extending a previously proposed efficient coding model for images, we trained a model to maximize mutual information between natural movies and neural firing rates. We found that the model produced four distinct types of RGC mosaics: One ON/OFF subgroup exhibited large, temporally precise filters, while the other exhibited spatially precise, temporally slow filters. These resemble the known distinctions between parasol and midget RGCs (in the monkey retina). Moreover, when given a larger population of neurons, the model produced additional sets of ON and OFF RF types/mosaics by subdividing the group of temporally precise filters. These results suggest a principle by which the visual system might specialize neural response types based on available channel capacity and further validate efficient coding as a theoretically grounded computational framework for reasoning about the roles of specialized RF types and their diversity.