Retinal ganglion cells (RGCs) show high convergence onto their downstream projections, which poses a problem for information transfer: how is it possible to preserve information through a synaptic layer that has significantly more inputs than outputs? There are an infinite number of ways to perform compression, and lossy compression suggests efficient yet computation-agnostic methods for reading out input stimuli or activity patterns. However, downstream neurons must perform essential computations on their inputs for the organism’s survival, rather than simply recapitulate input activity. In the visual system, one of those essential computations is motion prediction, which is necessary to overcome sensory and motor delays to avoid predators, catch prey, and navigate complex environments. RGC activity provides downstream areas with a near-optimal representation of the future, significantly narrowing down the number of plausible compressions by giving us insight into the limits of information transfer. Here we show that prediction-agnostic compression can, surprisingly, preserve predictive information on short timescales, but it is sub-optimal and does not generalize across scenes. This suggests that retinal projection areas such as the optic tectum, which is key for making fast reactions to visual stimuli, are unlikely to use such a compression. Instead, we find evidence that downstream areas may solve a generalized optimization problem for predicting in natural scenes in order to compress their retinal inputs. Other sensory systems also exhibit compression in their processing hierarchies, and we hope that our framework will be useful in cases where it is not yet known how information about a specific computation is maintained under compression.