Adaptation is ubiquitous in sensory systems. Sensory neurons change their activity depending on the past stimulus history, which they integrate at various timescales. Short- and long-term adaptations have been mostly studied using simple, artificial stimuli, but rarely with natural scenes. Even at the level of the retina, it is unclear how these adaptive processes impact ganglion cells, the retinal output, and how they change their feature selectivity to natural scenes as a function of the recent stimulus history. This raises a major challenge for our understanding of how ganglion cells process the continuous stream of natural stimuli. This is also a challenge for modeling: if we don’t model the process of adaptation, we cannot have models that generalize across stimulus statistics. Here we measured and modeled how short-term adaptation impacts ganglion cell selectivity to natural scenes. We recorded ganglion cells while stimulating them with natural images perturbed with noise patterns of small amplitudes. By repeating the same natural image, but with different noise patterns, we could determine the selectivity of each ganglion cell to changes in the natural image (Goldin et al., 2022). We then presented new stimuli where the same perturbed images were now preceded by another flashed stimulus (400 ms before), that triggers short-term adaptation. This stimulus altered the response of ganglion cells to the natural image, and also drastically modified the selectivity to changes in the image. Short-term adaptation has thus a major effect on ganglion cell feature selectivity to natural scenes. We designed models that could reproduce quantitatively both the responses of ganglion cells to natural scenes and their selectivity to changes in the image. A two-layer network could predict the responses to flashed natural images, but a gain control mechanism at the end of the first layer was necessary to predict the observed effects of short-term adaptation. Our results suggest that these gain control mechanisms are the key components that need to be taken into account to allow models to generalize across stimulus statistics.