The involvement of early visual cortex during the short-term maintenance of visual information has been demonstrated in human fMRI studies [1,2]. However, evidence for sustained, stimulus-related neuronal firing in primate V1 is limited [3,4]. Recently, fading traces of visual information have been shown in the population code of primate V1 neurons for ~1.5 seconds after visual stimulus presentation in delayed match to sample and attention tasks [5]. The relatively long time-course of these fading memories make it unlikely to be a neural correlate of iconic memory [6], but may be related to “fragile working memory”, a possible high-capacity memory buffer, that does not depend on sustained attention, but gradually degrades over time [7,8]. Here we show that the previously reported fading memory traces 5 are the inverse of the population code during stimulus processing, as evident from negative cross-temporal generalization from the stimulus epoch to the delay. This code reversal can be accounted for by a reversal in stimulus preferences at the single unit level. This inversion was observable from spontaneous neuronal spiking activity during the delay, which was at or below baseline activity levels, and became more apparent during visually evoked neural activity: when presenting a task irrelevant visual “impulse stimulus” during the delay, which has been shown to evoke a memory-specific neural response [9], neuronal firing rates were transiently increased. This evoked “impulse response” resulted in relatively fewer evoked spikes when the unit’s preferred stimulus had been shown previously, and more when a non-preferred stimulus was shown. These results can be explained by short-term synaptic depression, which we verified with a simple computational neural network. We speculate that this property of sensory neurons provides an efficient short-term memory buffer that can be used for fine-grained visual working memory.