Recordings from neural populations can be biased towards cells that are spontaneously active, and it is often implicitly assumed that stimulus responses will be unrelated to spontaneous activity. To avoid such biases [1] performed blind whole-cell patch clamp recordings of olfactory bulb mitral cells. Intriguingly, these recordings revealed an inverse relationship between spontaneous and odor-evoked activity: cells that were highly active at baseline were often quiet during odor presentation, while conversely, `silent' cells, quiet at baseline, dominated odor responses. In this study we use a probabilistic model of early olfactory processing to show how such an inverse relationship between baseline and odor-evoked activity can emerge through an odor-triggered variant of olfactory background subtraction we call `common feature subtraction.' In this computation, the olfactory system subtracts features common to recently encountered odors to preferentially infer the features unique to each odor. We suggest that the common features are not modeled directly, but are accounted for through the negative of their expected effects on receptor inputs. In a neural implementation of our probabilistic model we show that adjusting the background activation of mitral cells towards the negative of their recent activity results in their background rates reflecting the negative of the expected effect of the common features on receptor inputs. This negative expectation cancels the contribution of common features when odors are present, allowing preferential inference of odor-specific features by downstream neurons. Our model qualitatively reproduces the experimentally observed relationship between baseline and odor-evoked responses of mitral cells in [1] and provides a normative explanation for them. It is also readily testable through the direct link it predicts between baseline responses and the common features of recent odors.