A hallmark of natural intelligence is the ability to sustain stable memories while flexibly learning new associations, allowing animals to adapt to dynamic environments while executing precise behaviors years after they were first learned. Yet, recent experiments have revealed that neural representations of fixed stimuli change continuously over time, contravening the classical assumption that learned features should remain static to maintain task proficiency. The phenomenon of representational drift can be reconciled with normative principles for neural computation within the framework of probabilistic inference: drift in neural responses arises naturally during Bayesian sampling, which is a minimal model for noisy synaptic updates in the brain. However, our theoretical understanding of representation learning in deep Bayesian neural networks (BNNs) is generally poor. Here, we take the first step towards developing a rigorous theory of representation learning and drift by characterizing the statistics of the representational similarity kernels of each layer of large but finite BNNs. We show that network size controls the tradeoff between representational stability and flexibility: infinite BNNs are stable but have inflexible internal representations, while finite networks are flexible, but their kernels inevitably drift. In linear networks, we obtain a precise analytical description of how network architecture and the similarity between input stimuli affect the statistics of learned representations. During equilibrium sampling, representations of dissimilar stimuli drift more over time than representations of similar stimuli. Taken together, our results begin to elucidate how stimulus-dependent representational drift can arise in normative Bayesian models for neural computation. Moreover, they provide experimentally testable predictions for the structure of drift.