Representational drift has been observed in different parts of the nervous system. Nevertheless a complete mechanistic understanding about it is missing. In a recent experimental study in the olfactory system, recordings from the piriform cortex demonstrate drift in the representation of a single olfactory stimulus, despite apparently stable odor identification. Such drift is characterized by a decay in the self-similarity of the stimulus representation vector as a function of the time between the recordings. Additionally, the rate of the decay was shown to be smaller for more frequent stimuli. In this work, we study whether a diffusion process driven by noise during learning can explain some or all of the features observed in the experiments. We first show that a constrained diffusion process could explain the decay in the representation self-similarity. Next, we demonstrate how such process could occur as a result of noisy learning in a simple biologically relevant model of two-layer linear neural network, with the constraint being determined by the manifold of solution. We analytically derive the diffusion tensor on the manifold for the high-dimensional representation due to both online learning stochasticity and synaptic noise. Finally, using the current assumptions in the model, we quantify the change in the diffusion due to the application of a frequently applied stimulus.