In the mouse visual cortex, neural representations of stimuli fluctuate across repetitions (Schölvinck et al. 2015, Deitch et al. 2021). The single-trial dynamics, often termed as representational drift, were also found in other cortical structures (Rokni et al. 2007, Low et al. 2021, Schoonover et al. 2021). However, most of these studies focused on slow drift over days. Here, we demonstrated that representational drift occurs at a fast timescale of several minutes and is prevalent regardless of stimulus type and independent of behavioral state. Within sessions of the same day, repeats of the same stimulus in different blocks were separated in the low-dimensional neural manifold, indicating larger changes in neural activity across blocks than within blocks. The trial-by-trial variability was partially associated with behavior variables but did not account for fast drift. It suggests that the fast representational drift was more than just fluctuations in behavioral state. Moreover, in line with previous work (Xia et al. 2021), we found that decoding remained stable despite representational drift, indicating that fast drift is restricted to a coding subspace orthogonal to the stimulus dimension. To disentangle the representational drift component from behavior-relevant variability, we adopted an unsupervised dimensionality reduction method called Tensor Component Analysis (TCA) to identify underlying factors. TCA decomposed the trial-by-trial variability into behavior-related and fast drift components. Furthermore, we found that subcortical areas also displayed representational drift across blocks. Canonical Correlation Analysis (CCA) showed a strong correlation in drift between brain structures. Using a generative model called Inter-Battery Factor Analysis (IBFA), we identified components that were shared between visual cortex and hippocampus. The shared drift factors were highly correlated with the disentangled tensor components, indicating that the fast representational drift signal is propagated among different brain structures.