Arousal influences episodic memory retrieval, improving performance for arousing to-be-remembered stimuli, but its effect on later memory processes remains unclear. This functional Magnetic Resonance Imaging (fMRI) study investigated the effect of negative and positive arousal on memory of unrelated, neutral information presented following arousal exposure. Considering a decline in episodic memory retrieval and alterations in emotion regulation processes that occur in healthy ageing, we aimed to examine the effect of arousal on memory in young and old adults. Our findings indicate that both groups exhibited poorer performance in retrieving episodic memories related to a neutral video when it was preceded by a high-arousing negative image. A voxel-to-voxel whole-brain multivariate pattern analysis (MVPA) identified age-related differences in the frontal, occipital, and limbic cortices during memory retrieval following arousal. Using these data-driven differences as seeds, we examined the connectivity patterns triggered by positive and negative arousal. We found age-related differences in the occipital cortex, particularly in response to negative compared to positive and neutral stimuli. Our findings add to existing literature by highlighting that negative arousal exerts a more pronounced disruptive impact on later memory processes, and these effects are linked to multiple changes in functional connectivity with age. This work informs the understanding of the heterogeneous observation of how arousal affects episodic memory in different age groups and guides approaches towards the enhancement of memory in healthy ageing.