Prism Adaptation (PA) is a visuomanual adaptation paradigm, recently shown to induce multimodal cognitive aftereffects affecting visuospatial representation, auditory representations or auditory divided attention. The present study aimed at investigating whether PA can lower the auditory detection threshold in healthy young and older adults. Fifty-eight healthy subjects (30 young adults aged 24.4 ± 4.6 years and 28 older adults aged 70.9 ± 6.2 years) performed an audiometry test to determine their hearing threshold. Then, 1000 Hz pure tones were played randomly in their left or right ear at the intensity of their hearing threshold, and they had to indicate rapidly in which ear they perceived the sound. This task was performed before and following a rightward (RPA) or a leftward PA (LPA). We focused on the percentage of sounds correctly detected (accuracy), the Reaction Time (RT) and fatigue was assessed with a Visual Analog Scale at the beginning and at the end of the protocol. In the younger group, we observed no effect of PA on accuracy and RT. The older group exhibited a lowering of accuracy (p = .006) and an increase of RT (p = .014) with an interaction of RT and fatigue following LPA (p = .014), but no auditory aftereffect of RPA. These asymmetrical aftereffects of PA in older adults could be interpreted as the reflect of a greater cognitive cost of LPA, that could lead to an increase in fatigue resulting in a decrease in performance.