Hearing loss is highly prevalent among the elderly and significantly alters peripheral and central auditory processing, which are essential for music perception. A literature review from our lab suggests that musical activities can positively influence music perception in older adults. However, most of the studies surveyed measured neither their participants’ hearing threshold nor their musical experience. The goal of this project was to investigate the impact of musical experience, hearing loss and age and the interaction between these variables on music perception. 54 older adults aged 60 to 95 years were recruited. Following a hearing screening, they completed a musical experience survey and two tasks measuring music perception: a short version of the Montreal Battery Evaluation of Amusia (MBEA) measuring pitch and rhythm and a task measuring emotion recognition in music. A multiple regression analysis revealed a negative association between MBEA scores and hearing loss, which was moderated by musical experience (R2 = 0.514, p < 0.001). Additionally, scores for emotion recognition showed a negative association with age, but a positive association with musical experience (R2 = 0.475, p < 0.001). This suggests that between hearing loss and age, the former has a stronger negative association with MBEA performance while the latter is most strongly negatively associated with emotion recognition in music. Yet, musical experience has a protective effect on performance for both tasks, suggesting that it has a positive impact on music perception in the elderly.