AUDIOVISUAL INTEGRATION IN CORTICAL SPEECH TRACKING EVOKED BY TALKING AVATARS

Visual cues from a speaker’s face can substantially enhance speech comprehension in noisy environments. This benefit relies on audiovisual integration in the brain and includes neural tracking of speech-related rhythms. Recently, digitally generated avatars have emerged as a potential tool to support speech comprehension; however, how the brain integrates these artificial visual signals with natural speech remains poorly understood. In this study, we used magnetoencephalography (MEG) to measure neural responses to speech accompanied by a natural talking face, a deep–learning–based avatar, as well as a degraded video serving as a control condition [1]. Behaviorally, the avatar improved speech-in-noise comprehension to a similar extent as the degraded video, but less than the natural video. Neurally, we identified a late response around 600 ms in the auditory cortex, reflected in neural tracking within the high-delta band (1–4 Hz), that reliably predicted audiovisual speech comprehension. In contrast, neural tracking in the low-delta band (0.5–1 Hz) was associated with silent lip-reading performance. Notably, avatars elicited markedly weaker and earlier low-delta tracking than the other audiovisual stimuli. Neural tracking in the theta band (4–8 Hz) did not contribute to audiovisual integration. Together, these findings demonstrate distinct functional roles of low- and high-delta neural tracking in visual-only and audiovisual speech processing and highlight targets for improving avatar-based support of speech comprehension.

[1] Riegel et al., Talking avatars can differentially modulate cortical speech tracking in the high and in the low delta band, in review; bioRxiv, https://doi.org/10.64898/2026.01.07.695461