DELTA-BAND CORTICAL SPEECH TRACKING PREDICTS AUDIOVISUAL SPEECH-IN-NOISE BENEFIT FROM NATURAL AND SIMPLIFIED VISUAL CUES

Visual speech cues improve speech-in-noise comprehension, but it remains unclear which visual features are necessary and how they shape audiovisual (AV) integration. Prior work has tested both naturalistic faces and highly envelope-driven visualisations, with the latter yielding mixed results. We aimed to determine how structural detail in visual signals modulates behavioural AV benefit and its neural correlates.
Two experiments were conducted. In a behavioural study, participants performed a speech-in-noise comprehension task across conditions varying in visual detail. In an EEG experiment, the participants viewed audiovisual narratives from a single speaker, and neural envelope tracking was quantified using temporal response functions.
Behaviourally, AV benefit scaled with visual fidelity: natural faces produced strongest improvements, degraded faces yielded weaker benefits, while geometric animations produced none. Neurally, natural AV speech elicited a delta-band enhancement of auditory envelope tracking at ~180ms, with similar trends for degraded faces. This enhancement correlated strongly with individual behavioural AV gains (r=0.52, p<0.001).
Facial structural detail is critical for AV speech benefit: simplified faces provide measurable gains, whereas abstract envelope-driven animations do not. Neurally, AV integration manifested as an increase in auditory envelope tracking and a reduction in visual envelope tracking. The results suggest a role of AV speech integration in enhancing auditory cortical encoding of word-rate dynamics. The findings could provide direction for the design of visual hearing aids or expressive virtual avatars.
Varano, E., Thornton, M., Kolossa, D., Zeiler, S., Reichenbach, T. (2025). Delta-band cortical speech tracking predicts audiovisual speech-in-noise benefit from natural and simplified visual cues. Neuroimage.