POSTER DETAILS RELATED POSTERS

POSTER DETAILS

Reconstructing voice from fMRI using deep neural networks

Charly Lamothe, Etienne Thoret, Stéphane Ayache, Régis Trapeau, Bruno L Giordano, Sylvain Takerkart, Thierry Artières, Pascal Belin

Date / Location: Sunday, 10 July 2022 / S02-510
Add this poster event to your 🗓
Recently, neural decoding has been used in conjunction with deep neural networks, leading to important advances in our understanding of higher-level visual cerebral representations, allowing the reconstruction of visual stimuli from fMRI patterns (van Gerven MAJ et al., 2019; Dado et al., 2020). We are interested in establishing a linear relationship between a deep-derived 'Voice latent space' and neural activity measured by fMRI. The goodness of fit in different cerebral regions will be probed via brain-based voice reconstruction. The main idea is to learn a deep auto-encoder to reconstruct spectrograms of human voice, then to learn a linear mapping from the encoding space of the autoencoder to the fMRI space, a strategy assumed to be more relevant for learning such a mapping between fMRI maps and sounds. Then, an fMRI acquisition of test stimuli is mapped to the autoencoder space and subsequently decoded as a spectrogram, enabling stimulus reconstruction. We developed and proposed a simple framework to perform brain voice decoding from fMRI patterns using DNNs. We applied our voice decoding method using different functionally-defined ROIs. We found that the decoded spectrograms only contained human voice when the ROI consisted of the fMRI-defined Temporal Voicer Areas. Ongoing analyses aim to further refine reconstruction quality and systematically test it as a function of anatomical cerebral location. The reconstruction obtained by the linear regression will allow us to better understand the functional representation of voice stimuli in the brain and potential analogies to the DNN-generated latent representations.

POSTERS YOU MIGHT BE INTERESTED IN