ePoster

GPT-4 can recognize Theory of Mind in natural conversations: fMRI evidence

Camilla Di Pasquasio, Marc Cavazza, Thierry Chaminade
FENS Forum 2024(2024)
Messe Wien Exhibition & Congress Center, Vienna, Austria

Conference

FENS Forum 2024

Messe Wien Exhibition & Congress Center, Vienna, Austria

Resources

Authors & Affiliations

Camilla Di Pasquasio, Marc Cavazza, Thierry Chaminade

Abstract

The success of Large Language Models (LLM) has led to controversial hypotheses about emergent intelligence beyond language competence, amongst which Theory of Mind (ToM) has become the most debated. While some reported gpt-4 to pass the Sally-Anne test, others reported limitations. Here we aim to validate LLM ability to recognise ToM occurrences in natural human dialogue using the activity of ToM-associated brain regions.The current analysis uses an existing corpus of 25 participants recorded with functional MRI while carrying conversations in 4 sessions x 6 trials alternating a Human and a Robot interlocutor. The conversation is considered natural as participants are provided with a cover story hiding the actual objective of the experiment to study social interactions, and is known to elicit ToM phenomena.We analysed the resulting 100 hours of transcribed conversations synchronised with fMRI recordings by a bespoke gpt-4-turbo prompt. Each conversation turn from the participant and its interlocutors were tagged as containing (ToM+) or not (ToM-) references to mental states by the LLM. Speakers (2) and ToM category (2) defined the four conditions used in a general linear model analysis of whole-brain response.Contrast between conversation turns tagged ToM+ versus ToM- by gpt-4 revealed significant activations (p<0.1 FWE-corrected at the cluster-level ) in the left TemporoParietal Junction, Medial PreFrontal Cortex and lateral OrbitoFrontal Cortex, aligning with the implementation of mentalizing mechanisms in the human brain. Our contribution differs from previous work in the ecological nature of the task, which contrasts with studies based on dedicated ToM tests.

Unique ID: fens-24/gpt-4-recognize-theory-mind-natural-conversations-08694a45