Semantics seminars

This webinar convened researchers at the intersection of Artificial Intelligence and Neuroscience to investigate how large language models (LLMs) can serve as valuable “model organisms” for understanding human language processing. Presenters showcased evidence that brain recordings (fMRI, MEG, ECoG) acquired while participants read or listened to unconstrained speech can be predicted by representations extracted from state-of-the-art text- and speech-based LLMs. In particular, text-based LLMs tend to align better with higher-level language regions, capturing more semantic aspects, while speech-based LLMs excel at explaining early auditory cortical responses. However, purely low-level features can drive part of these alignments, complicating interpretations. New methods, including perturbation analyses, highlight which linguistic variables matter for each cortical area and time scale. Further, “brain tuning” of LLMs—fine-tuning on measured neural signals—can improve semantic representations and downstream language tasks. Despite open questions about interpretability and exact neural mechanisms, these results demonstrate that LLMs provide a promising framework for probing the computations underlying human language comprehension and production at multiple spatiotemporal scales.

Non-human great apes inform one another in ways that can seem very humanlike. Especially in the gestural domain, their behavior exhibits many similarities with human communication, meeting widely used empirical criteria for intentionality. At the same time, there remain some manifest differences. How to account for these similarities and differences in a unified way remains a major challenge. This presentation will summarise the arguments developed in a recent paper with Christophe Heintz. We make a key distinction between the expression of intentions (Ladyginian) and the expression of specifically informative intentions (Gricean), and we situate this distinction within a ‘special case of’ framework for classifying different modes of attention manipulation. The paper also argues that the attested tendencies of great ape interaction—for instance, to be dyadic rather than triadic, to be about the here-and-now rather than ‘displaced’—are products of its Ladyginian but not Gricean character. I will reinterpret video footage of great ape gesture as Ladyginian but not Gricean, and distinguish several varieties of meaning that are continuous with one another. We conclude that the evolutionary origins of linguistic meaning lie in gradual changes in not communication systems as such, but rather in social cognition, and specifically in what modes of attention manipulation are enabled by a species’ cognitive phenotype: first Ladyginian and in turn Gricean. The second of these shifts rendered humans, and only humans, ‘language ready’.

The ability to build, store, and manipulate semantic representations lies at the core of all our (inter)actions. Combining evidence from cognitive neuroimaging and experimental neuropsychology, I study the neurocognitive correlates of semantic knowledge in relation to other cognitive functions, chiefly language. In this talk, I will start by reviewing neuroimaging findings supporting the idea that semantic representations are encoded in distributed yet specialized cortical areas (1), and rapidly recovered (2) according to the requirement of the task at hand (3). I will then focus on studies conducted in neurodegenerative patients, offering a unique window on the key role played by a structurally and functionally heterogeneous piece of cortex: the anterior temporal lobe (4,5). I will present pathological, neuroimaging, cognitive, and behavioral data illustrating how damages to language-related networks can affect or spare semantic knowledge as well as possible paths to functional compensation (6,7). Time permitting, we will discuss the neurocognitive dissociation between nouns and verbs (8) and how verb production is differentially impacted by specific language impairments (9).

Metaphor is a pervasive phenomenon in language and cognition. To date, the vast majority of psycholinguistic research on metaphor has focused on noun-noun metaphors of the form An X is a Y (e.g., My job is a jail). Yet there is evidence that verb metaphor (e.g., I sailed through my exams) is more common. Despite this, comparatively little work has examined how verb metaphors are processed. In this talk, I will propose a novel account for verb metaphor comprehension: verb metaphors are understood in the same way that analogies are—as comparisons processed via structure-mapping. I will discuss the predictions that arise from applying the analogical framework to verb metaphor and present a series of experiments showing that verb metaphoric extension is consistent with those predictions.

In this talk, I’ll consider how new ideas emerge from old ones via the process of conceptual blending. I’ll start by considering analogical reasoning in problem solving and the role conceptual blending plays in these problem-solving contexts. Then I’ll consider blending in multi-modal contexts, including timelines, memes (viz. image macros), and, if time allows, zoom meetings. I suggest mappings analogy researchers have traditionally considered superficial are often important for the development of novel abstractions. Likewise, the analogue portion of multimodal blends anchors their generative capacity. Overall, these observations underscore the extent to which meaning is a socially distributed process whose intermediate products are stored in cognitive artifacts such as text and digital images.

We will explore how neuropsychology can be leveraged to directly test cognitive neuroscience theories using the case of frontotemporal dementias affecting the language network. Specifically, we will focus on pathological, neuroimaging, and cognitive data from primary progressive aphasia. We will see how they can help us investigate the reading network, semantic knowledge organisation, and grammatical categories processing. Time permitting, the end of the talk will cover the temporal dynamics of semantic dimensions recovery and the role played by the task.

What do people do when they comprehend language in discourse? According to many psychologists, they build and maintain cognitive representations of utterances in four complementary mental models for discourse that interact with each other: the surface text, the text base, the situation model, and the context model. When people encounter metaphors in these utterances, they need to incorporate them into each of these mental representations for the discourse. Since influential metaphor theories define metaphor as a form of (figurative) analogy, involving cross-domain mapping of a smaller or greater extent, the general expectation has been that metaphor comprehension is also based on analogizing. This expectation, however, has been partly borne out by the data, but not completely. There is no one-to-one relationship between metaphor as (conceptual) structure (analogy) and metaphor as (psychological) process (analogizing). According to Deliberate Metaphor Theory (DMT), only some metaphors are handled by analogy. Instead, most metaphors are presumably handled by lexical disambiguation. This is a hypothesis that brings together most metaphor research in a provocatively new way: it means that most metaphors are not processed metaphorically, which produces a paradox of metaphor. In this talk I will sketch out how this paradox arises and how it can be resolved by a new version of DMT, which I have described in my forthcoming book Slowing metaphor down: Updating Deliberate Metaphor Theory (currently under review). In this theory, the distinction between, but also the relation between, analogy in metaphorical structure versus analogy in metaphorical process is of central importance.

Prior studies of A:B::C:D verbal analogies have identified several factors that affect performance, including the semantic similarity between source and target domains (semantic distance), the semantic association between the C-term and incorrect answers (distracter salience), and the type of relations between word pairs (e.g., categorical, compositional, and causal). However, it is unclear how these stimulus properties affect performance when utilized together. Moreover, how do these item factors interact with individual differences such as crystallized intelligence and creative thinking? Several studies reveal interactions among these item and individual difference factors impacting verbal analogy performance. For example, a three-way interaction demonstrated that the effects of semantic distance and distracter salience had a greater impact on performance for compositional and causal relations than for categorical ones (Jones, Kmiecik, Irwin, & Morrison, 2022). Implications for analogy theories and future directions are discussed.

Children need inference in order to learn the meanings of words. They must infer the referent from the situation in which a target word is said. Furthermore, to be able to use the word in other situations, they also need to infer what other referents the word can be generalized to. As verbs refer to relations between arguments, verb learning requires relational analogical inference, something which is challenging to young children. To overcome this difficulty, young children recruit a diverse range of cues in their inference of verb meanings, including, but not limited to, syntactic cues and social and pragmatic cues as well as statistical cues. They also utilize perceptual similarity (object similarity) in progressive alignment to extract relational verb meanings and further to gain insights about relational verb meanings. However, just having a list of these cues is not useful: the cues must be selected, combined, and coordinated to produce the optimal interpretation in a particular context. This process involves abductive reasoning, similar to what scientists do to form hypotheses from a range of facts or evidence. In this talk, I discuss how children use a chain of inferences to learn meanings of verbs. I consider not only the process of analogical mapping and progressive alignment, but also how children use abductive inference to find the source of analogy and gain insights into the general principles underlying verb learning. I also present recent findings from my laboratory that show that prelinguistic human infants use a rudimentary form of abductive reasoning, which enables the first step of word learning.

It is well established that the opportunity to compare learning stimuli in a novel word learning/extension task elicits a larger number of conceptually relevant generalizations than standard no-comparison conditions. I will present results suggesting that the effectiveness of comparison depends on factors such as semantic distance, number of training items, dimension distinctiveness and interactions with age. I will address these issues in the case of familiar and unfamiliar object nouns and relational nouns. The alignment strategies followed by children during learning and at test (i.e., when learning items are compared and how children reach a solution) will be described with eye-tracking data. We will also assess the extent to which children’s performance in these tasks are associated with executive functions (inhibition and flexibility) and world knowledge. Finally, we will consider these issues in children with cognitive deficits (Intellectual deficiency, DLD)

The tight relationship between attention and conscious perception has been extensively researched in the past decades. However, whether attentional modulation extended to unconscious processes remained largely unknown, particularly when it came to abstract and high-level processing. I will talk about a recent study where we utilized the Stroop paradigm to show that task load gates unconscious semantic processing. In a series of psychophysical experiments, the unconscious word semantics influenced conscious task performance only under the low task load condition, but not the high task load condition. Intriguingly, with enough practice in the high task load condition, the unconscious effect reemerged. These findings suggest a competition of attentional resources between unconscious and conscious processes, challenging the automaticity account of unconscious processing.

In this talk, I will present the results of two recent systematic reviews and meta-analyses related to analogical reasoning and metaphor processing in autism, together with the results of a study that investigated verbal analogical reasoning and metaphor processing in the same sample of participants. Both metaphors and analogies rely on exploiting similarities, and they necessitate contextual processing. Nevertheless, our findings relating to metaphor processing and analogical reasoning showed distinct patterns. Whereas analogical reasoning emerged as a relative strength in autism, metaphor processing was found to be a relative weakness. Additionally, both meta-analytic studies investigated the relations between the level of intelligence of participants included in the studies, and the effect size of group differences between the autistic and typically developing (TD) samples. These analyses suggested in the case of analogical reasoning that the relative advantage of ASD participants might only be present in the case of individuals with lower levels of intelligence. By contrast, impairments in metaphor processing appeared to be more pronounced in the case of individuals with relatively lower levels of (verbal) intelligence. In our experimental study, we administered both verbal analogies and metaphors to the same sample of high-functioning autistic participants and TD controls. The two groups were matched on age, verbal IQ, working memory and educational background. Our aim was to understand better the similarities and differences between processing analogies and metaphors, and to see whether the advantage in analogical reasoning and disadvantage in metaphor processing is universal in autism.

Theoretical linguistics postulates abstract structures that successfully explain key aspects of language. However, the precise relation between abstract theoretical ideas and empirical data from language use is not always apparent. Here, we propose to empirically test abstract semantic theories through the lens of probabilistic pragmatic modelling. We consider the historically important case of quantity words (e.g., `some', `all'). Data from a large-scale production study seem to suggest that quantity words are understood via prototypes. But based on statistical and empirical model comparison, we show that a probabilistic pragmatic model that embeds a strict truth-conditional notion of meaning explains the data just as well as a model that encodes prototypes into the meaning of quantity words.

Previous work suggests that children’s ability to understand metaphors emerges late in development. Researchers argue that children’s initial failure to understand metaphors is due to an inability to reason about shared relational structures between concepts. However, recent work demonstrates that preschoolers, toddlers, and even infants are already capable of relational reasoning. Might preschoolers also be capable of understanding metaphors, given more sensitive experimental paradigms? I explore whether preschoolers (N = 200, ages 4-5) understand functional metaphors, namely metaphors based on functional similarities. In Experiment 1a, preschoolers rated functional metaphors (e.g. “Roofs are hats”; “Clouds are sponges”) as “smarter” than nonsense statements. In Experiment 1b, adults (N = 48) also rated functional metaphors as “smarter” than nonsense statements (e.g. “Dogs are scissors”; “Boats are skirts”). In Experiment 2, preschoolers preferred functional explanations (e.g. “Both hold water”) over perceptual explanations (e.g. “Both are fluffy”) when interpreting a functional metaphor (e.g. “Clouds are sponges”). In Experiment 3, preschoolers preferred functional metaphors over nonsense statements in a dichotomous-choice task. Overall, this work demonstrates preschoolers’ early-emerging ability to understand functional metaphors.

The evolution of speech is considered to be one of the hardest problems in science. Studies of the communicative abilities of our closest living relatives, the nonhuman primates, aim to contribute to a better understanding of the emergence of this uniquely human capability. Following a brief introduction over the key building blocks that make up the human speech faculty, I will focus on the question of meaning in nonhuman primate vocalizations. While nonhuman primate calls may be highly context specific, thus giving rise to the notion of ‘referentiality’, comparisons across closely related species suggest that this specificity is evolved rather than learned. Yet, as in humans, the structure of calls varies with arousal and affective state, and there is some evidence for effects of sensory-motor integration in vocal production. Thus, the vocal production of nonhuman primates bears little resemblance to the symbolic and combinatorial features of human speech, while basic production mechanisms are shared. Listeners, in contrast, are able learning the meaning of new sounds. A recent study using artificial predator shows that this learning may be extremely rapid. Furthermore, listeners are able to integrate information from multiple sources to make adaptive decisions, which renders the vocal communication system as a whole relatively flexible and powerful. In conclusion, constraints at the side of vocal production, including limits in social cognition and motivation to share experiences, rather than constraints at the side of the recipient explain the differences in communicative abilities between humans and other animals.

Abstract semantic relations (e.g., category membership, part-whole, antonymy, cause-effect) are central to human intelligence, underlying the distinctively human ability to reason by analogy. I will describe a computational project (Bayesian Analogy with Relational Transformations) that aims to extract explicit representations of abstract semantic relations from non-relational inputs automatically generated by machine learning. BART’s representations predict patterns of typicality and similarity for semantic relations, as well as similarity of neural signals triggered by semantic relations during analogical reasoning. In this approach, analogy emerges from the ability to learn and compare relations; mapping emerges later from the ability to compare patterns of relations.

In this talk, I will present some data showing that people’s similarity judgments among word pairs reflect distinctions between abstract semantic relations like contrast, cause-effect, or part-whole. Further, the extent that individual participants’ similarity judgments discriminate between abstract semantic relations was linearly associated with both fluid and crystallized verbal intelligence, albeit more strongly with fluid intelligence. Finally, I will compare three models according to their ability to predict these similarity judgments. All models take as input vector representations of individual word meanings, but they differ in their representation of relations: one model does not represent relations at all, a second model represents relations implicitly, and a third model represents relations explicitly. Across the three models, the third model served as the best predictor of human similarity judgments suggesting the importance of explicit relation representation to fully account for human semantic cognition.