Facial expressions are inherently dynamic, and our visual system is sensitive to subtle changes in their temporal sequence. However, researchers often use dynamic morphs of photographs—simplified, linear representations of motion—to study the neural correlates of dynamic face perception. To explore the brain's sensitivity to natural facial motion, we constructed a novel dynamic face database using generative neural networks, trained on a verified set of video-recorded emotional expressions. The resulting deepfakes, consciously indistinguishable from videos, enabled us to separate biological motion from photorealistic form. Results showed that conventional dynamic morphs elicit distinct responses in the brain compared to videos and photos, suggesting they violate expectations (n400) and have reduced social salience (late positive potential). This suggests that dynamic morphs misrepresent facial dynamism, resulting in misleading insights about the neural and behavioural correlates of face perception. Deepfakes and videos elicited largely similar neural responses, suggesting they could be used as a proxy for real faces in vision research, where video recordings cannot be experimentally manipulated. And yet, despite being consciously undetectable as fake, deepfakes elicited an expectation violation response in the brain. This points to a neural sensitivity to naturalistic facial motion, beyond conscious awareness. Despite some differences in neural responses, the realism and manipulability of deepfakes make them a valuable asset for research where videos are unfeasible. Using these stimuli, we proposed a novel marker for the conscious perception of naturalistic facial motion – Frontal delta activity – which was elevated for videos and deepfakes, but not for photos or dynamic morphs.

Humans and other animals approach reward and avoid punishment and pay attention to cues predicting these events. Such motivated behavior thus appears to be guided by value, which directs behavior towards or away from positively or negatively valenced outcomes. Moreover, it is facilitated by (top-down) salience, which enhances attention to behaviorally relevant learned cues predicting the occurrence of valenced outcomes. Using human neuroimaging, we recently separated value (ventral striatum, posterior ventromedial prefrontal cortex) from salience (anterior ventromedial cortex, occipital cortex) in the domain of liquid reward and punishment. Moreover, we investigated potential drivers of learned salience: the probability and uncertainty with which valenced and non-valenced outcomes occur. We find that the brain dissociates valenced from non-valenced probability and uncertainty, which indicates that reinforcement matters for the brain, in addition to information provided by probability and uncertainty alone, regardless of valence. Finally, we assessed learning signals (unsigned prediction errors) that may underpin the acquisition of salience. Particularly the insula appears to be central for this function, encoding a subjective salience prediction error, similarly at the time of positively and negatively valenced outcomes. However, it appears to employ domain-specific time constants, leading to stronger salience signals in the aversive than the appetitive domain at the time of cues. These findings explain why previous research associated the insula with both valence-independent salience processing and with preferential encoding of the aversive domain. More generally, the distinction of value and salience appears to provide a useful framework for capturing the neural basis of motivated behavior.

Prior studies of A:B::C:D verbal analogies have identified several factors that affect performance, including the semantic similarity between source and target domains (semantic distance), the semantic association between the C-term and incorrect answers (distracter salience), and the type of relations between word pairs (e.g., categorical, compositional, and causal). However, it is unclear how these stimulus properties affect performance when utilized together. Moreover, how do these item factors interact with individual differences such as crystallized intelligence and creative thinking? Several studies reveal interactions among these item and individual difference factors impacting verbal analogy performance. For example, a three-way interaction demonstrated that the effects of semantic distance and distracter salience had a greater impact on performance for compositional and causal relations than for categorical ones (Jones, Kmiecik, Irwin, & Morrison, 2022). Implications for analogy theories and future directions are discussed.

Executive control processes and flexible behaviors rely on the integrity of, and dynamic interactions between, large-scale functional brain networks. The right insular cortex is a critical component of a salience/midcingulo-insular network that is thought to mediate interactions between brain networks involved in externally oriented (central executive/lateral frontoparietal network) and internally oriented (default mode/medial frontoparietal network) processes. How these brain systems reconfigure with development is a critical question for cognitive neuroscience, with implications for neurodevelopmental pathologies affecting brain connectivity. I will describe studies examining how brain network dynamics support flexible behaviors in typical and atypical development, presenting evidence suggesting a unique role for the dorsal anterior insular from studies of meta-analytic connectivity modeling, dynamic functional connectivity, and structural connectivity. These findings from adults, typically developing children, and children with autism suggest that structural and functional maturation of insular pathways is a critical component of the process by which human brain networks mature to support complex, flexible cognitive processes throughout the lifespan.

Detection of motion is essential for survival, but how the visual system processes moving stimuli is not fully understood. Here, based on a detailed analysis of glutamate release from bipolar cells, we outline the rules that govern the representation of object motion in the early processing stages. Our main findings are as follows: (1) Motion processing begins already at the first retinal synapse. (2) The shape and the amplitude of motion responses cannot be reliably predicted from bipolar cell responses to stationary objects. (3) Enhanced representation of novel objects - particularly in bipolar cells with transient dynamics. (4) Response amplitude in bipolar cells matches visual salience reported in humans: suddenly appearing objects > novel motion > existing motion. These findings can be explained by antagonistic interactions in the center-surround receptive field, demonstrate that despite their simple operational concepts, classical center-surround receptive fields enable sophisticated visual computations.

Transient changes in dopamine activity in response to reward and punishment have been known to regulate reward-related learning. However, the cellular basis that detects the transient dopamine signaling has long been unclear. Using two-photon microscopy and optogenetics, I have shown that transient increases and decreases of dopamine modulate plasticity of dopamine D1 and D2 receptor-expressing cells in the nucleus accumbens, respectively. At the behavioral level, I characterized that these D1 and D2 cells cooperatively tune learning by generalization and discrimination learning. Interestingly, disturbance of the dopamine signaling impaired D2 cell plasticity and discrimination learning, which was analogous to salience misattribution seen in subjects with schizophrenia.

Current computational models of visual salience accurately predict the distribution of fixations on isolated visual stimuli. It is not known, however, whether the global salience of a stimulus, that is its effectiveness in the competition for attention with other stimuli, is a function of the local salience or an independent measure. Further, do task and familiarity with the competing images influence eye movements? In this talk, I will present the analysis of a computational model of the global salience of natural images. We trained a machine learning algorithm to learn the direction of the first saccade of participants who freely observed pairs of images. The pairs balanced the combinations of new and already seen images, as well as task and task-free trials. The coefficients of the model provided a reliable measure of the likelihood of each image to attract the first fixation when seen next to another image, that is their global salience. For example, images of close-up faces and images containing humans were consistently looked first and were assigned higher global salience. Interestingly, we found that global salience cannot be explained by the feature-driven local salience of images, the influence of task and familiarity was rather small and we reproduced the previously reported left-sided bias. This computational model of global salience allows to analyse multiple other aspects of human visual perception of competing stimuli. In the talk, I will also present our latest results from analysing the saccadic reaction time as a function of the global salience of the pair of images.

Drug addiction is a chronically relapsing disorder characterized by compulsive drug use despite catastrophic personal consequences (e.g., loss of family, job) and even when the substance is no longer perceived as pleasurable. In this talk, I will present results of human neuroimaging studies, utilizing a multimodal approach (neuropsychology, functional magnetic resonance imaging, event-related potentials recordings), to explore the neurobiology underlying the core psychological impairments in drug addiction (impulsivity, drive/motivation, insight/awareness) as associated with its clinical symptomatology (intoxication, craving, bingeing, withdrawal). The focus of this talk is on understanding the role of the dopaminergic mesocorticolimbic circuit, and especially the prefrontal cortex, in higher-order executive dysfunction (e.g., disadvantageous decision-making such as trading a car for a couple of cocaine hits) in drug addicted individuals. The theoretical model that guides the presented research is called iRISA (Impaired Response Inhibition and Salience Attribution), postulating that abnormalities in the orbitofrontal cortex and anterior cingulate cortex, as related to dopaminergic dysfunction, contribute to the core clinical symptoms in drug addiction. Specifically, our multi-modality program of research is guided by the underlying working hypothesis that drug addicted individuals disproportionately attribute reward value to their drug of choice at the expense of other potentially but no-longer-rewarding stimuli, with a concomitant decrease in the ability to inhibit maladaptive drug use. In this talk I will also explore whether treatment (as usual) and 6-month abstinence enhance recovery in these brain-behavior compromises in treatment seeking cocaine addicted individuals. Promising neuroimaging studies, which combine pharmacological (i.e., oral methylphenidate, or RitalinTM) and salient cognitive tasks or functional connectivity during resting-state, will be discussed as examples for using neuroimaging for empirically guiding the development of effective neurorehabilitation strategies (encompassing cognitive reappraisal and transcranial direct current stimulation) in drug addiction.

Many visual systems can process information in dynamically changing environments. In general, visual perception scales with changes in the visual stimulus, or contrast, irrespective of background illumination. This is achieved by adaptation. However, visual perception is challenged when adaptation is not fast enough to deal with sudden changes in overall illumination, for example when gaze follows a moving object from bright sunlight into a shaded area. We have recently shown that the visual system of the fly found a solution by propagating a corrective luminance-sensitive signal to higher processing stages. Using in vivo two-photon imaging and behavioural analyses we showed that distinct OFF-pathway inputs encode contrast and luminance. The luminance-sensitive pathway is particularly required when processing visual motion in contextual dim light, when pure contrast sensitivity underestimates the salience of a stimulus. Recent work in the lab has addressed the question how two visual pathways obtain such fundamentally different sensitivities, given common photoreceptor input. We are furthermore currently working out the network-based strategies by which luminance- and contrast-sensitive signals are combined to guide appropriate visual behaviour. Together, I will discuss the molecular, cellular, and circuit mechanisms that ensure contrast computation, and therefore robust vision, in fast changing visual scenes.