Multimodal
multimodal
From Spiking Predictive Coding to Learning Abstract Object Representation
In a first part of the talk, I will present Predictive Coding Light (PCL), a novel unsupervised learning architecture for spiking neural networks. In contrast to conventional predictive coding approaches, which only transmit prediction errors to higher processing stages, PCL learns inhibitory lateral and top-down connectivity to suppress the most predictable spikes and passes a compressed representation of the input to higher processing stages. We show that PCL reproduces a range of biological findings and exhibits a favorable tradeoff between energy consumption and downstream classification performance on challenging benchmarks. A second part of the talk will feature our lab’s efforts to explain how infants and toddlers might learn abstract object representations without supervision. I will present deep learning models that exploit the temporal and multimodal structure of their sensory inputs to learn representations of individual objects, object categories, or abstract super-categories such as „kitchen object“ in a fully unsupervised fashion. These models offer a parsimonious account of how abstract semantic knowledge may be rooted in children's embodied first-person experiences.
Harnessing Big Data in Neuroscience: From Mapping Brain Connectivity to Predicting Traumatic Brain Injury
Neuroscience is experiencing unprecedented growth in dataset size both within individual brains and across populations. Large-scale, multimodal datasets are transforming our understanding of brain structure and function, creating opportunities to address previously unexplored questions. However, managing this increasing data volume requires new training and technology approaches. Modern data technologies are reshaping neuroscience by enabling researchers to tackle complex questions within a Ph.D. or postdoctoral timeframe. I will discuss cloud-based platforms such as brainlife.io, that provide scalable, reproducible, and accessible computational infrastructure. Modern data technology can democratize neuroscience, accelerate discovery and foster scientific transparency and collaboration. Concrete examples will illustrate how these technologies can be applied to mapping brain connectivity, studying human learning and development, and developing predictive models for traumatic brain injury (TBI). By integrating cloud computing and scalable data-sharing frameworks, neuroscience can become more impactful, inclusive, and data-driven..
Structural & Functional Neuroplasticity in Children with Hemiplegia
About 30% of children with cerebral palsy have congenital hemiplegia, resulting from periventricular white matter injury, which impairs the use of one hand and disrupts bimanual co-ordination. Congenital hemiplegia has a profound effect on each child's life and, thus, is of great importance to the public health. Changes in brain organization (neuroplasticity) often occur following periventricular white matter injury. These changes vary widely depending on the timing, location, and extent of the injury, as well as the functional system involved. Currently, we have limited knowledge of neuroplasticity in children with congenital hemiplegia. As a result, we provide rehabilitation treatment to these children almost blindly based exclusively on behavioral data. In this talk, I will present recent research evidence of my team on understanding neuroplasticity in children with congenital hemiplegia by using a multimodal neuroimaging approach that combines data from structural and functional neuroimaging methods. I will further present preliminary data regarding functional improvements of upper extremities motor and sensory functions as a result of rehabilitation with a robotic system that involves active participation of the child in a video-game setup. Our research is essential for the development of novel or improved neurological rehabilitation strategies for children with congenital hemiplegia.
Neural architectures: what are they good for anyway?
The brain has a highly complex structure in terms of cell types and wiring between different regions. What is it for, if anything? I'll start this talk by asking what might an answer to this question even look like given that we can't run an alternative universe where our brains are structured differently. (Preview: we can do this with models!) I'll then talk about some of our work in two areas: (1) does the modular structure of the brain contribute to specialisation of function? (2) how do different cell types and architectures contribute to multimodal sensory processing?
“Open Raman Microscopy (ORM): A modular Raman spectroscopy setup with an open-source controller”
Raman spectroscopy is a powerful technique for identifying chemical species by probing their vibrational energy levels, offering exceptional specificity with a relatively simple setup involving a laser source, spectrometer, and microscope/probe. However, the high cost of Raman systems lacking modularity often limits exploratory research hindering broader adoption. To address the need for an affordable, modular microscopy platform for multimodal imaging, we present a customizable confocal Raman spectroscopy setup alongside an open-source acquisition software, ORM (Open Raman Microscopy) Controller, developed in Python. This solution bridges the gap between expensive commercial systems and complex, custom-built setups used by specialist research groups. In this presentation, we will cover the components of the setup, the design rationale, assembly methods, limitations, and its modular potential for expanding functionality. Additionally, we will demonstrate ORM’s capabilities for instrument control, 2D and 3D Raman mapping, region-of-interest selection, and its adaptability to various instrument configurations. We will conclude by showcasing practical applications of this setup across different research fields.
Decision and Behavior
This webinar addressed computational perspectives on how animals and humans make decisions, spanning normative, descriptive, and mechanistic models. Sam Gershman (Harvard) presented a capacity-limited reinforcement learning framework in which policies are compressed under an information bottleneck constraint. This approach predicts pervasive perseveration, stimulus‐independent “default” actions, and trade-offs between complexity and reward. Such policy compression reconciles observed action stochasticity and response time patterns with an optimal balance between learning capacity and performance. Jonathan Pillow (Princeton) discussed flexible descriptive models for tracking time-varying policies in animals. He introduced dynamic Generalized Linear Models (Sidetrack) and hidden Markov models (GLM-HMMs) that capture day-to-day and trial-to-trial fluctuations in choice behavior, including abrupt switches between “engaged” and “disengaged” states. These models provide new insights into how animals’ strategies evolve under learning. Finally, Kenji Doya (OIST) highlighted the importance of unifying reinforcement learning with Bayesian inference, exploring how cortical-basal ganglia networks might implement model-based and model-free strategies. He also described Japan’s Brain/MINDS 2.0 and Digital Brain initiatives, aiming to integrate multimodal data and computational principles into cohesive “digital brains.”
Mitochondrial diversity in the mouse and human brain
The basis of the mind, of mental states, and complex behaviors is the flow of energy through microscopic and macroscopic brain structures. Energy flow through brain circuits is powered by thousands of mitochondria populating the inside of every neuron, glial, and other nucleated cell across the brain-body unit. This seminar will cover emerging approaches to study the mind-mitochondria connection and present early attempts to map the distribution and diversity of mitochondria across brain tissue. In rodents, I will present convergent multimodal evidence anchored in enzyme activities, gene expression, and animal behavior that distinct behaviorally-relevant mitochondrial phenotypes exist across large-scale mouse brain networks. Extending these findings to the human brain, I will present a developing systematic biochemical and molecular map of mitochondrial variation across cortical and subcortical brain structures, representing a foundation to understand the origin of complex energy patterns that give rise to the human mind.
Learning with multimodal enrichment
Prosody in the voice, face, and hands changes which words you hear
Speech may be characterized as conveying both segmental information (i.e., about vowels and consonants) as well as suprasegmental information - cued through pitch, intensity, and duration - also known as the prosody of speech. In this contribution, I will argue that prosody shapes low-level speech perception, changing which speech sounds we hear. Perhaps the most notable example of how prosody guides word recognition is the phenomenon of lexical stress, whereby suprasegmental F0, intensity, and duration cues can distinguish otherwise segmentally identical words, such as "PLAto" vs. "plaTEAU" in Dutch. Work from our group showcases the vast variability in how different talkers produce stressed vs. unstressed syllables, while also unveiling the remarkable flexibility with which listeners can learn to handle this between-talker variability. It also emphasizes that lexical stress is a multimodal linguistic phenomenon, with the voice, lips, and even hands conveying stress in concert. In turn, human listeners actively weigh these multisensory cues to stress depending on the listening conditions at hand. Finally, lexical stress is presented as having a robust and lasting impact on low-level speech perception, even down to changing vowel perception. Thus, prosody - in all its multisensory forms - is a potent factor in speech perception, determining what speech sounds we hear.
Why is 7T MRI indispensable in epilepsy now?
Identifying a structural brain lesion on MRI is the most important factor that correlates with seizure freedom after surgery in patients suffering from drug-resistant focal epilepsy. By providing better image contrast and higher spatial resolution, structural MRI at 7 Tesla (7T) can lead to lesion detection in about 25% of patients presenting with negative MRI at lower fields. In addition to a better detection/delineation/phenotyping of epileptogenic lesions, higher signal at ultra-high field also facilitates more detailed analyses of several functional and molecular alterations of tissues, susceptible to detect epileptogenic properties even in absence of visible lesions. These advantages but also the technical challenges of 7T MRI in practice will be presented and discussed.
Multimodal Blending
In this talk, I’ll consider how new ideas emerge from old ones via the process of conceptual blending. I’ll start by considering analogical reasoning in problem solving and the role conceptual blending plays in these problem-solving contexts. Then I’ll consider blending in multi-modal contexts, including timelines, memes (viz. image macros), and, if time allows, zoom meetings. I suggest mappings analogy researchers have traditionally considered superficial are often important for the development of novel abstractions. Likewise, the analogue portion of multimodal blends anchors their generative capacity. Overall, these observations underscore the extent to which meaning is a socially distributed process whose intermediate products are stored in cognitive artifacts such as text and digital images.
Direction-selective ganglion cells in primate retina: a subcortical substrate for reflexive gaze stabilization?
To maintain a stable and clear image of the world, our eyes reflexively follow the direction in which a visual scene is moving. Such gaze stabilization mechanisms reduce image blur as we move in the environment. In non-primate mammals, this behavior is initiated by ON-type direction-selective ganglion cells (ON-DSGCs), which detect the direction of image motion and transmit signals to brainstem nuclei that drive compensatory eye movements. However, ON-DSGCs have not yet been functionally identified in primates, raising the possibility that the visual inputs that drive this behavior instead arise in the cortex. In this talk, I will present molecular, morphological and functional evidence for identification of an ON-DSGC in macaque retina. The presence of ON-DSGCs highlights the need to examine the contribution of subcortical retinal mechanisms to normal and aberrant gaze stabilization in the developing and mature visual system. More generally, our findings demonstrate the power of a multimodal approach to study sparsely represented primate RGC types.
A multimodal perspective on learning to read
Neural circuits for vector processing in the insect brain
Several species of insects have been observed to perform accurate path integration, constantly updating a vector memory of their location relative to a starting position, which they can use to take a direct return path. Foraging insects such as bees and ants are also able to store and recall the vectors to return to food locations, and to take novel shortcuts between these locations. Other insects, such as dung beetles, are observed to integrate multimodal directional cues in a manner well described by vector addition. All these processes appear to be functions of the Central Complex, a highly conserved and strongly structured circuit in the insect brain. Modelling this circuit, at the single neuron level, suggests it has general capabilities for vector encoding, vector memory, vector addition and vector rotation that can support a wide range of directed and navigational behaviours.
Driving human visual cortex, visually and electrically
The development of circuit-based therapeutics to treat neurological and neuropsychiatric diseases require detailed localization and understanding of electrophysiological signals in the human brain. Electrodes can record and stimulate circuits in many ways, and we often rely on non-invasive imaging methods to predict the location to implant electrodes. However, electrophysiological and imaging signals measure the underlying tissue in a fundamentally different manner. To integrate multimodal data and benefit from these complementary measurements, I will describe an approach that considers how different measurements integrate signals across the underlying tissue. I will show how this approach helps relate fMRI and intracranial EEG measurements and provides new insights into how electrical stimulation influences human brain networks.
The multimodal number sense: spanning space, time, sensory modality, and action
Humans and other animals can estimate rapidly the number of items in a scene, flashes or tones in a sequence and motor actions. Adaptation techniques provide clear evidence in humans for the existence of specialized numerosity mechanisms that make up the numbersense. This sense of number is truly general, encoding the numerosity of both spatial arrays and sequential sets, in vision and audition, and interacting strongly with action. The adaptation (cross-sensory and cross-format) acts on sensory mechanisms rather than decisional processes, pointing to a truly general sense.
Pynapple: a light-weight python package for neural data analysis - webinar + tutorial
In systems neuroscience, datasets are multimodal and include data-streams of various origins: multichannel electrophysiology, 1- or 2-p calcium imaging, behavior, etc. Often, the exact nature of data streams are unique to each lab, if not each project. Analyzing these datasets in an efficient and open way is crucial for collaboration and reproducibility. In this combined webinar and tutorial, Adrien Peyrache and Guillaume Viejo will present Pynapple, a Python-based data analysis pipeline for systems neuroscience. Designed for flexibility and versatility, Pynapple allows users to perform cross-modal neural data analysis via a common programming approach which facilitates easy sharing of both analysis code and data.
Pynapple: a light-weight python package for neural data analysis - webinar + tutorial
In systems neuroscience, datasets are multimodal and include data-streams of various origins: multichannel electrophysiology, 1- or 2-p calcium imaging, behavior, etc. Often, the exact nature of data streams are unique to each lab, if not each project. Analyzing these datasets in an efficient and open way is crucial for collaboration and reproducibility. In this combined webinar and tutorial, Adrien Peyrache and Guillaume Viejo will present Pynapple, a Python-based data analysis pipeline for systems neuroscience. Designed for flexibility and versatility, Pynapple allows users to perform cross-modal neural data analysis via a common programming approach which facilitates easy sharing of both analysis code and data.
Multimodal tracking of motor activity, sleep and mood
This talk will (1) describe patterns and correlates of objectively assessed motor activity (2) present findings on the inter-relationships among motor activity, sleep and circadian rhythms and mood disorders; (3) describe potential of cross species studies of motor activity and related systems to inform human chronobiology research
Multimodal investigation of the associations between sleep and Alzheimer's disease neuropathology in healthy individuals
Alterations in sleep are hallmarks of the ageing process and emerges as risk factors for Alzheimer’s disease (AD). While the fine-tuned coalescence of sleep microstructure elements may influence age-related cognitive trajectories, its association with AD-related processes is not fully established. We investigated whether sleep arousals and the coupling of spindles and slow waves, key elements of sleep microstructure, are associated with early amyloid-beta (Aβ) brain burden, hallmark of AD neuropathology, and cognitive change at 2 years in 100 late-midlife healthy individuals. We first found that arousals interrupting sleep continuity were positively linked to Aβ burden, while, by contrast, the more prevalent arousals upholding sleep continuity were associated with lower Aβ burden and better cognition. We further found that young-like co-occurrence of spindles and slow-depolarisation slow waves is associated to lower burden of Aβ over the medial prefrontal cortex and is predictive of memory decline at 2-year follow-up. We provide empirical evidence that arousals are diverse and differently associated with early AD-related neuropathology and cognition. We further show the altered coupling of sleep microstructure elements that are key to its mnesic functions may contribute to poorer brain and cognitive trajectories. The presentation will end with preliminary data show that activity of the locus coeruleus, essential to sleep and showing some of the earliest signs of AD-related pathological processes, is associated with sleep quality. These preliminary findings are the first of a project ailed at link sleep and AD through the locus coeruleus.
Multi-modal biomarkers improve prediction of memory function in cognitively unimpaired older adults
Identifying biomarkers that predict current and future cognition may improve estimates of Alzheimer’s disease risk among cognitively unimpaired older adults (CU). In vivo measures of amyloid and tau protein burden and task-based functional MRI measures of core memory mechanisms, such as the strength of cortical reinstatement during remembering, have each been linked to individual differences in memory in CU. This study assesses whether combining CSF biomarkers with fMRI indices of cortical reinstatement improves estimation of memory function in CU, assayed using three unique tests of hippocampal-dependent memory. Participants were 158 CU (90F, aged 60-88 years, CDR=0) enrolled in the Stanford Aging and Memory Study (SAMS). Cortical reinstatement was quantified using multivoxel pattern analysis of fMRI data collected during completion of a paired associate cued recall task. Memory was assayed by associative cued recall, a delayed recall composite, and a mnemonic discrimination task that involved discrimination between studied ‘target’ objects, novel ‘foil’ objects, and perceptually similar ‘lure’ objects. CSF Aβ42, Aβ40, and p-tau181 were measured with the automated Lumipulse G system (N=115). Regression analyses examined cross-sectional relationships between memory performance in each task and a) the strength of cortical reinstatement in the Default Network (comprised of posterior medial, medial frontal, and lateral parietal regions) during associative cued recall and b) CSF Aβ42/Aβ40 and p-tau181, controlling for age, sex, and education. For mnemonic discrimination, linear mixed effects models were used to examine the relationship between discrimination (d’) and each predictor as a function of target-lure similarity. Stronger cortical reinstatement was associated with better performance across all three memory assays. Age and higher CSF p-tau181 were each associated with poorer associative memory and a diminished improvement in mnemonic discrimination as target-lure similarity decreased. When combined in a single model, CSF p-tau181 and Default Network reinstatement strength, but not age, explained unique variance in associative memory and mnemonic discrimination performance, outperforming the single-modality models. Combining fMRI measures of core memory functions with protein biomarkers of Alzheimer’s disease significantly improved prediction of individual differences in memory performance in CU. Leveraging multimodal biomarkers may enhance future prediction of risk for cognitive decline.
Multimodal imaging in Dementia with Lewy bodies
Dementia with Lewy bodies (DLB) is a synucleinopathy but more than half of patients with DLB also have varying degrees of tau and amyloid-β co-pathology. Identifying and tracking the pathologic heterogeneity of DLB with multi-modal biomarkers is critical for the design of clinical trials that target each pathology early in the disease at a time when prevention or delaying the transition to dementia is possible. Furthermore, longitudinal evaluation of multi-modal biomarkers contributes to our understanding of the type and extent of the pathologic progression and serves to characterize the temporal emergence of the associated phenotypic expression. This talk will focus on the utility of multi-modal imaging in DLB.
Multimodal framework and fusion of EEG, graph theory and sentiment analysis for the prediction and interpretation of consumer decision
The application of neuroimaging methods to marketing has recently gained lots of attention. In analyzing consumer behaviors, the inclusion of neuroimaging tools and methods is improving our understanding of consumer’s preferences. Human emotions play a significant role in decision making and critical thinking. Emotion classification using EEG data and machine learning techniques has been on the rise in the recent past. We evaluate different feature extraction techniques, feature selection techniques and propose the optimal set of features and electrodes for emotion recognition.Affective neuroscience research can help in detecting emotions when a consumer responds to an advertisement. Successful emotional elicitation is a verification of the effectiveness of an advertisement. EEG provides a cost effective alternative to measure advertisement effectiveness while eliminating several drawbacks of the existing market research tools which depend on self-reporting. We used Graph theoretical principles to differentiate brain connectivity graphs when a consumer likes a logo versus a consumer disliking a logo. The fusion of EEG and sentiment analysis can be a real game changer and this combination has the power and potential to provide innovative tools for market research.
The vestibular system: a multimodal sense
The vestibular system plays an essential role in everyday life, contributing to a surprising range of functions from reflexes to the highest levels of perception and consciousness. Three orthogonal semicircular canals detect rotational movements of the head and the otolith organs sense translational acceleration, including the gravitational vertical. But, how vestibular signals are encoded by the human brain? We have recently combined innovative methods for eliciting virtual rotation and translation sensations with fMRI to identify brain areas representing vestibular signals. We have identified a bilateral inferior parietal, ventral premotor/anterior insula and prefrontal network and confirmed that these areas reliably possess information about the rotation and translation. We have also investigated how vestibular signals are integrated with other sensory cues to generate our perception of the external environment.
NMC4 Short Talk: The complete connectome of an insect brain
Brains must integrate complex sensory information and compare to past events to generate appropriate behavioral responses. The neural circuit basis of these computations is unclear and the underlying structure unknown. Here, we mapped the comprehensive synaptic wiring diagram of the fruit fly larva brain, which contains 3,013 neurons and 544K synaptic sites. It is the most complete insect connectome to date: 1) Both brain hemispheres are reconstructed, allowing investigation of neural pathways that include contralateral axons, which we found in 37% of brain neurons. 2) All sensory neurons and descending neurons are reconstructed, allowing one to follow signals in an uninterrupted chain—from the sensory periphery, through the brain, to motor neurons in the nerve cord. We developed novel computational tools, allowing us to cluster the brain and investigate how information flows through it. We discovered that feedforward pathways from sensory to descending neurons are multilayered and highly multimodal. Robust feedback was observed at almost all levels of the brain, including descending neurons. We investigated how the brain hemispheres communicate with each other and the nerve cord, leading to identification of novel circuit motifs. This work provides the complete blueprint of a brain and a strong foundation to study the structure-function relationship of neural circuits.
NMC4 Short Talk: Image embeddings informed by natural language improve predictions and understanding of human higher-level visual cortex
To better understand human scene understanding, we extracted features from images using CLIP, a neural network model of visual concept trained with supervision from natural language. We then constructed voxelwise encoding models to explain whole brain responses arising from viewing natural images from the Natural Scenes Dataset (NSD) - a large-scale fMRI dataset collected at 7T. Our results reveal that CLIP, as compared to convolution based image classification models such as ResNet or AlexNet, as well as language models such as BERT, gives rise to representations that enable better prediction performance - up to a 0.86 correlation with test data and an r-square of 0.75 - in higher-level visual cortex in humans. Moreover, CLIP representations explain distinctly unique variance in these higher-level visual areas as compared to models trained with only images or text. Control experiments show that the improvement in prediction observed with CLIP is not due to architectural differences (transformer vs. convolution) or to the encoding of image captions per se (vs. single object labels). Together our results indicate that CLIP and, more generally, multimodal models trained jointly on images and text, may serve as better candidate models of representation in human higher-level visual cortex. The bridge between language and vision provided by jointly trained models such as CLIP also opens up new and more semantically-rich ways of interpreting the visual brain.
Neural mechanisms of altered states of consciousness under psychedelics
Interest in psychedelic compounds is growing due to their remarkable potential for understanding altered neural states and their breakthrough status to treat various psychiatric disorders. However, there are major knowledge gaps regarding how psychedelics affect the brain. The Computational Neuroscience Laboratory at the Turner Institute for Brain and Mental Health, Monash University, uses multimodal neuroimaging to test hypotheses of the brain’s functional reorganisation under psychedelics, informed by the accounts of hierarchical predictive processing, using dynamic causal modelling (DCM). DCM is a generative modelling technique which allows to infer the directed connectivity among brain regions using functional brain imaging measurements. In this webinar, Associate Professor Adeel Razi and PhD candidate Devon Stoliker will showcase a series of previous and new findings of how changes to synaptic mechanisms, under the control of serotonin receptors, across the brain hierarchy influence sensory and associative brain connectivity. Understanding these neural mechanisms of subjective and therapeutic effects of psychedelics is critical for rational development of novel treatments and for the design and success of future clinical trials. Associate Professor Adeel Razi is a NHMRC Investigator Fellow and CIFAR Azrieli Global Scholar at the Turner Institute of Brain and Mental Health, Monash University. He performs cross-disciplinary research combining engineering, physics, and machine-learning. Devon Stoliker is a PhD candidate at the Turner Institute for Brain and Mental Health, Monash University. His interest in consciousness and psychiatry has led him to investigate the neural mechanisms of classic psychedelic effects in the brain.
Neural circuits that support robust and flexible navigation in dynamic naturalistic environments
Tracking heading within an environment is a fundamental requirement for flexible, goal-directed navigation. In insects, a head-direction representation that guides the animal’s movements is maintained in a conserved brain region called the central complex. Two-photon calcium imaging of genetically targeted neural populations in the central complex of tethered fruit flies behaving in virtual reality (VR) environments has shown that the head-direction representation is updated based on self-motion cues and external sensory information, such as visual features and wind direction. Thus far, the head direction representation has mainly been studied in VR settings that only give flies control of the angular rotation of simple sensory cues. How the fly’s head direction circuitry enables the animal to navigate in dynamic, immersive and naturalistic environments is largely unexplored. I have developed a novel setup that permits imaging in complex VR environments that also accommodate flies’ translational movements. I have previously demonstrated that flies perform visually-guided navigation in such an immersive VR setting, and also that they learn to associate aversive optogenetically-generated heat stimuli with specific visual landmarks. A stable head direction representation is likely necessary to support such behaviors, but the underlying neural mechanisms are unclear. Based on a connectomic analysis of the central complex, I identified likely circuit mechanisms for prioritizing and combining different sensory cues to generate a stable head direction representation in complex, multimodal environments. I am now testing these predictions using calcium imaging in genetically targeted cell types in flies performing 2D navigation in immersive VR.
Spatio-temporal large-scale organization of the trimodal connectome derived from concurrent EEG-fMRI and diffusion MRI
While time-averaged dynamics of brain functional connectivity are known to reflect the underlying structural connections, the exact relationship between large-scale function and structure remains an unsolved issue in network neuroscience. Large-scale networks are traditionally observed by correlation of fMRI timecourses, and connectivity of source-reconstructed electrophysiological measures are less prominent. Accessing the brain by using multimodal recordings combining EEG, fMRI and diffusion MRI (dMRI) can help to refine the understanding of the spatio-temporal organization of both static and dynamic brain connectivity. In this talk I will discuss our prior findings that whole-brain connectivity derived from source-reconstructed resting-state (rs) EEG is both linked to the rs-fMRI and dMRI connectome. The EEG connectome provides complimentary information to link function to structure as compared to an fMRI-only perspective. I will present an approach extending the multimodal data integration of concurrent rs-EEG-fMRI to the temporal domain by combining dynamic functional connectivity of both modalities to better understand the neural basis of functional connectivity dynamics. The close relationship between time-varying changes in EEG and fMRI whole-brain connectivity patterns provide evidence for spontaneous reconfigurations of the brain’s functional processing architecture. Finally, I will talk about data quality of connectivity derived from concurrent EEG-fMRI recordings and how the presented multimodal framework could be applied to better understand focal epilepsy. In summary this talk will give an overview of how to integrate large-scale EEG networks with MRI-derived brain structure and function. In conclusion EEG-based connectivity measures not only are closely linked to MRI-based measures of brain structure and function over different time-scales, but also provides complimentary information on the function of underlying brain organization.
AI-guided solutions for early detection of neurodegenerative disorders
Despite the importance of early diagnosis of dementia for prognosis and personalised interventions, we still lack robust tools for predicting individual progression to dementia. We propose a trajectory modelling approach that mines multimodal data from patients at early dementia stages to derive individualised prognostic scores of cognitive decline Our approach has potential to facilitate effective stratification of individuals based on prognostic disease trajectories, reducing patient misclassification with important implications for clinical practice.
From 1D to 5D: Data-driven Discovery of Whole-brain Dynamic Connectivity in fMRI Data
The analysis of functional magnetic resonance imaging (fMRI) data can greatly benefit from flexible analytic approaches. In particular, the advent of data-driven approaches to identify whole-brain time-varying connectivity and activity has revealed a number of interesting relevant variation in the data which, when ignored, can provide misleading information. In this lecture I will provide a comparative introduction of a range of data-driven approaches to estimating time-varying connectivity. I will also present detailed examples where studies of both brain health and disorder have been advanced by approaches designed to capture and estimate time-varying information in resting fMRI data. I will review several exemplar data sets analyzed in different ways to demonstrate the complementarity as well as trade-offs of various modeling approaches to answer questions about brain function. Finally, I will review and provide examples of strategies for validating time-varying connectivity including simulations, multimodal imaging, and comparative prediction within clinical populations, among others. As part of the interactive aspect I will provide a hands-on guide to the dynamic functional network connectivity toolbox within the GIFT software, including an online didactic analytic decision tree to introduce the various concepts and decisions that need to be made when using such tools
Life of Pain and Pleasure
The ability to experience pain is old in evolutionary terms. It is an experience shared across species. Acute pain is the body’s alarm system, and as such it is a good thing. Pain that persists beyond normal tissue healing time (3-4 months) is defined as chronic – it is the system gone wrong and it is not a good thing. Chronic pain has recently been classified as both a symptom and disease in its own right. It is one of the largest medical health problems worldwide with one in five adults diagnosed with the condition. The brain is key to the experience of pain and pain relief. This is the place where pain emerges as a perception. So, relating specific brain measures using advanced neuroimaging to the change patients describe in their pain perception induced by peripheral or central sensitization (i.e. amplification), psychological or pharmacological mechanisms has tremendous value. Identifying where amplification or attenuation processes occur along the journey from injury to the brain (i.e. peripheral nerves, spinal cord, brainstem and brain) for an individual and relating these neural mechanisms to specific pain experiences, measures of pain relief, persistence of pain states, degree of injury and the subject's underlying genetics, has neuroscientific and potential diagnostic relevance. This is what neuroimaging has afforded – a better understanding and explanation of why someone’s pain is the way it is. We can go ‘behind the scenes’ of the subjective report to find out what key changes and mechanisms make up an individual’s particular pain experience. A key area of development has been pharmacological imaging where objective evidence of drugs reaching the target and working can be obtained. We even now understand the mechanisms of placebo analgesia – a powerful phenomenon known about for millennia. More recently, researchers have been investigating through brain imaging whether there is a pre-disposing vulnerability in brain networks towards developing chronic pain. So, advanced neuroimaging studies can powerfully aid explanation of a subject’s multidimensional pain experience, pain relief (analgesia) and even what makes them vulnerable to developing chronic pain. The application of this goes beyond the clinic and has relevance in courts of law, and other areas of society, such as in veterinary care. Relatively far less work has been directed at understanding what changes in the brain occur during altered states of consciousness induced either endogenously (e.g. sleep) or exogenously (e.g. anaesthesia). However, that situation is changing rapidly. Our recent multimodal neuroimaging work explores how anaesthetic agents produce altered states of consciousness such that perceptual experiences of pain and awareness are degraded. This is bringing us fascinating insights into the complex phenomenon of anaesthesia, consciousness and even the concept of self-hood. These topics will be discussed in my talk alongside my ‘side-story’ of life as a scientist combining academic leadership roles with doing science and raising a family.
Mapping early brain network changes in neurodegenerative and cerebrovascular disorders: a longitudinal perspective
The spatial patterning of each neurodegenerative disease relates closely to a distinct structural and functional network in the human brain. This talk will mainly describe how brain network-sensitive neuroimaging methods such as resting-state fMRI and diffusion MRI can shed light on brain network dysfunctions associated with pathology and cognitive decline from preclinical to clinical dementia. I will first present our findings from two independent datasets on how amyloid and cerebrovascular pathology influence brain functional networks cross-sectionally and longitudinally in individuals with mild cognitive impairment and dementia. Evidence on longitudinal functional network organizational changes in healthy older adults and the influence of APOE genotype will be presented. In the second part, I will describe our work on how different pathology influences brain structural network and white matter microstructure. I will also touch on some new data on how brain network integrity contributes to behavior and disease progression using multivariate or machine learning approaches. These findings underscore the importance of studying selective brain network vulnerability instead of individual region and longitudinal design. Further developed with machine learning approaches, multimodal network-specific imaging signatures will help reveal disease mechanisms and facilitate early detection, prognosis and treatment search of neuropsychiatric disorders.
The Logic of Depth Cue Combination for Multimodal 3D Perception
Defining new multimodal neuroimaging marker for grey matter characterization
The human cortical ribbon varies during the lifespan, from childhood to senescence. To study the effects of genetic and environmental factors on these dynamics, one needs to measure specific phenotypes (cortical volume, surface area, thickness, new neuroimaging phenotypes such as intracortical myelination or multimodal ones based on their combination, or their asymmetries) that characterize the cerebral grey matter accurately
Multimodal brain imaging to predict progression of Alzheimer’s disease
Cross-sectional and longitudinal multimodal brain imaging studies using positron emission tomography (PET) and magnetic resonance imaging (MRI) have provided detailed insight into the pathophysiological progression of Alzheimer’s disease. It starts at an asymptomatic stage with widespread gradual accumulation of beta-amyloid and spread of pathological tau deposits. Subsequently changes of functional connectivity and glucose metabolism associated with mild cognitive impairment and brain atrophy may develop. However, the rate of progression to a symptomatic stage and ultimately dementia varies considerably between individuals. Mathematical models have been developed to describe disease progression, which may be used to identify markers that determine the current stage and likely rate of progression. Both are very important to improve the efficacy of clinical trials. In this lecture, I will provide an overview on current research and future perspectives in this area.
Learning Neurobiology with electric fish
Electric Gymnotiform fish live in muddy, shallow waters near the shore – hiding in the dense filamentous roots of floating plants such as Eichornia crassipes (“camalote”). They explore their surroundings by using a series of electric pulses that serve as self emitted carrier of electrosensory signals. This propagates at the speed of light through this spongiform habitat and is barely sensed by the lateral line of predators and prey. The emitted field polarizes the surroundings according to the difference in impedance with water which in turn modifies the profile of transcutaneous currents considered as an electrosensory image. Using this system, pulse Gymnotiformes create an electrosensory bubble where an object’s location, impedance, size and other characteristics are discriminated and probably recognized. Although consciousness is still not well-proven, cognitive functions as volition, attention, and path integration have been shown. Here I will summarize different aspects of the electromotor electrosensory loop of pulse Gymnotiforms. First, I will address how objects are polarized with a stereotyped but temporospatially complex electric field, consisting of brief pulses emitted at regular intervals. This relies on complex electric organs quasi periodically activated through an electromotor coordination system by a pacemaker in the medulla. Second, I will deal with the imaging mechanisms of pulse gymnotiform fish and the presence of two regions in the electrosensory field, a rostral region where the field time course is coherent and field vector direction is constant all along the electric organ discharge and a lateral region where the field time course is site specific and field vector direction describes a stereotyped 3D trajectory. Third, I will describe the electrosensory mosaic and their characteristics. Receptor and primary afferents correspond one to one showing subtypes optimally responding to the time course of the self generated pulse with a characteristic train of spikes. While polarized objects at the rostral region project their electric images on the perioral region where electrosensory receptor density, subtypes and central projection are maximal, the image of objects on the side recruit a single type of scattered receptors. Therefore, the rostral mosaic has been likened to an electrosensory fovea and its receptive field referred to as foveal field. The rest of the mosaic and field are referred to as peripheral. Finally, I will describe ongoing work on early processing structures. I will try to generate an integrated view, including anatomical and functional data obtained in vitro, acute experiments, and unitary recordings in freely moving fish. We have recently shown have shown that these fish tract allo-generated fields and the virtual fields generated by nearby objects in the presence of self-generated fields to explore the nearby environment. These data together with the presence of a multimodal receptor mosaic at the cutaneous surface particularly surrounding the mouth and an important role of proprioception in early sensory processing suggests the hypothesis that the active electrosensory system is part of a multimodal haptic sense.
Emergent scientists discuss Alzheimer's disease
This seminar is part of our “Emergent Scientists” series, an initiative that provides a platform for scientists at the critical PhD/postdoc transition period to share their work with a broad audience and network. Summary: These talks cover Alzheimer’s disease (AD) research in both mice and humans. Christiana will discuss in particular the translational aspects of applying mouse work to humans and the importance of timing in disease pathology and intervention (e.g. timing between AD biomarkers vs. symptom onset, timing of therapy, etc.). Siddharth will discuss a rare variant of Alzheimer’s disease called “Logopenic Progressive Aphasia”, which presents with temporo-parietal atrophy yet relative sparing of hippocampal circuitry. Siddharth will discuss how, despite the unusual anatomical basis underlying this AD variant, degeneration of the angular gyrus in the left inferior parietal lobule contributes to memory deficits similar to those of typical amnesic Alzheimer’s disease. Christiana’s abstract: Alzheimer’s disease (AD) is a debilitating neurodegenerative disorder that causes severe deterioration of memory, cognition, behavior, and the ability to perform daily activities. The disease is characterized by the accumulation of two proteins in fibrillar form; Amyloid-β forms fibrils that accumulate as extracellular plaques while tau fibrils form intracellular tangles. Here we aim to translate findings from a commonly used AD mouse model to AD patients. Here we initiate and chronically inhibit neuropathology in lateral entorhinal cortex (LEC) layer two neurons in an AD mouse model. This is achieved by over-expressing P301L tau virally and chronically activating hM4Di DREADDs intracranially using the ligand dechloroclozapine. Biomarkers in cerebrospinal fluid (CSF) is measured longitudinally in the model using microdialysis, and we use this same system to intracranially administer drugs aimed at halting AD-related neuropathology. The models are additionally tested in a novel contextual memory task. Preliminary findings indicate that viral injections of P301L tau into LEC layer two reveal direct projections between this region and the outer molecular layer of dentate gyrus and the rest of hippocampus. Additionally, phosphorylated tau co-localize with ‘starter cells’ and appear to spread from the injection site. Preliminary microdialysis results suggest that the concentrations of CSF amyloid-β and tau proteins mirror changes observed along the disease cascade in patients. The disease-modifying drugs appear to halt neuropathological development in this preclincial model. These findings will lead to a novel platform for translational AD research, linking the extensive research done in rodents to clinical applications. Siddharth’s abstract: A distributed brain network supports our ability to remember past events. The parietal cortex is a critical member of this network, yet, its exact contributions to episodic remembering remain unclear. Neurodegenerative syndromes affecting the posterior neocortex offer a unique opportunity to understand the importance and role of parietal regions to episodic memory. In this talk, I introduce and explore the rare neurodegenerative syndrome of Logopenic Progressive Aphasia (LPA), an aphasic variant of Alzheimer’s disease presenting with early, left-lateralized temporo-parietal atrophy, amidst relatively spared hippocampal integrity. I then discuss two key studies from my recent Ph.D. work showcasing pervasive episodic and autobiographical memory dysfunction in LPA, to a level comparable to typical, amnesic Alzheimer’s disease. Using multimodal neuroimaging, I demonstrate how degeneration of the angular gyrus in the left inferior parietal lobule, and its structural connections to the hippocampus, contribute to amnesic profiles in this syndrome. I finally evaluate these findings in the context of memory profiles in other posterior cortical neurodegenerative syndromes as well as recent theoretical models underscoring the importance of the parietal cortex in the integration and representation of episodic contextual information.
Carnosine negatively modulates pro-oxidant activities of M1 peripheral macrophages and prevents neuroinflammation induced by amyloid-β in microglial cells
Carnosine is a natural dipeptide widely distributed in mammalian tissues and exists at particularly high concentrations in skeletal and cardiac muscles and brain. A growing body of evidence shows that carnosine is involved in many cellular defense mechanisms against oxidative stress, including inhibition of amyloid-β (Aβ) aggregation, modulation of nitric oxide (NO) metabolism, and scavenging both reactive nitrogen and oxygen species. Different types of cells are involved in the innate immune response, with macrophage cells representing those primarily activated, especially under different diseases characterized by oxidative stress and systemic inflammation such as depression and cardiovascular disorders. Microglia, the tissue-resident macrophages of the brain, are emerging as a central player in regulating key pathways in central nervous system inflammation; with specific regard to Alzheimer’s disease (AD) these cells exert a dual role: on one hand promoting the clearance of Aβ via phagocytosis, on the other hand increasing neuroinflammation through the secretion of inflammatory mediators and free radicals. The activity of carnosine was tested in an in vitro model of macrophage activation (M1) (RAW 264.7 cells stimulated with LPS + IFN-γ) and in a well-validated model of Aβ-induced neuroinflammation (BV-2 microglia treated with Aβ oligomers). An ample set of techniques/assays including MTT assay, trypan blue exclusion test, high performance liquid chromatography, high-throughput real-time PCR, western blot, atomic force microscopy, microchip electrophoresis coupled to laser-induced fluorescence, and ELISA aimed to evaluate the antioxidant and anti-inflammatory activities of carnosine was employed. In our experimental model of macrophage activation (M1), therapeutic concentrations of carnosine exerted the following effects: 1) an increased degradation rate of NO into its non-toxic end-products nitrite and nitrate; 2) the amelioration of the macrophage energy state, by restoring nucleoside triphosphates and counterbalancing the changes in ATP/ADP, NAD+/NADH and NADP+/NADPH ratio obtained by LPS + IFN-γ induction; 3) a reduced expression of pro-oxidant enzymes (NADPH oxidase, Cyclooxygenase-2) and of the lipid peroxidation product malondialdehyde; 4) the rescue of antioxidant enzymes expression (Glutathione peroxidase 1, Superoxide dismutase 2, Catalase); 5) an increased synthesis of transforming growth factor-β1 (TGF-β1) combined with the negative modulation of interleukines 1β and 6 (IL-1β and IL-6), and 6) the induction of nuclear factor erythroid-derived 2-like 2 (Nrf2) and heme oxygenase-1 (HO-1). In our experimental model of Aβ-induced neuroinflammation, carnosine: 1) prevented cell death in BV-2 cells challenged with Aβ oligomers; 2) lowered oxidative stress by decreasing the expression of inducible nitric oxide synthase and NADPH oxidase, and the concentrations of nitric oxide and superoxide anion; 3) decreased the secretion of pro-inflammatory cytokines such as IL-1β simultaneously rescuing IL-10 levels and increasing the expression and the release of TGF-β1; 4) prevented Aβ-induced neurodegeneration in primary mixed neuronal cultures challenged with Aβ oligomers and these neuroprotective effects was completely abolished by SB431542, a selective inhibitor of type-1 TGF-β receptor. Overall, our data suggest a novel multimodal mechanism of action of carnosine underlying its protective effects in macrophages and microglia and the therapeutic potential of this dipeptide in counteracting pro-oxidant and pro-inflammatory phenomena observed in different disorders characterized by elevated levels of oxidative stress and inflammation such as depression, cardiovascular disorders, and Alzheimer’s disease.
Fluoxetine and vortioxetine reverse depressive-like phenotype and memory deficits induced by amyloid-β (1-42) oligomers in mice: implication of transforming growth factor-β1 and oxidative stress
A long-term treatment with antidepressants reduces the risk to develop AD and different second-generation antidepressants such as selective serotonin reuptake inhibitors (SSRIs) are currently studied for their neuroprotective properties in AD. An impairment of neurotrophic factors signaling seems to be a common pathophysiological event in depression and AD. In particular a deficit of transforming growth factor-β1 (TGF-β1) and increased oxidative stress have been found both in depression and AD. In the present work the SSRI fluoxetine and the new multimodal antidepressant vortioxetine were tested for their ability to prevent memory deficits and depressive-like phenotype in a non-transgenic mouse model of AD (i.c.v. Aβ1-42 injection) by rescue of TGF-β1 signaling. The same drugs were also tested for their ability to modulate the expression of pro-oxidant genes as well as of genes related to the antioxidant machinery.
Neuroimaging in human drug addiction: an eye towards intervention development
Drug addiction is a chronically relapsing disorder characterized by compulsive drug use despite catastrophic personal consequences (e.g., loss of family, job) and even when the substance is no longer perceived as pleasurable. In this talk, I will present results of human neuroimaging studies, utilizing a multimodal approach (neuropsychology, functional magnetic resonance imaging, event-related potentials recordings), to explore the neurobiology underlying the core psychological impairments in drug addiction (impulsivity, drive/motivation, insight/awareness) as associated with its clinical symptomatology (intoxication, craving, bingeing, withdrawal). The focus of this talk is on understanding the role of the dopaminergic mesocorticolimbic circuit, and especially the prefrontal cortex, in higher-order executive dysfunction (e.g., disadvantageous decision-making such as trading a car for a couple of cocaine hits) in drug addicted individuals. The theoretical model that guides the presented research is called iRISA (Impaired Response Inhibition and Salience Attribution), postulating that abnormalities in the orbitofrontal cortex and anterior cingulate cortex, as related to dopaminergic dysfunction, contribute to the core clinical symptoms in drug addiction. Specifically, our multi-modality program of research is guided by the underlying working hypothesis that drug addicted individuals disproportionately attribute reward value to their drug of choice at the expense of other potentially but no-longer-rewarding stimuli, with a concomitant decrease in the ability to inhibit maladaptive drug use. In this talk I will also explore whether treatment (as usual) and 6-month abstinence enhance recovery in these brain-behavior compromises in treatment seeking cocaine addicted individuals. Promising neuroimaging studies, which combine pharmacological (i.e., oral methylphenidate, or RitalinTM) and salient cognitive tasks or functional connectivity during resting-state, will be discussed as examples for using neuroimaging for empirically guiding the development of effective neurorehabilitation strategies (encompassing cognitive reappraisal and transcranial direct current stimulation) in drug addiction.
An Attention-based Multimodal Decoder for Hybrid Brain-Computer Interface Control Systems
Bernstein Conference 2024
Multimodal cues displayed by submissive rats facilitate prosocial choices by dominants
COSYNE 2022
Multimodal cues displayed by submissive rats facilitate prosocial choices by dominants
COSYNE 2022
Optimal Multimodal Integration Supports Course Control Under Uncertainty in Walking Drosophila
COSYNE 2022
Optimal Multimodal Integration Supports Course Control Under Uncertainty in Walking Drosophila
COSYNE 2022
Neuroformer: A Transformer Framework for Multimodal Neural Data Analysis
COSYNE 2023
Jointly modeling neural activity and behavior via multimodal masked modeling
COSYNE 2025
Multimodal Ising-based connectomics reveals an excitation-inhibition imbalance in Alzheimer's Risk
COSYNE 2025
Nonlinear Dynamical Modeling of Behavior and Multimodal Neural Data
COSYNE 2025
In vivo cell-type and brain region classification via multimodal contrastive learning
COSYNE 2025
Bayesian perceptual adaptation in auditory motion perception: A multimodal approach with EEG and pupillometry
FENS Forum 2024
Effects of multimodal sensory integration in D. melanogaster optokinetic response
FENS Forum 2024
Emotion regulation across dimensions of emotional response: A multimodal comparison of emotion regulation strategies
FENS Forum 2024
Exploring the combinatorial, diagnostic utility of multimodal biomarkers in differentially diagnosing Dementia with Lewy Bodies from Alzheimer’s through predictive statistical modelling
FENS Forum 2024
Exploring the Neurological Mechanisms of Phantom Limb Pain with High-Density EEG: A Multimodal Analysis
FENS Forum 2024
HBK-10, a multimodal compound, selectively blocks 5-HT1A receptor-mediated β-arrestin recruitment and mitigates memory deficits in mice
FENS Forum 2024
HBK-15, a multimodal compound, mitigates cognitive flexibility deficits in mice
FENS Forum 2024
Multimodal activity of mouse auditory cortex during audio-visual-motor virtual reality
FENS Forum 2024
A multimodal approach to investigate human brain mesoscale circuits with cellular and subcellular resolution
FENS Forum 2024
Multimodal sensory cue based novelty detection in CA1
FENS Forum 2024
Multimodal signal complexity in somatosensory cortex revealed through high-dimensional representations
FENS Forum 2024
Multiscale brain dynamic patterns change with glutamate modulation in obsessive-compulsive disorder: A comprehensive multimodal brain mapping approach
FENS Forum 2024
The neurophysiological mechanisms of impaired manual dexterity in Parkinson's disease: A multimodal study using PET/CT, EEG, and BDNF
FENS Forum 2024
A novel MRI-compatible restrain setup for awake rat multimodal experiments
FENS Forum 2024
Portable and turn-key multimodal multiphoton microscopy for easy-to-access label-free and intravital imaging
FENS Forum 2024
Rapid formation of new visual concepts in human early visual cortex assessed with multimodal MRI
FENS Forum 2024
Rescuing efficacy of cognitive and multimodal stimulation on neuroinflammation in two mouse models of Parkinson’s disease
FENS Forum 2024
Role of ionotropic glutamate receptors in multimodal learning of pheromone locations
FENS Forum 2024
The Tabulae Paralytica: Multimodal single-cell atlas of spinal cord injury
FENS Forum 2024
Transcranial magnetic stimulation neurofeedback – A multimodal, multiphase approach to stroke rehabilitation using EEG BCI
FENS Forum 2024
Year-long multimodal access to neuronal activity in large brains
FENS Forum 2024