Multimodal Data
multimodal data
Jochen Triesch
We solicit applications for a PhD position to develop machine learning techniques for personalized prediction of psychopathology. The position will be part of a large new center aiming to develop a novel dynamic network approach of mental health. This center, the 'LOEWE center DYNAMIC', brings together scientists from a range of disciplines, including psychology, psychiatry, computer science and machine learning, with a shared goal of advancing our understanding of mental disorders and developing new treatment options. The center’s research focuses on the application of dynamic network models at various levels (neurobiological, psychological and psychopathological) to mental disorder research. It brings together researchers from the Universities of Marburg, Giessen, Frankfurt and Darmstadt, as well as the Leibniz Institute for Research and Information in Education DIPF and the Ernst Strüngmann Institute for Neurosciences ESI. The respective university hospitals and the psychotherapy outpatient clinics of the psychological university institutes are also involved, facilitating the rapid transfer of research results into practice. The present opening will be associated with the Department of Computer Science at the University of Frankfurt. The objective of the project is to develop a personalized prediction model for changes in psychopathology (new depressive episodes), behavioral patterns and biological parameters. Many mental illnesses are characterized by changes in the network structure of the brain that affect observable patterns of activity or behavior in the future. Early detection and especially prediction of changes in behavioral parameters, psychopathology and biomarkers could enable targeted, personalized interventions to offer special (additional, more specific) therapies to patients with poor prognosis. The objective of this project is to develop methods for the early and reliable detection and prediction of changes in multimodal data.
Prof. Dr. Yee Lee Shing, Prof. Dr. Gemma Roig
The DFG funded project Learning From Environment Through the Eyes of Children within SPP 2431 New Data Spaces for the Social Sciences, situated at Goethe University Frankfurt, is looking for candidates for two positions: 1 PostDoc position in Psychology and 1 PhD or PostDoc position in Computer Science. The project aims to establish a new mode of data acquisition capturing young children’s first-person experience in naturalistic settings and develop AI systems to characterize the nature and complexity of these experiences. This interdisciplinary project involves collaboration between the psychology and computer science departments, contributing to the SPP programme's goals of establishing a new multimodal data approach in social science studies.
Harnessing Big Data in Neuroscience: From Mapping Brain Connectivity to Predicting Traumatic Brain Injury
Neuroscience is experiencing unprecedented growth in dataset size both within individual brains and across populations. Large-scale, multimodal datasets are transforming our understanding of brain structure and function, creating opportunities to address previously unexplored questions. However, managing this increasing data volume requires new training and technology approaches. Modern data technologies are reshaping neuroscience by enabling researchers to tackle complex questions within a Ph.D. or postdoctoral timeframe. I will discuss cloud-based platforms such as brainlife.io, that provide scalable, reproducible, and accessible computational infrastructure. Modern data technology can democratize neuroscience, accelerate discovery and foster scientific transparency and collaboration. Concrete examples will illustrate how these technologies can be applied to mapping brain connectivity, studying human learning and development, and developing predictive models for traumatic brain injury (TBI). By integrating cloud computing and scalable data-sharing frameworks, neuroscience can become more impactful, inclusive, and data-driven..
Decision and Behavior
This webinar addressed computational perspectives on how animals and humans make decisions, spanning normative, descriptive, and mechanistic models. Sam Gershman (Harvard) presented a capacity-limited reinforcement learning framework in which policies are compressed under an information bottleneck constraint. This approach predicts pervasive perseveration, stimulus‐independent “default” actions, and trade-offs between complexity and reward. Such policy compression reconciles observed action stochasticity and response time patterns with an optimal balance between learning capacity and performance. Jonathan Pillow (Princeton) discussed flexible descriptive models for tracking time-varying policies in animals. He introduced dynamic Generalized Linear Models (Sidetrack) and hidden Markov models (GLM-HMMs) that capture day-to-day and trial-to-trial fluctuations in choice behavior, including abrupt switches between “engaged” and “disengaged” states. These models provide new insights into how animals’ strategies evolve under learning. Finally, Kenji Doya (OIST) highlighted the importance of unifying reinforcement learning with Bayesian inference, exploring how cortical-basal ganglia networks might implement model-based and model-free strategies. He also described Japan’s Brain/MINDS 2.0 and Digital Brain initiatives, aiming to integrate multimodal data and computational principles into cohesive “digital brains.”
Driving human visual cortex, visually and electrically
The development of circuit-based therapeutics to treat neurological and neuropsychiatric diseases require detailed localization and understanding of electrophysiological signals in the human brain. Electrodes can record and stimulate circuits in many ways, and we often rely on non-invasive imaging methods to predict the location to implant electrodes. However, electrophysiological and imaging signals measure the underlying tissue in a fundamentally different manner. To integrate multimodal data and benefit from these complementary measurements, I will describe an approach that considers how different measurements integrate signals across the underlying tissue. I will show how this approach helps relate fMRI and intracranial EEG measurements and provides new insights into how electrical stimulation influences human brain networks.
Spatio-temporal large-scale organization of the trimodal connectome derived from concurrent EEG-fMRI and diffusion MRI
While time-averaged dynamics of brain functional connectivity are known to reflect the underlying structural connections, the exact relationship between large-scale function and structure remains an unsolved issue in network neuroscience. Large-scale networks are traditionally observed by correlation of fMRI timecourses, and connectivity of source-reconstructed electrophysiological measures are less prominent. Accessing the brain by using multimodal recordings combining EEG, fMRI and diffusion MRI (dMRI) can help to refine the understanding of the spatio-temporal organization of both static and dynamic brain connectivity. In this talk I will discuss our prior findings that whole-brain connectivity derived from source-reconstructed resting-state (rs) EEG is both linked to the rs-fMRI and dMRI connectome. The EEG connectome provides complimentary information to link function to structure as compared to an fMRI-only perspective. I will present an approach extending the multimodal data integration of concurrent rs-EEG-fMRI to the temporal domain by combining dynamic functional connectivity of both modalities to better understand the neural basis of functional connectivity dynamics. The close relationship between time-varying changes in EEG and fMRI whole-brain connectivity patterns provide evidence for spontaneous reconfigurations of the brain’s functional processing architecture. Finally, I will talk about data quality of connectivity derived from concurrent EEG-fMRI recordings and how the presented multimodal framework could be applied to better understand focal epilepsy. In summary this talk will give an overview of how to integrate large-scale EEG networks with MRI-derived brain structure and function. In conclusion EEG-based connectivity measures not only are closely linked to MRI-based measures of brain structure and function over different time-scales, but also provides complimentary information on the function of underlying brain organization.
AI-guided solutions for early detection of neurodegenerative disorders
Despite the importance of early diagnosis of dementia for prognosis and personalised interventions, we still lack robust tools for predicting individual progression to dementia. We propose a trajectory modelling approach that mines multimodal data from patients at early dementia stages to derive individualised prognostic scores of cognitive decline Our approach has potential to facilitate effective stratification of individuals based on prognostic disease trajectories, reducing patient misclassification with important implications for clinical practice.