multimodal data
Latest
Multimodal computational models for early prediction of peritoneal recurrence in gastric cancer
ABSTRACT Gastric cancer represents a significant disease burden and is a leading cause of cancer-related deaths in the United States and globally. Approximately 80% of gastric cancer patients are diagnosed at an advanced stage, with the peritoneum being the most common site of relapse (peritoneal recurrence) after radical surgery. Nearly 50% of patients with advanced-stage gastric cancer develop peritoneal recurrence post-surgery, resulting in a median survival of only 3–6 months and a markedly reduced quality of life. Early peritoneal recurrence is primarily characterized by micro-metastasis, which traditional imaging techniques struggle to detect due to the small size of metastatic nodules. Predicting the likelihood and timing of peritoneal recurrence is crucial for identifying at- risk patients, enabling timely interventions that could improve survival rates and quality of life. Unfortunately, reliable predictive biomarkers and models for peritoneal recurrence in gastric cancer are lacking in clinical practice, highlighting an urgent need for innovative predictive tools. This proposal aims to develop and validate novel predictive models for early peritoneal recurrence in gastric cancer, leveraging advanced deep learning techniques and multimodal integration of clinical, radiological (CT), and histopathological (hematoxylin and eosin, H&E) data. In Aim 1, we will develop a rational approach for predicting peritoneal recurrence by creating a novel deep learning multimodal method guided by genomics knowledge. Additionally, we will integrate both deep learning-extracted features and traditional hand-crafted radiomics features with clinical data to improve prediction accuracy. Aim 2 focuses on developing a robust prediction model of peritoneal recurrence utilizing a pre-trained foundation model from large-scale H&E image data. Aim 3 will combine CT, H&E, and clinical data to further enhance predictive capabilities, employing an innovative cross-modal collaborative optimization approach for multimodal data integration. All models will be trained and internally validated using a retrospective cohort from Atrium Health Wake Forest Baptist Comprehensive Cancer Center and externally validated in two independent cohorts from additional institutions to ensure robustness across populations and imaging protocols. Additionally, we will compare our models with existing methods, including clinical staging and alternative fusion strategies. If successful, these models will enhance risk stratification and prediction of peritoneal recurrence in gastric cancer patients, significantly improving survival rates and quality of life by identifying those likely to develop peritoneal recurrence post-surgery and facilitating timely intervention. Furthermore, they can help avoid the risk of complications and extra medical costs associated with overtreatment. Since the information is derived from routinely examined CT, H&E and clinical data, they could be seamlessly integrated into current clinical workflows. The AI technology developed through this project has the potential to benefit underserved populations in low- resource settings and reduce healthcare disparities in the U.S.
Harnessing Big Data in Neuroscience: From Mapping Brain Connectivity to Predicting Traumatic Brain Injury
Neuroscience is experiencing unprecedented growth in dataset size both within individual brains and across populations. Large-scale, multimodal datasets are transforming our understanding of brain structure and function, creating opportunities to address previously unexplored questions. However, managing this increasing data volume requires new training and technology approaches. Modern data technologies are reshaping neuroscience by enabling researchers to tackle complex questions within a Ph.D. or postdoctoral timeframe. I will discuss cloud-based platforms such as brainlife.io, that provide scalable, reproducible, and accessible computational infrastructure. Modern data technology can democratize neuroscience, accelerate discovery and foster scientific transparency and collaboration. Concrete examples will illustrate how these technologies can be applied to mapping brain connectivity, studying human learning and development, and developing predictive models for traumatic brain injury (TBI). By integrating cloud computing and scalable data-sharing frameworks, neuroscience can become more impactful, inclusive, and data-driven..
Decision and Behavior
This webinar addressed computational perspectives on how animals and humans make decisions, spanning normative, descriptive, and mechanistic models. Sam Gershman (Harvard) presented a capacity-limited reinforcement learning framework in which policies are compressed under an information bottleneck constraint. This approach predicts pervasive perseveration, stimulus‐independent “default” actions, and trade-offs between complexity and reward. Such policy compression reconciles observed action stochasticity and response time patterns with an optimal balance between learning capacity and performance. Jonathan Pillow (Princeton) discussed flexible descriptive models for tracking time-varying policies in animals. He introduced dynamic Generalized Linear Models (Sidetrack) and hidden Markov models (GLM-HMMs) that capture day-to-day and trial-to-trial fluctuations in choice behavior, including abrupt switches between “engaged” and “disengaged” states. These models provide new insights into how animals’ strategies evolve under learning. Finally, Kenji Doya (OIST) highlighted the importance of unifying reinforcement learning with Bayesian inference, exploring how cortical-basal ganglia networks might implement model-based and model-free strategies. He also described Japan’s Brain/MINDS 2.0 and Digital Brain initiatives, aiming to integrate multimodal data and computational principles into cohesive “digital brains.”
Driving human visual cortex, visually and electrically
The development of circuit-based therapeutics to treat neurological and neuropsychiatric diseases require detailed localization and understanding of electrophysiological signals in the human brain. Electrodes can record and stimulate circuits in many ways, and we often rely on non-invasive imaging methods to predict the location to implant electrodes. However, electrophysiological and imaging signals measure the underlying tissue in a fundamentally different manner. To integrate multimodal data and benefit from these complementary measurements, I will describe an approach that considers how different measurements integrate signals across the underlying tissue. I will show how this approach helps relate fMRI and intracranial EEG measurements and provides new insights into how electrical stimulation influences human brain networks.
AI-guided solutions for early detection of neurodegenerative disorders
Despite the importance of early diagnosis of dementia for prognosis and personalised interventions, we still lack robust tools for predicting individual progression to dementia. We propose a trajectory modelling approach that mines multimodal data from patients at early dementia stages to derive individualised prognostic scores of cognitive decline Our approach has potential to facilitate effective stratification of individuals based on prognostic disease trajectories, reducing patient misclassification with important implications for clinical practice.
multimodal data coverage
5 items
Add content
Have a seminar, talk, or paper on multimodal data? Post it so others working in this area can find it.
Post content