45 min
Seminar, Hosted by Institute for Pure and Applied Mathematics (IPAM), UCLA
When more isn't better: A Lesson from Rethinking Scale in Single-cell Foundation Models
Microsoft Research New England
Recording
Abstract
The success of transformer-based foundation models on natural language and images has motivated their use in single-cell transcriptomics. This talk assesses how pre-training dataset size and diversity affect the performance of single-cell foundation models. Using a corpus of 22.2 million cells, 400 models were pre-trained and more than 6,400 experiments evaluated. Current methods tend to plateau in performance with pre-training datasets that are only a fraction of the full size, challenging the assumption that ever-larger datasets are required for optimal generalization.
The second half of the talk evaluates how training-data composition affects model performance. Focusing on human hematopoiesis, deep generative models were trained and analyzed with a variety of training datasets, including cells from adult and developing tissues, disease states and perturbation atlases. Deep generative models generalize poorly to unseen cell types, and adding malignant or perturbed cells to healthy corpora does not consistently improve the modeling of novel states. These findings highlight the nuanced roles of dataset size and heterogeneity and suggest that strategic curation, rather than indiscriminate scaling, is key to optimizing single-cell foundation models.
Topics
- single-cell transcriptomics
- foundation models
- Hematopoiesis, search World Wide
- single-cell foundation models, broader topic: foundation models
- transformer-based foundation models, broader topic: foundation models
- pre-training dataset size, search World Wide
- generalization
- deep generative models
- human hematopoiesis, search World Wide
- unseen cell types, broader topic: cell types
- training-data composition, search World Wide