Skip to contentSkip to filters

We use cookies for analytics.

When more isn't better: A Lesson from Rethinking Scale in Single-cell Foundation Models

Lorin Crawford

Microsoft Research New England

Recording

45 min

Abstract

The success of transformer-based foundation models on natural language and images has motivated their use in single-cell transcriptomics. This talk assesses how pre-training dataset size and diversity affect the performance of single-cell foundation models. Using a corpus of 22.2 million cells, 400 models were pre-trained and more than 6,400 experiments evaluated. Current methods tend to plateau in performance with pre-training datasets that are only a fraction of the full size, challenging the assumption that ever-larger datasets are required for optimal generalization.

The second half of the talk evaluates how training-data composition affects model performance. Focusing on human hematopoiesis, deep generative models were trained and analyzed with a variety of training datasets, including cells from adult and developing tissues, disease states and perturbation atlases. Deep generative models generalize poorly to unseen cell types, and adding malignant or perturbed cells to healthy corpora does not consistently improve the modeling of novel states. These findings highlight the nuanced roles of dataset size and heterogeneity and suggest that strategic curation, rather than indiscriminate scaling, is key to optimizing single-cell foundation models.

Topics