Skip to contentSkip to filters

We use cookies for analytics.

Seminar, Hosted by Stanford MedAI

Merlin: A Vision Language Foundation Model for 3D Computed Tomography

Ashwin Kumar

Stanford University

Recording

1 h 7 min

Abstract

More than 85 million computed tomography (CT) scans are performed each year in the US, about a quarter of them of the abdomen. With shortages of general and specialized radiologists, there is strong impetus to use AI both to ease the burden of interpreting these complex studies and to extract new physiological insights. Existing medical vision-language models are generally limited to 2D images and short reports. This talk introduces Merlin, a 3D vision-language model for abdominal CT that is pretrained on structured electronic health record diagnosis codes and unstructured radiology reports without additional manual annotation, using paired data from 15,331 CTs (more than 6 million images), 1.8 million diagnosis codes and over 6 million report tokens. Merlin is evaluated on 6 task types and 752 individual tasks: zero-shot findings classification, phenotype classification and cross-modal retrieval, and adapted tasks including 5-year chronic disease prediction, report generation and 3D segmentation of 20 organs, with internal validation on 5,137 CTs and external validation on 7,000 clinical CTs and two public datasets. The talk compares architectures and training strategies, derives data scaling laws for downstream performance, and shows that all training runs on a single GPU, so health systems can train their own foundation models with modest compute.

Topics