1 h
Seminar, Hosted by Stanford MedAI
The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models
Carnegie Mellon University
Recording
Abstract
Several recent works adapt general-purpose large language models (LLMs) and vision-language models (VLMs) to medicine through continued pretraining on public biomedical corpora, claiming improved performance on downstream medical tasks such as medical exam questions. This work compares ten medical LLMs and two medical VLMs against their corresponding base models and reaches a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models for medical question answering in zero- and few-shot prompting and supervised fine-tuning. On clinical-note-based question answering in the 3-shot setting, for example, medical LLMs outperform their base models in only 26.7% of cases, tie in 16.7% and perform significantly worse in the remaining 56.7%. The conclusions rest on direct comparison of each medical model with its base model, separate prompt optimization for each model, and accounting for statistical uncertainty. The findings suggest that state-of-the-art general-domain models may already have strong medical knowledge and reasoning, and the talk offers recommendations for strengthening future studies.