Med-Flamingo: a Multimodal Medical Few-shot Learner
Medicine, by its nature, is a multifaceted domain that requires the synthesis
of information across various modalities. Medical generative vision-language
models (VLMs) make a first step in this direction and promise many exciting
clinical applications. However, existing models typically have to be fine-tuned
on sizeable down-stream datasets, which poses a significant limitation as in
many medical applications data is scarce, necessitating models that are capable
of learning from few examples in real-time. Here we propose Med-Flamingo, a
multimodal few-shot learner adapted to the medical domain. Based on
OpenFlamingo-9B, we continue pre-training on paired and interleaved medical
image-text data from publications and textbooks. Med-Flamingo unlocks few-shot
generative medical visual question answering (VQA) abilities, which we evaluate
on several datasets including a novel challenging open-ended VQA dataset of
visual USMLE-style problems. Furthermore, we conduct the first human evaluation
for generative medical VQA where physicians review the problems and blinded
generations in an interactive app. Med-Flamingo improves performance in
generative medical VQA by up to 20\% in clinician's rating and firstly enables
multimodal medical few-shot adaptations, such as rationale generation. We
release our model, code, and evaluation app under
https://github.com/snap-stanford/med-flamingo.
64c7437571947b03ff2a629f
Michael Moor
claimed_verified
2023-08-03T09:47:12.382Z
6438d1d843d932c462404500
https://cdn-avatars.huggingface.co/v1/production/uploads/6438d1d843d932c462404500/6UrtJbed9N5ETbImgIh-C.png
Michael Moor
user
mdmoor
64c7437571947b03ff2a62a0
Qian Huang
admin_assigned
2023-07-31T06:51:26.340Z
641ab10ea63c4e80623168c9
/avatars/64bbfe8e09d957ae8e9bd4c862333da7.svg
Qian Huang
user
qhwang
64c7437571947b03ff2a62a1
Shirley Wu
claimed_verified
2023-08-24T08:09:40.296Z
63641db9a1503c61efd140de
https://cdn-avatars.huggingface.co/v1/production/uploads/63641db9a1503c61efd140de/tt3yQ27kJKZD5067k2JRb.jpeg
Shirley Wu
user
shirwu
64c7437571947b03ff2a62a2
Michihiro Yasunaga
admin_assigned
2023-07-31T06:52:06.467Z
621e9388345a1d9ab65391c3
/avatars/d009c811dc63224f8eba1889fefad04b.svg
Michihiro Yasunaga
user
michiyasunaga
64c7437571947b03ff2a62a3
Cyril Zakka
claimed_verified
2025-06-10T15:20:53.255Z
66ba71a4447411b9c0e19d71
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4f93ZrYdaKfK3F53IB51x.jpeg
Cyril
user
cyrilzakka
64c7437571947b03ff2a62a4
Yash Dalmia
64c7437571947b03ff2a62a5
Eduardo Pontes Reis
admin_assigned
2023-07-31T06:53:00.830Z
60bc341d85b60a3459be9cac
/avatars/e0d3d2907504176451a0d22885eef997.svg
Eduardo Pontes Reis
user
edureisMD
64c7437571947b03ff2a62a6
Pranav Rajpurkar
64c7437571947b03ff2a62a7
Jure Leskovec
Med-Flamingo: a Multimodal Medical Few-shot Learner
Medicine, by its nature, is a multifaceted domain that requires the synthesis
of information across various modalities. Medical generative vision-language
models (VLMs) make a first step in this direction and promise many exciting
clinical applications. However, existing models typically have to be fine-tuned
on sizeable down-stream datasets, which poses a significant limitation as in
many medical applications data is scarce, necessitating models that are capable
of learning from few examples in real-time. Here we propose Med-Flamingo, a
multimodal few-shot learner adapted to the medical domain. Based on
OpenFlamingo-9B, we continue pre-training on paired and interleaved medical
image-text data from publications and textbooks. Med-Flamingo unlocks few-shot
generative medical visual question answering (VQA) abilities, which we evaluate
on several datasets including a novel challenging open-ended VQA dataset of
visual USMLE-style problems. Furthermore, we conduct the first human evaluation
for generative medical VQA where physicians review the problems and blinded
generations in an interactive app. Med-Flamingo improves performance in
generative medical VQA by up to 20\% in clinician's rating and firstly enables
multimodal medical few-shot adaptations, such as rationale generation. We
release our model, code, and evaluation app under
https://github.com/snap-stanford/med-flamingo.
Med-Flamingo, an adaptation of OpenFlamingo-9B for the medical domain, demonstrates few-shot capabilities in generative visual question answering with significant performance improvements as evaluated by clinicians.
medical generative vision-language models few-shot learner multimodal fine-tuning paired and interleaved medical image-text data visual question answering USMLE-style problems human evaluation physicians rationale generation