title: Med-Flamingo: a Multimodal Medical Few-shot Learner
summary: Medicine, by its nature, is a multifaceted domain that requires the synthesis
  of information across various modalities. Medical generative vision-language
  models (VLMs) make a first step in this direction and promise many exciting
  clinical applications. However, existing models typically have to be fine-tuned
  on sizeable down-stream datasets, which poses a significant limitation as in
  many medical applications data is scarce, necessitating models that are capable
  of learning from few examples in real-time. Here we propose Med-Flamingo, a
  multimodal few-shot learner adapted to the medical domain. Based on
  OpenFlamingo-9B, we continue pre-training on paired and interleaved medical
  image-text data from publications and textbooks. Med-Flamingo unlocks few-shot
  generative medical visual question answering (VQA) abilities, which we evaluate
  on several datasets including a novel challenging open-ended VQA dataset of
  visual USMLE-style problems. Furthermore, we conduct the first human evaluation
  for generative medical VQA where physicians review the problems and blinded
  generations in an interactive app. Med-Flamingo improves performance in
  generative medical VQA by up to 20\% in clinician's rating and firstly enables
  multimodal medical few-shot adaptations, such as rationale generation. We
  release our model, code, and evaluation app under
  https://github.com/snap-stanford/med-flamingo.
paper_authors._id: 64c7437571947b03ff2a629f
paper_authors.name: Michael Moor
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-03T09:47:12.382Z
paper_authors._id: 6438d1d843d932c462404500
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6438d1d843d932c462404500/6UrtJbed9N5ETbImgIh-C.png
paper_authors.fullname: Michael Moor
paper_authors.type: user
paper_authors.user: mdmoor
paper_authors._id: 64c7437571947b03ff2a62a0
paper_authors.name: Qian Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:51:26.340Z
paper_authors._id: 641ab10ea63c4e80623168c9
paper_authors.avatarUrl: /avatars/64bbfe8e09d957ae8e9bd4c862333da7.svg
paper_authors.fullname: Qian Huang
paper_authors.type: user
paper_authors.user: qhwang
paper_authors._id: 64c7437571947b03ff2a62a1
paper_authors.name: Shirley Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-24T08:09:40.296Z
paper_authors._id: 63641db9a1503c61efd140de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63641db9a1503c61efd140de/tt3yQ27kJKZD5067k2JRb.jpeg
paper_authors.fullname: Shirley Wu
paper_authors.type: user
paper_authors.user: shirwu
paper_authors._id: 64c7437571947b03ff2a62a2
paper_authors.name: Michihiro Yasunaga
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:52:06.467Z
paper_authors._id: 621e9388345a1d9ab65391c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e9388345a1d9ab65391c3/RxurNzyAWJOUdgeSHQi1R.jpeg
paper_authors.fullname: Michihiro Yasunaga
paper_authors.type: user
paper_authors.user: michiyasunaga
paper_authors._id: 64c7437571947b03ff2a62a3
paper_authors.name: Cyril Zakka
paper_authors._id: 64c7437571947b03ff2a62a4
paper_authors.name: Yash Dalmia
paper_authors._id: 64c7437571947b03ff2a62a5
paper_authors.name: Eduardo Pontes Reis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:53:00.830Z
paper_authors._id: 60bc341d85b60a3459be9cac
paper_authors.avatarUrl: /avatars/e0d3d2907504176451a0d22885eef997.svg
paper_authors.fullname: Eduardo Pontes Reis
paper_authors.type: user
paper_authors.user: edureisMD
paper_authors._id: 64c7437571947b03ff2a62a6
paper_authors.name: Pranav Rajpurkar
paper_authors._id: 64c7437571947b03ff2a62a7
paper_authors.name: Jure Leskovec
paper_title: Med-Flamingo: a Multimodal Medical Few-shot Learner
paper_summary: Medicine, by its nature, is a multifaceted domain that requires the synthesis
  of information across various modalities. Medical generative vision-language
  models (VLMs) make a first step in this direction and promise many exciting
  clinical applications. However, existing models typically have to be fine-tuned
  on sizeable down-stream datasets, which poses a significant limitation as in
  many medical applications data is scarce, necessitating models that are capable
  of learning from few examples in real-time. Here we propose Med-Flamingo, a
  multimodal few-shot learner adapted to the medical domain. Based on
  OpenFlamingo-9B, we continue pre-training on paired and interleaved medical
  image-text data from publications and textbooks. Med-Flamingo unlocks few-shot
  generative medical visual question answering (VQA) abilities, which we evaluate
  on several datasets including a novel challenging open-ended VQA dataset of
  visual USMLE-style problems. Furthermore, we conduct the first human evaluation
  for generative medical VQA where physicians review the problems and blinded
  generations in an interactive app. Med-Flamingo improves performance in
  generative medical VQA by up to 20\% in clinician's rating and firstly enables
  multimodal medical few-shot adaptations, such as rationale generation. We
  release our model, code, and evaluation app under
  https://github.com/snap-stanford/med-flamingo.
paper_ai_summary: Med-Flamingo, an adaptation of OpenFlamingo-9B for the medical domain, demonstrates few-shot capabilities in generative visual question answering with significant performance improvements as evaluated by clinicians.
paper_ai_keywords: medical generative vision-language models few-shot learner multimodal fine-tuning paired and interleaved medical image-text data visual question answering USMLE-style problems human evaluation physicians rationale generation