SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context
  Learning
Multimodal in-context learning (ICL) remains underexplored despite
significant potential for domains such as medicine. Clinicians routinely
encounter diverse, specialized tasks requiring adaptation from limited
examples, such as drawing insights from a few relevant prior cases or
considering a constrained set of differential diagnoses. While multimodal large
language models (MLLMs) have shown advances in medical visual question
answering (VQA), their ability to learn multimodal tasks from context is
largely unknown. We introduce SMMILE, the first expert-driven multimodal ICL
benchmark for medical tasks. Eleven medical experts curated problems, each
including a multimodal query and multimodal in-context examples as task
demonstrations. SMMILE encompasses 111 problems (517 question-image-answer
triplets) covering 6 medical specialties and 13 imaging modalities. We further
introduce SMMILE++, an augmented variant with 1038 permuted problems. A
comprehensive evaluation of 15 MLLMs demonstrates that most models exhibit
moderate to poor multimodal ICL ability in medical tasks. In open-ended
evaluations, ICL contributes only 8% average improvement over zero-shot on
SMMILE and 9.4% on SMMILE++. We observe a susceptibility for irrelevant
in-context examples: even a single noisy or irrelevant example can degrade
performance by up to 9.5%. Moreover, example ordering exhibits a recency bias,
i.e., placing the most relevant example last can lead to substantial
performance improvements by up to 71%. Our findings highlight critical
limitations and biases in current MLLMs when learning multimodal medical tasks
from context.
6862b6b43b7fc785c7ca5e41
Melanie Rieff
6862b6b43b7fc785c7ca5e42
Maya Varma
6862b6b43b7fc785c7ca5e43
Ossian Rabow
6862b6b43b7fc785c7ca5e44
Subathra Adithan
6862b6b43b7fc785c7ca5e45
Julie Kim
6862b6b43b7fc785c7ca5e46
Ken Chang
6862b6b43b7fc785c7ca5e47
Hannah Lee
6862b6b43b7fc785c7ca5e48
Nidhi Rohatgi
6862b6b43b7fc785c7ca5e49
Christian Bluethgen
6862b6b43b7fc785c7ca5e4a
Mohamed S. Muneer
6862b6b43b7fc785c7ca5e4b
Jean-Benoit Delbrouck
claimed_verified
2025-07-09T08:52:55.426Z
62716952bcef985363db8485
https://cdn-avatars.huggingface.co/v1/production/uploads/62716952bcef985363db8485/zJPPo5xlwZRJdEuwYsYKp.jpeg
JB D.
user
IAMJB
6862b6b43b7fc785c7ca5e4c
Michael Moor
claimed_verified
2025-07-02T07:41:02.677Z
6438d1d843d932c462404500
https://cdn-avatars.huggingface.co/v1/production/uploads/6438d1d843d932c462404500/6UrtJbed9N5ETbImgIh-C.png
Michael Moor
user
mdmoor
SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context
  Learning
Multimodal in-context learning (ICL) remains underexplored despite
significant potential for domains such as medicine. Clinicians routinely
encounter diverse, specialized tasks requiring adaptation from limited
examples, such as drawing insights from a few relevant prior cases or
considering a constrained set of differential diagnoses. While multimodal large
language models (MLLMs) have shown advances in medical visual question
answering (VQA), their ability to learn multimodal tasks from context is
largely unknown. We introduce SMMILE, the first expert-driven multimodal ICL
benchmark for medical tasks. Eleven medical experts curated problems, each
including a multimodal query and multimodal in-context examples as task
demonstrations. SMMILE encompasses 111 problems (517 question-image-answer
triplets) covering 6 medical specialties and 13 imaging modalities. We further
introduce SMMILE++, an augmented variant with 1038 permuted problems. A
comprehensive evaluation of 15 MLLMs demonstrates that most models exhibit
moderate to poor multimodal ICL ability in medical tasks. In open-ended
evaluations, ICL contributes only 8% average improvement over zero-shot on
SMMILE and 9.4% on SMMILE++. We observe a susceptibility for irrelevant
in-context examples: even a single noisy or irrelevant example can degrade
performance by up to 9.5%. Moreover, example ordering exhibits a recency bias,
i.e., placing the most relevant example last can lead to substantial
performance improvements by up to 71%. Our findings highlight critical
limitations and biases in current MLLMs when learning multimodal medical tasks
from context.
Current multimodal large language models show moderate to poor performance in multimodal in-context learning for medical tasks, with sensitivity to example relevance and ordering.
multimodal in-context learning MLLMs medical visual question answering in-context examples multimodal benchmark medical specialties imaging modalities expert-driven recency bias zero-shot learning