title: On the Compositional Generalization of Multimodal LLMs for Medical
    Imaging
summary: Multimodal large language models (MLLMs) hold significant potential in the
  medical field, but their capabilities are often limited by insufficient data in
  certain medical domains, highlighting the need for understanding what kinds of
  images can be used by MLLMs for generalization. Current research suggests that
  multi-task training outperforms single-task as different tasks can benefit each
  other, but they often overlook the internal relationships within these tasks,
  providing limited guidance on selecting datasets to enhance specific tasks. To
  analyze this phenomenon, we attempted to employ compositional generalization
  (CG)-the ability of models to understand novel combinations by recombining
  learned elements-as a guiding framework. Since medical images can be precisely
  defined by Modality, Anatomical area, and Task, naturally providing an
  environment for exploring CG. Therefore, we assembled 106 medical datasets to
  create Med-MAT for comprehensive experiments. The experiments confirmed that
  MLLMs can use CG to understand unseen medical images and identified CG as one
  of the main drivers of the generalization observed in multi-task training.
  Additionally, further studies demonstrated that CG effectively supports
  datasets with limited data and delivers consistent performance across different
  backbones, highlighting its versatility and broad applicability. Med-MAT is
  publicly available at https://github.com/FreedomIntelligence/Med-MAT.
paper_authors._id: 67735f9f9cc5d33bf6af3cef
paper_authors.name: Zhenyang Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:13:04.716Z
paper_authors._id: 64f1a34f2c5c8b767916447e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f1a34f2c5c8b767916447e/uak2CsMAnxW8q4dwyAOBN.jpeg
paper_authors.fullname: Zhenyang Cai
paper_authors.type: user
paper_authors.user: Eric3200
paper_authors._id: 67735f9f9cc5d33bf6af3cf0
paper_authors.name: Junying Chen
paper_authors._id: 67735f9f9cc5d33bf6af3cf1
paper_authors.name: Rongsheng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:13:02.316Z
paper_authors._id: 63ca949b04c979828315389d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ca949b04c979828315389d/HS5xWNAYjjHeyAAwWJ11l.jpeg
paper_authors.fullname: wangrongsheng
paper_authors.type: user
paper_authors.user: wangrongsheng
paper_authors._id: 67735f9f9cc5d33bf6af3cf2
paper_authors.name: Weihong Wang
paper_authors._id: 67735f9f9cc5d33bf6af3cf3
paper_authors.name: Yonglin Deng
paper_authors._id: 67735f9f9cc5d33bf6af3cf4
paper_authors.name: Dingjie Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:12:59.736Z
paper_authors._id: 619f01b8cc04eadf54fa5d5d
paper_authors.avatarUrl: /avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
paper_authors.fullname: Song Dingjie
paper_authors.type: user
paper_authors.user: songdj
paper_authors._id: 67735f9f9cc5d33bf6af3cf5
paper_authors.name: Yize Chen
paper_authors._id: 67735f9f9cc5d33bf6af3cf6
paper_authors.name: Zixu Zhang
paper_authors._id: 67735f9f9cc5d33bf6af3cf7
paper_authors.name: Benyou Wang
paper_title: On the Compositional Generalization of Multimodal LLMs for Medical
    Imaging
paper_summary: Multimodal large language models (MLLMs) hold significant potential in the
  medical field, but their capabilities are often limited by insufficient data in
  certain medical domains, highlighting the need for understanding what kinds of
  images can be used by MLLMs for generalization. Current research suggests that
  multi-task training outperforms single-task as different tasks can benefit each
  other, but they often overlook the internal relationships within these tasks,
  providing limited guidance on selecting datasets to enhance specific tasks. To
  analyze this phenomenon, we attempted to employ compositional generalization
  (CG)-the ability of models to understand novel combinations by recombining
  learned elements-as a guiding framework. Since medical images can be precisely
  defined by Modality, Anatomical area, and Task, naturally providing an
  environment for exploring CG. Therefore, we assembled 106 medical datasets to
  create Med-MAT for comprehensive experiments. The experiments confirmed that
  MLLMs can use CG to understand unseen medical images and identified CG as one
  of the main drivers of the generalization observed in multi-task training.
  Additionally, further studies demonstrated that CG effectively supports
  datasets with limited data and delivers consistent performance across different
  backbones, highlighting its versatility and broad applicability. Med-MAT is
  publicly available at https://github.com/FreedomIntelligence/Med-MAT.
paper_ai_summary: Compositional generalization aids multimodal large language models in understanding and generalizing to unseen medical images across various domains and tasks.
paper_ai_keywords: multimodal large language models MLLMs medical field multi-task training single-task training compositional generalization CG Med-MAT medical datasets