title: MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular
    Annotations for Medicine
summary: This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal
  dataset for medicine, covering over 25 million images across 10 modalities,
  with multigranular annotations for more than 65 diseases. These enriched
  annotations encompass both global textual information, such as disease/lesion
  type, modality, region-specific descriptions, and inter-regional relationships,
  as well as detailed local annotations for regions of interest (ROIs), including
  bounding boxes, segmentation masks. Unlike existing approach which is limited
  by the availability of image-text pairs, we have developed the first automated
  pipeline that scales up multimodal data by generating multigranular visual and
  texual annotations (in the form of image-ROI-description triplets) without the
  need for any paired text descriptions. Specifically, data from over 90
  different sources have been collected, preprocessed, and grounded using
  domain-specific expert models to identify ROIs related to abnormal regions. We
  then build a comprehensive knowledge base and prompt multimodal large language
  models to perform retrieval-augmented generation with the identified ROIs as
  guidance, resulting in multigranular texual descriptions. Compared to existing
  datasets, MedTrinity-25M provides the most enriched annotations, supporting a
  comprehensive range of multimodal tasks such as captioning and report
  generation, as well as vision-centric tasks like classification and
  segmentation. Pretraining on MedTrinity-25M, our model achieves
  state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal
  large language models and other representative SoTA approaches. This dataset
  can also be utilized to support large-scale pre-training of multimodal medical
  AI models, contributing to the development of future foundation models in the
  medical domain.
paper_authors._id: 66b2d7e8fa7069c7f1f0887b
paper_authors.name: Yunfei Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:20:04.588Z
paper_authors._id: 65f6afa36ea40b9a29dd45ca
paper_authors.avatarUrl: /avatars/a80a25fa70a89a3bf3d50ea2866f0df1.svg
paper_authors.fullname: Yunfei Xie
paper_authors.type: user
paper_authors.user: yunfeixie
paper_authors._id: 66b2d7e8fa7069c7f1f0887c
paper_authors.name: Ce Zhou
paper_authors._id: 66b2d7e8fa7069c7f1f0887d
paper_authors.name: Lang Gao
paper_authors._id: 66b2d7e8fa7069c7f1f0887e
paper_authors.name: Juncheng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:18:59.704Z
paper_authors._id: 641a58d791e3376a0579f011
paper_authors.avatarUrl: /avatars/bb4745bb83ac0ee094b43e1e5562d743.svg
paper_authors.fullname: Juncheng Wu
paper_authors.type: user
paper_authors.user: JunchengWu
paper_authors._id: 66b2d7e8fa7069c7f1f0887f
paper_authors.name: Xianhang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:18:51.477Z
paper_authors._id: 645e72d4b5c9a8666d0ddc1c
paper_authors.avatarUrl: /avatars/f0f1684731eb5ae148eadf841cceb18e.svg
paper_authors.fullname: xhl
paper_authors.type: user
paper_authors.user: Xianhang
paper_authors._id: 66b2d7e8fa7069c7f1f08880
paper_authors.name: Hong-Yu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:18:41.441Z
paper_authors._id: 63983ea1ec07edfe479b9967
paper_authors.avatarUrl: /avatars/2e4fcdac836beb1829bfea335f285fd1.svg
paper_authors.fullname: Hong-Yu Zhou
paper_authors.type: user
paper_authors.user: hyzhou
paper_authors._id: 66b2d7e8fa7069c7f1f08881
paper_authors.name: Sheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:18:35.018Z
paper_authors._id: 653585666141b3927a083b4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653585666141b3927a083b4f/EpMnW_gKz9FV3OvogvUzi.jpeg
paper_authors.fullname: Sheng Liu
paper_authors.type: user
paper_authors.user: shengliu66
paper_authors._id: 66b2d7e8fa7069c7f1f08882
paper_authors.name: Lei Xing
paper_authors._id: 66b2d7e8fa7069c7f1f08883
paper_authors.name: James Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:18:01.076Z
paper_authors._id: 648a769003fc4a3938bb7943
paper_authors.avatarUrl: /avatars/7647f99abdcca4251fcac7783b6fcc8d.svg
paper_authors.fullname: zou
paper_authors.type: user
paper_authors.user: jameszou707
paper_authors._id: 66b2d7e8fa7069c7f1f08884
paper_authors.name: Cihang Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:17:55.313Z
paper_authors._id: 645eb61da3c5cd8a16efffff
paper_authors.avatarUrl: /avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
paper_authors.fullname: Cihang Xie
paper_authors.type: user
paper_authors.user: cihangxie
paper_authors._id: 66b2d7e8fa7069c7f1f08885
paper_authors.name: Yuyin Zhou
paper_title: MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular
    Annotations for Medicine
paper_summary: This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal
  dataset for medicine, covering over 25 million images across 10 modalities,
  with multigranular annotations for more than 65 diseases. These enriched
  annotations encompass both global textual information, such as disease/lesion
  type, modality, region-specific descriptions, and inter-regional relationships,
  as well as detailed local annotations for regions of interest (ROIs), including
  bounding boxes, segmentation masks. Unlike existing approach which is limited
  by the availability of image-text pairs, we have developed the first automated
  pipeline that scales up multimodal data by generating multigranular visual and
  texual annotations (in the form of image-ROI-description triplets) without the
  need for any paired text descriptions. Specifically, data from over 90
  different sources have been collected, preprocessed, and grounded using
  domain-specific expert models to identify ROIs related to abnormal regions. We
  then build a comprehensive knowledge base and prompt multimodal large language
  models to perform retrieval-augmented generation with the identified ROIs as
  guidance, resulting in multigranular texual descriptions. Compared to existing
  datasets, MedTrinity-25M provides the most enriched annotations, supporting a
  comprehensive range of multimodal tasks such as captioning and report
  generation, as well as vision-centric tasks like classification and
  segmentation. Pretraining on MedTrinity-25M, our model achieves
  state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal
  large language models and other representative SoTA approaches. This dataset
  can also be utilized to support large-scale pre-training of multimodal medical
  AI models, contributing to the development of future foundation models in the
  medical domain.