title: GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A
    Comprehensive Multimodal Dataset Towards General Medical AI
summary: Despite significant advancements in general artificial intelligence, such as
  GPT-4, their effectiveness in the medical domain (general medical AI, GMAI)
  remains constrained due to the absence of specialized medical knowledge. To
  address this challenge, we present GMAI-VL-5.5M, a comprehensive multimodal
  medical dataset created by converting hundreds of specialized medical datasets
  into meticulously constructed image-text pairs. This dataset features
  comprehensive task coverage, diverse modalities, and high-quality image-text
  data. Building upon this multimodal dataset, we propose GMAI-VL, a general
  medical vision-language model with a progressively three-stage training
  strategy. This approach significantly enhances the model's ability by
  integrating visual and textual information, thereby improving its ability to
  process multimodal data and support accurate diagnosis and clinical
  decision-making. Experimental evaluations demonstrate that GMAI-VL achieves
  state-of-the-art results across a wide range of multimodal medical tasks, such
  as visual question answering and medical image diagnosis. Our contributions
  include the development of the GMAI-VL-5.5M dataset, the introduction of the
  GMAI-VL model, and the establishment of new benchmarks in multiple medical
  domains. Code and dataset will be released at
  https://github.com/uni-medical/GMAI-VL.
paper_authors._id: 6744834547d61b15da9ac6b4
paper_authors.name: Tianbin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T20:38:04.236Z
paper_authors._id: 64aff95a1e39d3cd14efe6d5
paper_authors.avatarUrl: /avatars/77efc2db6462dce2561116aa3a9d5b54.svg
paper_authors.fullname: li
paper_authors.type: user
paper_authors.user: foreverbeliever
paper_authors._id: 6744834547d61b15da9ac6b5
paper_authors.name: Yanzhou Su
paper_authors._id: 6744834547d61b15da9ac6b6
paper_authors.name: Wei Li
paper_authors._id: 6744834547d61b15da9ac6b7
paper_authors.name: Bin Fu
paper_authors._id: 6744834547d61b15da9ac6b8
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:48:02.229Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 6744834547d61b15da9ac6b9
paper_authors.name: Ziyan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:45:26.988Z
paper_authors._id: 643fddd6388f31f64ccb9344
paper_authors.avatarUrl: /avatars/c029b59cfbbc4124489e72fa8e71ee9a.svg
paper_authors.fullname: Ziyan Huang
paper_authors.type: user
paper_authors.user: ziyanhuang
paper_authors._id: 6744834547d61b15da9ac6ba
paper_authors.name: Guoan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:45:21.788Z
paper_authors._id: 6409b7eb96aae649740e2920
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678358480751-noauth.png
paper_authors.fullname: Guoan Wang
paper_authors.type: user
paper_authors.user: SolooooMan
paper_authors._id: 6744834547d61b15da9ac6bb
paper_authors.name: Chenglong Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:45:15.333Z
paper_authors._id: 666432a35adc67db967560d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/u7-8F9-aH7uDAPgEYq12E.jpeg
paper_authors.fullname: Chenglong Ma
paper_authors.type: user
paper_authors.user: ChenglongMa
paper_authors._id: 6744834547d61b15da9ac6bc
paper_authors.name: Ying Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:16.620Z
paper_authors._id: 64c0aaa4a8a2dcaa167ab5b3
paper_authors.avatarUrl: /avatars/04f20baf200712f781b1a338a11644bf.svg
paper_authors.fullname: Ying Chen
paper_authors.type: user
paper_authors.user: cying
paper_authors._id: 6744834547d61b15da9ac6bd
paper_authors.name: Ming Hu
paper_authors._id: 6744834547d61b15da9ac6be
paper_authors.name: Yanjun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:46:01.774Z
paper_authors._id: 6741c26133e10fc6d4526c32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/rsFgSp1qogazSOEcMHN7L.png
paper_authors.fullname: Yanjun Li
paper_authors.type: user
paper_authors.user: ballinli
paper_authors._id: 6744834547d61b15da9ac6bf
paper_authors.name: Pengcheng Chen
paper_authors._id: 6744834547d61b15da9ac6c0
paper_authors.name: Xiaowei Hu
paper_authors._id: 6744834547d61b15da9ac6c1
paper_authors.name: Zhongying Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:44:21.069Z
paper_authors._id: 66b7a5d1b4958a65e26e1d05
paper_authors.avatarUrl: /avatars/10ceeea495628731c056be4afe5e10a4.svg
paper_authors.fullname: Zhongying Deng
paper_authors.type: user
paper_authors.user: zdcam294
paper_authors._id: 6744834547d61b15da9ac6c2
paper_authors.name: Yuanfeng Ji
paper_authors._id: 6744834547d61b15da9ac6c3
paper_authors.name: Jin Ye
paper_authors._id: 6744834547d61b15da9ac6c4
paper_authors.name: Yu Qiao
paper_authors._id: 6744834547d61b15da9ac6c5
paper_authors.name: Junjun He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:44:05.179Z
paper_authors._id: 66b593026cef22e6ba6adb8a
paper_authors.avatarUrl: /avatars/8a94d55b85177e84d65dd0bd537e335f.svg
paper_authors.fullname: JunjunHe
paper_authors.type: user
paper_authors.user: JunjunHe
paper_title: GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A
    Comprehensive Multimodal Dataset Towards General Medical AI
paper_summary: Despite significant advancements in general artificial intelligence, such as
  GPT-4, their effectiveness in the medical domain (general medical AI, GMAI)
  remains constrained due to the absence of specialized medical knowledge. To
  address this challenge, we present GMAI-VL-5.5M, a comprehensive multimodal
  medical dataset created by converting hundreds of specialized medical datasets
  into meticulously constructed image-text pairs. This dataset features
  comprehensive task coverage, diverse modalities, and high-quality image-text
  data. Building upon this multimodal dataset, we propose GMAI-VL, a general
  medical vision-language model with a progressively three-stage training
  strategy. This approach significantly enhances the model's ability by
  integrating visual and textual information, thereby improving its ability to
  process multimodal data and support accurate diagnosis and clinical
  decision-making. Experimental evaluations demonstrate that GMAI-VL achieves
  state-of-the-art results across a wide range of multimodal medical tasks, such
  as visual question answering and medical image diagnosis. Our contributions
  include the development of the GMAI-VL-5.5M dataset, the introduction of the
  GMAI-VL model, and the establishment of new benchmarks in multiple medical
  domains. Code and dataset will be released at
  https://github.com/uni-medical/GMAI-VL.
paper_ai_summary: GMAI-VL, a vision-language model trained on the GMAI-VL-5.5M dataset, achieves top performance in multimodal medical tasks by integrating visual and textual information.
paper_ai_keywords: multimodal image-text pairs general medical vision-language model progressively three-stage training strategy visual question answering medical image diagnosis benchmarks GMAI-VL GMAI-VL-5.5M