GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A
  Comprehensive Multimodal Dataset Towards General Medical AI
Despite significant advancements in general artificial intelligence, such as
GPT-4, their effectiveness in the medical domain (general medical AI, GMAI)
remains constrained due to the absence of specialized medical knowledge. To
address this challenge, we present GMAI-VL-5.5M, a comprehensive multimodal
medical dataset created by converting hundreds of specialized medical datasets
into meticulously constructed image-text pairs. This dataset features
comprehensive task coverage, diverse modalities, and high-quality image-text
data. Building upon this multimodal dataset, we propose GMAI-VL, a general
medical vision-language model with a progressively three-stage training
strategy. This approach significantly enhances the model's ability by
integrating visual and textual information, thereby improving its ability to
process multimodal data and support accurate diagnosis and clinical
decision-making. Experimental evaluations demonstrate that GMAI-VL achieves
state-of-the-art results across a wide range of multimodal medical tasks, such
as visual question answering and medical image diagnosis. Our contributions
include the development of the GMAI-VL-5.5M dataset, the introduction of the
GMAI-VL model, and the establishment of new benchmarks in multiple medical
domains. Code and dataset will be released at
https://github.com/uni-medical/GMAI-VL.
6744834547d61b15da9ac6b4
Tianbin Li
claimed_verified
2024-11-25T20:38:04.236Z
64aff95a1e39d3cd14efe6d5
/avatars/77efc2db6462dce2561116aa3a9d5b54.svg
li
user
foreverbeliever
6744834547d61b15da9ac6b5
Yanzhou Su
6744834547d61b15da9ac6b6
Wei Li
6744834547d61b15da9ac6b7
Bin Fu
6744834547d61b15da9ac6b8
Zhe Chen
claimed_verified
2024-12-09T14:48:02.229Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
6744834547d61b15da9ac6b9
Ziyan Huang
admin_assigned
2024-11-26T22:45:26.988Z
643fddd6388f31f64ccb9344
/avatars/c029b59cfbbc4124489e72fa8e71ee9a.svg
Ziyan Huang
user
ziyanhuang
6744834547d61b15da9ac6ba
Guoan Wang
admin_assigned
2024-11-26T22:45:21.788Z
6409b7eb96aae649740e2920
https://cdn-avatars.huggingface.co/v1/production/uploads/1678358480751-noauth.png
Guoan Wang
user
SolooooMan
6744834547d61b15da9ac6bb
Chenglong Ma
admin_assigned
2024-11-26T22:45:15.333Z
666432a35adc67db967560d3
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/u7-8F9-aH7uDAPgEYq12E.jpeg
Chenglong Ma
user
ChenglongMa
6744834547d61b15da9ac6bc
Ying Chen
claimed_verified
2024-11-26T21:21:16.620Z
64c0aaa4a8a2dcaa167ab5b3
/avatars/04f20baf200712f781b1a338a11644bf.svg
Ying Chen
user
cying
6744834547d61b15da9ac6bd
Ming Hu
6744834547d61b15da9ac6be
Yanjun Li
admin_assigned
2024-11-26T22:46:01.774Z
6741c26133e10fc6d4526c32
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/rsFgSp1qogazSOEcMHN7L.png
Yanjun Li
user
ballinli
6744834547d61b15da9ac6bf
Pengcheng Chen
6744834547d61b15da9ac6c0
Xiaowei Hu
6744834547d61b15da9ac6c1
Zhongying Deng
admin_assigned
2024-11-26T22:44:21.069Z
66b7a5d1b4958a65e26e1d05
/avatars/10ceeea495628731c056be4afe5e10a4.svg
Zhongying Deng
user
zdcam294
6744834547d61b15da9ac6c2
Yuanfeng Ji
6744834547d61b15da9ac6c3
Jin Ye
6744834547d61b15da9ac6c4
Yu Qiao
6744834547d61b15da9ac6c5
Junjun He
admin_assigned
2024-11-26T22:44:05.179Z
66b593026cef22e6ba6adb8a
/avatars/8a94d55b85177e84d65dd0bd537e335f.svg
JunjunHe
user
JunjunHe
GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A
  Comprehensive Multimodal Dataset Towards General Medical AI
Despite significant advancements in general artificial intelligence, such as
GPT-4, their effectiveness in the medical domain (general medical AI, GMAI)
remains constrained due to the absence of specialized medical knowledge. To
address this challenge, we present GMAI-VL-5.5M, a comprehensive multimodal
medical dataset created by converting hundreds of specialized medical datasets
into meticulously constructed image-text pairs. This dataset features
comprehensive task coverage, diverse modalities, and high-quality image-text
data. Building upon this multimodal dataset, we propose GMAI-VL, a general
medical vision-language model with a progressively three-stage training
strategy. This approach significantly enhances the model's ability by
integrating visual and textual information, thereby improving its ability to
process multimodal data and support accurate diagnosis and clinical
decision-making. Experimental evaluations demonstrate that GMAI-VL achieves
state-of-the-art results across a wide range of multimodal medical tasks, such
as visual question answering and medical image diagnosis. Our contributions
include the development of the GMAI-VL-5.5M dataset, the introduction of the
GMAI-VL model, and the establishment of new benchmarks in multiple medical
domains. Code and dataset will be released at
https://github.com/uni-medical/GMAI-VL.
GMAI-VL, a vision-language model trained on the GMAI-VL-5.5M dataset, achieves top performance in multimodal medical tasks by integrating visual and textual information.
multimodal image-text pairs general medical vision-language model progressively three-stage training strategy visual question answering medical image diagnosis benchmarks GMAI-VL GMAI-VL-5.5M