title: GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards
    General Medical AI
summary: Large Vision-Language Models (LVLMs) are capable of handling diverse data
  types such as imaging, text, and physiological signals, and can be applied in
  various fields. In the medical field, LVLMs have a high potential to offer
  substantial assistance for diagnosis and treatment. Before that, it is crucial
  to develop benchmarks to evaluate LVLMs' effectiveness in various medical
  applications. Current benchmarks are often built upon specific academic
  literature, mainly focusing on a single domain, and lacking varying perceptual
  granularities. Thus, they face specific challenges, including limited clinical
  relevance, incomplete evaluations, and insufficient guidance for interactive
  LVLMs. To address these limitations, we developed the GMAI-MMBench, the most
  comprehensive general medical AI benchmark with well-categorized data structure
  and multi-perceptual granularity to date. It is constructed from 285 datasets
  across 39 medical image modalities, 18 clinical-related tasks, 18 departments,
  and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  Additionally, we implemented a lexical tree structure that allows users to
  customize evaluation tasks, accommodating various assessment needs and
  substantially supporting medical AI research and applications. We evaluated 50
  LVLMs, and the results show that even the advanced GPT-4o only achieves an
  accuracy of 52%, indicating significant room for improvement. Moreover, we
  identified five key insufficiencies in current cutting-edge LVLMs that need to
  be addressed to advance the development of better medical applications. We
  believe that GMAI-MMBench will stimulate the community to build the next
  generation of LVLMs toward GMAI.
    Project Page: https://uni-medical.github.io/GMAI-MMBench.github.io/
paper_authors._id: 66b57e63e1f790d2510cd344
paper_authors.name: Pengcheng Chen
paper_authors._id: 66b57e63e1f790d2510cd345
paper_authors.name: Jin Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T20:39:46.118Z
paper_authors._id: 656292f6412dcdebf2c1c6a8
paper_authors.avatarUrl: /avatars/310604f8c7d701565e9a659e8e85621d.svg
paper_authors.fullname: Jin Ye
paper_authors.type: user
paper_authors.user: EugeneYonng
paper_authors._id: 66b57e63e1f790d2510cd346
paper_authors.name: Guoan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:47:30.273Z
paper_authors._id: 6409b7eb96aae649740e2920
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678358480751-noauth.png
paper_authors.fullname: Guoan Wang
paper_authors.type: user
paper_authors.user: SolooooMan
paper_authors._id: 66b57e63e1f790d2510cd347
paper_authors.name: Yanjun Li
paper_authors._id: 66b57e63e1f790d2510cd348
paper_authors.name: Zhongying Deng
paper_authors._id: 66b57e63e1f790d2510cd349
paper_authors.name: Wei Li
paper_authors._id: 66b57e63e1f790d2510cd34a
paper_authors.name: Tianbin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T09:34:10.951Z
paper_authors._id: 64aff8ecf808e8d2023804ce
paper_authors.avatarUrl: /avatars/13d32347626814801c0f9b2d893b6741.svg
paper_authors.fullname: li
paper_authors.type: user
paper_authors.user: tianbin
paper_authors._id: 66b57e63e1f790d2510cd34b
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T09:29:22.932Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 66b57e63e1f790d2510cd34c
paper_authors.name: Ziyan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T09:29:29.120Z
paper_authors._id: 643fddd6388f31f64ccb9344
paper_authors.avatarUrl: /avatars/c029b59cfbbc4124489e72fa8e71ee9a.svg
paper_authors.fullname: Ziyan Huang
paper_authors.type: user
paper_authors.user: ziyanhuang
paper_authors._id: 66b57e63e1f790d2510cd34d
paper_authors.name: Yanzhou Su
paper_authors._id: 66b57e63e1f790d2510cd34e
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T09:29:45.299Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 66b57e63e1f790d2510cd34f
paper_authors.name: Shaoting Zhang
paper_authors._id: 66b57e63e1f790d2510cd350
paper_authors.name: Bin Fu
paper_authors._id: 66b57e63e1f790d2510cd351
paper_authors.name: Jianfei Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T09:30:13.163Z
paper_authors._id: 656fda082f058b368c27d1b9
paper_authors.avatarUrl: /avatars/cba33f6b20fc93ac9d3d579cbd7f839a.svg
paper_authors.fullname: cai jianfei
paper_authors.type: user
paper_authors.user: genify
paper_authors._id: 66b57e63e1f790d2510cd352
paper_authors.name: Bohan Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T09:30:19.255Z
paper_authors._id: 65a8a828ef14f9e6037dfad2
paper_authors.avatarUrl: /avatars/55fdc0393b2e37ac998ddbdb1e1ff636.svg
paper_authors.fullname: Bohan Zhuang
paper_authors.type: user
paper_authors.user: BohanZ
paper_authors._id: 66b57e63e1f790d2510cd353
paper_authors.name: Eric J Seibel
paper_authors._id: 66b57e63e1f790d2510cd354
paper_authors.name: Junjun He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T09:30:32.614Z
paper_authors._id: 66b593026cef22e6ba6adb8a
paper_authors.avatarUrl: /avatars/8a94d55b85177e84d65dd0bd537e335f.svg
paper_authors.fullname: JunjunHe
paper_authors.type: user
paper_authors.user: JunjunHe
paper_authors._id: 66b57e63e1f790d2510cd355
paper_authors.name: Yu Qiao
paper_title: GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards
    General Medical AI
paper_summary: Large Vision-Language Models (LVLMs) are capable of handling diverse data
  types such as imaging, text, and physiological signals, and can be applied in
  various fields. In the medical field, LVLMs have a high potential to offer
  substantial assistance for diagnosis and treatment. Before that, it is crucial
  to develop benchmarks to evaluate LVLMs' effectiveness in various medical
  applications. Current benchmarks are often built upon specific academic
  literature, mainly focusing on a single domain, and lacking varying perceptual
  granularities. Thus, they face specific challenges, including limited clinical
  relevance, incomplete evaluations, and insufficient guidance for interactive
  LVLMs. To address these limitations, we developed the GMAI-MMBench, the most
  comprehensive general medical AI benchmark with well-categorized data structure
  and multi-perceptual granularity to date. It is constructed from 285 datasets
  across 39 medical image modalities, 18 clinical-related tasks, 18 departments,
  and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  Additionally, we implemented a lexical tree structure that allows users to
  customize evaluation tasks, accommodating various assessment needs and
  substantially supporting medical AI research and applications. We evaluated 50
  LVLMs, and the results show that even the advanced GPT-4o only achieves an
  accuracy of 52%, indicating significant room for improvement. Moreover, we
  identified five key insufficiencies in current cutting-edge LVLMs that need to
  be addressed to advance the development of better medical applications. We
  believe that GMAI-MMBench will stimulate the community to build the next
  generation of LVLMs toward GMAI.
    Project Page: https://uni-medical.github.io/GMAI-MMBench.github.io/
paper_ai_keywords: Large Vision-Language Models (LVLMs) imaging text physiological signals medical field diagnosis treatment benchmarks medical applicative perceptual granularities clinical relevance evaluations interactive LVLMs GMAI-MMBench datasets medical image modalities clinical-robust tasks departments perceptual granularities Visual Question Answer (VQA) lexical tree structure accuracy key insufficiencies next generation of LVLMs GMAI