title: HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into
    Multimodal LLMs at Scale
summary: The rapid development of multimodal large language models (MLLMs), such as
  GPT-4V, has led to significant advancements. However, these models still face
  challenges in medical multimodal capabilities due to limitations in the
  quantity and quality of medical vision-text data, stemming from data privacy
  concerns and high annotation costs. While pioneering approaches utilize
  PubMed's large-scale, de-identified medical image-text pairs to address these
  limitations, they still fall short due to inherent data noise. To tackle this,
  we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in
  an 'unblinded' capacity to denoise and reformat the data, resulting in the
  creation of the PubMedVision dataset with 1.3 million medical VQA samples. Our
  validation demonstrates that: (1) PubMedVision can significantly enhance the
  medical multimodal capabilities of current MLLMs, showing significant
  improvement in benchmarks including the MMMU Health & Medicine track; (2)
  manual checks by medical experts and empirical results validate the superior
  data quality of our dataset compared to other data construction methods. Using
  PubMedVision, we train a 34B medical MLLM HuatuoGPT-Vision, which shows
  superior performance in medical multimodal scenarios among open-source MLLMs.
paper_authors._id: 667e437a3a685cd30f9c0f84
paper_authors.name: Junying Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-30T20:57:35.571Z
paper_authors._id: 64097dd1b6a334f53e2b3e4c
paper_authors.avatarUrl: /avatars/18d036aab5e096054a8706bc78027126.svg
paper_authors.fullname: Junying Chen
paper_authors.type: user
paper_authors.user: jymcc
paper_authors._id: 667e437a3a685cd30f9c0f85
paper_authors.name: Ruyi Ouyang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T07:56:39.424Z
paper_authors._id: 662ccc10e93bb738042a33b0
paper_authors.avatarUrl: /avatars/3ae69531fb03e522647e134688459fe7.svg
paper_authors.fullname: Ruyi Ouyang
paper_authors.type: user
paper_authors.user: OBB1028
paper_authors._id: 667e437a3a685cd30f9c0f86
paper_authors.name: Anningzhe Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-25T11:07:34.463Z
paper_authors._id: 66a2271fc56fba8f76bc12ed
paper_authors.avatarUrl: /avatars/ef0e9b600d5dac7f160a86fcc08e0b7d.svg
paper_authors.fullname: Anningzhe
paper_authors.type: user
paper_authors.user: ganz1994
paper_authors._id: 667e437a3a685cd30f9c0f87
paper_authors.name: Shunian Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T07:56:56.176Z
paper_authors._id: 623be9e1d1eb227788764959
paper_authors.avatarUrl: /avatars/b6521b795a59754dbb40123fd4f63b8c.svg
paper_authors.fullname: Shunian Chen
paper_authors.type: user
paper_authors.user: Shunian
paper_authors._id: 667e437a3a685cd30f9c0f88
paper_authors.name: Guiming Hardy Chen
paper_authors._id: 667e437a3a685cd30f9c0f89
paper_authors.name: Xidong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T07:57:32.221Z
paper_authors._id: 640ed3e9f2d7c41a1e9a9fde
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678693486092-640ed3e9f2d7c41a1e9a9fde.jpeg
paper_authors.fullname: Xidong Wang
paper_authors.type: user
paper_authors.user: Xidong
paper_authors._id: 667e437a3a685cd30f9c0f8a
paper_authors.name: Ruifei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T07:57:38.854Z
paper_authors._id: 6520d42d728c0b6dc7032d7f
paper_authors.avatarUrl: /avatars/5acb8548cfbc1a46e0f2c59094e1c30f.svg
paper_authors.fullname: Ruifei Zhang
paper_authors.type: user
paper_authors.user: ReaFly
paper_authors._id: 667e437a3a685cd30f9c0f8b
paper_authors.name: Zhenyang Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T07:57:45.400Z
paper_authors._id: 64f1a34f2c5c8b767916447e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f1a34f2c5c8b767916447e/uak2CsMAnxW8q4dwyAOBN.jpeg
paper_authors.fullname: Zhenyang Cai
paper_authors.type: user
paper_authors.user: Eric3200
paper_authors._id: 667e437a3a685cd30f9c0f8c
paper_authors.name: Ke Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-01T09:45:22.865Z
paper_authors._id: 64c24a1b3499e4f3a2fb680c
paper_authors.avatarUrl: /avatars/7859c3164071a1797c4d9271c8c6ac2e.svg
paper_authors.fullname: Ke Ji
paper_authors.type: user
paper_authors.user: kehorizon
paper_authors._id: 667e437a3a685cd30f9c0f8d
paper_authors.name: Guangjun Yu
paper_authors._id: 667e437a3a685cd30f9c0f8e
paper_authors.name: Xiang Wan
paper_authors._id: 667e437a3a685cd30f9c0f8f
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T07:58:18.395Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_title: HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into
    Multimodal LLMs at Scale
paper_summary: The rapid development of multimodal large language models (MLLMs), such as
  GPT-4V, has led to significant advancements. However, these models still face
  challenges in medical multimodal capabilities due to limitations in the
  quantity and quality of medical vision-text data, stemming from data privacy
  concerns and high annotation costs. While pioneering approaches utilize
  PubMed's large-scale, de-identified medical image-text pairs to address these
  limitations, they still fall short due to inherent data noise. To tackle this,
  we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in
  an 'unblinded' capacity to denoise and reformat the data, resulting in the
  creation of the PubMedVision dataset with 1.3 million medical VQA samples. Our
  validation demonstrates that: (1) PubMedVision can significantly enhance the
  medical multimodal capabilities of current MLLMs, showing significant
  improvement in benchmarks including the MMMU Health & Medicine track; (2)
  manual checks by medical experts and empirical results validate the superior
  data quality of our dataset compared to other data construction methods. Using
  PubMedVision, we train a 34B medical MLLM HuatuoGPT-Vision, which shows
  superior performance in medical multimodal scenarios among open-source MLLMs.
paper_ai_keywords: multimodal large language models (MLLMs) multilingual large language models (MLLMs) (contextual correction to multimodal) GPT-4V medical multimodal capabilities medical image-text pairs denoise reformat PubMedVision VQA samples MMMU Health & Medicine track medical MLLM HuatuoGPT-Vision