title: LLaVA-Med: Training a Large Language-and-Vision Assistant for
    Biomedicine in One Day
summary: Conversational generative AI has demonstrated remarkable promise for
  empowering biomedical practitioners, but current investigations focus on
  unimodal text. Multimodal conversational AI has seen rapid progress by
  leveraging billions of image-text pairs from the public web, but such
  general-domain vision-language models still lack sophistication in
  understanding and conversing about biomedical images. In this paper, we propose
  a cost-efficient approach for training a vision-language conversational
  assistant that can answer open-ended research questions of biomedical images.
  The key idea is to leverage a large-scale, broad-coverage biomedical
  figure-caption dataset extracted from PubMed Central, use GPT-4 to
  self-instruct open-ended instruction-following data from the captions, and then
  fine-tune a large general-domain vision-language model using a novel curriculum
  learning method. Specifically, the model first learns to align biomedical
  vocabulary using the figure-caption pairs as is, then learns to master
  open-ended conversational semantics using GPT-4 generated instruction-following
  data, broadly mimicking how a layperson gradually acquires biomedical
  knowledge. This enables us to train a Large Language and Vision Assistant for
  BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med
  exhibits excellent multimodal conversational capability and can follow
  open-ended instruction to assist with inquiries about a biomedical image. On
  three standard biomedical visual question answering datasets, LLaVA-Med
  outperforms previous supervised state-of-the-art on certain metrics. To
  facilitate biomedical multimodal research, we will release our
  instruction-following data and the LLaVA-Med model.
paper_authors._id: 647940a83b7f8b1f624c07ec
paper_authors.name: Chunyuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-02T07:16:44.404Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 647940a83b7f8b1f624c07ed
paper_authors.name: Cliff Wong
paper_authors._id: 647940a83b7f8b1f624c07ee
paper_authors.name: Sheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:32.281Z
paper_authors._id: 6234c11b7d5de9839bc44163
paper_authors.avatarUrl: /avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
paper_authors.fullname: Sheng Zhang
paper_authors.type: user
paper_authors.user: shengz
paper_authors._id: 647940a83b7f8b1f624c07ef
paper_authors.name: Naoto Usuyama
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:06:32.327Z
paper_authors._id: 5f243561e923d665e616263f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
paper_authors.fullname: Naoto Usuyama
paper_authors.type: user
paper_authors.user: naotous
paper_authors._id: 647940a83b7f8b1f624c07f0
paper_authors.name: Haotian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:23.514Z
paper_authors._id: 63898b61ec1f539adc0f4da2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
paper_authors.fullname: Haotian Liu
paper_authors.type: user
paper_authors.user: liuhaotian
paper_authors._id: 647940a83b7f8b1f624c07f1
paper_authors.name: Jianwei Yang
paper_authors._id: 647940a83b7f8b1f624c07f2
paper_authors.name: Tristan Naumann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T08:55:11.048Z
paper_authors._id: 5e5870466bc35159a08ca572
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e5870466bc35159a08ca572/pT6gEEs8RLRJGeM-faNWj.jpeg
paper_authors.fullname: Tristan Naumann
paper_authors.type: user
paper_authors.user: tnaumann
paper_authors._id: 647940a83b7f8b1f624c07f3
paper_authors.name: Hoifung Poon
paper_authors._id: 647940a83b7f8b1f624c07f4
paper_authors.name: Jianfeng Gao
paper_title: LLaVA-Med: Training a Large Language-and-Vision Assistant for
    Biomedicine in One Day
paper_summary: Conversational generative AI has demonstrated remarkable promise for
  empowering biomedical practitioners, but current investigations focus on
  unimodal text. Multimodal conversational AI has seen rapid progress by
  leveraging billions of image-text pairs from the public web, but such
  general-domain vision-language models still lack sophistication in
  understanding and conversing about biomedical images. In this paper, we propose
  a cost-efficient approach for training a vision-language conversational
  assistant that can answer open-ended research questions of biomedical images.
  The key idea is to leverage a large-scale, broad-coverage biomedical
  figure-caption dataset extracted from PubMed Central, use GPT-4 to
  self-instruct open-ended instruction-following data from the captions, and then
  fine-tune a large general-domain vision-language model using a novel curriculum
  learning method. Specifically, the model first learns to align biomedical
  vocabulary using the figure-caption pairs as is, then learns to master
  open-ended conversational semantics using GPT-4 generated instruction-following
  data, broadly mimicking how a layperson gradually acquires biomedical
  knowledge. This enables us to train a Large Language and Vision Assistant for
  BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med
  exhibits excellent multimodal conversational capability and can follow
  open-ended instruction to assist with inquiries about a biomedical image. On
  three standard biomedical visual question answering datasets, LLaVA-Med
  outperforms previous supervised state-of-the-art on certain metrics. To
  facilitate biomedical multimodal research, we will release our
  instruction-following data and the LLaVA-Med model.
paper_ai_keywords: GPT-4 self-instruct fine-tune curriculum system figure-caption dataset Large Language and Vision Assistant (LLaVA-Med) biomedical visual question answering datasets multimodal conversational capability instruction-following data