title: An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
summary: Visual instruction tuning has recently shown encouraging progress with
  open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However,
  most existing studies of open-source LMM are performed using models with 13B
  parameters or smaller. In this paper we present an empirical study of scaling
  LLaVA up to 33B and 65B/70B, and share our findings from our explorations in
  image resolution, data mixing and parameter-efficient training methods such as
  LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language
  capabilities when completing real-world tasks in the wild.
    We find that scaling LMM consistently enhances model performance and improves
  language capabilities, and performance of LoRA/QLoRA tuning of LMM are
  comparable to the performance of full-model fine-tuning. Additionally, the
  study highlights the importance of higher image resolutions and mixing
  multimodal-language data to improve LMM performance, and visual instruction
  tuning can sometimes improve LMM's pure language capability. We hope that this
  study makes state-of-the-art LMM research at a larger scale more accessible,
  thus helping establish stronger baselines for future research. Code and
  checkpoints will be made public.
paper_authors._id: 6508fc9a3499309131a1501b
paper_authors.name: Yadong Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:14:13.931Z
paper_authors._id: 64f570ff96185818de7a1ccb
paper_authors.avatarUrl: /avatars/5d4e0a3d21fabccc7c02c441f5786a88.svg
paper_authors.fullname: Yadong Lu
paper_authors.type: user
paper_authors.user: Adong17
paper_authors._id: 6508fc9a3499309131a1501c
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:14:20.971Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 6508fc9a3499309131a1501d
paper_authors.name: Haotian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:14:38.578Z
paper_authors._id: 63898b61ec1f539adc0f4da2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
paper_authors.fullname: Haotian Liu
paper_authors.type: user
paper_authors.user: liuhaotian
paper_authors._id: 6508fc9a3499309131a1501e
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:15:33.011Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 6508fc9a3499309131a1501f
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:15:39.494Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 6508fc9a3499309131a15020
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:15:45.392Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_title: An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
paper_summary: Visual instruction tuning has recently shown encouraging progress with
  open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However,
  most existing studies of open-source LMM are performed using models with 13B
  parameters or smaller. In this paper we present an empirical study of scaling
  LLaVA up to 33B and 65B/70B, and share our findings from our explorations in
  image resolution, data mixing and parameter-efficient training methods such as
  LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language
  capabilities when completing real-world tasks in the wild.
    We find that scaling LMM consistently enhances model performance and improves
  language capabilities, and performance of LoRA/QLoRA tuning of LMM are
  comparable to the performance of full-model fine-tuning. Additionally, the
  study highlights the importance of higher image resolutions and mixing
  multimodal-language data to improve LMM performance, and visual instruction
  tuning can sometimes improve LMM's pure language capability. We hope that this
  study makes state-of-the-art LMM research at a larger scale more accessible,
  thus helping establish stronger baselines for future research. Code and
  checkpoints will be made public.
paper_ai_keywords: LLaVA MiniGPT-4 visual instruction tuning large multimodal models parameter-efficient training methods LoRA QLoRA full-model fine-tuning image resolution data mixing