title: Improved Baselines with Visual Instruction Tuning
summary: Large multimodal models (LMM) have recently shown encouraging progress with
  visual instruction tuning. In this note, we show that the fully-connected
  vision-language cross-modal connector in LLaVA is surprisingly powerful and
  data-efficient. With simple modifications to LLaVA, namely, using
  CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA
  data with simple response formatting prompts, we establish stronger baselines
  that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint
  uses merely 1.2M publicly available data, and finishes full training in ~1 day
  on a single 8-A100 node. We hope this can make state-of-the-art LMM research
  more accessible. Code and model will be publicly available.
paper_authors._id: 651f6912da9dcb165cbfd133
paper_authors.name: Haotian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-09T14:12:57.025Z
paper_authors._id: 63898b61ec1f539adc0f4da2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
paper_authors.fullname: Haotian Liu
paper_authors.type: user
paper_authors.user: liuhaotian
paper_authors._id: 651f6912da9dcb165cbfd134
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:47:09.450Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 651f6912da9dcb165cbfd135
paper_authors.name: Yuheng Li
paper_authors._id: 651f6912da9dcb165cbfd136
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:46:09.628Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_title: Improved Baselines with Visual Instruction Tuning
paper_summary: Large multimodal models (LMM) have recently shown encouraging progress with
  visual instruction tuning. In this note, we show that the fully-connected
  vision-language cross-modal connector in LLaVA is surprisingly powerful and
  data-efficient. With simple modifications to LLaVA, namely, using
  CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA
  data with simple response formatting prompts, we establish stronger baselines
  that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint
  uses merely 1.2M publicly available data, and finishes full training in ~1 day
  on a single 8-A100 node. We hope this can make state-of-the-art LMM research
  more accessible. Code and model will be publicly available.
paper_ai_summary: Modifications to LLaVA using CLIP-ViT-L-336px and academic VQA data improve multimodal performance, achieving state-of-the-art results with limited training data.
paper_ai_keywords: LLaVA CLIP-ViT-L-336px MLP projection VQA benchmarks