title: SVIT: Scaling up Visual Instruction Tuning
summary: Thanks to the emerging of foundation models, the large language and vision
  models are integrated to acquire the multimodal ability of visual captioning,
  dialogue, question answering, etc. Although existing multimodal models present
  impressive performance of visual understanding and reasoning, their limits are
  still largely under-explored due to the scarcity of high-quality instruction
  tuning data. To push the limits of multimodal capability, we Sale up Visual
  Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
  instruction tuning data including 1.6M conversation question-answer (QA) pairs
  and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
  Besides the volume, the proposed dataset is also featured by the high quality
  and rich diversity, which is generated by prompting GPT-4 with the abundant
  manual annotations of images. We empirically verify that training multimodal
  models on SVIT can significantly improve the multimodal performance in terms of
  visual perception, reasoning and planing.
paper_authors._id: 64acaf5d6864362a7efb154d
paper_authors.name: Bo Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T08:39:28.836Z
paper_authors._id: 64acafac1aee69ece03af05a
paper_authors.avatarUrl: /avatars/6380777852546dbd432fed5544d5db0b.svg
paper_authors.fullname: Bo Zhao
paper_authors.type: user
paper_authors.user: Z-LIRA
paper_authors._id: 64acaf5d6864362a7efb154e
paper_authors.name: Boya Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-12T07:43:17.918Z
paper_authors._id: 64acaeccb575c5e272f5f124
paper_authors.avatarUrl: /avatars/29b0f9d0ad38755a52db1e69d112caa9.svg
paper_authors.fullname: BoyaWu
paper_authors.type: user
paper_authors.user: BoyaWu10
paper_authors._id: 64acaf5d6864362a7efb154f
paper_authors.name: Tiejun Huang
paper_title: SVIT: Scaling up Visual Instruction Tuning
paper_summary: Thanks to the emerging of foundation models, the large language and vision
  models are integrated to acquire the multimodal ability of visual captioning,
  dialogue, question answering, etc. Although existing multimodal models present
  impressive performance of visual understanding and reasoning, their limits are
  still largely under-explored due to the scarcity of high-quality instruction
  tuning data. To push the limits of multimodal capability, we Sale up Visual
  Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
  instruction tuning data including 1.6M conversation question-answer (QA) pairs
  and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
  Besides the volume, the proposed dataset is also featured by the high quality
  and rich diversity, which is generated by prompting GPT-4 with the abundant
  manual annotations of images. We empirically verify that training multimodal
  models on SVIT can significantly improve the multimodal performance in terms of
  visual perception, reasoning and planing.
paper_ai_summary: A new dataset of 3.2 million visual instruction tuning pairs improves multimodal performance in visual perception, reasoning, and planning by training on high-quality, diverse manual annotations.
paper_ai_keywords: foundation models large language models large vision models multimodal ability visual captioning dialogue question answering speech instruction tuning SVIT conversation question-answer pairs complex reasoning question-answer pairs detailed image descriptions GPT-4 visual perception visual reasoning planning