title: SVIT: Scaling up Visual Instruction Tuning
summary: Thanks to the emerging of foundation models, the large language and vision
  models are integrated to acquire the multimodal ability of visual captioning,
  dialogue, question answering, etc. Although existing multimodal models present
  impressive performance of visual understanding and reasoning, their limits are
  still largely under-explored due to the scarcity of high-quality instruction
  tuning data. To push the limits of multimodal capability, we Sale up Visual
  Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
  instruction tuning data including 1.6M conversation question-answer (QA) pairs
  and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
  Besides the volume, the proposed dataset is also featured by the high quality
  and rich diversity, which is generated by prompting GPT-4 with the abundant
  manual annotations of images. We empirically verify that training multimodal
  models on SVIT can significantly improve the multimodal performance in terms of
  visual perception, reasoning and planing.
paper_authors._id: 64acaf5d6864362a7efb154d
paper_authors.name: Bo Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T08:39:28.836Z
paper_authors._id: 64acafac1aee69ece03af05a
paper_authors.avatarUrl: /avatars/6380777852546dbd432fed5544d5db0b.svg
paper_authors.fullname: Bo Zhao
paper_authors.type: user
paper_authors.user: BoZhaoHuggingFace
paper_authors._id: 64acaf5d6864362a7efb154e
paper_authors.name: Boya Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-12T07:43:17.918Z
paper_authors._id: 64acaeccb575c5e272f5f124
paper_authors.avatarUrl: /avatars/29b0f9d0ad38755a52db1e69d112caa9.svg
paper_authors.fullname: BoyaWu
paper_authors.type: user
paper_authors.user: BoyaWu10
paper_authors._id: 64acaf5d6864362a7efb154f
paper_authors.name: Tiejun Huang
paper_title: SVIT: Scaling up Visual Instruction Tuning
paper_summary: Thanks to the emerging of foundation models, the large language and vision
  models are integrated to acquire the multimodal ability of visual captioning,
  dialogue, question answering, etc. Although existing multimodal models present
  impressive performance of visual understanding and reasoning, their limits are
  still largely under-explored due to the scarcity of high-quality instruction
  tuning data. To push the limits of multimodal capability, we Sale up Visual
  Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
  instruction tuning data including 1.6M conversation question-answer (QA) pairs
  and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
  Besides the volume, the proposed dataset is also featured by the high quality
  and rich diversity, which is generated by prompting GPT-4 with the abundant
  manual annotations of images. We empirically verify that training multimodal
  models on SVIT can significantly improve the multimodal performance in terms of
  visual perception, reasoning and planing.
paper_ai_keywords: foundation models large language models vision models visual captioning dialogue question answering visual understanding visual reasoning multimodal capability Visual Instruction Tuning (SVIT) conversation question-answer (QA) pairs complex reasoning QA pairs detailed image descriptions prompting GPT-4 multimodal performance visual perception reasoning planning