SVIT: Scaling up Visual Instruction Tuning
Thanks to the emerging of foundation models, the large language and vision
models are integrated to acquire the multimodal ability of visual captioning,
dialogue, question answering, etc. Although existing multimodal models present
impressive performance of visual understanding and reasoning, their limits are
still largely under-explored due to the scarcity of high-quality instruction
tuning data. To push the limits of multimodal capability, we Sale up Visual
Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
instruction tuning data including 1.6M conversation question-answer (QA) pairs
and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
Besides the volume, the proposed dataset is also featured by the high quality
and rich diversity, which is generated by prompting GPT-4 with the abundant
manual annotations of images. We empirically verify that training multimodal
models on SVIT can significantly improve the multimodal performance in terms of
visual perception, reasoning and planing.
64acaf5d6864362a7efb154d
Bo Zhao
claimed_verified
2024-02-20T08:39:28.836Z
64acafac1aee69ece03af05a
/avatars/6380777852546dbd432fed5544d5db0b.svg
Bo Zhao
user
Z-LIRA
64acaf5d6864362a7efb154e
Boya Wu
admin_assigned
2023-07-12T07:43:17.918Z
64acaeccb575c5e272f5f124
/avatars/29b0f9d0ad38755a52db1e69d112caa9.svg
BoyaWu
user
BoyaWu10
64acaf5d6864362a7efb154f
Tiejun Huang
SVIT: Scaling up Visual Instruction Tuning
Thanks to the emerging of foundation models, the large language and vision
models are integrated to acquire the multimodal ability of visual captioning,
dialogue, question answering, etc. Although existing multimodal models present
impressive performance of visual understanding and reasoning, their limits are
still largely under-explored due to the scarcity of high-quality instruction
tuning data. To push the limits of multimodal capability, we Sale up Visual
Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
instruction tuning data including 1.6M conversation question-answer (QA) pairs
and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
Besides the volume, the proposed dataset is also featured by the high quality
and rich diversity, which is generated by prompting GPT-4 with the abundant
manual annotations of images. We empirically verify that training multimodal
models on SVIT can significantly improve the multimodal performance in terms of
visual perception, reasoning and planing.
A new dataset of 3.2 million visual instruction tuning pairs improves multimodal performance in visual perception, reasoning, and planning by training on high-quality, diverse manual annotations.
foundation models large language models large vision models multimodal ability visual captioning dialogue question answering speech instruction tuning SVIT conversation question-answer pairs complex reasoning question-answer pairs detailed image descriptions GPT-4 visual perception visual reasoning planning