Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
Despite vision-language models' (VLMs) remarkable capabilities as versatile
visual assistants, two substantial challenges persist within the existing VLM
frameworks: (1) lacking task diversity in pretraining and visual instruction
tuning, and (2) annotation error and bias in GPT-4 synthesized instruction
tuning data. Both challenges lead to issues such as poor generalizability,
hallucination, and catastrophic forgetting. To address these challenges, we
construct Vision-Flan, the most diverse publicly available visual instruction
tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances
sourced from academic datasets, and each task is accompanied by an
expert-written instruction. In addition, we propose a two-stage instruction
tuning framework, in which VLMs are firstly finetuned on Vision-Flan and
further tuned on GPT-4 synthesized data. We find this two-stage tuning
framework significantly outperforms the traditional single-stage visual
instruction tuning framework and achieves the state-of-the-art performance
across a wide range of multi-modal evaluation benchmarks. Finally, we conduct
in-depth analyses to understand visual instruction tuning and our findings
reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs'
capabilities but rather modulates the model's responses to human-preferred
formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can
effectively align VLM responses with human-preference; (3) Visual instruction
tuning mainly helps large-language models (LLMs) to understand visual features.
65d42b34bd2ebbf3020e69cf
Zhiyang Xu
admin_assigned
2024-02-20T10:48:56.074Z
64b6c686cf5117d7962d8f62
/avatars/96ed7a9602aa4c21b3a3d89608e76dc8.svg
Zhiyang Xu
user
Zhiyang03
65d42b34bd2ebbf3020e69d0
Chao Feng
claimed_verified
2024-02-21T08:30:36.837Z
645ab0b7c266796265baefa4
/avatars/bdac661996b63c4b2a56881707afa01f.svg
Chao Feng
user
chfeng
65d42b34bd2ebbf3020e69d1
Rulin Shao
admin_assigned
2024-02-20T10:49:29.692Z
6334a0bd31a2be3938c59537
https://cdn-avatars.huggingface.co/v1/production/uploads/6334a0bd31a2be3938c59537/kSetFUWAmJbPQ1KSlNKBr.jpeg
Rulin Shao
user
rulins
65d42b34bd2ebbf3020e69d2
Trevor Ashby
admin_assigned
2024-02-20T10:50:05.317Z
628415624695ad4fb6f883ef
https://cdn-avatars.huggingface.co/v1/production/uploads/1652913118915-628415624695ad4fb6f883ef.jpeg
Trevor Ashby
user
TrevorAshby
65d42b34bd2ebbf3020e69d3
Ying Shen
65d42b34bd2ebbf3020e69d4
Di Jin
65d42b34bd2ebbf3020e69d5
Yu Cheng
claimed_verified
2024-10-09T16:26:46.658Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
65d42b34bd2ebbf3020e69d6
Qifan Wang
admin_assigned
2024-02-20T10:51:19.836Z
64b65c84c991daa61404abd7
/avatars/3592aa8774ea4e2836099e0701b4b78d.svg
WangQifan
user
Bayees
65d42b34bd2ebbf3020e69d7
Lifu Huang
Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
Despite vision-language models' (VLMs) remarkable capabilities as versatile
visual assistants, two substantial challenges persist within the existing VLM
frameworks: (1) lacking task diversity in pretraining and visual instruction
tuning, and (2) annotation error and bias in GPT-4 synthesized instruction
tuning data. Both challenges lead to issues such as poor generalizability,
hallucination, and catastrophic forgetting. To address these challenges, we
construct Vision-Flan, the most diverse publicly available visual instruction
tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances
sourced from academic datasets, and each task is accompanied by an
expert-written instruction. In addition, we propose a two-stage instruction
tuning framework, in which VLMs are firstly finetuned on Vision-Flan and
further tuned on GPT-4 synthesized data. We find this two-stage tuning
framework significantly outperforms the traditional single-stage visual
instruction tuning framework and achieves the state-of-the-art performance
across a wide range of multi-modal evaluation benchmarks. Finally, we conduct
in-depth analyses to understand visual instruction tuning and our findings
reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs'
capabilities but rather modulates the model's responses to human-preferred
formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can
effectively align VLM responses with human-preference; (3) Visual instruction
tuning mainly helps large-language models (LLMs) to understand visual features.
A two-stage instruction tuning framework using Vision-Flan and GPT-4 data enhances vision-language models' performance and understanding of visual features.
vision-language models visual instruction tuning pretraining annotation error bias catastrophic forgetting Vision-Flan two-stage tuning framework finetuning multi-modal evaluation benchmarks human-preferred formats large-language models