title: Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
summary: Despite vision-language models' (VLMs) remarkable capabilities as versatile
  visual assistants, two substantial challenges persist within the existing VLM
  frameworks: (1) lacking task diversity in pretraining and visual instruction
  tuning, and (2) annotation error and bias in GPT-4 synthesized instruction
  tuning data. Both challenges lead to issues such as poor generalizability,
  hallucination, and catastrophic forgetting. To address these challenges, we
  construct Vision-Flan, the most diverse publicly available visual instruction
  tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances
  sourced from academic datasets, and each task is accompanied by an
  expert-written instruction. In addition, we propose a two-stage instruction
  tuning framework, in which VLMs are firstly finetuned on Vision-Flan and
  further tuned on GPT-4 synthesized data. We find this two-stage tuning
  framework significantly outperforms the traditional single-stage visual
  instruction tuning framework and achieves the state-of-the-art performance
  across a wide range of multi-modal evaluation benchmarks. Finally, we conduct
  in-depth analyses to understand visual instruction tuning and our findings
  reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs'
  capabilities but rather modulates the model's responses to human-preferred
  formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can
  effectively align VLM responses with human-preference; (3) Visual instruction
  tuning mainly helps large-language models (LLMs) to understand visual features.
paper_authors._id: 65d42b34bd2ebbf3020e69cf
paper_authors.name: Zhiyang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:48:56.074Z
paper_authors._id: 64b6c686cf5117d7962d8f62
paper_authors.avatarUrl: /avatars/96ed7a9602aa4c21b3a3d89608e76dc8.svg
paper_authors.fullname: Zhiyang Xu
paper_authors.type: user
paper_authors.user: Zhiyang03
paper_authors._id: 65d42b34bd2ebbf3020e69d0
paper_authors.name: Chao Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T08:30:36.837Z
paper_authors._id: 645ab0b7c266796265baefa4
paper_authors.avatarUrl: /avatars/bdac661996b63c4b2a56881707afa01f.svg
paper_authors.fullname: Chao Feng
paper_authors.type: user
paper_authors.user: chfeng
paper_authors._id: 65d42b34bd2ebbf3020e69d1
paper_authors.name: Rulin Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:49:29.692Z
paper_authors._id: 6334a0bd31a2be3938c59537
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6334a0bd31a2be3938c59537/kSetFUWAmJbPQ1KSlNKBr.jpeg
paper_authors.fullname: Rulin Shao
paper_authors.type: user
paper_authors.user: rulins
paper_authors._id: 65d42b34bd2ebbf3020e69d2
paper_authors.name: Trevor Ashby
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:50:05.317Z
paper_authors._id: 628415624695ad4fb6f883ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652913118915-628415624695ad4fb6f883ef.jpeg
paper_authors.fullname: Trevor Ashby
paper_authors.type: user
paper_authors.user: TrevorAshby
paper_authors._id: 65d42b34bd2ebbf3020e69d3
paper_authors.name: Ying Shen
paper_authors._id: 65d42b34bd2ebbf3020e69d4
paper_authors.name: Di Jin
paper_authors._id: 65d42b34bd2ebbf3020e69d5
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T16:26:46.658Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 65d42b34bd2ebbf3020e69d6
paper_authors.name: Qifan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:51:19.836Z
paper_authors._id: 64b65c84c991daa61404abd7
paper_authors.avatarUrl: /avatars/3592aa8774ea4e2836099e0701b4b78d.svg
paper_authors.fullname: WangQifan
paper_authors.type: user
paper_authors.user: Bayees
paper_authors._id: 65d42b34bd2ebbf3020e69d7
paper_authors.name: Lifu Huang
paper_title: Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
paper_summary: Despite vision-language models' (VLMs) remarkable capabilities as versatile
  visual assistants, two substantial challenges persist within the existing VLM
  frameworks: (1) lacking task diversity in pretraining and visual instruction
  tuning, and (2) annotation error and bias in GPT-4 synthesized instruction
  tuning data. Both challenges lead to issues such as poor generalizability,
  hallucination, and catastrophic forgetting. To address these challenges, we
  construct Vision-Flan, the most diverse publicly available visual instruction
  tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances
  sourced from academic datasets, and each task is accompanied by an
  expert-written instruction. In addition, we propose a two-stage instruction
  tuning framework, in which VLMs are firstly finetuned on Vision-Flan and
  further tuned on GPT-4 synthesized data. We find this two-stage tuning
  framework significantly outperforms the traditional single-stage visual
  instruction tuning framework and achieves the state-of-the-art performance
  across a wide range of multi-modal evaluation benchmarks. Finally, we conduct
  in-depth analyses to understand visual instruction tuning and our findings
  reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs'
  capabilities but rather modulates the model's responses to human-preferred
  formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can
  effectively align VLM responses with human-preference; (3) Visual instruction
  tuning mainly helps large-language models (LLMs) to understand visual features.
paper_ai_summary: A two-stage instruction tuning framework using Vision-Flan and GPT-4 data enhances vision-language models' performance and understanding of visual features.
paper_ai_keywords: vision-language models visual instruction tuning pretraining annotation error bias catastrophic forgetting Vision-Flan two-stage tuning framework finetuning multi-modal evaluation benchmarks human-preferred formats large-language models