title: To See is to Believe: Prompting GPT-4V for Better Visual Instruction
    Tuning
summary: Existing visual instruction tuning methods typically prompt large language
  models with textual descriptions to generate instruction-following data.
  Despite the promising performance achieved, these descriptions are derived from
  image annotations, which are oftentimes coarse-grained. Furthermore, the
  instructions might even contradict the visual content without observing the
  entire visual context. To address this challenge, we introduce a fine-grained
  visual instruction dataset, LVIS-Instruct4V, which contains 220K visually
  aligned and context-aware instructions produced by prompting the powerful
  GPT-4V with images from LVIS. Through experimental validation and case studies,
  we demonstrate that high-quality visual instructional data could improve the
  performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a
  wide spectrum of benchmarks by clear margins. Notably, by simply replacing the
  LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA
  on most challenging LMM benchmarks, e.g., LLaVA^w (76.7 vs. 70.7) and MM-Vet
  (40.2 vs. 35.4). We release our data and model at
  https://github.com/X2FD/LVIS-INSTRUCT4V.
paper_authors._id: 6552f439ecf367738e0e47c8
paper_authors.name: Junke Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:42:50.054Z
paper_authors._id: 642e1e7147833318f0eb3755
paper_authors.avatarUrl: /avatars/7ef4f5a099d3eb0fab99b589f33903fa.svg
paper_authors.fullname: Junke Wang
paper_authors.type: user
paper_authors.user: Daniel0724
paper_authors._id: 6552f439ecf367738e0e47c9
paper_authors.name: Lingchen Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:42:58.013Z
paper_authors._id: 64d852a4bab152b2470bf96e
paper_authors.avatarUrl: /avatars/074ef416ea865ca97c208ab2ba55d622.svg
paper_authors.fullname: Lingchen Meng
paper_authors.type: user
paper_authors.user: menglc
paper_authors._id: 6552f439ecf367738e0e47ca
paper_authors.name: Zejia Weng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:43:09.163Z
paper_authors._id: 6437cc89961bb61e4639bc47
paper_authors.avatarUrl: /avatars/05f5dc29cade3bb5d3146072dadce41c.svg
paper_authors.fullname: wengzejia
paper_authors.type: user
paper_authors.user: wengzejia
paper_authors._id: 6552f439ecf367738e0e47cb
paper_authors.name: Bo He
paper_authors._id: 6552f439ecf367738e0e47cc
paper_authors.name: Zuxuan Wu
paper_authors._id: 6552f439ecf367738e0e47cd
paper_authors.name: Yu-Gang Jiang
paper_title: To See is to Believe: Prompting GPT-4V for Better Visual Instruction
    Tuning
paper_summary: Existing visual instruction tuning methods typically prompt large language
  models with textual descriptions to generate instruction-following data.
  Despite the promising performance achieved, these descriptions are derived from
  image annotations, which are oftentimes coarse-grained. Furthermore, the
  instructions might even contradict the visual content without observing the
  entire visual context. To address this challenge, we introduce a fine-grained
  visual instruction dataset, LVIS-Instruct4V, which contains 220K visually
  aligned and context-aware instructions produced by prompting the powerful
  GPT-4V with images from LVIS. Through experimental validation and case studies,
  we demonstrate that high-quality visual instructional data could improve the
  performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a
  wide spectrum of benchmarks by clear margins. Notably, by simply replacing the
  LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA
  on most challenging LMM benchmarks, e.g., LLaVA^w (76.7 vs. 70.7) and MM-Vet
  (40.2 vs. 35.4). We release our data and model at
  https://github.com/X2FD/LVIS-INSTRUCT4V.
paper_ai_summary: A fine-grained visual instruction dataset, LVIS-Instruct4V, improves the performance of LLaVA-1.5 across benchmarks by generating context-aware instructions from images using GPT-4V.
paper_ai_keywords: GPT-4V LVIS-Instruct4V LLaVA-1.5 multimodal model LMM benchmarks LLaVA$^w$ MM-Vet