title: InstructBLIP: Towards General-purpose Vision-Language Models with
    Instruction Tuning
summary: General-purpose language models that can solve various language-domain tasks
  have emerged driven by the pre-training and instruction-tuning pipeline.
  However, building general-purpose vision-language models is challenging due to
  the increased task discrepancy introduced by the additional visual input.
  Although vision-language pre-training has been widely studied, vision-language
  instruction tuning remains relatively less explored. In this paper, we conduct
  a systematic and comprehensive study on vision-language instruction tuning
  based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly
  available datasets, transform them into instruction tuning format and
  categorize them into two clusters for held-in instruction tuning and held-out
  zero-shot evaluation. Additionally, we introduce instruction-aware visual
  feature extraction, a crucial method that enables the model to extract
  informative features tailored to the given instruction. The resulting
  InstructBLIP models achieve state-of-the-art zero-shot performance across all
  13 held-out datasets, substantially outperforming BLIP-2 and the larger
  Flamingo. Our models also lead to state-of-the-art performance when finetuned
  on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG).
  Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over
  concurrent multimodal models. All InstructBLIP models have been open-sourced at
  https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.
paper_authors._id: 645dafd058f9ee3151487b9c
paper_authors.name: Wenliang Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-12T12:56:22.584Z
paper_authors._id: 61bcb994a4d85c3406ad2c23
paper_authors.avatarUrl: /avatars/8ff55c0329d3c0986d54af08f8c0c7fa.svg
paper_authors.fullname: Wenliang
paper_authors.type: user
paper_authors.user: wdaiai
paper_authors._id: 645dafd058f9ee3151487b9d
paper_authors.name: Junnan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-12T05:04:29.001Z
paper_authors._id: 61f9d3b54ac99e8a1bae85f4
paper_authors.avatarUrl: /avatars/ac47d13204dd22452e4bc46e280842d5.svg
paper_authors.fullname: JunnanLi
paper_authors.type: user
paper_authors.user: JunnanLi
paper_authors._id: 645dafd058f9ee3151487b9e
paper_authors.name: Dongxu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-12T12:28:20.292Z
paper_authors._id: 6220388b354ac333eb998319
paper_authors.avatarUrl: /avatars/e04ebeb57f9dc9aa2f6d4732737f6cb8.svg
paper_authors.fullname: Dongxu Li
paper_authors.type: user
paper_authors.user: dxli
paper_authors._id: 645dafd058f9ee3151487b9f
paper_authors.name: Anthony Meng Huat Tiong
paper_authors._id: 645dafd058f9ee3151487ba0
paper_authors.name: Junqi Zhao
paper_authors._id: 645dafd058f9ee3151487ba1
paper_authors.name: Weisheng Wang
paper_authors._id: 645dafd058f9ee3151487ba2
paper_authors.name: Boyang Li
paper_authors._id: 645dafd058f9ee3151487ba3
paper_authors.name: Pascale Fung
paper_authors._id: 645dafd058f9ee3151487ba4
paper_authors.name: Steven Hoi
paper_title: InstructBLIP: Towards General-purpose Vision-Language Models with
    Instruction Tuning
paper_summary: General-purpose language models that can solve various language-domain tasks
  have emerged driven by the pre-training and instruction-tuning pipeline.
  However, building general-purpose vision-language models is challenging due to
  the increased task discrepancy introduced by the additional visual input.
  Although vision-language pre-training has been widely studied, vision-language
  instruction tuning remains relatively less explored. In this paper, we conduct
  a systematic and comprehensive study on vision-language instruction tuning
  based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly
  available datasets, transform them into instruction tuning format and
  categorize them into two clusters for held-in instruction tuning and held-out
  zero-shot evaluation. Additionally, we introduce instruction-aware visual
  feature extraction, a crucial method that enables the model to extract
  informative features tailored to the given instruction. The resulting
  InstructBLIP models achieve state-of-the-art zero-shot performance across all
  13 held-out datasets, substantially outperforming BLIP-2 and the larger
  Flamingo. Our models also lead to state-of-the-art performance when finetuned
  on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG).
  Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over
  concurrent multimodal models. All InstructBLIP models have been open-sourced at
  https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.
paper_ai_summary: InstructBLIP models, built through instruction-aware visual feature extraction on BLIP-2, achieve top performance in vision-language tasks, including zero-shot evaluation and fine-tuned downstream tasks.
paper_ai_keywords: vision-language models pre-training instruction-tuning BLIP-2 InstructBLIP instruction-aware visual feature extraction zero-shot performance downstream tasks ScienceQA IMG multimodal models