title: MiniGPT-v2: large language model as a unified interface for
    vision-language multi-task learning
summary: Large language models have shown their remarkable capabilities as a general
  interface for various language-related applications. Motivated by this, we
  target to build a unified interface for completing many vision-language tasks
  including image description, visual question answering, and visual grounding,
  among others. The challenge is to use a single model for performing diverse
  vision-language tasks effectively with simple multi-modal instructions. Towards
  this objective, we introduce MiniGPT-v2, a model that can be treated as a
  unified interface for better handling various vision-language tasks. We propose
  using unique identifiers for different tasks when training the model. These
  identifiers enable our model to better distinguish each task instruction
  effortlessly and also improve the model learning efficiency for each task.
  After the three-stage training, the experimental results show that MiniGPT-v2
  achieves strong performance on many visual question-answering and visual
  grounding benchmarks compared to other vision-language generalist models. Our
  model and codes are available at https://minigpt-v2.github.io/
paper_authors._id: 652dea01cc4609d64398ca85
paper_authors.name: Jun Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-17T07:30:56.960Z
paper_authors._id: 62f94ec077b722f186641192
paper_authors.avatarUrl: /avatars/37f4c7f2b6f59e2eac2d1b1b5cc88269.svg
paper_authors.fullname: jun chen
paper_authors.type: user
paper_authors.user: junchen14
paper_authors._id: 652dea01cc4609d64398ca86
paper_authors.name: Deyao Zhu
paper_authors._id: 652dea01cc4609d64398ca87
paper_authors.name: Xiaoqian Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:41:03.238Z
paper_authors._id: 620c5b20c33ca69c4c82145b
paper_authors.avatarUrl: /avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
paper_authors.fullname: Xiaoqian Shen
paper_authors.type: user
paper_authors.user: shenxq
paper_authors._id: 652dea01cc4609d64398ca88
paper_authors.name: Xiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-17T07:49:43.602Z
paper_authors._id: 63fb35b01b4b1bd4e7fd1c72
paper_authors.avatarUrl: /avatars/e3c77e8afff06af4ee6dd025c16feac7.svg
paper_authors.fullname: Xiang Li
paper_authors.type: user
paper_authors.user: xiang709
paper_authors._id: 652dea01cc4609d64398ca89
paper_authors.name: Zechun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:41:21.467Z
paper_authors._id: 620ae55fe190de4b8f27ba9d
paper_authors.avatarUrl: /avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Zechun
paper_authors._id: 652dea01cc4609d64398ca8a
paper_authors.name: Pengchuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:41:28.897Z
paper_authors._id: 640056f5261cfa61f39b4c35
paper_authors.avatarUrl: /avatars/25aa5dee42caa69327d158a669f6bf3a.svg
paper_authors.fullname: Pengchuan Zhang
paper_authors.type: user
paper_authors.user: pzzhang
paper_authors._id: 652dea01cc4609d64398ca8b
paper_authors.name: Raghuraman Krishnamoorthi
paper_authors._id: 652dea01cc4609d64398ca8c
paper_authors.name: Vikas Chandra
paper_authors._id: 652dea01cc4609d64398ca8d
paper_authors.name: Yunyang Xiong
paper_authors._id: 652dea01cc4609d64398ca8e
paper_authors.name: Mohamed Elhoseiny
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:42:01.590Z
paper_authors._id: 64a27d39649b0d08ca0a4ca6
paper_authors.avatarUrl: /avatars/e4446a875506c10de9ae28411dc6416d.svg
paper_authors.fullname: Mohamed Elhoseiny
paper_authors.type: user
paper_authors.user: mhelhoseiny
paper_title: MiniGPT-v2: large language model as a unified interface for
    vision-language multi-task learning
paper_summary: Large language models have shown their remarkable capabilities as a general
  interface for various language-related applications. Motivated by this, we
  target to build a unified interface for completing many vision-language tasks
  including image description, visual question answering, and visual grounding,
  among others. The challenge is to use a single model for performing diverse
  vision-language tasks effectively with simple multi-modal instructions. Towards
  this objective, we introduce MiniGPT-v2, a model that can be treated as a
  unified interface for better handling various vision-language tasks. We propose
  using unique identifiers for different tasks when training the model. These
  identifiers enable our model to better distinguish each task instruction
  effortlessly and also improve the model learning efficiency for each task.
  After the three-stage training, the experimental results show that MiniGPT-v2
  achieves strong performance on many visual question-answering and visual
  grounding benchmarks compared to other vision-language generalist models. Our
  model and codes are available at https://minigpt-v2.github.io/
paper_ai_summary: MiniGPT-v2 is a unified vision-language model that uses task-specific identifiers to perform diverse tasks with high performance.
paper_ai_keywords: MiniGPT-v2 vision-language tasks image description visual question answering visual grounding unique identifiers multistage training visual question-answering benchmarks visual grounding benchmarks