title: MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
summary: We present a vision and language model named MultiModal-GPT to conduct
  multi-round dialogue with humans. MultiModal-GPT can follow various
  instructions from humans, such as generating a detailed caption, counting the
  number of interested objects, and answering general questions from users.
  MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with
  Low-rank Adapter (LoRA) added both in the cross-attention part and the
  self-attention part of the language model. We first construct instruction
  templates with vision and language data for multi-modality instruction tuning
  to make the model understand and follow human instructions. We find the quality
  of training data is vital for the dialogue performance, where few data
  containing short answers can lead the model to respond shortly to any
  instructions. To further enhance the ability to chat with humans of the
  MultiModal-GPT, we utilize language-only instruction-following data to train
  the MultiModal-GPT jointly. The joint training of language-only and
  visual-language instructions with the same instruction template
  effectively improves dialogue performance. Various demos show the ability of
  continuous dialogue of MultiModal-GPT with humans. Code and demo are at
  https://github.com/open-mmlab/Multimodal-GPT
paper_authors._id: 6459a91fabdbb77c4c6d9990
paper_authors.name: Tao Gong
paper_authors._id: 6459a91fabdbb77c4c6d9991
paper_authors.name: Chengqi Lyu
paper_authors._id: 6459a91fabdbb77c4c6d9992
paper_authors.name: Shilong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:08:09.911Z
paper_authors._id: 6424ffce46d202ad3d918a67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6424ffce46d202ad3d918a67/gmYmOA072fP_5cJLc9Qs4.jpeg
paper_authors.fullname: Shilong Zhang
paper_authors.type: user
paper_authors.user: shilongz
paper_authors._id: 6459a91fabdbb77c4c6d9993
paper_authors.name: Yudong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:24:12.126Z
paper_authors._id: 63be286fb3b8c44f8cecc16f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63be286fb3b8c44f8cecc16f/1CIkfEKoTnBYdYDSuQ8AT.jpeg
paper_authors.fullname: Yudong Wang
paper_authors.type: user
paper_authors.user: BigDong
paper_authors._id: 6459a91fabdbb77c4c6d9994
paper_authors.name: Miao Zheng
paper_authors._id: 6459a91fabdbb77c4c6d9995
paper_authors.name: Qian Zhao
paper_authors._id: 6459a91fabdbb77c4c6d9996
paper_authors.name: Kuikun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-02T07:35:36.973Z
paper_authors._id: 63fd691794cc8f815d50c112
paper_authors.avatarUrl: /avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: Harold-lkk
paper_authors._id: 6459a91fabdbb77c4c6d9997
paper_authors.name: Wenwei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:36:50.932Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 6459a91fabdbb77c4c6d9998
paper_authors.name: Ping Luo
paper_authors._id: 6459a91fabdbb77c4c6d9999
paper_authors.name: Kai Chen
paper_title: MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
paper_summary: We present a vision and language model named MultiModal-GPT to conduct
  multi-round dialogue with humans. MultiModal-GPT can follow various
  instructions from humans, such as generating a detailed caption, counting the
  number of interested objects, and answering general questions from users.
  MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with
  Low-rank Adapter (LoRA) added both in the cross-attention part and the
  self-attention part of the language model. We first construct instruction
  templates with vision and language data for multi-modality instruction tuning
  to make the model understand and follow human instructions. We find the quality
  of training data is vital for the dialogue performance, where few data
  containing short answers can lead the model to respond shortly to any
  instructions. To further enhance the ability to chat with humans of the
  MultiModal-GPT, we utilize language-only instruction-following data to train
  the MultiModal-GPT jointly. The joint training of language-only and
  visual-language instructions with the same instruction template
  effectively improves dialogue performance. Various demos show the ability of
  continuous dialogue of MultiModal-GPT with humans. Code and demo are at
  https://github.com/open-mmlab/Multimodal-GPT
paper_ai_summary: MultiModal-GPT, fine-tuned with Low-rank Adapter, enhances dialogue quality by using mixed instruction templates from language-only and visual-language data.
paper_ai_keywords: parameter-efficient fine-tuning Low-rank Adapter LoRA cross-attention self-attention instruction templates multi-modality instruction tuning dialogue performance continuous dialogue