title: BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
summary: LLMs have demonstrated remarkable abilities at interacting with humans
  through language, especially with the usage of instruction-following data.
  Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further
  enlarge their abilities by incorporating multi-modal inputs, including image,
  video, and speech. Despite their effectiveness at generating precise and
  detailed language understanding of the given modality signal, these LLMs give
  up the ability to ground specific parts of inputs, thus only constructing a
  coarse-grained mapping. However, explicit and informative correspondence
  between text and other modalities will not only improve the user experience but
  also help to expand the application scenario of multi-modal LLMs. Therefore, we
  propose BuboGPT, a multi-modal LLM with visual grounding that can perform
  cross-modal interaction between vision, audio and language, providing
  fine-grained understanding of visual objects and other given modalities. As a
  result, BuboGPT is able to point out the specific location of an object in the
  image, when it is generating response or description for that object. Our
  contributions are two-fold: 1) An off-the-shelf visual grounding module based
  on SAM that extracts entities in a sentence and find corresponding masks in the
  image. 2) A two-stage training scheme and instruction dataset to endow joint
  text-image-audio understanding. Our experiments show that BuboGPT achieves
  impressive multi-modality understanding and visual grounding abilities during
  the interaction with human. It performs consistently well when provided by
  arbitrary modality combinations (either aligned or unaligned). Our code, model
  and dataset are available at https://bubo-gpt.github.io .
paper_authors._id: 64b5ec56a9509e0af729378d
paper_authors.name: Yang Zhao
paper_authors._id: 64b5ec56a9509e0af729378e
paper_authors.name: Zhijie Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T19:25:10.071Z
paper_authors._id: 64415957bd0c9726529802f6
paper_authors.avatarUrl: /avatars/1132d1ee68fb58ec635d57c8175caacd.svg
paper_authors.fullname: Zhijie Lin
paper_authors.type: user
paper_authors.user: Ikuinen
paper_authors._id: 64b5ec56a9509e0af729378f
paper_authors.name: Daquan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-18T11:11:10.172Z
paper_authors._id: 63fe0b160c1bbe8e29d2dd32
paper_authors.avatarUrl: /avatars/bc574036287170a77057893efaa48e2d.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: DaQuan21
paper_authors._id: 64b5ec56a9509e0af7293790
paper_authors.name: Zilong Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:38:12.355Z
paper_authors._id: 63a430ec84a6a25c65bff20c
paper_authors.avatarUrl: /avatars/3ab51214c491f28d8fe6b70c373d72aa.svg
paper_authors.fullname: Zilong Huang
paper_authors.type: user
paper_authors.user: speedinghzl
paper_authors._id: 64b5ec56a9509e0af7293791
paper_authors.name: Jiashi Feng
paper_authors._id: 64b5ec56a9509e0af7293792
paper_authors.name: Bingyi Kang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-18T06:43:54.607Z
paper_authors._id: 647b5fef6a79fbf5e996c47c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
paper_authors.fullname: Bingyi Kang
paper_authors.type: user
paper_authors.user: bykang
paper_title: BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
paper_summary: LLMs have demonstrated remarkable abilities at interacting with humans
  through language, especially with the usage of instruction-following data.
  Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further
  enlarge their abilities by incorporating multi-modal inputs, including image,
  video, and speech. Despite their effectiveness at generating precise and
  detailed language understanding of the given modality signal, these LLMs give
  up the ability to ground specific parts of inputs, thus only constructing a
  coarse-grained mapping. However, explicit and informative correspondence
  between text and other modalities will not only improve the user experience but
  also help to expand the application scenario of multi-modal LLMs. Therefore, we
  propose BuboGPT, a multi-modal LLM with visual grounding that can perform
  cross-modal interaction between vision, audio and language, providing
  fine-grained understanding of visual objects and other given modalities. As a
  result, BuboGPT is able to point out the specific location of an object in the
  image, when it is generating response or description for that object. Our
  contributions are two-fold: 1) An off-the-shelf visual grounding module based
  on SAM that extracts entities in a sentence and find corresponding masks in the
  image. 2) A two-stage training scheme and instruction dataset to endow joint
  text-image-audio understanding. Our experiments show that BuboGPT achieves
  impressive multi-modality understanding and visual grounding abilities during
  the interaction with human. It performs consistently well when provided by
  arbitrary modality combinations (either aligned or unaligned). Our code, model
  and dataset are available at https://bubo-gpt.github.io .
paper_ai_summary: BuboGPT is a multi-modal LLM with visual grounding that enhances understanding and interaction between vision, audio, and language.
paper_ai_keywords: LLMs MiniGPT-4 LLaVA X-LLM multi-modal inputs visual grounding cross-modal interaction vision audio language fine-grained understanding visual objects SAM entities masks two-stage training scheme instruction dataset multi-modality understanding