BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
LLMs have demonstrated remarkable abilities at interacting with humans
through language, especially with the usage of instruction-following data.
Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further
enlarge their abilities by incorporating multi-modal inputs, including image,
video, and speech. Despite their effectiveness at generating precise and
detailed language understanding of the given modality signal, these LLMs give
up the ability to ground specific parts of inputs, thus only constructing a
coarse-grained mapping. However, explicit and informative correspondence
between text and other modalities will not only improve the user experience but
also help to expand the application scenario of multi-modal LLMs. Therefore, we
propose BuboGPT, a multi-modal LLM with visual grounding that can perform
cross-modal interaction between vision, audio and language, providing
fine-grained understanding of visual objects and other given modalities. As a
result, BuboGPT is able to point out the specific location of an object in the
image, when it is generating response or description for that object. Our
contributions are two-fold: 1) An off-the-shelf visual grounding module based
on SAM that extracts entities in a sentence and find corresponding masks in the
image. 2) A two-stage training scheme and instruction dataset to endow joint
text-image-audio understanding. Our experiments show that BuboGPT achieves
impressive multi-modality understanding and visual grounding abilities during
the interaction with human. It performs consistently well when provided by
arbitrary modality combinations (either aligned or unaligned). Our code, model
and dataset are available at https://bubo-gpt.github.io .
64b5ec56a9509e0af729378d
Yang Zhao
64b5ec56a9509e0af729378e
Zhijie Lin
claimed_verified
2023-07-21T19:25:10.071Z
64415957bd0c9726529802f6
/avatars/1132d1ee68fb58ec635d57c8175caacd.svg
Zhijie Lin
user
Ikuinen
64b5ec56a9509e0af729378f
Daquan Zhou
claimed_verified
2023-07-18T11:11:10.172Z
63fe0b160c1bbe8e29d2dd32
/avatars/bc574036287170a77057893efaa48e2d.svg
Zhou
user
DaQuan21
64b5ec56a9509e0af7293790
Zilong Huang
claimed_verified
2025-05-14T07:38:12.355Z
63a430ec84a6a25c65bff20c
/avatars/3ab51214c491f28d8fe6b70c373d72aa.svg
Zilong Huang
user
speedinghzl
64b5ec56a9509e0af7293791
Jiashi Feng
64b5ec56a9509e0af7293792
Bingyi Kang
claimed_verified
2023-07-18T06:43:54.607Z
647b5fef6a79fbf5e996c47c
https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
Bingyi Kang
user
bykang
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
LLMs have demonstrated remarkable abilities at interacting with humans
through language, especially with the usage of instruction-following data.
Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further
enlarge their abilities by incorporating multi-modal inputs, including image,
video, and speech. Despite their effectiveness at generating precise and
detailed language understanding of the given modality signal, these LLMs give
up the ability to ground specific parts of inputs, thus only constructing a
coarse-grained mapping. However, explicit and informative correspondence
between text and other modalities will not only improve the user experience but
also help to expand the application scenario of multi-modal LLMs. Therefore, we
propose BuboGPT, a multi-modal LLM with visual grounding that can perform
cross-modal interaction between vision, audio and language, providing
fine-grained understanding of visual objects and other given modalities. As a
result, BuboGPT is able to point out the specific location of an object in the
image, when it is generating response or description for that object. Our
contributions are two-fold: 1) An off-the-shelf visual grounding module based
on SAM that extracts entities in a sentence and find corresponding masks in the
image. 2) A two-stage training scheme and instruction dataset to endow joint
text-image-audio understanding. Our experiments show that BuboGPT achieves
impressive multi-modality understanding and visual grounding abilities during
the interaction with human. It performs consistently well when provided by
arbitrary modality combinations (either aligned or unaligned). Our code, model
and dataset are available at https://bubo-gpt.github.io .
BuboGPT is a multi-modal LLM with visual grounding that enhances understanding and interaction between vision, audio, and language.
LLMs MiniGPT-4 LLaVA X-LLM multi-modal inputs visual grounding cross-modal interaction vision audio language fine-grained understanding visual objects SAM entities masks two-stage training scheme instruction dataset multi-modality understanding