LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
With the recent significant advancements in large multi-modal models (LMMs),
the importance of their grounding capability in visual chat is increasingly
recognized. Despite recent efforts to enable LMMs to support grounding, their
capabilities for grounding and chat are usually separate, and their chat
performance drops dramatically when asked to ground. The problem is the lack of
a dataset for grounded visual chat (GVC). Existing grounding datasets only
contain short captions. To address this issue, we have created GVC data that
allows for the combination of grounding and chat capabilities. To better
evaluate the GVC capabilities, we have introduced a benchmark called
Grounding-Bench. Additionally, we have proposed a model design that can support
GVC and various types of visual prompts by connecting segmentation models with
language models. Experimental results demonstrate that our model outperforms
other LMMs on Grounding-Bench. Furthermore, our model achieves competitive
performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K
Entities. Our code will be released at
https://github.com/UX-Decoder/LLaVA-Grounding .
656fd72a15709277eddaf527
Hao Zhang
656fd72a15709277eddaf528
Hongyang Li
admin_assigned
2023-12-06T13:08:17.125Z
6499b0184936457997180c90
/avatars/b8be7bfabf746639e30330f5f623f560.svg
Hongyang Li
user
compileme
656fd72a15709277eddaf529
Feng Li
656fd72a15709277eddaf52a
Tianhe Ren
admin_assigned
2023-12-06T13:09:03.794Z
6357ba47d3ec8a8aca6ea254
/avatars/2deb725505aa510a712d91bd0b6b1967.svg
Ren Tianhe
user
rentianhe
656fd72a15709277eddaf52b
Xueyan Zou
admin_assigned
2023-12-06T13:09:11.647Z
62520988818a5dc29ab91d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
Xueyan Zou
user
xueyanz
656fd72a15709277eddaf52c
Shilong Liu
admin_assigned
2023-12-06T13:09:18.215Z
638b13c0c1d591879698f4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
Shilong Liu
user
ShilongLiu
656fd72a15709277eddaf52d
Shijia Huang
admin_assigned
2023-12-06T13:09:25.588Z
65204fdf377bffdc59602a8d
/avatars/02d200f856f03696b183fd0b59e9f909.svg
Shijia Huang
user
slvjul
656fd72a15709277eddaf52e
Jianfeng Gao
admin_assigned
2023-12-06T13:09:31.897Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
656fd72a15709277eddaf52f
Lei Zhang
656fd72a15709277eddaf530
Chunyuan Li
admin_assigned
2023-12-06T13:09:55.385Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
656fd72a15709277eddaf531
Jianwei Yang
admin_assigned
2023-12-06T13:10:11.208Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
With the recent significant advancements in large multi-modal models (LMMs),
the importance of their grounding capability in visual chat is increasingly
recognized. Despite recent efforts to enable LMMs to support grounding, their
capabilities for grounding and chat are usually separate, and their chat
performance drops dramatically when asked to ground. The problem is the lack of
a dataset for grounded visual chat (GVC). Existing grounding datasets only
contain short captions. To address this issue, we have created GVC data that
allows for the combination of grounding and chat capabilities. To better
evaluate the GVC capabilities, we have introduced a benchmark called
Grounding-Bench. Additionally, we have proposed a model design that can support
GVC and various types of visual prompts by connecting segmentation models with
language models. Experimental results demonstrate that our model outperforms
other LMMs on Grounding-Bench. Furthermore, our model achieves competitive
performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K
Entities. Our code will be released at
https://github.com/UX-Decoder/LLaVA-Grounding .
A new dataset and benchmark for grounded visual chat improve LMMs performance by integrating segmentation and language models.
multi-modal models large multi-modal models grounded visual chat segmentation models language models Grounding-Bench RefCOCO/+/g Flickr30K Entities