title: LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
summary: With the recent significant advancements in large multi-modal models (LMMs),
  the importance of their grounding capability in visual chat is increasingly
  recognized. Despite recent efforts to enable LMMs to support grounding, their
  capabilities for grounding and chat are usually separate, and their chat
  performance drops dramatically when asked to ground. The problem is the lack of
  a dataset for grounded visual chat (GVC). Existing grounding datasets only
  contain short captions. To address this issue, we have created GVC data that
  allows for the combination of grounding and chat capabilities. To better
  evaluate the GVC capabilities, we have introduced a benchmark called
  Grounding-Bench. Additionally, we have proposed a model design that can support
  GVC and various types of visual prompts by connecting segmentation models with
  language models. Experimental results demonstrate that our model outperforms
  other LMMs on Grounding-Bench. Furthermore, our model achieves competitive
  performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K
  Entities. Our code will be released at
  https://github.com/UX-Decoder/LLaVA-Grounding .
paper_authors._id: 656fd72a15709277eddaf527
paper_authors.name: Hao Zhang
paper_authors._id: 656fd72a15709277eddaf528
paper_authors.name: Hongyang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:08:17.125Z
paper_authors._id: 6499b0184936457997180c90
paper_authors.avatarUrl: /avatars/b8be7bfabf746639e30330f5f623f560.svg
paper_authors.fullname: Hongyang Li
paper_authors.type: user
paper_authors.user: compileme
paper_authors._id: 656fd72a15709277eddaf529
paper_authors.name: Feng Li
paper_authors._id: 656fd72a15709277eddaf52a
paper_authors.name: Tianhe Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:09:03.794Z
paper_authors._id: 6357ba47d3ec8a8aca6ea254
paper_authors.avatarUrl: /avatars/2deb725505aa510a712d91bd0b6b1967.svg
paper_authors.fullname: Ren Tianhe
paper_authors.type: user
paper_authors.user: rentianhe
paper_authors._id: 656fd72a15709277eddaf52b
paper_authors.name: Xueyan Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:09:11.647Z
paper_authors._id: 62520988818a5dc29ab91d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
paper_authors.fullname: Xueyan Zou
paper_authors.type: user
paper_authors.user: xueyanz
paper_authors._id: 656fd72a15709277eddaf52c
paper_authors.name: Shilong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:09:18.215Z
paper_authors._id: 638b13c0c1d591879698f4e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
paper_authors.fullname: Shilong Liu
paper_authors.type: user
paper_authors.user: ShilongLiu
paper_authors._id: 656fd72a15709277eddaf52d
paper_authors.name: Shijia Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:09:25.588Z
paper_authors._id: 65204fdf377bffdc59602a8d
paper_authors.avatarUrl: /avatars/02d200f856f03696b183fd0b59e9f909.svg
paper_authors.fullname: Shijia Huang
paper_authors.type: user
paper_authors.user: slvjul
paper_authors._id: 656fd72a15709277eddaf52e
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:09:31.897Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 656fd72a15709277eddaf52f
paper_authors.name: Lei Zhang
paper_authors._id: 656fd72a15709277eddaf530
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:09:55.385Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 656fd72a15709277eddaf531
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:10:11.208Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_title: LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
paper_summary: With the recent significant advancements in large multi-modal models (LMMs),
  the importance of their grounding capability in visual chat is increasingly
  recognized. Despite recent efforts to enable LMMs to support grounding, their
  capabilities for grounding and chat are usually separate, and their chat
  performance drops dramatically when asked to ground. The problem is the lack of
  a dataset for grounded visual chat (GVC). Existing grounding datasets only
  contain short captions. To address this issue, we have created GVC data that
  allows for the combination of grounding and chat capabilities. To better
  evaluate the GVC capabilities, we have introduced a benchmark called
  Grounding-Bench. Additionally, we have proposed a model design that can support
  GVC and various types of visual prompts by connecting segmentation models with
  language models. Experimental results demonstrate that our model outperforms
  other LMMs on Grounding-Bench. Furthermore, our model achieves competitive
  performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K
  Entities. Our code will be released at
  https://github.com/UX-Decoder/LLaVA-Grounding .
paper_ai_summary: A new dataset and benchmark for grounded visual chat improve LMMs performance by integrating segmentation and language models.
paper_ai_keywords: multi-modal models large multi-modal models grounded visual chat segmentation models language models Grounding-Bench RefCOCO/+/g Flickr30K Entities