title: GLaMM: Pixel Grounding Large Multimodal Model
summary: Large Multimodal Models (LMMs) extend Large Language Models to the vision
  domain. Initial efforts towards LMMs used holistic images and text prompts to
  generate ungrounded textual responses. Very recently, region-level LMMs have
  been used to generate visually grounded responses. However, they are limited to
  only referring a single object category at a time, require users to specify the
  regions in inputs, or cannot offer dense pixel-wise object grounding. In this
  work, we present Grounding LMM (GLaMM), the first model that can generate
  natural language responses seamlessly intertwined with corresponding object
  segmentation masks. GLaMM not only grounds objects appearing in the
  conversations but is flexible enough to accept both textual and optional visual
  prompts (region of interest) as input. This empowers users to interact with the
  model at various levels of granularity, both in textual and visual domains. Due
  to the lack of standard benchmarks for the novel setting of generating visually
  grounded detailed conversations, we introduce a comprehensive evaluation
  protocol with our curated grounded conversations. Our proposed Grounded
  Conversation Generation (GCG) task requires densely grounded concepts in
  natural scenes at a large-scale. To this end, we propose a densely annotated
  Grounding-anything Dataset (GranD) using our proposed automated annotation
  pipeline that encompasses 7.5M unique concepts grounded in a total of 810M
  regions available with segmentation masks. Besides GCG, GLaMM also performs
  effectively on several downstream tasks e.g., referring expression
  segmentation, image and region-level captioning and vision-language
  conversations. Project Page: https://mbzuai-oryx.github.io/groundingLMM.
paper_authors._id: 6549e5283766361a20c546b7
paper_authors.name: Hanoona Rasheed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:48:43.698Z
paper_authors._id: 64636b2551fa6e6306046293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64636b2551fa6e6306046293/Uuz6z2MZb_LKLGM8uxF9s.jpeg
paper_authors.fullname: Hanoona Rasheed
paper_authors.type: user
paper_authors.user: Hanoona
paper_authors._id: 6549e5283766361a20c546b8
paper_authors.name: Muhammad Maaz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-07T14:58:34.582Z
paper_authors._id: 64807585856901b0edb8d68b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
paper_authors.fullname: Muhammad Maaz
paper_authors.type: user
paper_authors.user: mmaaz60
paper_authors._id: 6549e5283766361a20c546b9
paper_authors.name: Sahal Shaji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:49:35.305Z
paper_authors._id: 62e23c7f555a866437a53cd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e23c7f555a866437a53cd0/UaAsYZQXuwb4NSG5WnvdG.jpeg
paper_authors.fullname: Sahal Shaji
paper_authors.type: user
paper_authors.user: sahalshajim
paper_authors._id: 6549e5283766361a20c546ba
paper_authors.name: Abdelrahman Shaker
paper_authors._id: 6549e5283766361a20c546bb
paper_authors.name: Salman Khan
paper_authors._id: 6549e5283766361a20c546bc
paper_authors.name: Hisham Cholakkal
paper_authors._id: 6549e5283766361a20c546bd
paper_authors.name: Rao M. Anwer
paper_authors._id: 6549e5283766361a20c546be
paper_authors.name: Erix Xing
paper_authors._id: 6549e5283766361a20c546bf
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 6549e5283766361a20c546c0
paper_authors.name: Fahad S. Khan
paper_title: GLaMM: Pixel Grounding Large Multimodal Model
paper_summary: Large Multimodal Models (LMMs) extend Large Language Models to the vision
  domain. Initial efforts towards LMMs used holistic images and text prompts to
  generate ungrounded textual responses. Very recently, region-level LMMs have
  been used to generate visually grounded responses. However, they are limited to
  only referring a single object category at a time, require users to specify the
  regions in inputs, or cannot offer dense pixel-wise object grounding. In this
  work, we present Grounding LMM (GLaMM), the first model that can generate
  natural language responses seamlessly intertwined with corresponding object
  segmentation masks. GLaMM not only grounds objects appearing in the
  conversations but is flexible enough to accept both textual and optional visual
  prompts (region of interest) as input. This empowers users to interact with the
  model at various levels of granularity, both in textual and visual domains. Due
  to the lack of standard benchmarks for the novel setting of generating visually
  grounded detailed conversations, we introduce a comprehensive evaluation
  protocol with our curated grounded conversations. Our proposed Grounded
  Conversation Generation (GCG) task requires densely grounded concepts in
  natural scenes at a large-scale. To this end, we propose a densely annotated
  Grounding-anything Dataset (GranD) using our proposed automated annotation
  pipeline that encompasses 7.5M unique concepts grounded in a total of 810M
  regions available with segmentation masks. Besides GCG, GLaMM also performs
  effectively on several downstream tasks e.g., referring expression
  segmentation, image and region-level captioning and vision-language
  conversations. Project Page: https://mbzuai-oryx.github.io/groundingLMM.
paper_ai_summary: GLaMM is a multimodal model that generates visually grounded language responses with object segmentation masks from both text and optional visual prompts.
paper_ai_keywords: Large Multimodal Models Large Language Models region-level LMMs visually grounded responses object segmentation masks Grounding LMM Grounded Conversation Generation Grounding-anything Dataset automated annotation pipeline referring expression segmentation image and region-level captioning vision-language conversations