GLaMM: Pixel Grounding Large Multimodal Model
Large Multimodal Models (LMMs) extend Large Language Models to the vision
domain. Initial efforts towards LMMs used holistic images and text prompts to
generate ungrounded textual responses. Very recently, region-level LMMs have
been used to generate visually grounded responses. However, they are limited to
only referring a single object category at a time, require users to specify the
regions in inputs, or cannot offer dense pixel-wise object grounding. In this
work, we present Grounding LMM (GLaMM), the first model that can generate
natural language responses seamlessly intertwined with corresponding object
segmentation masks. GLaMM not only grounds objects appearing in the
conversations but is flexible enough to accept both textual and optional visual
prompts (region of interest) as input. This empowers users to interact with the
model at various levels of granularity, both in textual and visual domains. Due
to the lack of standard benchmarks for the novel setting of generating visually
grounded detailed conversations, we introduce a comprehensive evaluation
protocol with our curated grounded conversations. Our proposed Grounded
Conversation Generation (GCG) task requires densely grounded concepts in
natural scenes at a large-scale. To this end, we propose a densely annotated
Grounding-anything Dataset (GranD) using our proposed automated annotation
pipeline that encompasses 7.5M unique concepts grounded in a total of 810M
regions available with segmentation masks. Besides GCG, GLaMM also performs
effectively on several downstream tasks e.g., referring expression
segmentation, image and region-level captioning and vision-language
conversations. Project Page: https://mbzuai-oryx.github.io/groundingLMM.
6549e5283766361a20c546b7
Hanoona Rasheed
admin_assigned
2023-11-07T14:48:43.698Z
64636b2551fa6e6306046293
https://cdn-avatars.huggingface.co/v1/production/uploads/64636b2551fa6e6306046293/Uuz6z2MZb_LKLGM8uxF9s.jpeg
Hanoona Rasheed
user
Hanoona
6549e5283766361a20c546b8
Muhammad Maaz
claimed_verified
2023-11-07T14:58:34.582Z
64807585856901b0edb8d68b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
Muhammad Maaz
user
mmaaz60
6549e5283766361a20c546b9
Sahal Shaji
admin_assigned
2023-11-07T14:49:35.305Z
62e23c7f555a866437a53cd0
https://cdn-avatars.huggingface.co/v1/production/uploads/62e23c7f555a866437a53cd0/UaAsYZQXuwb4NSG5WnvdG.jpeg
Sahal Shaji
user
sahalshajim
6549e5283766361a20c546ba
Abdelrahman Shaker
6549e5283766361a20c546bb
Salman Khan
6549e5283766361a20c546bc
Hisham Cholakkal
6549e5283766361a20c546bd
Rao M. Anwer
6549e5283766361a20c546be
Erix Xing
6549e5283766361a20c546bf
Ming-Hsuan Yang
6549e5283766361a20c546c0
Fahad S. Khan
GLaMM: Pixel Grounding Large Multimodal Model
Large Multimodal Models (LMMs) extend Large Language Models to the vision
domain. Initial efforts towards LMMs used holistic images and text prompts to
generate ungrounded textual responses. Very recently, region-level LMMs have
been used to generate visually grounded responses. However, they are limited to
only referring a single object category at a time, require users to specify the
regions in inputs, or cannot offer dense pixel-wise object grounding. In this
work, we present Grounding LMM (GLaMM), the first model that can generate
natural language responses seamlessly intertwined with corresponding object
segmentation masks. GLaMM not only grounds objects appearing in the
conversations but is flexible enough to accept both textual and optional visual
prompts (region of interest) as input. This empowers users to interact with the
model at various levels of granularity, both in textual and visual domains. Due
to the lack of standard benchmarks for the novel setting of generating visually
grounded detailed conversations, we introduce a comprehensive evaluation
protocol with our curated grounded conversations. Our proposed Grounded
Conversation Generation (GCG) task requires densely grounded concepts in
natural scenes at a large-scale. To this end, we propose a densely annotated
Grounding-anything Dataset (GranD) using our proposed automated annotation
pipeline that encompasses 7.5M unique concepts grounded in a total of 810M
regions available with segmentation masks. Besides GCG, GLaMM also performs
effectively on several downstream tasks e.g., referring expression
segmentation, image and region-level captioning and vision-language
conversations. Project Page: https://mbzuai-oryx.github.io/groundingLMM.
GLaMM is a multimodal model that generates visually grounded language responses with object segmentation masks from both text and optional visual prompts.
Large Multimodal Models Large Language Models region-level LMMs visually grounded responses object segmentation masks Grounding LMM Grounded Conversation Generation Grounding-anything Dataset automated annotation pipeline referring expression segmentation image and region-level captioning vision-language conversations