Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
We present Set-of-Mark (SoM), a new visual prompting method, to unleash the
visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
As illustrated in Fig. 1 (right), we employ off-the-shelf interactive
segmentation models, such as SAM, to partition an image into regions at
different levels of granularity, and overlay these regions with a set of marks
e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can
answer the questions that require visual grounding. We perform a comprehensive
empirical study to validate the effectiveness of SoM on a wide range of
fine-grained vision and multimodal tasks. For example, our experiments show
that GPT-4V with SoM outperforms the state-of-the-art fully-finetuned referring
segmentation model on RefCOCOg in a zero-shot setting.
652f476f660516114942f5f7
Jianwei Yang
admin_assigned
2023-10-18T09:09:39.700Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
652f476f660516114942f5f8
Hao Zhang
652f476f660516114942f5f9
Feng Li
652f476f660516114942f5fa
Xueyan Zou
admin_assigned
2023-10-18T09:11:24.220Z
62520988818a5dc29ab91d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
Xueyan Zou
user
xueyanz
652f476f660516114942f5fb
Chunyuan Li
admin_assigned
2023-10-18T09:11:31.483Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
652f476f660516114942f5fc
Jianfeng Gao
admin_assigned
2023-10-18T09:11:38.024Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
We present Set-of-Mark (SoM), a new visual prompting method, to unleash the
visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
As illustrated in Fig. 1 (right), we employ off-the-shelf interactive
segmentation models, such as SAM, to partition an image into regions at
different levels of granularity, and overlay these regions with a set of marks
e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can
answer the questions that require visual grounding. We perform a comprehensive
empirical study to validate the effectiveness of SoM on a wide range of
fine-grained vision and multimodal tasks. For example, our experiments show
that GPT-4V with SoM outperforms the state-of-the-art fully-finetuned referring
segmentation model on RefCOCOg in a zero-shot setting.
Set-of-Mark (SoM) enhances visual grounding in large multimodal models using segmented and marked images, demonstrating superior performance on fine-grained vision tasks without full fine-tuning.
Set-of-Mark (SoM) visual grounding large multimodal models (LMMs) GPT-4V SAM interactive segmentation models marking referring segmentation RefCOCOg zero-shot