title: Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
summary: We present Set-of-Mark (SoM), a new visual prompting method, to unleash the
  visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
  As illustrated in Fig. 1 (right), we employ off-the-shelf interactive
  segmentation models, such as SAM, to partition an image into regions at
  different levels of granularity, and overlay these regions with a set of marks
  e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can
  answer the questions that require visual grounding. We perform a comprehensive
  empirical study to validate the effectiveness of SoM on a wide range of
  fine-grained vision and multimodal tasks. For example, our experiments show
  that GPT-4V with SoM outperforms the state-of-the-art fully-finetuned referring
  segmentation model on RefCOCOg in a zero-shot setting.
paper_authors._id: 652f476f660516114942f5f7
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:09:39.700Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 652f476f660516114942f5f8
paper_authors.name: Hao Zhang
paper_authors._id: 652f476f660516114942f5f9
paper_authors.name: Feng Li
paper_authors._id: 652f476f660516114942f5fa
paper_authors.name: Xueyan Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:11:24.220Z
paper_authors._id: 62520988818a5dc29ab91d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
paper_authors.fullname: Xueyan Zou
paper_authors.type: user
paper_authors.user: xueyanz
paper_authors._id: 652f476f660516114942f5fb
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:11:31.483Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 652f476f660516114942f5fc
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:11:38.024Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_title: Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
paper_summary: We present Set-of-Mark (SoM), a new visual prompting method, to unleash the
  visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
  As illustrated in Fig. 1 (right), we employ off-the-shelf interactive
  segmentation models, such as SAM, to partition an image into regions at
  different levels of granularity, and overlay these regions with a set of marks
  e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can
  answer the questions that require visual grounding. We perform a comprehensive
  empirical study to validate the effectiveness of SoM on a wide range of
  fine-grained vision and multimodal tasks. For example, our experiments show
  that GPT-4V with SoM outperforms the state-of-the-art fully-finetuned referring
  segmentation model on RefCOCOg in a zero-shot setting.
paper_ai_keywords: Set-of-Mark (SoM) visual prompting method visual grounding abilities large multimodal models (LMMs) GPT-4V off-the-shelf interactive segmentation models SAM (Segment Anything Model) granularity referring segmentation model RefCOCOg zero-shot setting