title: List Items One by One: A New Data Source and Learning Paradigm for
    Multimodal LLMs
summary: Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of
  GPT-4V, by enabling the model to associate visual objects with tags inserted on
  the image. These tags, marked with alphanumerics, can be indexed via text
  tokens for easy reference. Despite the extraordinary performance from GPT-4V,
  we observe that other Multimodal Large Language Models (MLLMs) struggle to
  understand these visual tags. To promote the learning of SoM prompting for
  open-source models, we propose a new learning paradigm: "list items one by
  one," which asks the model to enumerate and describe all visual tags placed on
  the image following the alphanumeric orders of tags. By integrating our curated
  dataset with other visual instruction tuning datasets, we are able to equip
  existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our
  finetuned SoM models on five MLLM benchmarks. We find that this new dataset,
  even in a relatively small size (10k-30k images with tags), significantly
  enhances visual reasoning capabilities and reduces hallucinations for MLLMs.
  Perhaps surprisingly, these improvements persist even when the visual tags are
  omitted from input images during inference. This suggests the potential of
  "list items one by one" as a new paradigm for training MLLMs, which strengthens
  the object-text alignment through the use of visual tags in the training stage.
  Finally, we conduct analyses by probing trained models to understand the
  working mechanism of SoM. Our code and data are available at
  https://github.com/zzxslp/SoM-LLaVA.
paper_authors._id: 662b2d274149b1a3b0090f66
paper_authors.name: An Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:30:08.776Z
paper_authors._id: 634dfb9bbe5a827d48749f39
paper_authors.avatarUrl: /avatars/a8df0eeb1df39f3935dd686a94768f30.svg
paper_authors.fullname: An Yan
paper_authors.type: user
paper_authors.user: zzxslp
paper_authors._id: 662b2d274149b1a3b0090f67
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:16:42.867Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 662b2d274149b1a3b0090f68
paper_authors.name: Junda Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T08:32:12.372Z
paper_authors._id: 616c0cd640e2f69baa1c7adf
paper_authors.avatarUrl: /avatars/9ad6ada73b3fb2d5fb696ec91f8edb1b.svg
paper_authors.fullname: Junda Wu
paper_authors.type: user
paper_authors.user: JoshuaW1997
paper_authors._id: 662b2d274149b1a3b0090f69
paper_authors.name: Wanrong Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:17:31.249Z
paper_authors._id: 633a7e91c8eb7b089034fb3c
paper_authors.avatarUrl: /avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
paper_authors.fullname: Wanrong Zhu
paper_authors.type: user
paper_authors.user: VegB
paper_authors._id: 662b2d274149b1a3b0090f6a
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:18:00.719Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 662b2d274149b1a3b0090f6b
paper_authors.name: Linjie Li
paper_authors._id: 662b2d274149b1a3b0090f6c
paper_authors.name: Kevin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:18:42.974Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 662b2d274149b1a3b0090f6d
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:19:04.731Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 662b2d274149b1a3b0090f6e
paper_authors.name: Julian McAuley
paper_authors._id: 662b2d274149b1a3b0090f6f
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:19:16.215Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 662b2d274149b1a3b0090f70
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:19:22.481Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_title: List Items One by One: A New Data Source and Learning Paradigm for
    Multimodal LLMs
paper_summary: Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of
  GPT-4V, by enabling the model to associate visual objects with tags inserted on
  the image. These tags, marked with alphanumerics, can be indexed via text
  tokens for easy reference. Despite the extraordinary performance from GPT-4V,
  we observe that other Multimodal Large Language Models (MLLMs) struggle to
  understand these visual tags. To promote the learning of SoM prompting for
  open-source models, we propose a new learning paradigm: "list items one by
  one," which asks the model to enumerate and describe all visual tags placed on
  the image following the alphanumeric orders of tags. By integrating our curated
  dataset with other visual instruction tuning datasets, we are able to equip
  existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our
  finetuned SoM models on five MLLM benchmarks. We find that this new dataset,
  even in a relatively small size (10k-30k images with tags), significantly
  enhances visual reasoning capabilities and reduces hallucinations for MLLMs.
  Perhaps surprisingly, these improvements persist even when the visual tags are
  omitted from input images during inference. This suggests the potential of
  "list items one by one" as a new paradigm for training MLLMs, which strengthens
  the object-text alignment through the use of visual tags in the training stage.
  Finally, we conduct analyses by probing trained models to understand the
  working mechanism of SoM. Our code and data are available at
  https://github.com/zzxslp/SoM-LLaVA.
paper_ai_summary: A new paradigm, "list items one by one," enhances multimodal large language models' visual reasoning capability through the use of visual tags in training, even when these tags are omitted during inference.
paper_ai_keywords: GPT-4V Multimodal Large Language Models (MLLMs) SoM prompting list items one by one visual tags alphanumeric orders visual instruction tuning MLLM benchmarks visual reasoning capabilities hallucinations object-text alignment