title: Groma: Localized Visual Tokenization for Grounding Multimodal Large
    Language Models
summary: We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded
  and fine-grained visual perception ability. Beyond holistic image
  understanding, Groma is adept at region-level tasks such as region captioning
  and visual grounding. Such capabilities are built upon a localized visual
  tokenization mechanism, where an image input is decomposed into regions of
  interest and subsequently encoded into region tokens. By integrating region
  tokens into user instructions and model responses, we seamlessly enable Groma
  to understand user-specified region inputs and ground its textual output to
  images. Besides, to enhance the grounded chat ability of Groma, we curate a
  visually grounded instruction dataset by leveraging the powerful GPT-4V and
  visual prompting techniques. Compared with MLLMs that rely on the language
  model or external module for localization, Groma consistently demonstrates
  superior performances in standard referring and grounding benchmarks,
  highlighting the advantages of embedding localization into image tokenization.
  Project page: https://groma-mllm.github.io/.
paper_authors._id: 6625f042629de7157f849f25
paper_authors.name: Chuofan Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T07:32:19.607Z
paper_authors._id: 6620cc3ef44be495f133e86d
paper_authors.avatarUrl: /avatars/b9f1dafaba803db74b16ad64cc57419e.svg
paper_authors.fullname: Chuofan Ma
paper_authors.type: user
paper_authors.user: machuofan
paper_authors._id: 6625f042629de7157f849f26
paper_authors.name: Yi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T09:19:21.357Z
paper_authors._id: 6344dcb1cd37e44d9ed46508
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
paper_authors.fullname: Yi Jiang
paper_authors.type: user
paper_authors.user: JiangYi
paper_authors._id: 6625f042629de7157f849f27
paper_authors.name: Jiannan Wu
paper_authors._id: 6625f042629de7157f849f28
paper_authors.name: Zehuan Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:01:41.640Z
paper_authors._id: 661a80af3557013b638061d5
paper_authors.avatarUrl: /avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
paper_authors.fullname: Zehuan Yuan
paper_authors.type: user
paper_authors.user: sweetrabor
paper_authors._id: 6625f042629de7157f849f29
paper_authors.name: Xiaojuan Qi
paper_title: Groma: Localized Visual Tokenization for Grounding Multimodal Large
    Language Models
paper_summary: We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded
  and fine-grained visual perception ability. Beyond holistic image
  understanding, Groma is adept at region-level tasks such as region captioning
  and visual grounding. Such capabilities are built upon a localized visual
  tokenization mechanism, where an image input is decomposed into regions of
  interest and subsequently encoded into region tokens. By integrating region
  tokens into user instructions and model responses, we seamlessly enable Groma
  to understand user-specified region inputs and ground its textual output to
  images. Besides, to enhance the grounded chat ability of Groma, we curate a
  visually grounded instruction dataset by leveraging the powerful GPT-4V and
  visual prompting techniques. Compared with MLLMs that rely on the language
  model or external module for localization, Groma consistently demonstrates
  superior performances in standard referring and grounding benchmarks,
  highlighting the advantages of embedding localization into image tokenization.
  Project page: https://groma-mllm.github.io/.
paper_ai_keywords: Multimodal Large Language Model (MLLM) grounded and fine-grained visual perception ability region-level tasks region captioning visual grounding localized visual tokenization mechanism region tokens timestep images visually grounded instruction dataset GPT-4V visual prompting techniques standard referring and grounding benchmarks image effectivation