Groma: Localized Visual Tokenization for Grounding Multimodal Large
  Language Models
We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded
and fine-grained visual perception ability. Beyond holistic image
understanding, Groma is adept at region-level tasks such as region captioning
and visual grounding. Such capabilities are built upon a localized visual
tokenization mechanism, where an image input is decomposed into regions of
interest and subsequently encoded into region tokens. By integrating region
tokens into user instructions and model responses, we seamlessly enable Groma
to understand user-specified region inputs and ground its textual output to
images. Besides, to enhance the grounded chat ability of Groma, we curate a
visually grounded instruction dataset by leveraging the powerful GPT-4V and
visual prompting techniques. Compared with MLLMs that rely on the language
model or external module for localization, Groma consistently demonstrates
superior performances in standard referring and grounding benchmarks,
highlighting the advantages of embedding localization into image tokenization.
Project page: https://groma-mllm.github.io/.
6625f042629de7157f849f25
Chuofan Ma
claimed_verified
2024-05-02T07:32:19.607Z
6620cc3ef44be495f133e86d
/avatars/b9f1dafaba803db74b16ad64cc57419e.svg
Chuofan Ma
user
machuofan
6625f042629de7157f849f26
Yi Jiang
claimed_verified
2024-04-22T09:19:21.357Z
6344dcb1cd37e44d9ed46508
https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
Yi Jiang
user
JiangYi
6625f042629de7157f849f27
Jiannan Wu
6625f042629de7157f849f28
Zehuan Yuan
admin_assigned
2024-04-22T14:01:41.640Z
661a80af3557013b638061d5
/avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
Zehuan Yuan
user
sweetrabor
6625f042629de7157f849f29
Xiaojuan Qi
claimed_verified
2025-07-15T19:15:19.425Z
6875266f9cd3191dfddc7071
/avatars/64c581910833b111e9a7bae5b8740229.svg
xiaojuan qi
user
xjqi
Groma: Localized Visual Tokenization for Grounding Multimodal Large
  Language Models
We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded
and fine-grained visual perception ability. Beyond holistic image
understanding, Groma is adept at region-level tasks such as region captioning
and visual grounding. Such capabilities are built upon a localized visual
tokenization mechanism, where an image input is decomposed into regions of
interest and subsequently encoded into region tokens. By integrating region
tokens into user instructions and model responses, we seamlessly enable Groma
to understand user-specified region inputs and ground its textual output to
images. Besides, to enhance the grounded chat ability of Groma, we curate a
visually grounded instruction dataset by leveraging the powerful GPT-4V and
visual prompting techniques. Compared with MLLMs that rely on the language
model or external module for localization, Groma consistently demonstrates
superior performances in standard referring and grounding benchmarks,
highlighting the advantages of embedding localization into image tokenization.
Project page: https://groma-mllm.github.io/.
Groma, a Multimodal Large Language Model, utilizes region-level visual tokenization to enhance visual understanding and localization accuracy in image-grounded tasks.
Multimodal Large Language Model MLLM region captioning visual grounding region tokens visually grounded instruction dataset GPT-4V visual prompting standard referring and grounding benchmarks