Pixel Aligned Language Models
Large language models have achieved great success in recent years, so as
their variants in vision. Existing vision-language models can describe images
in natural languages, answer visual-related questions, or perform complex
reasoning about the image. However, it is yet unclear how localization tasks,
such as word grounding or referring localization, can be performed using large
language models. In this work, we aim to develop a vision-language model that
can take locations, for example, a set of points or boxes, as either inputs or
outputs. When taking locations as inputs, the model performs
location-conditioned captioning, which generates captions for the indicated
object or region. When generating locations as outputs, our model regresses
pixel coordinates for each output word generated by the language model, and
thus performs dense word grounding. Our model is pre-trained on the Localized
Narrative dataset, which contains pixel-word-aligned captioning from human
attention. We show our model can be applied to various location-aware
vision-language tasks, including referring localization, location-conditioned
captioning, and dense object captioning, archiving state-of-the-art performance
on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .
657bce5e79cf6182f036d94b
Jiarui Xu
claimed_verified
2024-06-09T20:07:21.342Z
61f7671a481b345bd1059092
https://cdn-avatars.huggingface.co/v1/production/uploads/1643603726045-noauth.jpeg
Jiarui Xu
user
xvjiarui
657bce5e79cf6182f036d94c
Xingyi Zhou
admin_assigned
2023-12-15T13:23:53.598Z
629ee11846b4826be2d94676
/avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
Xingyi Zhou
user
zhouxy
657bce5e79cf6182f036d94d
Shen Yan
657bce5e79cf6182f036d94e
Xiuye Gu
657bce5e79cf6182f036d94f
Anurag Arnab
admin_assigned
2023-12-15T13:24:20.247Z
63d7f0466ac3104e50c5311e
/avatars/2421bfb7747664d1acefbac1c00c3475.svg
Anurag Arnab
user
anurag-arnab
657bce5e79cf6182f036d950
Chen Sun
657bce5e79cf6182f036d951
Xiaolong Wang
657bce5e79cf6182f036d952
Cordelia Schmid
Pixel Aligned Language Models
Large language models have achieved great success in recent years, so as
their variants in vision. Existing vision-language models can describe images
in natural languages, answer visual-related questions, or perform complex
reasoning about the image. However, it is yet unclear how localization tasks,
such as word grounding or referring localization, can be performed using large
language models. In this work, we aim to develop a vision-language model that
can take locations, for example, a set of points or boxes, as either inputs or
outputs. When taking locations as inputs, the model performs
location-conditioned captioning, which generates captions for the indicated
object or region. When generating locations as outputs, our model regresses
pixel coordinates for each output word generated by the language model, and
thus performs dense word grounding. Our model is pre-trained on the Localized
Narrative dataset, which contains pixel-word-aligned captioning from human
attention. We show our model can be applied to various location-aware
vision-language tasks, including referring localization, location-conditioned
captioning, and dense object captioning, archiving state-of-the-art performance
on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .
The developed vision-language model can perform location-aware tasks by handling inputs and outputs as pixel coordinates or locations, achieving state-of-the-art performance on several benchmarks.
vision-language models location-conditioned captioning dense word grounding pixel coordinates local narrative dataset referring localization dense object captioning RefCOCO Visual Genome