title: Pixel Aligned Language Models
summary: Large language models have achieved great success in recent years, so as
  their variants in vision. Existing vision-language models can describe images
  in natural languages, answer visual-related questions, or perform complex
  reasoning about the image. However, it is yet unclear how localization tasks,
  such as word grounding or referring localization, can be performed using large
  language models. In this work, we aim to develop a vision-language model that
  can take locations, for example, a set of points or boxes, as either inputs or
  outputs. When taking locations as inputs, the model performs
  location-conditioned captioning, which generates captions for the indicated
  object or region. When generating locations as outputs, our model regresses
  pixel coordinates for each output word generated by the language model, and
  thus performs dense word grounding. Our model is pre-trained on the Localized
  Narrative dataset, which contains pixel-word-aligned captioning from human
  attention. We show our model can be applied to various location-aware
  vision-language tasks, including referring localization, location-conditioned
  captioning, and dense object captioning, archiving state-of-the-art performance
  on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .
paper_authors._id: 657bce5e79cf6182f036d94b
paper_authors.name: Jiarui Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:07:21.342Z
paper_authors._id: 61f7671a481b345bd1059092
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643603726045-noauth.jpeg
paper_authors.fullname: Jiarui Xu
paper_authors.type: user
paper_authors.user: xvjiarui
paper_authors._id: 657bce5e79cf6182f036d94c
paper_authors.name: Xingyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:23:53.598Z
paper_authors._id: 629ee11846b4826be2d94676
paper_authors.avatarUrl: /avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
paper_authors.fullname: Xingyi Zhou
paper_authors.type: user
paper_authors.user: zhouxy
paper_authors._id: 657bce5e79cf6182f036d94d
paper_authors.name: Shen Yan
paper_authors._id: 657bce5e79cf6182f036d94e
paper_authors.name: Xiuye Gu
paper_authors._id: 657bce5e79cf6182f036d94f
paper_authors.name: Anurag Arnab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:24:20.247Z
paper_authors._id: 63d7f0466ac3104e50c5311e
paper_authors.avatarUrl: /avatars/2421bfb7747664d1acefbac1c00c3475.svg
paper_authors.fullname: Anurag Arnab
paper_authors.type: user
paper_authors.user: anurag-arnab
paper_authors._id: 657bce5e79cf6182f036d950
paper_authors.name: Chen Sun
paper_authors._id: 657bce5e79cf6182f036d951
paper_authors.name: Xiaolong Wang
paper_authors._id: 657bce5e79cf6182f036d952
paper_authors.name: Cordelia Schmid
paper_title: Pixel Aligned Language Models
paper_summary: Large language models have achieved great success in recent years, so as
  their variants in vision. Existing vision-language models can describe images
  in natural languages, answer visual-related questions, or perform complex
  reasoning about the image. However, it is yet unclear how localization tasks,
  such as word grounding or referring localization, can be performed using large
  language models. In this work, we aim to develop a vision-language model that
  can take locations, for example, a set of points or boxes, as either inputs or
  outputs. When taking locations as inputs, the model performs
  location-conditioned captioning, which generates captions for the indicated
  object or region. When generating locations as outputs, our model regresses
  pixel coordinates for each output word generated by the language model, and
  thus performs dense word grounding. Our model is pre-trained on the Localized
  Narrative dataset, which contains pixel-word-aligned captioning from human
  attention. We show our model can be applied to various location-aware
  vision-language tasks, including referring localization, location-conditioned
  captioning, and dense object captioning, archiving state-of-the-art performance
  on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .
paper_ai_summary: The developed vision-language model can perform location-aware tasks by handling inputs and outputs as pixel coordinates or locations, achieving state-of-the-art performance on several benchmarks.
paper_ai_keywords: vision-language models location-conditioned captioning dense word grounding pixel coordinates local narrative dataset referring localization dense object captioning RefCOCO Visual Genome