title: FINECAPTION: Compositional Image Captioning Focusing on Wherever You
    Want at Any Granularity
summary: The advent of large Vision-Language Models (VLMs) has significantly advanced
  multimodal tasks, enabling more sophisticated and accurate reasoning across
  various applications, including image and video captioning, visual question
  answering, and cross-modal retrieval. Despite their superior capabilities, VLMs
  struggle with fine-grained image regional composition information perception.
  Specifically, they have difficulty accurately aligning the segmentation masks
  with the corresponding semantics and precisely describing the compositional
  aspects of the referred regions.
    However, compositionality - the ability to understand and generate novel
  combinations of known visual and textual components - is critical for
  facilitating coherent reasoning and understanding across modalities by VLMs. To
  address this issue, we propose FINECAPTION, a novel VLM that can recognize
  arbitrary masks as referential inputs and process high-resolution images for
  compositional image captioning at different granularity levels. To support this
  endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region
  compositional image captioning, which introduces the task of compositional
  attribute-aware regional image captioning.
    Empirical results demonstrate the effectiveness of our proposed model
  compared to other state-of-the-art VLMs. Additionally, we analyze the
  capabilities of current VLMs in recognizing various visual prompts for
  compositional region image captioning, highlighting areas for improvement in
  VLM design and training.
paper_authors._id: 674689ccfe2f63e2200c520c
paper_authors.name: Hang Hua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:30:18.387Z
paper_authors._id: 639f8277beb95d698de007dd
paper_authors.avatarUrl: /avatars/57f223ccd9d3cb03166ccf0e41361c58.svg
paper_authors.fullname: HangHua
paper_authors.type: user
paper_authors.user: hhua2
paper_authors._id: 674689ccfe2f63e2200c520d
paper_authors.name: Qing Liu
paper_authors._id: 674689ccfe2f63e2200c520e
paper_authors.name: Lingzhi Zhang
paper_authors._id: 674689ccfe2f63e2200c520f
paper_authors.name: Jing Shi
paper_authors._id: 674689ccfe2f63e2200c5210
paper_authors.name: Zhifei Zhang
paper_authors._id: 674689ccfe2f63e2200c5211
paper_authors.name: Yilin Wang
paper_authors._id: 674689ccfe2f63e2200c5212
paper_authors.name: Jianming Zhang
paper_authors._id: 674689ccfe2f63e2200c5213
paper_authors.name: Jiebo Luo
paper_title: FINECAPTION: Compositional Image Captioning Focusing on Wherever You
    Want at Any Granularity
paper_summary: The advent of large Vision-Language Models (VLMs) has significantly advanced
  multimodal tasks, enabling more sophisticated and accurate reasoning across
  various applications, including image and video captioning, visual question
  answering, and cross-modal retrieval. Despite their superior capabilities, VLMs
  struggle with fine-grained image regional composition information perception.
  Specifically, they have difficulty accurately aligning the segmentation masks
  with the corresponding semantics and precisely describing the compositional
  aspects of the referred regions.
    However, compositionality - the ability to understand and generate novel
  combinations of known visual and textual components - is critical for
  facilitating coherent reasoning and understanding across modalities by VLMs. To
  address this issue, we propose FINECAPTION, a novel VLM that can recognize
  arbitrary masks as referential inputs and process high-resolution images for
  compositional image captioning at different granularity levels. To support this
  endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region
  compositional image captioning, which introduces the task of compositional
  attribute-aware regional image captioning.
    Empirical results demonstrate the effectiveness of our proposed model
  compared to other state-of-the-art VLMs. Additionally, we analyze the
  capabilities of current VLMs in recognizing various visual prompts for
  compositional region image captioning, highlighting areas for improvement in
  VLM design and training.
paper_ai_keywords: Vision-Language Models (VLMs) multimodal tasks image and video captioning visual question answering cross-modal retrieval fine-grained image regional composition information segmentation masks compositional aspects referential inputs compositional image captioning multi-grained region compositional image captioning compositional attribute-aware regional image captioning COMPOSITIONCAP empirical results current VLMs