title: MMCOMPOSITION: Revisiting the Compositionality of Pre-trained
    Vision-Language Models
summary: The advent of large Vision-Language Models (VLMs) has significantly advanced
  multimodal understanding, enabling more sophisticated and accurate integration
  of visual and textual information across various tasks, including image and
  video captioning, visual question answering, and cross-modal retrieval. Despite
  VLMs' superior capabilities, researchers lack a comprehensive understanding of
  their compositionality -- the ability to understand and produce novel
  combinations of known visual and textual components. Prior benchmarks provide
  only a relatively rough compositionality evaluation from the perspectives of
  objects, relations, and attributes while neglecting deeper reasoning about
  object interactions, counting, and complex compositions. However,
  compositionality is a critical ability that facilitates coherent reasoning and
  understanding across modalities for VLMs. To address this limitation, we
  propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively
  and accurately evaluating VLMs' compositionality. Our proposed benchmark serves
  as a complement to these earlier works. With MMCOMPOSITION, we can quantify and
  explore the compositionality of the mainstream VLMs. Surprisingly, we find
  GPT-4o's compositionality inferior to the best open-source model, and we
  analyze the underlying reasons. Our experimental analysis reveals the
  limitations of VLMs in fine-grained compositional perception and reasoning, and
  points to areas for improvement in VLM design and training. Resources available
  at: https://hanghuacs.github.io/MMComposition/
paper_authors._id: 670df88bd2e43d76064fbfd0
paper_authors.name: Hang Hua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T14:09:46.462Z
paper_authors._id: 639f8277beb95d698de007dd
paper_authors.avatarUrl: /avatars/57f223ccd9d3cb03166ccf0e41361c58.svg
paper_authors.fullname: HangHua
paper_authors.type: user
paper_authors.user: hhua2
paper_authors._id: 670df88bd2e43d76064fbfd1
paper_authors.name: Yunlong Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:43.559Z
paper_authors._id: 6344c87f0f69ad8aa61dfcf6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6344c87f0f69ad8aa61dfcf6/RJfMH9KaBIBubyk1TN5OM.jpeg
paper_authors.fullname: Yunlong Tang
paper_authors.type: user
paper_authors.user: yunlong10
paper_authors._id: 670df88bd2e43d76064fbfd2
paper_authors.name: Ziyun Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T08:07:48.782Z
paper_authors._id: 65e6a9a4c799256ec5f7ef77
paper_authors.avatarUrl: /avatars/539a455841675539a62b44ff9803a948.svg
paper_authors.fullname: ziyun zeng
paper_authors.type: user
paper_authors.user: zengziyun
paper_authors._id: 670df88bd2e43d76064fbfd3
paper_authors.name: Liangliang Cao
paper_authors._id: 670df88bd2e43d76064fbfd4
paper_authors.name: Zhengyuan Yang
paper_authors._id: 670df88bd2e43d76064fbfd5
paper_authors.name: Hangfeng He
paper_authors._id: 670df88bd2e43d76064fbfd6
paper_authors.name: Chenliang Xu
paper_authors._id: 670df88bd2e43d76064fbfd7
paper_authors.name: Jiebo Luo
paper_title: MMCOMPOSITION: Revisiting the Compositionality of Pre-trained
    Vision-Language Models
paper_summary: The advent of large Vision-Language Models (VLMs) has significantly advanced
  multimodal understanding, enabling more sophisticated and accurate integration
  of visual and textual information across various tasks, including image and
  video captioning, visual question answering, and cross-modal retrieval. Despite
  VLMs' superior capabilities, researchers lack a comprehensive understanding of
  their compositionality -- the ability to understand and produce novel
  combinations of known visual and textual components. Prior benchmarks provide
  only a relatively rough compositionality evaluation from the perspectives of
  objects, relations, and attributes while neglecting deeper reasoning about
  object interactions, counting, and complex compositions. However,
  compositionality is a critical ability that facilitates coherent reasoning and
  understanding across modalities for VLMs. To address this limitation, we
  propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively
  and accurately evaluating VLMs' compositionality. Our proposed benchmark serves
  as a complement to these earlier works. With MMCOMPOSITION, we can quantify and
  explore the compositionality of the mainstream VLMs. Surprisingly, we find
  GPT-4o's compositionality inferior to the best open-source model, and we
  analyze the underlying reasons. Our experimental analysis reveals the
  limitations of VLMs in fine-grained compositional perception and reasoning, and
  points to areas for improvement in VLM design and training. Resources available
  at: https://hanghuacs.github.io/MMComposition/
paper_ai_keywords: Vision-Language Models (VLMs) multimodal understanding image and video captioning visual question answering cross-modal retrieval compositionality objects relations attributes object interactions counting complex compositions MMCOMPOSITION fine-grained compositional perception reasoning