MMCOMPOSITION: Revisiting the Compositionality of Pre-trained
  Vision-Language Models
The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal understanding, enabling more sophisticated and accurate integration
of visual and textual information across various tasks, including image and
video captioning, visual question answering, and cross-modal retrieval. Despite
VLMs' superior capabilities, researchers lack a comprehensive understanding of
their compositionality -- the ability to understand and produce novel
combinations of known visual and textual components. Prior benchmarks provide
only a relatively rough compositionality evaluation from the perspectives of
objects, relations, and attributes while neglecting deeper reasoning about
object interactions, counting, and complex compositions. However,
compositionality is a critical ability that facilitates coherent reasoning and
understanding across modalities for VLMs. To address this limitation, we
propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively
and accurately evaluating VLMs' compositionality. Our proposed benchmark serves
as a complement to these earlier works. With MMCOMPOSITION, we can quantify and
explore the compositionality of the mainstream VLMs. Surprisingly, we find
GPT-4o's compositionality inferior to the best open-source model, and we
analyze the underlying reasons. Our experimental analysis reveals the
limitations of VLMs in fine-grained compositional perception and reasoning, and
points to areas for improvement in VLM design and training. Resources available
at: https://hanghuacs.github.io/MMComposition/
670df88bd2e43d76064fbfd0
Hang Hua
claimed_verified
2024-10-15T14:09:46.462Z
639f8277beb95d698de007dd
/avatars/57f223ccd9d3cb03166ccf0e41361c58.svg
HangHua
user
hhua2
670df88bd2e43d76064fbfd1
Yunlong Tang
claimed_verified
2024-10-21T07:48:43.559Z
6344c87f0f69ad8aa61dfcf6
https://cdn-avatars.huggingface.co/v1/production/uploads/6344c87f0f69ad8aa61dfcf6/m4QiQ_4c6GfOIZZFICy7T.jpeg
Yolo Y. Tang
user
yunlong10
670df88bd2e43d76064fbfd2
Ziyun Zeng
claimed_verified
2024-10-15T08:07:48.782Z
65e6a9a4c799256ec5f7ef77
/avatars/539a455841675539a62b44ff9803a948.svg
ziyun zeng
user
zengziyun
670df88bd2e43d76064fbfd3
Liangliang Cao
670df88bd2e43d76064fbfd4
Zhengyuan Yang
670df88bd2e43d76064fbfd5
Hangfeng He
670df88bd2e43d76064fbfd6
Chenliang Xu
670df88bd2e43d76064fbfd7
Jiebo Luo
MMCOMPOSITION: Revisiting the Compositionality of Pre-trained
  Vision-Language Models
The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal understanding, enabling more sophisticated and accurate integration
of visual and textual information across various tasks, including image and
video captioning, visual question answering, and cross-modal retrieval. Despite
VLMs' superior capabilities, researchers lack a comprehensive understanding of
their compositionality -- the ability to understand and produce novel
combinations of known visual and textual components. Prior benchmarks provide
only a relatively rough compositionality evaluation from the perspectives of
objects, relations, and attributes while neglecting deeper reasoning about
object interactions, counting, and complex compositions. However,
compositionality is a critical ability that facilitates coherent reasoning and
understanding across modalities for VLMs. To address this limitation, we
propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively
and accurately evaluating VLMs' compositionality. Our proposed benchmark serves
as a complement to these earlier works. With MMCOMPOSITION, we can quantify and
explore the compositionality of the mainstream VLMs. Surprisingly, we find
GPT-4o's compositionality inferior to the best open-source model, and we
analyze the underlying reasons. Our experimental analysis reveals the
limitations of VLMs in fine-grained compositional perception and reasoning, and
points to areas for improvement in VLM design and training. Resources available
at: https://hanghuacs.github.io/MMComposition/
A benchmark named MMCOMPOSITION evaluates the compositionality of Vision-Language Models, revealing their limitations in handling complex images and images with fine-grained interactions.
Vision-Language Models multimodal understanding image and video captioning visual question answering cross-modal retrieval compositionality objects relations attributes object interactions counting complex compositions GPT-4o open-source model fine-grained compositional perception