CompCap: Improving Multimodal Large Language Models with Composite
  Captions
How well can Multimodal Large Language Models (MLLMs) understand composite
images? Composite images (CIs) are synthetic visuals created by merging
multiple visual elements, such as charts, posters, or screenshots, rather than
being captured directly by a camera. While CIs are prevalent in real-world
applications, recent MLLM developments have primarily focused on interpreting
natural images (NIs). Our research reveals that current MLLMs face significant
challenges in accurately understanding CIs, often struggling to extract
information or perform complex reasoning based on these images. We find that
existing training data for CIs are mostly formatted for question-answer tasks
(e.g., in datasets like ChartQA and ScienceQA), while high-quality
image-caption datasets, critical for robust vision-language alignment, are only
available for NIs. To bridge this gap, we introduce Composite Captions
(CompCap), a flexible framework that leverages Large Language Models (LLMs) and
automation tools to synthesize CIs with accurate and detailed captions. Using
CompCap, we curate CompCap-118K, a dataset containing 118K image-caption pairs
across six CI types. We validate the effectiveness of CompCap-118K by
supervised fine-tuning MLLMs of three sizes: xGen-MM-inst.-4B and
LLaVA-NeXT-Vicuna-7B/13B. Empirical results show that CompCap-118K
significantly enhances MLLMs' understanding of CIs, yielding average gains of
1.7%, 2.0%, and 2.9% across eleven benchmarks, respectively.
67570c25bb1d50918a3c8c03
Xiaohui Chen
claimed_verified
2024-12-11T09:20:54.990Z
648015c6b82adfa7cc1a9565
/avatars/e630e72a9d80a4d73bbf979133a0e07a.svg
Xiaohui Chen
user
xchen16
67570c25bb1d50918a3c8c04
Satya Narayan Shukla
67570c25bb1d50918a3c8c05
Mahmoud Azab
claimed_verified
2024-12-10T10:25:41.245Z
64c05582d71cdcdcc80b7658
/avatars/8ab4e2b0a70f657556e8e09ece1c496a.svg
Mahmoud Azab
user
azab
67570c25bb1d50918a3c8c06
Aashu Singh
67570c25bb1d50918a3c8c07
Qifan Wang
67570c25bb1d50918a3c8c08
David Yang
67570c25bb1d50918a3c8c09
ShengYun Peng
67570c25bb1d50918a3c8c0a
Hanchao Yu
67570c25bb1d50918a3c8c0b
Shen Yan
67570c25bb1d50918a3c8c0c
Xuewen Zhang
67570c25bb1d50918a3c8c0d
Baosheng He
CompCap: Improving Multimodal Large Language Models with Composite
  Captions
How well can Multimodal Large Language Models (MLLMs) understand composite
images? Composite images (CIs) are synthetic visuals created by merging
multiple visual elements, such as charts, posters, or screenshots, rather than
being captured directly by a camera. While CIs are prevalent in real-world
applications, recent MLLM developments have primarily focused on interpreting
natural images (NIs). Our research reveals that current MLLMs face significant
challenges in accurately understanding CIs, often struggling to extract
information or perform complex reasoning based on these images. We find that
existing training data for CIs are mostly formatted for question-answer tasks
(e.g., in datasets like ChartQA and ScienceQA), while high-quality
image-caption datasets, critical for robust vision-language alignment, are only
available for NIs. To bridge this gap, we introduce Composite Captions
(CompCap), a flexible framework that leverages Large Language Models (LLMs) and
automation tools to synthesize CIs with accurate and detailed captions. Using
CompCap, we curate CompCap-118K, a dataset containing 118K image-caption pairs
across six CI types. We validate the effectiveness of CompCap-118K by
supervised fine-tuning MLLMs of three sizes: xGen-MM-inst.-4B and
LLaVA-NeXT-Vicuna-7B/13B. Empirical results show that CompCap-118K
significantly enhances MLLMs' understanding of CIs, yielding average gains of
1.7%, 2.0%, and 2.9% across eleven benchmarks, respectively.
A new framework called Composite Captions (CompCap) enhances the understanding of composite images by MLLMs through the creation of a large, detailed dataset, leading to significant performance improvements.
Multimodal Large Language Models composite images natural images ChartQA ScienceQA Composite Captions CompCap image-caption datasets supervised fine-tuning vision-language alignment