title: CompCap: Improving Multimodal Large Language Models with Composite
    Captions
summary: How well can Multimodal Large Language Models (MLLMs) understand composite
  images? Composite images (CIs) are synthetic visuals created by merging
  multiple visual elements, such as charts, posters, or screenshots, rather than
  being captured directly by a camera. While CIs are prevalent in real-world
  applications, recent MLLM developments have primarily focused on interpreting
  natural images (NIs). Our research reveals that current MLLMs face significant
  challenges in accurately understanding CIs, often struggling to extract
  information or perform complex reasoning based on these images. We find that
  existing training data for CIs are mostly formatted for question-answer tasks
  (e.g., in datasets like ChartQA and ScienceQA), while high-quality
  image-caption datasets, critical for robust vision-language alignment, are only
  available for NIs. To bridge this gap, we introduce Composite Captions
  (CompCap), a flexible framework that leverages Large Language Models (LLMs) and
  automation tools to synthesize CIs with accurate and detailed captions. Using
  CompCap, we curate CompCap-118K, a dataset containing 118K image-caption pairs
  across six CI types. We validate the effectiveness of CompCap-118K by
  supervised fine-tuning MLLMs of three sizes: xGen-MM-inst.-4B and
  LLaVA-NeXT-Vicuna-7B/13B. Empirical results show that CompCap-118K
  significantly enhances MLLMs' understanding of CIs, yielding average gains of
  1.7%, 2.0%, and 2.9% across eleven benchmarks, respectively.
paper_authors._id: 67570c25bb1d50918a3c8c03
paper_authors.name: Xiaohui Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:20:54.990Z
paper_authors._id: 648015c6b82adfa7cc1a9565
paper_authors.avatarUrl: /avatars/e630e72a9d80a4d73bbf979133a0e07a.svg
paper_authors.fullname: Xiaohui Chen
paper_authors.type: user
paper_authors.user: xchen16
paper_authors._id: 67570c25bb1d50918a3c8c04
paper_authors.name: Satya Narayan Shukla
paper_authors._id: 67570c25bb1d50918a3c8c05
paper_authors.name: Mahmoud Azab
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T10:25:41.245Z
paper_authors._id: 64c05582d71cdcdcc80b7658
paper_authors.avatarUrl: /avatars/8ab4e2b0a70f657556e8e09ece1c496a.svg
paper_authors.fullname: Mahmoud Azab
paper_authors.type: user
paper_authors.user: azab
paper_authors._id: 67570c25bb1d50918a3c8c06
paper_authors.name: Aashu Singh
paper_authors._id: 67570c25bb1d50918a3c8c07
paper_authors.name: Qifan Wang
paper_authors._id: 67570c25bb1d50918a3c8c08
paper_authors.name: David Yang
paper_authors._id: 67570c25bb1d50918a3c8c09
paper_authors.name: ShengYun Peng
paper_authors._id: 67570c25bb1d50918a3c8c0a
paper_authors.name: Hanchao Yu
paper_authors._id: 67570c25bb1d50918a3c8c0b
paper_authors.name: Shen Yan
paper_authors._id: 67570c25bb1d50918a3c8c0c
paper_authors.name: Xuewen Zhang
paper_authors._id: 67570c25bb1d50918a3c8c0d
paper_authors.name: Baosheng He
paper_title: CompCap: Improving Multimodal Large Language Models with Composite
    Captions
paper_summary: How well can Multimodal Large Language Models (MLLMs) understand composite
  images? Composite images (CIs) are synthetic visuals created by merging
  multiple visual elements, such as charts, posters, or screenshots, rather than
  being captured directly by a camera. While CIs are prevalent in real-world
  applications, recent MLLM developments have primarily focused on interpreting
  natural images (NIs). Our research reveals that current MLLMs face significant
  challenges in accurately understanding CIs, often struggling to extract
  information or perform complex reasoning based on these images. We find that
  existing training data for CIs are mostly formatted for question-answer tasks
  (e.g., in datasets like ChartQA and ScienceQA), while high-quality
  image-caption datasets, critical for robust vision-language alignment, are only
  available for NIs. To bridge this gap, we introduce Composite Captions
  (CompCap), a flexible framework that leverages Large Language Models (LLMs) and
  automation tools to synthesize CIs with accurate and detailed captions. Using
  CompCap, we curate CompCap-118K, a dataset containing 118K image-caption pairs
  across six CI types. We validate the effectiveness of CompCap-118K by
  supervised fine-tuning MLLMs of three sizes: xGen-MM-inst.-4B and
  LLaVA-NeXT-Vicuna-7B/13B. Empirical results show that CompCap-118K
  significantly enhances MLLMs' understanding of CIs, yielding average gains of
  1.7%, 2.0%, and 2.9% across eleven benchmarks, respectively.
paper_ai_summary: A new framework called Composite Captions (CompCap) enhances the understanding of composite images by MLLMs through the creation of a large, detailed dataset, leading to significant performance improvements.
paper_ai_keywords: Multimodal Large Language Models composite images natural images ChartQA ScienceQA Composite Captions CompCap image-caption datasets supervised fine-tuning vision-language alignment