title: Img-Diff: Contrastive Data Synthesis for Multimodal Large Language
    Models
summary: High-performance Multimodal Large Language Models (MLLMs) rely heavily on
  data quality. This study introduces a novel dataset named Img-Diff, designed to
  enhance fine-grained image recognition in MLLMs by leveraging insights from
  contrastive learning and image difference captioning. By analyzing object
  differences between similar images, we challenge models to identify both
  matching and distinct components. We utilize the Stable-Diffusion-XL model and
  advanced image editing techniques to create pairs of similar images that
  highlight object replacements. Our methodology includes a Difference Area
  Generator for object differences identifying, followed by a Difference Captions
  Generator for detailed difference descriptions. The result is a relatively
  small but high-quality dataset of "object replacement" samples. We use the the
  proposed dataset to fine-tune state-of-the-art (SOTA) MLLMs such as MGM-7B,
  yielding comprehensive improvements of performance scores over SOTA models that
  trained with larger-scale datasets, in numerous image difference and Visual
  Question Answering tasks. For instance, our trained models notably surpass the
  SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate
  alternative methods for generating image difference data through "object
  removal" and conduct thorough evaluation to confirm the dataset's diversity,
  quality, and robustness, presenting several insights on synthesis of such
  contrastive dataset. To encourage further research and advance the field of
  multimodal data synthesis and enhancement of MLLMs' fundamental capabilities
  for image understanding, we release our codes and dataset at
  https://github.com/modelscope/data-juicer/tree/ImgDiff.
paper_authors._id: 66b590d88a19c5c75ecce3e7
paper_authors.name: Qirui Jiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T14:45:26.811Z
paper_authors._id: 63944182872e49c02f58ef88
paper_authors.avatarUrl: /avatars/8110b88632234e588916f049598c402f.svg
paper_authors.fullname: Qirui_Jiao
paper_authors.type: user
paper_authors.user: LuckyBanana
paper_authors._id: 66b590d88a19c5c75ecce3e8
paper_authors.name: Daoyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:05:40.823Z
paper_authors._id: 6530ca70c1b8f265a431fc42
paper_authors.avatarUrl: /avatars/7b1ec5b75398c8434ccd43cdce7fee53.svg
paper_authors.fullname: Daoyuan Chen
paper_authors.type: user
paper_authors.user: yxdyc
paper_authors._id: 66b590d88a19c5c75ecce3e9
paper_authors.name: Yilun Huang
paper_authors._id: 66b590d88a19c5c75ecce3ea
paper_authors.name: Yaliang Li
paper_authors._id: 66b590d88a19c5c75ecce3eb
paper_authors.name: Ying Shen
paper_title: Img-Diff: Contrastive Data Synthesis for Multimodal Large Language
    Models
paper_summary: High-performance Multimodal Large Language Models (MLLMs) rely heavily on
  data quality. This study introduces a novel dataset named Img-Diff, designed to
  enhance fine-grained image recognition in MLLMs by leveraging insights from
  contrastive learning and image difference captioning. By analyzing object
  differences between similar images, we challenge models to identify both
  matching and distinct components. We utilize the Stable-Diffusion-XL model and
  advanced image editing techniques to create pairs of similar images that
  highlight object replacements. Our methodology includes a Difference Area
  Generator for object differences identifying, followed by a Difference Captions
  Generator for detailed difference descriptions. The result is a relatively
  small but high-quality dataset of "object replacement" samples. We use the the
  proposed dataset to fine-tune state-of-the-art (SOTA) MLLMs such as MGM-7B,
  yielding comprehensive improvements of performance scores over SOTA models that
  trained with larger-scale datasets, in numerous image difference and Visual
  Question Answering tasks. For instance, our trained models notably surpass the
  SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate
  alternative methods for generating image difference data through "object
  removal" and conduct thorough evaluation to confirm the dataset's diversity,
  quality, and robustness, presenting several insights on synthesis of such
  contrastive dataset. To encourage further research and advance the field of
  multimodal data synthesis and enhancement of MLLMs' fundamental capabilities
  for image understanding, we release our codes and dataset at
  https://github.com/modelscope/data-juicer/tree/ImgDiff.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) Img-Diff fine-grained image-recognition contrastive learning image-present captioning _difference Area Generators Difference Captions Generator MGM-7B GPT-4V Gemini MMVP benchmark object replacement object removal multimodal data synthesis