Img-Diff: Contrastive Data Synthesis for Multimodal Large Language
  Models
High-performance Multimodal Large Language Models (MLLMs) rely heavily on
data quality. This study introduces a novel dataset named Img-Diff, designed to
enhance fine-grained image recognition in MLLMs by leveraging insights from
contrastive learning and image difference captioning. By analyzing object
differences between similar images, we challenge models to identify both
matching and distinct components. We utilize the Stable-Diffusion-XL model and
advanced image editing techniques to create pairs of similar images that
highlight object replacements. Our methodology includes a Difference Area
Generator for object differences identifying, followed by a Difference Captions
Generator for detailed difference descriptions. The result is a relatively
small but high-quality dataset of "object replacement" samples. We use the the
proposed dataset to fine-tune state-of-the-art (SOTA) MLLMs such as MGM-7B,
yielding comprehensive improvements of performance scores over SOTA models that
trained with larger-scale datasets, in numerous image difference and Visual
Question Answering tasks. For instance, our trained models notably surpass the
SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate
alternative methods for generating image difference data through "object
removal" and conduct thorough evaluation to confirm the dataset's diversity,
quality, and robustness, presenting several insights on synthesis of such
contrastive dataset. To encourage further research and advance the field of
multimodal data synthesis and enhancement of MLLMs' fundamental capabilities
for image understanding, we release our codes and dataset at
https://github.com/modelscope/data-juicer/tree/ImgDiff.
66b590d88a19c5c75ecce3e7
Qirui Jiao
claimed_verified
2024-08-16T14:45:26.811Z
63944182872e49c02f58ef88
/avatars/8110b88632234e588916f049598c402f.svg
Qirui_Jiao
user
LuckyBanana
66b590d88a19c5c75ecce3e8
Daoyuan Chen
admin_assigned
2024-08-09T10:05:40.823Z
6530ca70c1b8f265a431fc42
/avatars/7b1ec5b75398c8434ccd43cdce7fee53.svg
Daoyuan Chen
user
yxdyc
66b590d88a19c5c75ecce3e9
Yilun Huang
claimed_verified
2025-07-07T11:27:36.843Z
64213e9d9628d83384af2632
/avatars/46a34fe25b139acf59f8e5a665999ba1.svg
Dylan Huang
user
hylcool
66b590d88a19c5c75ecce3ea
Yaliang Li
66b590d88a19c5c75ecce3eb
Ying Shen
Img-Diff: Contrastive Data Synthesis for Multimodal Large Language
  Models
High-performance Multimodal Large Language Models (MLLMs) rely heavily on
data quality. This study introduces a novel dataset named Img-Diff, designed to
enhance fine-grained image recognition in MLLMs by leveraging insights from
contrastive learning and image difference captioning. By analyzing object
differences between similar images, we challenge models to identify both
matching and distinct components. We utilize the Stable-Diffusion-XL model and
advanced image editing techniques to create pairs of similar images that
highlight object replacements. Our methodology includes a Difference Area
Generator for object differences identifying, followed by a Difference Captions
Generator for detailed difference descriptions. The result is a relatively
small but high-quality dataset of "object replacement" samples. We use the the
proposed dataset to fine-tune state-of-the-art (SOTA) MLLMs such as MGM-7B,
yielding comprehensive improvements of performance scores over SOTA models that
trained with larger-scale datasets, in numerous image difference and Visual
Question Answering tasks. For instance, our trained models notably surpass the
SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate
alternative methods for generating image difference data through "object
removal" and conduct thorough evaluation to confirm the dataset's diversity,
quality, and robustness, presenting several insights on synthesis of such
contrastive dataset. To encourage further research and advance the field of
multimodal data synthesis and enhancement of MLLMs' fundamental capabilities
for image understanding, we release our codes and dataset at
https://github.com/modelscope/data-juicer/tree/ImgDiff.
A novel dataset, Img-Diff, enhances fine-grained image recognition in MLLMs using contrastive learning and image difference captioning, improving performance over larger datasets.
multimodal large language models Img-Diff contrastive learning image difference captioning Stable-Diffusion-XL Difference Area Generator Difference Captions Generator MGM-7B MMVP benchmark GPT-4V Gemini object replacement object removal multimodal data synthesis