Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM
  Data Contamination
The rapid progression of multimodal large language models (MLLMs) has
demonstrated superior performance on various multimodal benchmarks. However,
the issue of data contamination during training creates challenges in
performance evaluation and comparison. While numerous methods exist for
detecting dataset contamination in large language models (LLMs), they are less
effective for MLLMs due to their various modalities and multiple training
phases. In this study, we introduce a multimodal data contamination detection
framework, MM-Detect, designed for MLLMs. Our experimental results indicate
that MM-Detect is sensitive to varying degrees of contamination and can
highlight significant performance improvements due to leakage of the training
set of multimodal benchmarks. Furthermore, We also explore the possibility of
contamination originating from the pre-training phase of LLMs used by MLLMs and
the fine-tuning phase of MLLMs, offering new insights into the stages at which
contamination may be introduced.
672c2ca7dbc16559ba1d8f94
Dingjie Song
claimed_verified
2024-11-15T09:36:10.352Z
619f01b8cc04eadf54fa5d5d
/avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
Song Dingjie
user
songdj
672c2ca7dbc16559ba1d8f95
Sicheng Lai
claimed_verified
2024-11-15T13:49:43.481Z
662bad8d52e194d5d416558d
/avatars/b5392479b6b22fef553b4a0c5dae0c13.svg
Sicheng Lai
user
LeoLai689
672c2ca7dbc16559ba1d8f96
Shunian Chen
672c2ca7dbc16559ba1d8f97
Lichao Sun
672c2ca7dbc16559ba1d8f98
Benyou Wang
Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM
  Data Contamination
The rapid progression of multimodal large language models (MLLMs) has
demonstrated superior performance on various multimodal benchmarks. However,
the issue of data contamination during training creates challenges in
performance evaluation and comparison. While numerous methods exist for
detecting dataset contamination in large language models (LLMs), they are less
effective for MLLMs due to their various modalities and multiple training
phases. In this study, we introduce a multimodal data contamination detection
framework, MM-Detect, designed for MLLMs. Our experimental results indicate
that MM-Detect is sensitive to varying degrees of contamination and can
highlight significant performance improvements due to leakage of the training
set of multimodal benchmarks. Furthermore, We also explore the possibility of
contamination originating from the pre-training phase of LLMs used by MLLMs and
the fine-tuning phase of MLLMs, offering new insights into the stages at which
contamination may be introduced.
A framework called MM-Detect is introduced to detect data contamination in multimodal large language models, demonstrating sensitivity across different contamination levels and identifying potential contamination stages during pre-training and fine-tuning.
multimodal large language models MLLMs dataset contamination data contamination MM-Detect pre-training phase fine-tuning phase