title: Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM
    Data Contamination
summary: The rapid progression of multimodal large language models (MLLMs) has
  demonstrated superior performance on various multimodal benchmarks. However,
  the issue of data contamination during training creates challenges in
  performance evaluation and comparison. While numerous methods exist for
  detecting dataset contamination in large language models (LLMs), they are less
  effective for MLLMs due to their various modalities and multiple training
  phases. In this study, we introduce a multimodal data contamination detection
  framework, MM-Detect, designed for MLLMs. Our experimental results indicate
  that MM-Detect is sensitive to varying degrees of contamination and can
  highlight significant performance improvements due to leakage of the training
  set of multimodal benchmarks. Furthermore, We also explore the possibility of
  contamination originating from the pre-training phase of LLMs used by MLLMs and
  the fine-tuning phase of MLLMs, offering new insights into the stages at which
  contamination may be introduced.
paper_authors._id: 672c2ca7dbc16559ba1d8f94
paper_authors.name: Dingjie Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:36:10.352Z
paper_authors._id: 619f01b8cc04eadf54fa5d5d
paper_authors.avatarUrl: /avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
paper_authors.fullname: Song Dingjie
paper_authors.type: user
paper_authors.user: songdj
paper_authors._id: 672c2ca7dbc16559ba1d8f95
paper_authors.name: Sicheng Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T13:49:43.481Z
paper_authors._id: 662bad8d52e194d5d416558d
paper_authors.avatarUrl: /avatars/b5392479b6b22fef553b4a0c5dae0c13.svg
paper_authors.fullname: Sicheng Lai
paper_authors.type: user
paper_authors.user: LeoLai689
paper_authors._id: 672c2ca7dbc16559ba1d8f96
paper_authors.name: Shunian Chen
paper_authors._id: 672c2ca7dbc16559ba1d8f97
paper_authors.name: Lichao Sun
paper_authors._id: 672c2ca7dbc16559ba1d8f98
paper_authors.name: Benyou Wang
paper_title: Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM
    Data Contamination
paper_summary: The rapid progression of multimodal large language models (MLLMs) has
  demonstrated superior performance on various multimodal benchmarks. However,
  the issue of data contamination during training creates challenges in
  performance evaluation and comparison. While numerous methods exist for
  detecting dataset contamination in large language models (LLMs), they are less
  effective for MLLMs due to their various modalities and multiple training
  phases. In this study, we introduce a multimodal data contamination detection
  framework, MM-Detect, designed for MLLMs. Our experimental results indicate
  that MM-Detect is sensitive to varying degrees of contamination and can
  highlight significant performance improvements due to leakage of the training
  set of multimodal benchmarks. Furthermore, We also explore the possibility of
  contamination originating from the pre-training phase of LLMs used by MLLMs and
  the fine-tuning phase of MLLMs, offering new insights into the stages at which
  contamination may be introduced.
paper_ai_keywords: multimodal large language models (MLLMs) Visual Question framework MM-Detect Contamination detection Contamination Data contamination LLMs (Large Language Models) Pre-training phase Fine-tuning phase Multimodal benchmarks