title: Data Contamination Can Cross Language Barriers
summary: The opacity in developing large language models (LLMs) is raising growing
  concerns about the potential contamination of public benchmarks in the
  pre-training data. Existing contamination detection methods are typically based
  on the text overlap between training and evaluation data, which can be too
  superficial to reflect deeper forms of contamination. In this paper, we first
  present a cross-lingual form of contamination that inflates LLMs' performance
  while evading current detection methods, deliberately injected by overfitting
  LLMs on the translated versions of benchmark test sets. Then, we propose
  generalization-based approaches to unmask such deeply concealed contamination.
  Specifically, we examine the LLM's performance change after modifying the
  original benchmark by replacing the false answer choices with correct ones from
  other questions. Contaminated models can hardly generalize to such easier
  situations, where the false choices can be not even wrong, as all
  choices are correct in their memorization. Experimental results demonstrate
  that cross-lingual contamination can easily fool existing detection methods,
  but not ours. In addition, we discuss the potential utilization of
  cross-lingual contamination in interpreting LLMs' working mechanisms and in
  post-training LLMs for enhanced multilingual capabilities. The code and dataset
  we use can be obtained from https://github.com/ShangDataLab/Deep-Contam.
paper_authors._id: 6675f76925c64f9273c01335
paper_authors.name: Feng Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:30:07.241Z
paper_authors._id: 64c8b2c5c547ed5243e14a6e
paper_authors.avatarUrl: /avatars/96d4a9010f96001c8cff235915926390.svg
paper_authors.fullname: Feng Yao
paper_authors.type: user
paper_authors.user: fengyao1909
paper_authors._id: 6675f76925c64f9273c01336
paper_authors.name: Yufan Zhuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:05:15.032Z
paper_authors._id: 6438ccbb3b46237de3d052e8
paper_authors.avatarUrl: /avatars/baa624d417b0b905e82127dc66346478.svg
paper_authors.fullname: Yufan Zhuang
paper_authors.type: user
paper_authors.user: yzhuang
paper_authors._id: 6675f76925c64f9273c01337
paper_authors.name: Zihao Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:03:26.218Z
paper_authors._id: 65a0966a074bfce80244f6f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a0966a074bfce80244f6f8/cjDJ68RQvHuBtnji0vtjK.jpeg
paper_authors.fullname: Zihao Sun
paper_authors.type: user
paper_authors.user: ericsun153
paper_authors._id: 6675f76925c64f9273c01338
paper_authors.name: Sunan Xu
paper_authors._id: 6675f76925c64f9273c01339
paper_authors.name: Animesh Kumar
paper_authors._id: 6675f76925c64f9273c0133a
paper_authors.name: Jingbo Shang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:51:56.944Z
paper_authors._id: 660655119e3555d648f6c6b5
paper_authors.avatarUrl: /avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
paper_authors.fullname: Jingbo Shang
paper_authors.type: user
paper_authors.user: shangjingbo
paper_title: Data Contamination Can Cross Language Barriers
paper_summary: The opacity in developing large language models (LLMs) is raising growing
  concerns about the potential contamination of public benchmarks in the
  pre-training data. Existing contamination detection methods are typically based
  on the text overlap between training and evaluation data, which can be too
  superficial to reflect deeper forms of contamination. In this paper, we first
  present a cross-lingual form of contamination that inflates LLMs' performance
  while evading current detection methods, deliberately injected by overfitting
  LLMs on the translated versions of benchmark test sets. Then, we propose
  generalization-based approaches to unmask such deeply concealed contamination.
  Specifically, we examine the LLM's performance change after modifying the
  original benchmark by replacing the false answer choices with correct ones from
  other questions. Contaminated models can hardly generalize to such easier
  situations, where the false choices can be not even wrong, as all
  choices are correct in their memorization. Experimental results demonstrate
  that cross-lingual contamination can easily fool existing detection methods,
  but not ours. In addition, we discuss the potential utilization of
  cross-lingual contamination in interpreting LLMs' working mechanisms and in
  post-training LLMs for enhanced multilingual capabilities. The code and dataset
  we use can be obtained from https://github.com/ShangDataLab/Deep-Contam.
paper_ai_summary: Generalization-based approaches detect cross-lingual contamination in large language models by modifying benchmark tests to replaces false answers with correct ones, exposing models that overfit to translated versions of benchmark sets.
paper_ai_keywords: large language models cross-lingual contamination generalization-based approaches overfitting benchmark tests LLMs