Data Contamination Can Cross Language Barriers
The opacity in developing large language models (LLMs) is raising growing
concerns about the potential contamination of public benchmarks in the
pre-training data. Existing contamination detection methods are typically based
on the text overlap between training and evaluation data, which can be too
superficial to reflect deeper forms of contamination. In this paper, we first
present a cross-lingual form of contamination that inflates LLMs' performance
while evading current detection methods, deliberately injected by overfitting
LLMs on the translated versions of benchmark test sets. Then, we propose
generalization-based approaches to unmask such deeply concealed contamination.
Specifically, we examine the LLM's performance change after modifying the
original benchmark by replacing the false answer choices with correct ones from
other questions. Contaminated models can hardly generalize to such easier
situations, where the false choices can be not even wrong, as all
choices are correct in their memorization. Experimental results demonstrate
that cross-lingual contamination can easily fool existing detection methods,
but not ours. In addition, we discuss the potential utilization of
cross-lingual contamination in interpreting LLMs' working mechanisms and in
post-training LLMs for enhanced multilingual capabilities. The code and dataset
we use can be obtained from https://github.com/ShangDataLab/Deep-Contam.
6675f76925c64f9273c01335
Feng Yao
claimed_verified
2024-06-24T07:30:07.241Z
64c8b2c5c547ed5243e14a6e
/avatars/96d4a9010f96001c8cff235915926390.svg
Feng Yao
user
fengyao1909
6675f76925c64f9273c01336
Yufan Zhuang
claimed_verified
2024-06-26T08:05:15.032Z
6438ccbb3b46237de3d052e8
/avatars/baa624d417b0b905e82127dc66346478.svg
Yufan Zhuang
user
yzhuang
6675f76925c64f9273c01337
Zihao Sun
claimed_verified
2024-06-27T09:03:26.218Z
65a0966a074bfce80244f6f8
https://cdn-avatars.huggingface.co/v1/production/uploads/65a0966a074bfce80244f6f8/cjDJ68RQvHuBtnji0vtjK.jpeg
Zihao Sun
user
ericsun153
6675f76925c64f9273c01338
Sunan Xu
6675f76925c64f9273c01339
Animesh Kumar
6675f76925c64f9273c0133a
Jingbo Shang
claimed_verified
2025-02-10T09:51:56.944Z
660655119e3555d648f6c6b5
/avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
Jingbo Shang
user
shangjingbo
Data Contamination Can Cross Language Barriers
The opacity in developing large language models (LLMs) is raising growing
concerns about the potential contamination of public benchmarks in the
pre-training data. Existing contamination detection methods are typically based
on the text overlap between training and evaluation data, which can be too
superficial to reflect deeper forms of contamination. In this paper, we first
present a cross-lingual form of contamination that inflates LLMs' performance
while evading current detection methods, deliberately injected by overfitting
LLMs on the translated versions of benchmark test sets. Then, we propose
generalization-based approaches to unmask such deeply concealed contamination.
Specifically, we examine the LLM's performance change after modifying the
original benchmark by replacing the false answer choices with correct ones from
other questions. Contaminated models can hardly generalize to such easier
situations, where the false choices can be not even wrong, as all
choices are correct in their memorization. Experimental results demonstrate
that cross-lingual contamination can easily fool existing detection methods,
but not ours. In addition, we discuss the potential utilization of
cross-lingual contamination in interpreting LLMs' working mechanisms and in
post-training LLMs for enhanced multilingual capabilities. The code and dataset
we use can be obtained from https://github.com/ShangDataLab/Deep-Contam.
Generalization-based approaches detect cross-lingual contamination in large language models by modifying benchmark tests to replaces false answers with correct ones, exposing models that overfit to translated versions of benchmark sets.
large language models cross-lingual contamination generalization-based approaches overfitting benchmark tests LLMs