AntiLeak-Bench: Preventing Data Contamination by Automatically
  Constructing Benchmarks with Updated Real-World Knowledge
Data contamination hinders fair LLM evaluation by introducing test data into
newer models' training sets. Existing studies solve this challenge by updating
benchmarks with newly collected data. However, they fail to guarantee
contamination-free evaluation as the newly collected data may contain
pre-existing knowledge, and their benchmark updates rely on intensive human
labor. To address these issues, we in this paper propose AntiLeak-Bench, an
automated anti-leakage benchmarking framework. Instead of simply using newly
collected data, we construct samples with explicitly new knowledge absent from
LLMs' training sets, which thus ensures strictly contamination-free evaluation.
We further design a fully automated workflow to build and update our benchmark
without human labor. This significantly reduces the cost of benchmark
maintenance to accommodate emerging LLMs. Through extensive experiments, we
highlight that data contamination likely exists before LLMs' cutoff time and
demonstrate AntiLeak-Bench effectively overcomes this challenge.
6763e39859e17975d93819aa
Xiaobao Wu
claimed_verified
2024-12-19T18:01:00.226Z
64cb02869e30a46f7b80b355
/avatars/81ce4ba78826b54f0e1b53eeaff87ee6.svg
Xiaobao Wu
user
bobxwu
6763e39859e17975d93819ab
Liangming Pan
admin_assigned
2024-12-20T09:46:35.302Z
628cd837ef14f971b699e4d4
/avatars/cf64c88e71c730fc0a54df59e17bffbb.svg
Liangming Pan
user
Liangming
6763e39859e17975d93819ac
Yuxi Xie
admin_assigned
2024-12-20T09:46:29.379Z
647730d403fe88eff54d9b24
/avatars/66a41a42a4f9adaca7477b73ab1a0c0e.svg
Sigrid Xie
user
yuxixie
6763e39859e17975d93819ad
Ruiwen Zhou
admin_assigned
2024-12-20T09:46:13.476Z
66ad5084bcede3c1e3445395
https://cdn-avatars.huggingface.co/v1/production/uploads/66ad5084bcede3c1e3445395/egBqNuArOy0rz7aGVB-T6.jpeg
Ruiwen Zhou
user
rwzhou
6763e39859e17975d93819ae
Shuai Zhao
6763e39859e17975d93819af
Yubo Ma
admin_assigned
2024-12-20T09:45:51.684Z
63883d1cad6d6d6e93574705
/avatars/19f17abdcfae6ff17e16d0a478c6b87f.svg
Yubo Ma
user
yubo2333
6763e39859e17975d93819b0
Mingzhe Du
admin_assigned
2024-12-20T09:45:31.442Z
61711f02e0b1ddb56eb9b526
/avatars/3e2fdf774f5bc1f73b450486d6da42d4.svg
Mingzhe Du
user
Elfsong
6763e39859e17975d93819b1
Rui Mao
admin_assigned
2024-12-20T09:45:25.560Z
66163174509f9e084f61be12
https://cdn-avatars.huggingface.co/v1/production/uploads/66163174509f9e084f61be12/9kt21-2XlIRQ6FvbLp_Fc.jpeg
Rui Mao
user
RuiMao1988
6763e39859e17975d93819b2
Anh Tuan Luu
admin_assigned
2024-12-20T09:45:18.955Z
655722e80438e0854fae7554
/avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
Luu Anh Tuan
user
anhtuanluu36
6763e39859e17975d93819b3
William Yang Wang
admin_assigned
2024-12-20T09:45:09.822Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
AntiLeak-Bench: Preventing Data Contamination by Automatically
  Constructing Benchmarks with Updated Real-World Knowledge
Data contamination hinders fair LLM evaluation by introducing test data into
newer models' training sets. Existing studies solve this challenge by updating
benchmarks with newly collected data. However, they fail to guarantee
contamination-free evaluation as the newly collected data may contain
pre-existing knowledge, and their benchmark updates rely on intensive human
labor. To address these issues, we in this paper propose AntiLeak-Bench, an
automated anti-leakage benchmarking framework. Instead of simply using newly
collected data, we construct samples with explicitly new knowledge absent from
LLMs' training sets, which thus ensures strictly contamination-free evaluation.
We further design a fully automated workflow to build and update our benchmark
without human labor. This significantly reduces the cost of benchmark
maintenance to accommodate emerging LLMs. Through extensive experiments, we
highlight that data contamination likely exists before LLMs' cutoff time and
demonstrate AntiLeak-Bench effectively overcomes this challenge.
AntiLeak-Bench is an automated framework that constructs contamination-free benchmarks for LLM evaluation by ensuring samples contain new knowledge absent from training sets.
AntiLeak-Bench LLMs data contamination benchmarking framework