title: AntiLeak-Bench: Preventing Data Contamination by Automatically
    Constructing Benchmarks with Updated Real-World Knowledge
summary: Data contamination hinders fair LLM evaluation by introducing test data into
  newer models' training sets. Existing studies solve this challenge by updating
  benchmarks with newly collected data. However, they fail to guarantee
  contamination-free evaluation as the newly collected data may contain
  pre-existing knowledge, and their benchmark updates rely on intensive human
  labor. To address these issues, we in this paper propose AntiLeak-Bench, an
  automated anti-leakage benchmarking framework. Instead of simply using newly
  collected data, we construct samples with explicitly new knowledge absent from
  LLMs' training sets, which thus ensures strictly contamination-free evaluation.
  We further design a fully automated workflow to build and update our benchmark
  without human labor. This significantly reduces the cost of benchmark
  maintenance to accommodate emerging LLMs. Through extensive experiments, we
  highlight that data contamination likely exists before LLMs' cutoff time and
  demonstrate AntiLeak-Bench effectively overcomes this challenge.
paper_authors._id: 6763e39859e17975d93819aa
paper_authors.name: Xiaobao Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:00.226Z
paper_authors._id: 64cb02869e30a46f7b80b355
paper_authors.avatarUrl: /avatars/81ce4ba78826b54f0e1b53eeaff87ee6.svg
paper_authors.fullname: Xiaobao Wu
paper_authors.type: user
paper_authors.user: bobxwu
paper_authors._id: 6763e39859e17975d93819ab
paper_authors.name: Liangming Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:46:35.302Z
paper_authors._id: 628cd837ef14f971b699e4d4
paper_authors.avatarUrl: /avatars/cf64c88e71c730fc0a54df59e17bffbb.svg
paper_authors.fullname: Liangming Pan
paper_authors.type: user
paper_authors.user: Liangming
paper_authors._id: 6763e39859e17975d93819ac
paper_authors.name: Yuxi Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:46:29.379Z
paper_authors._id: 647730d403fe88eff54d9b24
paper_authors.avatarUrl: /avatars/66a41a42a4f9adaca7477b73ab1a0c0e.svg
paper_authors.fullname: Sigrid Xie
paper_authors.type: user
paper_authors.user: yuxixie
paper_authors._id: 6763e39859e17975d93819ad
paper_authors.name: Ruiwen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:46:13.476Z
paper_authors._id: 66ad5084bcede3c1e3445395
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66ad5084bcede3c1e3445395/egBqNuArOy0rz7aGVB-T6.jpeg
paper_authors.fullname: Ruiwen Zhou
paper_authors.type: user
paper_authors.user: rwzhou
paper_authors._id: 6763e39859e17975d93819ae
paper_authors.name: Shuai Zhao
paper_authors._id: 6763e39859e17975d93819af
paper_authors.name: Yubo Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:45:51.684Z
paper_authors._id: 63883d1cad6d6d6e93574705
paper_authors.avatarUrl: /avatars/19f17abdcfae6ff17e16d0a478c6b87f.svg
paper_authors.fullname: Yubo Ma
paper_authors.type: user
paper_authors.user: yubo2333
paper_authors._id: 6763e39859e17975d93819b0
paper_authors.name: Mingzhe Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:45:31.442Z
paper_authors._id: 61711f02e0b1ddb56eb9b526
paper_authors.avatarUrl: /avatars/3e2fdf774f5bc1f73b450486d6da42d4.svg
paper_authors.fullname: Mingzhe Du
paper_authors.type: user
paper_authors.user: Elfsong
paper_authors._id: 6763e39859e17975d93819b1
paper_authors.name: Rui Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:45:25.560Z
paper_authors._id: 66163174509f9e084f61be12
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66163174509f9e084f61be12/9kt21-2XlIRQ6FvbLp_Fc.jpeg
paper_authors.fullname: Rui Mao
paper_authors.type: user
paper_authors.user: RuiMao1988
paper_authors._id: 6763e39859e17975d93819b2
paper_authors.name: Anh Tuan Luu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:45:18.955Z
paper_authors._id: 655722e80438e0854fae7554
paper_authors.avatarUrl: /avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
paper_authors.fullname: Luu Anh Tuan
paper_authors.type: user
paper_authors.user: anhtuanluu36
paper_authors._id: 6763e39859e17975d93819b3
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:45:09.822Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_title: AntiLeak-Bench: Preventing Data Contamination by Automatically
    Constructing Benchmarks with Updated Real-World Knowledge
paper_summary: Data contamination hinders fair LLM evaluation by introducing test data into
  newer models' training sets. Existing studies solve this challenge by updating
  benchmarks with newly collected data. However, they fail to guarantee
  contamination-free evaluation as the newly collected data may contain
  pre-existing knowledge, and their benchmark updates rely on intensive human
  labor. To address these issues, we in this paper propose AntiLeak-Bench, an
  automated anti-leakage benchmarking framework. Instead of simply using newly
  collected data, we construct samples with explicitly new knowledge absent from
  LLMs' training sets, which thus ensures strictly contamination-free evaluation.
  We further design a fully automated workflow to build and update our benchmark
  without human labor. This significantly reduces the cost of benchmark
  maintenance to accommodate emerging LLMs. Through extensive experiments, we
  highlight that data contamination likely exists before LLMs' cutoff time and
  demonstrate AntiLeak-Bench effectively overcomes this challenge.
paper_ai_summary: AntiLeak-Bench is an automated framework that constructs contamination-free benchmarks for LLM evaluation by ensuring samples contain new knowledge absent from training sets.
paper_ai_keywords: AntiLeak-Bench LLMs data contamination benchmarking framework