Preference Leakage: A Contamination Problem in LLM-as-a-judge
Large Language Models (LLMs) as judges and LLM-based data synthesis have
emerged as two fundamental LLM-driven data annotation methods in model
development. While their combination significantly enhances the efficiency of
model training and evaluation, little attention has been given to the potential
contamination brought by this new model development paradigm. In this work, we
expose preference leakage, a contamination problem in LLM-as-a-judge caused by
the relatedness between the synthetic data generators and LLM-based evaluators.
To study this issue, we first define three common relatednesses between data
generator LLM and judge LLM: being the same model, having an inheritance
relationship, and belonging to the same model family. Through extensive
experiments, we empirically confirm the bias of judges towards their related
student models caused by preference leakage across multiple LLM baselines and
benchmarks. Further analysis suggests that preference leakage is a pervasive
issue that is harder to detect compared to previously identified biases in
LLM-as-a-judge scenarios. All of these findings imply that preference leakage
is a widespread and challenging problem in the area of LLM-as-a-judge. We
release all codes and data at:
https://github.com/David-Li0406/Preference-Leakage.
67a1ad77d797fac51fa80770
Dawei Li
claimed_verified
2025-02-18T09:34:59.780Z
6474e1afb68461d5cf7c41cc
https://cdn-avatars.huggingface.co/v1/production/uploads/6474e1afb68461d5cf7c41cc/bcoiD_qPrjHUBlB259djg.png
Dawei Li
user
wjldw
67a1ad77d797fac51fa80771
Renliang Sun
claimed_verified
2025-02-04T09:39:11.035Z
653a195b0da86d726c9c580c
/avatars/61649e1d600fdc1edc50ead0dfa99fdd.svg
Renliang Sun
user
RLSNLP
67a1ad77d797fac51fa80772
Yue Huang
67a1ad77d797fac51fa80773
Ming Zhong
admin_assigned
2025-02-04T14:54:39.212Z
61d53df2062444ea769d3b79
/avatars/fa771202368b6b2626a8fdf1c4369239.svg
Ming Zhong
user
MingZhong
67a1ad77d797fac51fa80774
Bohan Jiang
67a1ad77d797fac51fa80775
Jiawei Han
67a1ad77d797fac51fa80776
Xiangliang Zhang
admin_assigned
2025-02-04T17:14:47.621Z
605a97d9b54d35bc67a4ff12
/avatars/7a48a2dac4e6ebb9e775022e15ddc5a7.svg
zhangxiangliang
user
ZhangXiangliang
67a1ad77d797fac51fa80777
Wei Wang
admin_assigned
2025-02-04T17:14:54.615Z
62fa0ffe0697d224219a0cb7
/avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
Wei Wang
user
WeiWang
67a1ad77d797fac51fa80778
Huan Liu
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Large Language Models (LLMs) as judges and LLM-based data synthesis have
emerged as two fundamental LLM-driven data annotation methods in model
development. While their combination significantly enhances the efficiency of
model training and evaluation, little attention has been given to the potential
contamination brought by this new model development paradigm. In this work, we
expose preference leakage, a contamination problem in LLM-as-a-judge caused by
the relatedness between the synthetic data generators and LLM-based evaluators.
To study this issue, we first define three common relatednesses between data
generator LLM and judge LLM: being the same model, having an inheritance
relationship, and belonging to the same model family. Through extensive
experiments, we empirically confirm the bias of judges towards their related
student models caused by preference leakage across multiple LLM baselines and
benchmarks. Further analysis suggests that preference leakage is a pervasive
issue that is harder to detect compared to previously identified biases in
LLM-as-a-judge scenarios. All of these findings imply that preference leakage
is a widespread and challenging problem in the area of LLM-as-a-judge. We
release all codes and data at:
https://github.com/David-Li0406/Preference-Leakage.
Preference leakage, a bias in LLM-as-a-judge systems caused by the relatedness between synthetic data generators and evaluators, has been empirically confirmed and found to be pervasive and challenging.
LLMs LLM-as-a-judge LLM-based data synthesis data annotation preference leakage relatedness bias benchmarks