title: Beyond the Surface: Measuring Self-Preference in LLM Judgments
summary: Recent studies show that large language models (LLMs) exhibit self-preference
  bias when serving as judges, meaning they tend to favor their own responses
  over those generated by other models. Existing methods typically measure this
  bias by calculating the difference between the scores a judge model assigns to
  its own responses and those it assigns to responses from other models. However,
  this approach conflates self-preference bias with response quality, as
  higher-quality responses from the judge model may also lead to positive score
  differences, even in the absence of bias. To address this issue, we introduce
  gold judgments as proxies for the actual quality of responses and propose the
  DBG score, which measures self-preference bias as the difference between the
  scores assigned by the judge model to its own responses and the corresponding
  gold judgments. Since gold judgments reflect true response quality, the DBG
  score mitigates the confounding effect of response quality on bias measurement.
  Using the DBG score, we conduct comprehensive experiments to assess
  self-preference bias across LLMs of varying versions, sizes, and reasoning
  abilities. Additionally, we investigate two factors that influence and help
  alleviate self-preference bias: response text style and the post-training data
  of judge models. Finally, we explore potential underlying mechanisms of
  self-preference bias from an attention-based perspective. Our code and data are
  available at https://github.com/zhiyuanc2001/self-preference.
paper_authors._id: 684104c89ec96d9991484c24
paper_authors.name: Zhi-Yuan Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-06-05T02:45:29.221Z
paper_authors._id: 65309a1d657ae56cdb65e0e7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lHQI9RNjfz8E5v1uyCGeV.png
paper_authors.fullname: Zhi-Yuan Chen
paper_authors.type: user
paper_authors.user: JaxChen
paper_authors._id: 684104c89ec96d9991484c25
paper_authors.name: Hao Wang
paper_authors._id: 684104c89ec96d9991484c26
paper_authors.name: Xinyu Zhang
paper_authors._id: 684104c89ec96d9991484c27
paper_authors.name: Enrui Hu
paper_authors._id: 684104c89ec96d9991484c28
paper_authors.name: Yankai Lin
paper_title: Beyond the Surface: Measuring Self-Preference in LLM Judgments
paper_summary: Recent studies show that large language models (LLMs) exhibit self-preference
  bias when serving as judges, meaning they tend to favor their own responses
  over those generated by other models. Existing methods typically measure this
  bias by calculating the difference between the scores a judge model assigns to
  its own responses and those it assigns to responses from other models. However,
  this approach conflates self-preference bias with response quality, as
  higher-quality responses from the judge model may also lead to positive score
  differences, even in the absence of bias. To address this issue, we introduce
  gold judgments as proxies for the actual quality of responses and propose the
  DBG score, which measures self-preference bias as the difference between the
  scores assigned by the judge model to its own responses and the corresponding
  gold judgments. Since gold judgments reflect true response quality, the DBG
  score mitigates the confounding effect of response quality on bias measurement.
  Using the DBG score, we conduct comprehensive experiments to assess
  self-preference bias across LLMs of varying versions, sizes, and reasoning
  abilities. Additionally, we investigate two factors that influence and help
  alleviate self-preference bias: response text style and the post-training data
  of judge models. Finally, we explore potential underlying mechanisms of
  self-preference bias from an attention-based perspective. Our code and data are
  available at https://github.com/zhiyuanc2001/self-preference.
paper_ai_summary: The DBG score is introduced to measure self-preference bias in large language models by using gold judgments as proxies for response quality, addressing the confounding effect of response quality.
paper_ai_keywords: large language models self-preference bias judge model gold judgments DBG score response quality attention-based perspective