title: RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
summary: With significant efforts in recent studies, LLM-as-a-Judge has become a
  cost-effective alternative to human evaluation for assessing the text
  generation quality in a wide range of tasks. However, there still remains a
  reliability gap between LLM-as-a-Judge and human evaluation. One important
  reason is the lack of guided oracles in the evaluation process. Motivated by
  the role of reference pervasively used in classic text evaluation, we introduce
  RevisEval, a novel text generation evaluation paradigm via the response-adapted
  references. RevisEval is driven by the key observation that an ideal reference
  should maintain the necessary relevance to the response to be evaluated.
  Specifically, RevisEval leverages the text revision capabilities of large
  language models (LLMs) to adaptively revise the response, then treat the
  revised text as the reference (response-adapted reference) for the subsequent
  evaluation. Extensive experiments demonstrate that RevisEval outperforms
  traditional reference-free and reference-based evaluation paradigms that use
  LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks.
  More importantly, our response-adapted references can further boost the
  classical text metrics, e.g., BLEU and BERTScore, compared to traditional
  references and even rival the LLM-as-a-Judge. A detailed analysis is also
  conducted to confirm RevisEval's effectiveness in bias reduction, the impact of
  inference cost, and reference relevance.
paper_authors._id: 6704e69412e582928046c613
paper_authors.name: Qiyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:16:43.381Z
paper_authors._id: 62a42f22c683d02f5b63320c
paper_authors.avatarUrl: /avatars/bc611abe9c4ef8d378123cb8ac9fdbf2.svg
paper_authors.fullname: Qiyuan Zhang
paper_authors.type: user
paper_authors.user: DonJoey
paper_authors._id: 6704e69412e582928046c614
paper_authors.name: Yufei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T09:17:57.837Z
paper_authors._id: 667c1c0b18aae25ed39b96c1
paper_authors.avatarUrl: /avatars/3513e7ae5d2cc8767e09893605123ee2.svg
paper_authors.fullname: Yufei Wang
paper_authors.type: user
paper_authors.user: yufeiwang201217a
paper_authors._id: 6704e69412e582928046c615
paper_authors.name: Tiezheng YU
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T09:18:04.918Z
paper_authors._id: 61bd88aa94bd578f7286181a
paper_authors.avatarUrl: /avatars/0f40dfd7accc558f9201103e3e4ec007.svg
paper_authors.fullname: Tiezheng Yu
paper_authors.type: user
paper_authors.user: Tiezheng
paper_authors._id: 6704e69412e582928046c616
paper_authors.name: Yuxin Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T09:18:38.205Z
paper_authors._id: 63c20105726f62e411fbe882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c20105726f62e411fbe882/2UsU9O2psbDjJzz-sAmGH.jpeg
paper_authors.fullname: Yuxin Jiang
paper_authors.type: user
paper_authors.user: YuxinJiang
paper_authors._id: 6704e69412e582928046c617
paper_authors.name: Chuhan Wu
paper_authors._id: 6704e69412e582928046c618
paper_authors.name: Liangyou Li
paper_authors._id: 6704e69412e582928046c619
paper_authors.name: Yasheng Wang
paper_authors._id: 6704e69412e582928046c61a
paper_authors.name: Xin Jiang
paper_authors._id: 6704e69412e582928046c61b
paper_authors.name: Lifeng Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T09:20:33.974Z
paper_authors._id: 655b1360c11dee7f7e7cf794
paper_authors.avatarUrl: /avatars/efb4b91e9bb8ab531331c8e4296f754c.svg
paper_authors.fullname: lifengshang
paper_authors.type: user
paper_authors.user: lifengshang
paper_authors._id: 6704e69412e582928046c61c
paper_authors.name: Ruiming Tang
paper_authors._id: 6704e69412e582928046c61d
paper_authors.name: Fuyuan Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T12:54:01.875Z
paper_authors._id: 65d2bb5c6130ef7be012d235
paper_authors.avatarUrl: /avatars/1c1e3bbb2c683a5c9d1f792a2c13fc4a.svg
paper_authors.fullname: Fuyuan Lyu
paper_authors.type: user
paper_authors.user: silentspring2
paper_authors._id: 6704e69412e582928046c61e
paper_authors.name: Chen Ma
paper_title: RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
paper_summary: With significant efforts in recent studies, LLM-as-a-Judge has become a
  cost-effective alternative to human evaluation for assessing the text
  generation quality in a wide range of tasks. However, there still remains a
  reliability gap between LLM-as-a-Judge and human evaluation. One important
  reason is the lack of guided oracles in the evaluation process. Motivated by
  the role of reference pervasively used in classic text evaluation, we introduce
  RevisEval, a novel text generation evaluation paradigm via the response-adapted
  references. RevisEval is driven by the key observation that an ideal reference
  should maintain the necessary relevance to the response to be evaluated.
  Specifically, RevisEval leverages the text revision capabilities of large
  language models (LLMs) to adaptively revise the response, then treat the
  revised text as the reference (response-adapted reference) for the subsequent
  evaluation. Extensive experiments demonstrate that RevisEval outperforms
  traditional reference-free and reference-based evaluation paradigms that use
  LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks.
  More importantly, our response-adapted references can further boost the
  classical text metrics, e.g., BLEU and BERTScore, compared to traditional
  references and even rival the LLM-as-a-Judge. A detailed analysis is also
  conducted to confirm RevisEval's effectiveness in bias reduction, the impact of
  inference cost, and reference relevance.
paper_ai_keywords: RevisEval response-adapted references text revision capabilities large language models (LLMs) LLM-as-a-Judge NLG tasks open-ended instruction-following tasks reference-free reference-based evaluation paradigms BLEU BERTScore bias reduction