From Generation to Judgment: Opportunities and Challenges of
  LLM-as-a-judge
Assessment and evaluation have long been critical challenges in artificial
intelligence (AI) and natural language processing (NLP). However, traditional
methods, whether matching-based or embedding-based, often fall short of judging
subtle attributes and delivering satisfactory results. Recent advancements in
Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs
are leveraged to perform scoring, ranking, or selection across various tasks
and applications. This paper provides a comprehensive survey of LLM-based
judgment and assessment, offering an in-depth overview to advance this emerging
field. We begin by giving detailed definitions from both input and output
perspectives. Then we introduce a comprehensive taxonomy to explore
LLM-as-a-judge from three dimensions: what to judge, how to judge and where to
judge. Finally, we compile benchmarks for evaluating LLM-as-a-judge and
highlight key challenges and promising directions, aiming to provide valuable
insights and inspire future research in this promising research area. Paper
list and more resources about LLM-as-a-judge can be found at
https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge and
https://llm-as-a-judge.github.io.
674547c6cc0f907b7bb44733
Dawei Li
claimed_verified
2024-11-26T21:21:01.472Z
6474e1afb68461d5cf7c41cc
https://cdn-avatars.huggingface.co/v1/production/uploads/6474e1afb68461d5cf7c41cc/bcoiD_qPrjHUBlB259djg.png
Dawei Li
user
wjldw
674547c6cc0f907b7bb44734
Bohan Jiang
674547c6cc0f907b7bb44735
Liangjie Huang
674547c6cc0f907b7bb44736
Alimohammad Beigi
674547c6cc0f907b7bb44737
Chengshuai Zhao
admin_assigned
2024-11-26T22:31:33.087Z
65b2fae679954e21ac426aec
/avatars/c495c4ee9afb140a87ebc04602aa8c3e.svg
Chengshuai Zhao
user
chengshuaizhao
674547c6cc0f907b7bb44738
Zhen Tan
674547c6cc0f907b7bb44739
Amrita Bhattacharjee
admin_assigned
2024-11-26T22:32:39.034Z
64c2cecadeb9c5c29681ef63
/avatars/6386b86644e33a382dfcd27558e8c1c2.svg
Amrita Bhattacharjee
user
AmritaBh
674547c6cc0f907b7bb4473a
Yuxuan Jiang
674547c6cc0f907b7bb4473b
Canyu Chen
claimed_verified
2024-11-26T21:21:02.969Z
6483af58571c2dcfa98cae82
https://cdn-avatars.huggingface.co/v1/production/uploads/6483af58571c2dcfa98cae82/d8Vnh_EG1KxeEoNAAQXHZ.jpeg
Canyu Chen
user
canyuchen
674547c6cc0f907b7bb4473c
Tianhao Wu
674547c6cc0f907b7bb4473d
Kai Shu
674547c6cc0f907b7bb4473e
Lu Cheng
674547c6cc0f907b7bb4473f
Huan Liu
From Generation to Judgment: Opportunities and Challenges of
  LLM-as-a-judge
Assessment and evaluation have long been critical challenges in artificial
intelligence (AI) and natural language processing (NLP). However, traditional
methods, whether matching-based or embedding-based, often fall short of judging
subtle attributes and delivering satisfactory results. Recent advancements in
Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs
are leveraged to perform scoring, ranking, or selection across various tasks
and applications. This paper provides a comprehensive survey of LLM-based
judgment and assessment, offering an in-depth overview to advance this emerging
field. We begin by giving detailed definitions from both input and output
perspectives. Then we introduce a comprehensive taxonomy to explore
LLM-as-a-judge from three dimensions: what to judge, how to judge and where to
judge. Finally, we compile benchmarks for evaluating LLM-as-a-judge and
highlight key challenges and promising directions, aiming to provide valuable
insights and inspire future research in this promising research area. Paper
list and more resources about LLM-as-a-judge can be found at
https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge and
https://llm-as-a-judge.github.io.
A survey of LLM-based judgment and assessment explores definitions, taxonomy, benchmarks, challenges, and future directions for the LLM-as-a-judge paradigm.
LLM-as-a-judge Large Language Models LLM-based judgment assessment scoring ranking selection taxonomy benchmarks