title: From Generation to Judgment: Opportunities and Challenges of
    LLM-as-a-judge
summary: Assessment and evaluation have long been critical challenges in artificial
  intelligence (AI) and natural language processing (NLP). However, traditional
  methods, whether matching-based or embedding-based, often fall short of judging
  subtle attributes and delivering satisfactory results. Recent advancements in
  Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs
  are leveraged to perform scoring, ranking, or selection across various tasks
  and applications. This paper provides a comprehensive survey of LLM-based
  judgment and assessment, offering an in-depth overview to advance this emerging
  field. We begin by giving detailed definitions from both input and output
  perspectives. Then we introduce a comprehensive taxonomy to explore
  LLM-as-a-judge from three dimensions: what to judge, how to judge and where to
  judge. Finally, we compile benchmarks for evaluating LLM-as-a-judge and
  highlight key challenges and promising directions, aiming to provide valuable
  insights and inspire future research in this promising research area. Paper
  list and more resources about LLM-as-a-judge can be found at
  https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge and
  https://llm-as-a-judge.github.io.
paper_authors._id: 674547c6cc0f907b7bb44733
paper_authors.name: Dawei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:01.472Z
paper_authors._id: 6474e1afb68461d5cf7c41cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6474e1afb68461d5cf7c41cc/bcoiD_qPrjHUBlB259djg.png
paper_authors.fullname: Dawei Li
paper_authors.type: user
paper_authors.user: wjldw
paper_authors._id: 674547c6cc0f907b7bb44734
paper_authors.name: Bohan Jiang
paper_authors._id: 674547c6cc0f907b7bb44735
paper_authors.name: Liangjie Huang
paper_authors._id: 674547c6cc0f907b7bb44736
paper_authors.name: Alimohammad Beigi
paper_authors._id: 674547c6cc0f907b7bb44737
paper_authors.name: Chengshuai Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:31:33.087Z
paper_authors._id: 65b2fae679954e21ac426aec
paper_authors.avatarUrl: /avatars/c495c4ee9afb140a87ebc04602aa8c3e.svg
paper_authors.fullname: Chengshuai Zhao
paper_authors.type: user
paper_authors.user: chengshuaizhao
paper_authors._id: 674547c6cc0f907b7bb44738
paper_authors.name: Zhen Tan
paper_authors._id: 674547c6cc0f907b7bb44739
paper_authors.name: Amrita Bhattacharjee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:32:39.034Z
paper_authors._id: 64c2cecadeb9c5c29681ef63
paper_authors.avatarUrl: /avatars/6386b86644e33a382dfcd27558e8c1c2.svg
paper_authors.fullname: Amrita Bhattacharjee
paper_authors.type: user
paper_authors.user: AmritaBh
paper_authors._id: 674547c6cc0f907b7bb4473a
paper_authors.name: Yuxuan Jiang
paper_authors._id: 674547c6cc0f907b7bb4473b
paper_authors.name: Canyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:02.969Z
paper_authors._id: 6483af58571c2dcfa98cae82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483af58571c2dcfa98cae82/d8Vnh_EG1KxeEoNAAQXHZ.jpeg
paper_authors.fullname: Canyu Chen
paper_authors.type: user
paper_authors.user: canyuchen
paper_authors._id: 674547c6cc0f907b7bb4473c
paper_authors.name: Tianhao Wu
paper_authors._id: 674547c6cc0f907b7bb4473d
paper_authors.name: Kai Shu
paper_authors._id: 674547c6cc0f907b7bb4473e
paper_authors.name: Lu Cheng
paper_authors._id: 674547c6cc0f907b7bb4473f
paper_authors.name: Huan Liu
paper_title: From Generation to Judgment: Opportunities and Challenges of
    LLM-as-a-judge
paper_summary: Assessment and evaluation have long been critical challenges in artificial
  intelligence (AI) and natural language processing (NLP). However, traditional
  methods, whether matching-based or embedding-based, often fall short of judging
  subtle attributes and delivering satisfactory results. Recent advancements in
  Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs
  are leveraged to perform scoring, ranking, or selection across various tasks
  and applications. This paper provides a comprehensive survey of LLM-based
  judgment and assessment, offering an in-depth overview to advance this emerging
  field. We begin by giving detailed definitions from both input and output
  perspectives. Then we introduce a comprehensive taxonomy to explore
  LLM-as-a-judge from three dimensions: what to judge, how to judge and where to
  judge. Finally, we compile benchmarks for evaluating LLM-as-a-judge and
  highlight key challenges and promising directions, aiming to provide valuable
  insights and inspire future research in this promising research area. Paper
  list and more resources about LLM-as-a-judge can be found at
  https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge and
  https://llm-as-a-judge.github.io.
paper_ai_keywords: Large Language Models (LLMs) LLM-as-a-judge scoring ranking selection benchmarks