title: Quantitative LLM Judges
summary: LLM-as-a-judge is a framework in which a large language model (LLM)
  automatically evaluates the output of another LLM. We propose quantitative LLM
  judges, which align evaluation scores of existing LLM judges to human scores in
  a given domain using regression models. The models are trained to improve the
  score of the original judge by using the judge's textual evaluation and score.
  We present four quantitative judges for different types of absolute and
  relative feedback, which showcases the generality and versatility of our
  framework. Our framework is more computationally efficient than supervised
  fine-tuning and can be more statistically efficient when human feedback is
  limited, which is expected in most applications of our work. We validate these
  claims empirically on four datasets using two base judges. Our experiments show
  that quantitative judges can effectively improve the predictive power of
  existing judges through post-hoc modeling.
paper_authors._id: 6841008f2f66f731bf010feb
paper_authors.name: Aishwarya Sahoo
paper_authors._id: 6841008f2f66f731bf010fec
paper_authors.name: Jeevana Kruthi Karnuthala
paper_authors._id: 6841008f2f66f731bf010fed
paper_authors.name: Tushar Parmanand Budhwani
paper_authors._id: 6841008f2f66f731bf010fee
paper_authors.name: Pranchal Agarwal
paper_authors._id: 6841008f2f66f731bf010fef
paper_authors.name: Sankaran Vaidyanathan
paper_authors._id: 6841008f2f66f731bf010ff0
paper_authors.name: Alexa Siu
paper_authors._id: 6841008f2f66f731bf010ff1
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:26:55.170Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 6841008f2f66f731bf010ff2
paper_authors.name: Jennifer Healey
paper_authors._id: 6841008f2f66f731bf010ff3
paper_authors.name: Nedim Lipka
paper_authors._id: 6841008f2f66f731bf010ff4
paper_authors.name: Ryan Rossi
paper_authors._id: 6841008f2f66f731bf010ff5
paper_authors.name: Uttaran Bhattacharya
paper_authors._id: 6841008f2f66f731bf010ff6
paper_authors.name: Branislav Kveton
paper_title: Quantitative LLM Judges
paper_summary: LLM-as-a-judge is a framework in which a large language model (LLM)
  automatically evaluates the output of another LLM. We propose quantitative LLM
  judges, which align evaluation scores of existing LLM judges to human scores in
  a given domain using regression models. The models are trained to improve the
  score of the original judge by using the judge's textual evaluation and score.
  We present four quantitative judges for different types of absolute and
  relative feedback, which showcases the generality and versatility of our
  framework. Our framework is more computationally efficient than supervised
  fine-tuning and can be more statistically efficient when human feedback is
  limited, which is expected in most applications of our work. We validate these
  claims empirically on four datasets using two base judges. Our experiments show
  that quantitative judges can effectively improve the predictive power of
  existing judges through post-hoc modeling.
paper_ai_summary: A framework uses quantitative LLM judges to align existing LLM evaluation scores with human scores, improving predictive power and efficiency through regression models.
paper_ai_keywords: LLM-as-a-judge large language model quantitative LLM judges regression models score alignment predictive power post-hoc modeling