Quantitative LLM Judges
LLM-as-a-judge is a framework in which a large language model (LLM)
automatically evaluates the output of another LLM. We propose quantitative LLM
judges, which align evaluation scores of existing LLM judges to human scores in
a given domain using regression models. The models are trained to improve the
score of the original judge by using the judge's textual evaluation and score.
We present four quantitative judges for different types of absolute and
relative feedback, which showcases the generality and versatility of our
framework. Our framework is more computationally efficient than supervised
fine-tuning and can be more statistically efficient when human feedback is
limited, which is expected in most applications of our work. We validate these
claims empirically on four datasets using two base judges. Our experiments show
that quantitative judges can effectively improve the predictive power of
existing judges through post-hoc modeling.
6841008f2f66f731bf010feb
Aishwarya Sahoo
6841008f2f66f731bf010fec
Jeevana Kruthi Karnuthala
6841008f2f66f731bf010fed
Tushar Parmanand Budhwani
6841008f2f66f731bf010fee
Pranchal Agarwal
6841008f2f66f731bf010fef
Sankaran Vaidyanathan
6841008f2f66f731bf010ff0
Alexa Siu
6841008f2f66f731bf010ff1
Franck Dernoncourt
claimed_verified
2025-06-05T07:26:55.170Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
6841008f2f66f731bf010ff2
Jennifer Healey
6841008f2f66f731bf010ff3
Nedim Lipka
6841008f2f66f731bf010ff4
Ryan Rossi
6841008f2f66f731bf010ff5
Uttaran Bhattacharya
6841008f2f66f731bf010ff6
Branislav Kveton
Quantitative LLM Judges
LLM-as-a-judge is a framework in which a large language model (LLM)
automatically evaluates the output of another LLM. We propose quantitative LLM
judges, which align evaluation scores of existing LLM judges to human scores in
a given domain using regression models. The models are trained to improve the
score of the original judge by using the judge's textual evaluation and score.
We present four quantitative judges for different types of absolute and
relative feedback, which showcases the generality and versatility of our
framework. Our framework is more computationally efficient than supervised
fine-tuning and can be more statistically efficient when human feedback is
limited, which is expected in most applications of our work. We validate these
claims empirically on four datasets using two base judges. Our experiments show
that quantitative judges can effectively improve the predictive power of
existing judges through post-hoc modeling.
A framework uses quantitative LLM judges to align existing LLM evaluation scores with human scores, improving predictive power and efficiency through regression models.
LLM-as-a-judge large language model quantitative LLM judges regression models score alignment predictive power post-hoc modeling