Prometheus 2: An Open Source Language Model Specialized in Evaluating
  Other Language Models
Proprietary LMs such as GPT-4 are often employed to assess the quality of
responses from various LMs. However, concerns including transparency,
controllability, and affordability strongly motivate the development of
open-source LMs specialized in evaluations. On the other hand, existing open
evaluator LMs exhibit critical shortcomings: 1) they issue scores that
significantly diverge from those assigned by humans, and 2) they lack the
flexibility to perform both direct assessment and pairwise ranking, the two
most prevalent forms of assessment. Additionally, they do not possess the
ability to evaluate based on custom evaluation criteria, focusing instead on
general attributes like helpfulness and harmlessness. To address these issues,
we introduce Prometheus 2, a more powerful evaluator LM than its predecessor
that closely mirrors human and GPT-4 judgements. Moreover, it is capable of
processing both direct assessment and pair-wise ranking formats grouped with a
user-defined evaluation criteria. On four direct assessment benchmarks and four
pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and
agreement with humans and proprietary LM judges among all tested open evaluator
LMs. Our models, code, and data are all publicly available at
https://github.com/prometheus-eval/prometheus-eval.
66344a6f925dc6852bfcf78e
Seungone Kim
claimed_verified
2024-05-03T07:16:19.366Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
66344a6f925dc6852bfcf78f
Juyoung Suk
claimed_verified
2024-05-03T07:16:22.822Z
6138cc1306dd10833d2db64b
https://cdn-avatars.huggingface.co/v1/production/uploads/6138cc1306dd10833d2db64b/IRX4y-8M4YlzR_8jOwkKp.jpeg
Juyoung Suk
user
juyoungml
66344a6f925dc6852bfcf790
Shayne Longpre
admin_assigned
2024-05-03T07:22:50.127Z
61f4283a81c4d30f58140242
/avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
Shayne Longpre
user
Shayne
66344a6f925dc6852bfcf791
Bill Yuchen Lin
admin_assigned
2024-05-03T07:23:02.022Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
66344a6f925dc6852bfcf792
Jamin Shin
claimed_verified
2025-04-24T12:41:19.901Z
6188cf3293317afcd1c2df7f
/avatars/95621801d5b3f3c1a681f1ad6cc66c6a.svg
Jay Shin
user
jshin49
66344a6f925dc6852bfcf793
Sean Welleck
admin_assigned
2024-05-03T07:23:43.029Z
63e3f3c59db5da2dc1ef6889
/avatars/f7546f57a5fd69bc99ff1640cc4a4853.svg
Sean Welleck
user
wellecks
66344a6f925dc6852bfcf794
Graham Neubig
admin_assigned
2024-05-03T07:23:49.331Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
66344a6f925dc6852bfcf795
Moontae Lee
66344a6f925dc6852bfcf796
Kyungjae Lee
66344a6f925dc6852bfcf797
Minjoon Seo
admin_assigned
2024-05-03T07:24:28.206Z
621f05ba970615ad5861ceb1
/avatars/7e1902aa71369a524afda9b0a9e88e22.svg
Minjoon Seo
user
minjoon
Prometheus 2: An Open Source Language Model Specialized in Evaluating
  Other Language Models
Proprietary LMs such as GPT-4 are often employed to assess the quality of
responses from various LMs. However, concerns including transparency,
controllability, and affordability strongly motivate the development of
open-source LMs specialized in evaluations. On the other hand, existing open
evaluator LMs exhibit critical shortcomings: 1) they issue scores that
significantly diverge from those assigned by humans, and 2) they lack the
flexibility to perform both direct assessment and pairwise ranking, the two
most prevalent forms of assessment. Additionally, they do not possess the
ability to evaluate based on custom evaluation criteria, focusing instead on
general attributes like helpfulness and harmlessness. To address these issues,
we introduce Prometheus 2, a more powerful evaluator LM than its predecessor
that closely mirrors human and GPT-4 judgements. Moreover, it is capable of
processing both direct assessment and pair-wise ranking formats grouped with a
user-defined evaluation criteria. On four direct assessment benchmarks and four
pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and
agreement with humans and proprietary LM judges among all tested open evaluator
LMs. Our models, code, and data are all publicly available at
https://github.com/prometheus-eval/prometheus-eval.
Prometheus 2 is an advanced open-source evaluator LM that aligns closely with human and proprietary LM judgments, supporting both direct assessment and pairwise ranking with customizable criteria.
LMs GPT-4 open-source LMs evaluation pairwise ranking direct assessment custom evaluation criteria Prometheus 2