title: Prometheus 2: An Open Source Language Model Specialized in Evaluating
    Other Language Models
summary: Proprietary LMs such as GPT-4 are often employed to assess the quality of
  responses from various LMs. However, concerns including transparency,
  controllability, and affordability strongly motivate the development of
  open-source LMs specialized in evaluations. On the other hand, existing open
  evaluator LMs exhibit critical shortcomings: 1) they issue scores that
  significantly diverge from those assigned by humans, and 2) they lack the
  flexibility to perform both direct assessment and pairwise ranking, the two
  most prevalent forms of assessment. Additionally, they do not possess the
  ability to evaluate based on custom evaluation criteria, focusing instead on
  general attributes like helpfulness and harmlessness. To address these issues,
  we introduce Prometheus 2, a more powerful evaluator LM than its predecessor
  that closely mirrors human and GPT-4 judgements. Moreover, it is capable of
  processing both direct assessment and pair-wise ranking formats grouped with a
  user-defined evaluation criteria. On four direct assessment benchmarks and four
  pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and
  agreement with humans and proprietary LM judges among all tested open evaluator
  LMs. Our models, code, and data are all publicly available at
  https://github.com/prometheus-eval/prometheus-eval.
paper_authors._id: 66344a6f925dc6852bfcf78e
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-03T07:16:19.366Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 66344a6f925dc6852bfcf78f
paper_authors.name: Juyoung Suk
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-03T07:16:22.822Z
paper_authors._id: 6138cc1306dd10833d2db64b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6138cc1306dd10833d2db64b/IRX4y-8M4YlzR_8jOwkKp.jpeg
paper_authors.fullname: Juyoung Suk
paper_authors.type: user
paper_authors.user: juyoungml
paper_authors._id: 66344a6f925dc6852bfcf790
paper_authors.name: Shayne Longpre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:22:50.127Z
paper_authors._id: 61f4283a81c4d30f58140242
paper_authors.avatarUrl: /avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
paper_authors.fullname: Shayne Longpre
paper_authors.type: user
paper_authors.user: Shayne
paper_authors._id: 66344a6f925dc6852bfcf791
paper_authors.name: Bill Yuchen Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:23:02.022Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 66344a6f925dc6852bfcf792
paper_authors.name: Jamin Shin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-24T12:41:19.901Z
paper_authors._id: 6188cf3293317afcd1c2df7f
paper_authors.avatarUrl: /avatars/95621801d5b3f3c1a681f1ad6cc66c6a.svg
paper_authors.fullname: Jay Shin
paper_authors.type: user
paper_authors.user: jshin49
paper_authors._id: 66344a6f925dc6852bfcf793
paper_authors.name: Sean Welleck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:23:43.029Z
paper_authors._id: 63e3f3c59db5da2dc1ef6889
paper_authors.avatarUrl: /avatars/f7546f57a5fd69bc99ff1640cc4a4853.svg
paper_authors.fullname: Sean Welleck
paper_authors.type: user
paper_authors.user: wellecks
paper_authors._id: 66344a6f925dc6852bfcf794
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:23:49.331Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 66344a6f925dc6852bfcf795
paper_authors.name: Moontae Lee
paper_authors._id: 66344a6f925dc6852bfcf796
paper_authors.name: Kyungjae Lee
paper_authors._id: 66344a6f925dc6852bfcf797
paper_authors.name: Minjoon Seo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:24:28.206Z
paper_authors._id: 621f05ba970615ad5861ceb1
paper_authors.avatarUrl: /avatars/7e1902aa71369a524afda9b0a9e88e22.svg
paper_authors.fullname: Minjoon Seo
paper_authors.type: user
paper_authors.user: minjoon
paper_title: Prometheus 2: An Open Source Language Model Specialized in Evaluating
    Other Language Models
paper_summary: Proprietary LMs such as GPT-4 are often employed to assess the quality of
  responses from various LMs. However, concerns including transparency,
  controllability, and affordability strongly motivate the development of
  open-source LMs specialized in evaluations. On the other hand, existing open
  evaluator LMs exhibit critical shortcomings: 1) they issue scores that
  significantly diverge from those assigned by humans, and 2) they lack the
  flexibility to perform both direct assessment and pairwise ranking, the two
  most prevalent forms of assessment. Additionally, they do not possess the
  ability to evaluate based on custom evaluation criteria, focusing instead on
  general attributes like helpfulness and harmlessness. To address these issues,
  we introduce Prometheus 2, a more powerful evaluator LM than its predecessor
  that closely mirrors human and GPT-4 judgements. Moreover, it is capable of
  processing both direct assessment and pair-wise ranking formats grouped with a
  user-defined evaluation criteria. On four direct assessment benchmarks and four
  pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and
  agreement with humans and proprietary LM judges among all tested open evaluator
  LMs. Our models, code, and data are all publicly available at
  https://github.com/prometheus-eval/prometheus-eval.
paper_ai_summary: Prometheus 2 is an advanced open-source evaluator LM that aligns closely with human and proprietary LM judgments, supporting both direct assessment and pairwise ranking with customizable criteria.
paper_ai_keywords: LMs GPT-4 open-source LMs evaluation pairwise ranking direct assessment custom evaluation criteria Prometheus 2