title: Calibrating LLM-Based Evaluator
summary: Recent advancements in large language models (LLMs) on language modeling and
  emergent capabilities make them a promising reference-free evaluator of natural
  language generation quality, and a competent alternative to human evaluation.
  However, hindered by the closed-source or high computational demand to host and
  tune, there is a lack of practice to further calibrate an off-the-shelf
  LLM-based evaluator towards better human alignment. In this work, we propose
  AutoCalibrate, a multi-stage, gradient-free approach to automatically calibrate
  and align an LLM-based evaluator toward human preference. Instead of explicitly
  modeling human preferences, we first implicitly encompass them within a set of
  human labels. Then, an initial set of scoring criteria is drafted by the
  language model itself, leveraging in-context learning on different few-shot
  examples. To further calibrate this set of criteria, we select the best
  performers and re-draft them with self-refinement. Our experiments on multiple
  text quality evaluation datasets illustrate a significant improvement in
  correlation with expert evaluation through calibration. Our comprehensive
  qualitative analysis conveys insightful intuitions and observations on the
  essence of effective scoring criteria.
paper_authors._id: 651271be2f0aa026dd6733df
paper_authors.name: Yuxuan Liu
paper_authors._id: 651271be2f0aa026dd6733e0
paper_authors.name: Tianchi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:07:33.805Z
paper_authors._id: 63b640f20d5913eee48bff48
paper_authors.avatarUrl: /avatars/a3ab337cdccaaac0652dba60ce581b14.svg
paper_authors.fullname: yangtianchi
paper_authors.type: user
paper_authors.user: asinica
paper_authors._id: 651271be2f0aa026dd6733e1
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:07:47.142Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 651271be2f0aa026dd6733e2
paper_authors.name: Zihan Zhang
paper_authors._id: 651271be2f0aa026dd6733e3
paper_authors.name: Haizhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-23T09:16:23.761Z
paper_authors._id: 6302e8e57e137e3d6b3f5a06
paper_authors.avatarUrl: /avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Haizhen
paper_authors._id: 651271be2f0aa026dd6733e4
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:21:39.779Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 651271be2f0aa026dd6733e5
paper_authors.name: Weiwei Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:23:56.509Z
paper_authors._id: 642da49dc1adada5b345c72b
paper_authors.avatarUrl: /avatars/5fa8d345a7248246454444999756105b.svg
paper_authors.fullname: deng
paper_authors.type: user
paper_authors.user: weiweideng
paper_authors._id: 651271be2f0aa026dd6733e6
paper_authors.name: Feng Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:24:04.058Z
paper_authors._id: 64b88a7449bde5d9481e4755
paper_authors.avatarUrl: /avatars/45a963517ec2c54c81ee80449d655014.svg
paper_authors.fullname: feng sun
paper_authors.type: user
paper_authors.user: forserty
paper_authors._id: 651271be2f0aa026dd6733e7
paper_authors.name: Qi Zhang
paper_title: Calibrating LLM-Based Evaluator
paper_summary: Recent advancements in large language models (LLMs) on language modeling and
  emergent capabilities make them a promising reference-free evaluator of natural
  language generation quality, and a competent alternative to human evaluation.
  However, hindered by the closed-source or high computational demand to host and
  tune, there is a lack of practice to further calibrate an off-the-shelf
  LLM-based evaluator towards better human alignment. In this work, we propose
  AutoCalibrate, a multi-stage, gradient-free approach to automatically calibrate
  and align an LLM-based evaluator toward human preference. Instead of explicitly
  modeling human preferences, we first implicitly encompass them within a set of
  human labels. Then, an initial set of scoring criteria is drafted by the
  language model itself, leveraging in-context learning on different few-shot
  examples. To further calibrate this set of criteria, we select the best
  performers and re-draft them with self-refinement. Our experiments on multiple
  text quality evaluation datasets illustrate a significant improvement in
  correlation with expert evaluation through calibration. Our comprehensive
  qualitative analysis conveys insightful intuitions and observations on the
  essence of effective scoring criteria.
paper_ai_summary: AutoCalibrate uses an iterative, gradient-free approach to automatically align an off-the-shelf LLM-based evaluator with human preferences, improving its correlation with expert evaluations.
paper_ai_keywords: large language models LLMs language modeling emergent capabilities reference-free evaluator natural language generation human evaluation closed-source computational demand human labels in-context learning few-shot examples self-refinement text quality evaluation expert evaluation scoring criteria