title: PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning
    Optimization
summary: Instruction tuning large language models (LLMs) remains a challenging task,
  owing to the complexity of hyperparameter selection and the difficulty involved
  in evaluating the tuned models. To determine the optimal hyperparameters, an
  automatic, robust, and reliable evaluation benchmark is essential. However,
  establishing such a benchmark is not a trivial task due to the challenges
  associated with evaluation accuracy and privacy protection. In response to
  these challenges, we introduce a judge large language model, named PandaLM,
  which is trained to distinguish the superior model given several LLMs.
  PandaLM's focus extends beyond just the objective correctness of responses,
  which is the main focus of traditional evaluation datasets. It addresses vital
  subjective factors such as relative conciseness, clarity, adherence to
  instructions, comprehensiveness, and formality. To ensure the reliability of
  PandaLM, we collect a diverse human-annotated test dataset, where all contexts
  are generated by humans and labels are aligned with human preferences. Our
  results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation
  ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM
  enables the evaluation of LLM to be fairer but with less cost, evidenced by
  significant improvements achieved by models tuned through PandaLM compared to
  their counterparts trained with default Alpaca's hyperparameters. In addition,
  PandaLM does not depend on API-based evaluations, thus avoiding potential data
  leakage. All resources of PandaLM are released at
  https://github.com/WeOpenML/PandaLM.
paper_authors._id: 64827ce72e73ce18ad396ad3
paper_authors.name: Yidong Wang
paper_authors._id: 64827ce72e73ce18ad396ad4
paper_authors.name: Zhuohao Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T09:12:37.731Z
paper_authors._id: 61e24808b31e7cc38eb84d37
paper_authors.avatarUrl: /avatars/65fbea940fad211462ecc5ad725e0c28.svg
paper_authors.fullname: Zhuohao Yu
paper_authors.type: user
paper_authors.user: zhuohaoyu
paper_authors._id: 64827ce72e73ce18ad396ad5
paper_authors.name: Zhengran Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:24:19.832Z
paper_authors._id: 6438c233a5e10f6d586976a9
paper_authors.avatarUrl: /avatars/e5eb4d5580d1109de7e6a2769514a30d.svg
paper_authors.fullname: Zhengran Zeng
paper_authors.type: user
paper_authors.user: ZHENGRAN
paper_authors._id: 64827ce72e73ce18ad396ad6
paper_authors.name: Linyi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-14T09:29:34.202Z
paper_authors._id: 64895683f534abe18eec264b
paper_authors.avatarUrl: /avatars/73cc9e6db6db86793787750776b57c63.svg
paper_authors.fullname: Linyi Yang
paper_authors.type: user
paper_authors.user: linyiyang2023
paper_authors._id: 64827ce72e73ce18ad396ad7
paper_authors.name: Cunxiang Wang
paper_authors._id: 64827ce72e73ce18ad396ad8
paper_authors.name: Hao Chen
paper_authors._id: 64827ce72e73ce18ad396ad9
paper_authors.name: Chaoya Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:38:34.729Z
paper_authors._id: 61b9976efd429dff1ed3bd44
paper_authors.avatarUrl: /avatars/5cdaa04e970e1e3dcfbb38ba89c0660a.svg
paper_authors.fullname: jiangchaoya
paper_authors.type: user
paper_authors.user: jcy
paper_authors._id: 64827ce72e73ce18ad396ada
paper_authors.name: Rui Xie
paper_authors._id: 64827ce72e73ce18ad396adb
paper_authors.name: Jindong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:32:31.918Z
paper_authors._id: 6204cc0d522e40b4a18d86e2
paper_authors.avatarUrl: /avatars/18daf2de5671e711dc745388dd60569d.svg
paper_authors.fullname: Jindong Wang
paper_authors.type: user
paper_authors.user: jindongwang
paper_authors._id: 64827ce72e73ce18ad396adc
paper_authors.name: Xing Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:38:52.626Z
paper_authors._id: 64785ca82b3730b6b63b75a2
paper_authors.avatarUrl: /avatars/b0c4f7fb4d4c556118fa6b52d3615429.svg
paper_authors.fullname: Xing Xie
paper_authors.type: user
paper_authors.user: xingustc
paper_authors._id: 64827ce72e73ce18ad396add
paper_authors.name: Wei Ye
paper_authors._id: 64827ce72e73ce18ad396ade
paper_authors.name: Shikun Zhang
paper_authors._id: 64827ce72e73ce18ad396adf
paper_authors.name: Yue Zhang
paper_title: PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning
    Optimization
paper_summary: Instruction tuning large language models (LLMs) remains a challenging task,
  owing to the complexity of hyperparameter selection and the difficulty involved
  in evaluating the tuned models. To determine the optimal hyperparameters, an
  automatic, robust, and reliable evaluation benchmark is essential. However,
  establishing such a benchmark is not a trivial task due to the challenges
  associated with evaluation accuracy and privacy protection. In response to
  these challenges, we introduce a judge large language model, named PandaLM,
  which is trained to distinguish the superior model given several LLMs.
  PandaLM's focus extends beyond just the objective correctness of responses,
  which is the main focus of traditional evaluation datasets. It addresses vital
  subjective factors such as relative conciseness, clarity, adherence to
  instructions, comprehensiveness, and formality. To ensure the reliability of
  PandaLM, we collect a diverse human-annotated test dataset, where all contexts
  are generated by humans and labels are aligned with human preferences. Our
  results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation
  ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM
  enables the evaluation of LLM to be fairer but with less cost, evidenced by
  significant improvements achieved by models tuned through PandaLM compared to
  their counterparts trained with default Alpaca's hyperparameters. In addition,
  PandaLM does not depend on API-based evaluations, thus avoiding potential data
  leakage. All resources of PandaLM are released at
  https://github.com/WeOpenML/PandaLM.
paper_ai_keywords: large language models (LLMs) hyperparameter selection evaluation benchmark automatic evaluation robust evaluation reliable evaluation judge large language model PandaLM objective correctness relative conciseness clarity adherence to instructions comprehensiveness formality human-annotated test dataset F1-score API-based evaluations data leakage