PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning
  Optimization
Instruction tuning large language models (LLMs) remains a challenging task,
owing to the complexity of hyperparameter selection and the difficulty involved
in evaluating the tuned models. To determine the optimal hyperparameters, an
automatic, robust, and reliable evaluation benchmark is essential. However,
establishing such a benchmark is not a trivial task due to the challenges
associated with evaluation accuracy and privacy protection. In response to
these challenges, we introduce a judge large language model, named PandaLM,
which is trained to distinguish the superior model given several LLMs.
PandaLM's focus extends beyond just the objective correctness of responses,
which is the main focus of traditional evaluation datasets. It addresses vital
subjective factors such as relative conciseness, clarity, adherence to
instructions, comprehensiveness, and formality. To ensure the reliability of
PandaLM, we collect a diverse human-annotated test dataset, where all contexts
are generated by humans and labels are aligned with human preferences. Our
results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation
ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM
enables the evaluation of LLM to be fairer but with less cost, evidenced by
significant improvements achieved by models tuned through PandaLM compared to
their counterparts trained with default Alpaca's hyperparameters. In addition,
PandaLM does not depend on API-based evaluations, thus avoiding potential data
leakage. All resources of PandaLM are released at
https://github.com/WeOpenML/PandaLM.
64827ce72e73ce18ad396ad3
Yidong Wang
64827ce72e73ce18ad396ad4
Zhuohao Yu
claimed_verified
2023-06-12T09:12:37.731Z
61e24808b31e7cc38eb84d37
/avatars/65fbea940fad211462ecc5ad725e0c28.svg
Zhuohao Yu
user
zhuohaoyu
64827ce72e73ce18ad396ad5
Zhengran Zeng
admin_assigned
2023-06-09T16:24:19.832Z
6438c233a5e10f6d586976a9
/avatars/e5eb4d5580d1109de7e6a2769514a30d.svg
Zhengran Zeng
user
ZHENGRAN
64827ce72e73ce18ad396ad6
Linyi Yang
claimed_verified
2023-06-14T09:29:34.202Z
64895683f534abe18eec264b
/avatars/73cc9e6db6db86793787750776b57c63.svg
Linyi Yang
user
linyiyang2023
64827ce72e73ce18ad396ad7
Cunxiang Wang
64827ce72e73ce18ad396ad8
Hao Chen
64827ce72e73ce18ad396ad9
Chaoya Jiang
admin_assigned
2023-06-12T13:38:34.729Z
61b9976efd429dff1ed3bd44
/avatars/5cdaa04e970e1e3dcfbb38ba89c0660a.svg
jiangchaoya
user
jcy
64827ce72e73ce18ad396ada
Rui Xie
64827ce72e73ce18ad396adb
Jindong Wang
admin_assigned
2023-06-09T16:32:31.918Z
6204cc0d522e40b4a18d86e2
/avatars/18daf2de5671e711dc745388dd60569d.svg
Jindong Wang
user
jindongwang
64827ce72e73ce18ad396adc
Xing Xie
admin_assigned
2023-06-12T13:38:52.626Z
64785ca82b3730b6b63b75a2
/avatars/b0c4f7fb4d4c556118fa6b52d3615429.svg
Xing Xie
user
xingustc
64827ce72e73ce18ad396add
Wei Ye
64827ce72e73ce18ad396ade
Shikun Zhang
64827ce72e73ce18ad396adf
Yue Zhang
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning
  Optimization
Instruction tuning large language models (LLMs) remains a challenging task,
owing to the complexity of hyperparameter selection and the difficulty involved
in evaluating the tuned models. To determine the optimal hyperparameters, an
automatic, robust, and reliable evaluation benchmark is essential. However,
establishing such a benchmark is not a trivial task due to the challenges
associated with evaluation accuracy and privacy protection. In response to
these challenges, we introduce a judge large language model, named PandaLM,
which is trained to distinguish the superior model given several LLMs.
PandaLM's focus extends beyond just the objective correctness of responses,
which is the main focus of traditional evaluation datasets. It addresses vital
subjective factors such as relative conciseness, clarity, adherence to
instructions, comprehensiveness, and formality. To ensure the reliability of
PandaLM, we collect a diverse human-annotated test dataset, where all contexts
are generated by humans and labels are aligned with human preferences. Our
results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation
ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM
enables the evaluation of LLM to be fairer but with less cost, evidenced by
significant improvements achieved by models tuned through PandaLM compared to
their counterparts trained with default Alpaca's hyperparameters. In addition,
PandaLM does not depend on API-based evaluations, thus avoiding potential data
leakage. All resources of PandaLM are released at
https://github.com/WeOpenML/PandaLM.
PandaLM, a judge LLM, evaluates and distinguishes between LLMs based on both objective correctness and subjective factors, providing a fairer and more cost-effective evaluation method.
large language models LLMs hyperparameter selection evaluation benchmark PandaLM relative conciseness clarity instruction adherence comprehensiveness formality human-annotated test dataset F1-score Alpaca's hyperparameters API-based evaluations data leakage