Fusion-Eval: Integrating Evaluators with LLMs
Evaluating Large Language Models (LLMs) is a complex task, especially
considering the intricacies of natural language understanding and the
expectations for high-level reasoning. Traditional evaluations typically lean
on human-based, model-based, or automatic-metrics-based paradigms, each with
its own advantages and shortcomings. We introduce "Fusion-Eval", a system that
employs LLMs not solely for direct evaluations, but to skillfully integrate
insights from diverse evaluators. This gives Fusion-Eval flexibility, enabling
it to work effectively across diverse tasks and make optimal use of multiple
references. In testing on the SummEval dataset, Fusion-Eval achieved a Spearman
correlation of 0.96, outperforming other evaluators. The success of Fusion-Eval
underscores the potential of LLMs to produce evaluations that closely align
human perspectives, setting a new standard in the field of LLM evaluation.
6555abf289fd41f8aff7dac8
Lei Shu
6555abf289fd41f8aff7dac9
Nevan Wichers
6555abf289fd41f8aff7daca
Liangchen Luo
admin_assigned
2023-11-16T09:42:16.046Z
645a9ab321ab438e732a7b8d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bNp7XtXqPQoauh7CU5B7b.jpeg
Liangchen Luo
user
luolc
6555abf289fd41f8aff7dacb
Yun Zhu
6555abf289fd41f8aff7dacc
Yinxiao Liu
6555abf289fd41f8aff7dacd
Jindong Chen
admin_assigned
2023-11-16T09:42:53.957Z
635242b8b3678a43742baec6
/avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
chenjindong
user
chenjindong
6555abf289fd41f8aff7dace
Lei Meng
Fusion-Eval: Integrating Evaluators with LLMs
Evaluating Large Language Models (LLMs) is a complex task, especially
considering the intricacies of natural language understanding and the
expectations for high-level reasoning. Traditional evaluations typically lean
on human-based, model-based, or automatic-metrics-based paradigms, each with
its own advantages and shortcomings. We introduce "Fusion-Eval", a system that
employs LLMs not solely for direct evaluations, but to skillfully integrate
insights from diverse evaluators. This gives Fusion-Eval flexibility, enabling
it to work effectively across diverse tasks and make optimal use of multiple
references. In testing on the SummEval dataset, Fusion-Eval achieved a Spearman
correlation of 0.96, outperforming other evaluators. The success of Fusion-Eval
underscores the potential of LLMs to produce evaluations that closely align
human perspectives, setting a new standard in the field of LLM evaluation.
Fusion-Eval leverages LLMs to integrate insights from various evaluators, achieving high correlation and setting a new standard in LLM evaluation.
Fusion-Eval Large Language Models LLMs SummEval Spearman correlation