title: Fusion-Eval: Integrating Evaluators with LLMs
summary: Evaluating Large Language Models (LLMs) is a complex task, especially
  considering the intricacies of natural language understanding and the
  expectations for high-level reasoning. Traditional evaluations typically lean
  on human-based, model-based, or automatic-metrics-based paradigms, each with
  its own advantages and shortcomings. We introduce "Fusion-Eval", a system that
  employs LLMs not solely for direct evaluations, but to skillfully integrate
  insights from diverse evaluators. This gives Fusion-Eval flexibility, enabling
  it to work effectively across diverse tasks and make optimal use of multiple
  references. In testing on the SummEval dataset, Fusion-Eval achieved a Spearman
  correlation of 0.96, outperforming other evaluators. The success of Fusion-Eval
  underscores the potential of LLMs to produce evaluations that closely align
  human perspectives, setting a new standard in the field of LLM evaluation.
paper_authors._id: 6555abf289fd41f8aff7dac8
paper_authors.name: Lei Shu
paper_authors._id: 6555abf289fd41f8aff7dac9
paper_authors.name: Nevan Wichers
paper_authors._id: 6555abf289fd41f8aff7daca
paper_authors.name: Liangchen Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T09:42:16.046Z
paper_authors._id: 645a9ab321ab438e732a7b8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bNp7XtXqPQoauh7CU5B7b.jpeg
paper_authors.fullname: Liangchen Luo
paper_authors.type: user
paper_authors.user: luolc
paper_authors._id: 6555abf289fd41f8aff7dacb
paper_authors.name: Yun Zhu
paper_authors._id: 6555abf289fd41f8aff7dacc
paper_authors.name: Yinxiao Liu
paper_authors._id: 6555abf289fd41f8aff7dacd
paper_authors.name: Jindong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T09:42:53.957Z
paper_authors._id: 635242b8b3678a43742baec6
paper_authors.avatarUrl: /avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
paper_authors.fullname: chenjindong
paper_authors.type: user
paper_authors.user: chenjindong
paper_authors._id: 6555abf289fd41f8aff7dace
paper_authors.name: Lei Meng
paper_title: Fusion-Eval: Integrating Evaluators with LLMs
paper_summary: Evaluating Large Language Models (LLMs) is a complex task, especially
  considering the intricacies of natural language understanding and the
  expectations for high-level reasoning. Traditional evaluations typically lean
  on human-based, model-based, or automatic-metrics-based paradigms, each with
  its own advantages and shortcomings. We introduce "Fusion-Eval", a system that
  employs LLMs not solely for direct evaluations, but to skillfully integrate
  insights from diverse evaluators. This gives Fusion-Eval flexibility, enabling
  it to work effectively across diverse tasks and make optimal use of multiple
  references. In testing on the SummEval dataset, Fusion-Eval achieved a Spearman
  correlation of 0.96, outperforming other evaluators. The success of Fusion-Eval
  underscores the potential of LLMs to produce evaluations that closely align
  human perspectives, setting a new standard in the field of LLM evaluation.
paper_ai_summary: Fusion-Eval leverages LLMs to integrate insights from various evaluators, achieving high correlation and setting a new standard in LLM evaluation.
paper_ai_keywords: Fusion-Eval Large Language Models LLMs SummEval Spearman correlation