title: LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and
    Generative Fusion
summary: We present LLM-Blender, an ensembling framework designed to attain
  consistently superior performance by leveraging the diverse strengths of
  multiple open-source large language models (LLMs). Our framework consists of
  two modules: PairRanker and GenFuser, addressing the observation that optimal
  LLMs for different examples can significantly vary. PairRanker employs a
  specialized pairwise comparison method to distinguish subtle differences
  between candidate outputs. It jointly encodes the input text and a pair of
  candidates, using cross-attention encoders to determine the superior one. Our
  results demonstrate that PairRanker exhibits the highest correlation with
  ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates,
  generating an improved output by capitalizing on their strengths and mitigating
  their weaknesses. To facilitate large-scale evaluation, we introduce a
  benchmark dataset, MixInstruct, which is a mixture of multiple instruction
  datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly
  outperform individual LLMs and baseline methods across various metrics,
  establishing a substantial performance gap.
paper_authors._id: 647e8610e4d52fe0e02108bf
paper_authors.name: Dongfu Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-06T10:03:21.541Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 647e8610e4d52fe0e02108c0
paper_authors.name: Xiang Ren
paper_authors._id: 647e8610e4d52fe0e02108c1
paper_authors.name: Bill Yuchen Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T08:40:05.051Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_title: LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and
    Generative Fusion
paper_summary: We present LLM-Blender, an ensembling framework designed to attain
  consistently superior performance by leveraging the diverse strengths of
  multiple open-source large language models (LLMs). Our framework consists of
  two modules: PairRanker and GenFuser, addressing the observation that optimal
  LLMs for different examples can significantly vary. PairRanker employs a
  specialized pairwise comparison method to distinguish subtle differences
  between candidate outputs. It jointly encodes the input text and a pair of
  candidates, using cross-attention encoders to determine the superior one. Our
  results demonstrate that PairRanker exhibits the highest correlation with
  ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates,
  generating an improved output by capitalizing on their strengths and mitigating
  their weaknesses. To facilitate large-scale evaluation, we introduce a
  benchmark dataset, MixInstruct, which is a mixture of multiple instruction
  datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly
  outperform individual LLMs and baseline methods across various metrics,
  establishing a substantial performance gap.
paper_ai_keywords: ensembling framework large language models (LLMs) PairRanker GenFuser pairwise comparison cross-attention encoders ChatGPT-based ranking MixInstruct oracle pairwise comparisons