title: Routing to the Expert: Efficient Reward-guided Ensemble of Large
    Language Models
summary: The complementary potential of Large Language Models (LLM) assumes
  off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and
  tasks so that an ensemble of LLMs can achieve consistently better performance.
  Existing ensemble methods for LLMs mainly focus on reward model ranking of
  outputs, leading to significant computation overhead. To combat this issue, we
  revisit the complementary potential of LLMs and further elaborate it by mining
  latent expertise with off-the-shelf reward models. We propose Zooter, a
  reward-guided routing method distilling rewards on training queries to train a
  routing function, which can precisely distribute each query to the LLM with
  expertise about it. We also integrate a tag-based label enhancement to mitigate
  noise from uncertainty when using rewards as silver supervision. Zooter shows
  computation efficiency in inference as it introduces only a minor computation
  overhead of a routing function compared with reward model ranking methods. We
  evaluate Zooter on a comprehensive benchmark collection with 26 subsets on
  different domains and tasks. Zooter outperforms the best single model on
  average and ranks first on 44% of tasks, even surpassing multiple reward model
  ranking methods.
paper_authors._id: 6555a962f0cf8603f379f63c
paper_authors.name: Keming Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:36:09.260Z
paper_authors._id: 6453fa96ed6d7fede94408e0
paper_authors.avatarUrl: /avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
paper_authors.fullname: Keming Lu
paper_authors.type: user
paper_authors.user: keminglu
paper_authors._id: 6555a962f0cf8603f379f63d
paper_authors.name: Hongyi Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:36:24.403Z
paper_authors._id: 64b77b602f5a966b9731b14a
paper_authors.avatarUrl: /avatars/cdcf534bd451140dff47286095336535.svg
paper_authors.fullname: Hongyi Yuan
paper_authors.type: user
paper_authors.user: Yuan1997
paper_authors._id: 6555a962f0cf8603f379f63e
paper_authors.name: Runji Lin
paper_authors._id: 6555a962f0cf8603f379f63f
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:36:41.407Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 6555a962f0cf8603f379f640
paper_authors.name: Zheng Yuan
paper_authors._id: 6555a962f0cf8603f379f641
paper_authors.name: Chang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:37:34.817Z
paper_authors._id: 622892774323cef93a956a4a
paper_authors.avatarUrl: /avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
paper_authors.fullname: chang zhou
paper_authors.type: user
paper_authors.user: jiemizc
paper_authors._id: 6555a962f0cf8603f379f642
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:37:45.638Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: Routing to the Expert: Efficient Reward-guided Ensemble of Large
    Language Models
paper_summary: The complementary potential of Large Language Models (LLM) assumes
  off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and
  tasks so that an ensemble of LLMs can achieve consistently better performance.
  Existing ensemble methods for LLMs mainly focus on reward model ranking of
  outputs, leading to significant computation overhead. To combat this issue, we
  revisit the complementary potential of LLMs and further elaborate it by mining
  latent expertise with off-the-shelf reward models. We propose Zooter, a
  reward-guided routing method distilling rewards on training queries to train a
  routing function, which can precisely distribute each query to the LLM with
  expertise about it. We also integrate a tag-based label enhancement to mitigate
  noise from uncertainty when using rewards as silver supervision. Zooter shows
  computation efficiency in inference as it introduces only a minor computation
  overhead of a routing function compared with reward model ranking methods. We
  evaluate Zooter on a comprehensive benchmark collection with 26 subsets on
  different domains and tasks. Zooter outperforms the best single model on
  average and ranks first on 44% of tasks, even surpassing multiple reward model
  ranking methods.
paper_ai_keywords: Large LanguageModels (LLM) ensemble methods reward model ranking latent expertise reward-guided routing method routing function tag-based label enhancement silver supervision computational efficiency benchmark collection subsets domains tasks evaluation single model ranking methods