Routing to the Expert: Efficient Reward-guided Ensemble of Large
  Language Models
The complementary potential of Large Language Models (LLM) assumes
off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and
tasks so that an ensemble of LLMs can achieve consistently better performance.
Existing ensemble methods for LLMs mainly focus on reward model ranking of
outputs, leading to significant computation overhead. To combat this issue, we
revisit the complementary potential of LLMs and further elaborate it by mining
latent expertise with off-the-shelf reward models. We propose Zooter, a
reward-guided routing method distilling rewards on training queries to train a
routing function, which can precisely distribute each query to the LLM with
expertise about it. We also integrate a tag-based label enhancement to mitigate
noise from uncertainty when using rewards as silver supervision. Zooter shows
computation efficiency in inference as it introduces only a minor computation
overhead of a routing function compared with reward model ranking methods. We
evaluate Zooter on a comprehensive benchmark collection with 26 subsets on
different domains and tasks. Zooter outperforms the best single model on
average and ranks first on 44% of tasks, even surpassing multiple reward model
ranking methods.
6555a962f0cf8603f379f63c
Keming Lu
admin_assigned
2023-11-16T08:36:09.260Z
6453fa96ed6d7fede94408e0
/avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
Keming Lu
user
keminglu
6555a962f0cf8603f379f63d
Hongyi Yuan
admin_assigned
2023-11-16T08:36:24.403Z
64b77b602f5a966b9731b14a
/avatars/cdcf534bd451140dff47286095336535.svg
Hongyi Yuan
user
Yuan1997
6555a962f0cf8603f379f63e
Runji Lin
6555a962f0cf8603f379f63f
Junyang Lin
admin_assigned
2023-11-16T08:36:41.407Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
6555a962f0cf8603f379f640
Zheng Yuan
6555a962f0cf8603f379f641
Chang Zhou
admin_assigned
2023-11-16T08:37:34.817Z
622892774323cef93a956a4a
/avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
chang zhou
user
jiemizc
6555a962f0cf8603f379f642
Jingren Zhou
admin_assigned
2023-11-16T08:37:45.638Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
Routing to the Expert: Efficient Reward-guided Ensemble of Large
  Language Models
The complementary potential of Large Language Models (LLM) assumes
off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and
tasks so that an ensemble of LLMs can achieve consistently better performance.
Existing ensemble methods for LLMs mainly focus on reward model ranking of
outputs, leading to significant computation overhead. To combat this issue, we
revisit the complementary potential of LLMs and further elaborate it by mining
latent expertise with off-the-shelf reward models. We propose Zooter, a
reward-guided routing method distilling rewards on training queries to train a
routing function, which can precisely distribute each query to the LLM with
expertise about it. We also integrate a tag-based label enhancement to mitigate
noise from uncertainty when using rewards as silver supervision. Zooter shows
computation efficiency in inference as it introduces only a minor computation
overhead of a routing function compared with reward model ranking methods. We
evaluate Zooter on a comprehensive benchmark collection with 26 subsets on
different domains and tasks. Zooter outperforms the best single model on
average and ranks first on 44% of tasks, even surpassing multiple reward model
ranking methods.
Zooter, a reward-guided routing method for Large Language Models, improves efficiency and performance by precisely directing queries to the most suitable model, outperforming existing ensemble techniques.
Large Language Models ensemble methods reward model ranking latent expertise reward-guided routing Zooter routing function tag-based label enhancement silver supervision