Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via
  Reinforcement Learning
The rapid emergence of diverse large language models (LLMs) has spurred the
development of LLM routers that assign user queries to the most suitable model.
However, existing LLM routers typically perform a single-round, one-to-one
mapping (i.e., assigning each query to a single model in isolation),
which limits their capability to tackle complex tasks that demand the
complementary strengths of multiple LLMs. In this paper, we present
Router-R1, a reinforcement learning (RL)-based framework that
formulates multi-LLM routing and aggregation as a sequential decision process.
Router-R1 instantiates the router itself as a capable LLM, leveraging its
reasoning ability to interleave "think" actions (internal deliberation) with
"route" actions (dynamic model invocation), and integrates each response into
its evolving context. To guide learning, we employ a lightweight rule-based
reward comprising format rewards, final outcome rewards, and a novel cost
reward for performance and cost trade-off optimization, opening a pathway
toward optimizing performance-cost tradeoffs via RL. Router-R1 also conditions
only on simple model descriptors such as pricing, latency, and example
performance, enabling strong generalization to unseen model selection.
Experiments on seven general and multi-hop QA benchmarks show that Router-R1
outperforms over several strong baselines, achieving superior performance while
maintaining robust generalization and cost management.Code is available at
https://github.com/ulab-uiuc/Router-R1.
6852ac4a7eb90a35d35de5f1
Haozhen Zhang
6852ac4a7eb90a35d35de5f2
Tao Feng
6852ac4a7eb90a35d35de5f3
Jiaxuan You
Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via
  Reinforcement Learning
The rapid emergence of diverse large language models (LLMs) has spurred the
development of LLM routers that assign user queries to the most suitable model.
However, existing LLM routers typically perform a single-round, one-to-one
mapping (i.e., assigning each query to a single model in isolation),
which limits their capability to tackle complex tasks that demand the
complementary strengths of multiple LLMs. In this paper, we present
Router-R1, a reinforcement learning (RL)-based framework that
formulates multi-LLM routing and aggregation as a sequential decision process.
Router-R1 instantiates the router itself as a capable LLM, leveraging its
reasoning ability to interleave "think" actions (internal deliberation) with
"route" actions (dynamic model invocation), and integrates each response into
its evolving context. To guide learning, we employ a lightweight rule-based
reward comprising format rewards, final outcome rewards, and a novel cost
reward for performance and cost trade-off optimization, opening a pathway
toward optimizing performance-cost tradeoffs via RL. Router-R1 also conditions
only on simple model descriptors such as pricing, latency, and example
performance, enabling strong generalization to unseen model selection.
Experiments on seven general and multi-hop QA benchmarks show that Router-R1
outperforms over several strong baselines, achieving superior performance while
maintaining robust generalization and cost management.Code is available at
https://github.com/ulab-uiuc/Router-R1.
Router-R1, a reinforcement learning-based framework, improves multi-LLM routing by interleaving think and route actions, optimizing performance-cost trade-offs, and generalizing to unseen models.
LLMs reinforcement learning multi-LLM routing aggregation sequential decision process think actions route actions context integration reward shaping format rewards final outcome rewards cost reward performance optimization cost management generalization