Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement
  Learning
Recently, large language models (LLMs) have shown remarkable reasoning
capabilities via large-scale reinforcement learning (RL). However, leveraging
the RL algorithm to empower effective multi-tool collaborative reasoning in
LLMs remains an open challenge. In this paper, we introduce Tool-Star, an
RL-based framework designed to empower LLMs to autonomously invoke multiple
external tools during stepwise reasoning. Tool-Star integrates six types of
tools and incorporates systematic designs in both data synthesis and training.
To address the scarcity of tool-use data, we propose a general tool-integrated
reasoning data synthesis pipeline, which combines tool-integrated prompting
with hint-based sampling to automatically and scalably generate tool-use
trajectories. A subsequent quality normalization and difficulty-aware
classification process filters out low-quality samples and organizes the
dataset from easy to hard. Furthermore, we propose a two-stage training
framework to enhance multi-tool collaborative reasoning by: (1) cold-start
fine-tuning, which guides LLMs to explore reasoning patterns via
tool-invocation feedback; and (2) a multi-tool self-critic RL algorithm with
hierarchical reward design, which reinforces reward understanding and promotes
effective tool collaboration. Experimental analyses on over 10 challenging
reasoning benchmarks highlight the effectiveness and efficiency of Tool-Star.
The code is available at https://github.com/dongguanting/Tool-Star.
682fd6045e83dc325675312b
Guanting Dong
claimed_verified
2025-05-26T08:16:50.980Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
682fd6045e83dc325675312c
Yifei Chen
682fd6045e83dc325675312d
Xiaoxi Li
682fd6045e83dc325675312e
Jiajie Jin
682fd6045e83dc325675312f
Hongjin Qian
682fd6045e83dc3256753130
Yutao Zhu
682fd6045e83dc3256753131
Hangyu Mao
682fd6045e83dc3256753132
Guorui Zhou
682fd6045e83dc3256753133
Zhicheng Dou
682fd6045e83dc3256753134
Ji-Rong Wen
Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement
  Learning
Recently, large language models (LLMs) have shown remarkable reasoning
capabilities via large-scale reinforcement learning (RL). However, leveraging
the RL algorithm to empower effective multi-tool collaborative reasoning in
LLMs remains an open challenge. In this paper, we introduce Tool-Star, an
RL-based framework designed to empower LLMs to autonomously invoke multiple
external tools during stepwise reasoning. Tool-Star integrates six types of
tools and incorporates systematic designs in both data synthesis and training.
To address the scarcity of tool-use data, we propose a general tool-integrated
reasoning data synthesis pipeline, which combines tool-integrated prompting
with hint-based sampling to automatically and scalably generate tool-use
trajectories. A subsequent quality normalization and difficulty-aware
classification process filters out low-quality samples and organizes the
dataset from easy to hard. Furthermore, we propose a two-stage training
framework to enhance multi-tool collaborative reasoning by: (1) cold-start
fine-tuning, which guides LLMs to explore reasoning patterns via
tool-invocation feedback; and (2) a multi-tool self-critic RL algorithm with
hierarchical reward design, which reinforces reward understanding and promotes
effective tool collaboration. Experimental analyses on over 10 challenging
reasoning benchmarks highlight the effectiveness and efficiency of Tool-Star.
The code is available at https://github.com/dongguanting/Tool-Star.
Tool-Star, an RL-based framework, enables LLMs to autonomously use multiple tools for stepwise reasoning, leveraging data synthesis and hierarchical reward design.
large language models LLMs large-scale reinforcement learning RL multi-tool collaborative reasoning tool-use data tool-integrated reasoning tool-invocation feedback multi-tool self-critic hierarchical reward design