title: Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement
    Learning
summary: Recently, large language models (LLMs) have shown remarkable reasoning
  capabilities via large-scale reinforcement learning (RL). However, leveraging
  the RL algorithm to empower effective multi-tool collaborative reasoning in
  LLMs remains an open challenge. In this paper, we introduce Tool-Star, an
  RL-based framework designed to empower LLMs to autonomously invoke multiple
  external tools during stepwise reasoning. Tool-Star integrates six types of
  tools and incorporates systematic designs in both data synthesis and training.
  To address the scarcity of tool-use data, we propose a general tool-integrated
  reasoning data synthesis pipeline, which combines tool-integrated prompting
  with hint-based sampling to automatically and scalably generate tool-use
  trajectories. A subsequent quality normalization and difficulty-aware
  classification process filters out low-quality samples and organizes the
  dataset from easy to hard. Furthermore, we propose a two-stage training
  framework to enhance multi-tool collaborative reasoning by: (1) cold-start
  fine-tuning, which guides LLMs to explore reasoning patterns via
  tool-invocation feedback; and (2) a multi-tool self-critic RL algorithm with
  hierarchical reward design, which reinforces reward understanding and promotes
  effective tool collaboration. Experimental analyses on over 10 challenging
  reasoning benchmarks highlight the effectiveness and efficiency of Tool-Star.
  The code is available at https://github.com/dongguanting/Tool-Star.
paper_authors._id: 682fd6045e83dc325675312b
paper_authors.name: Guanting Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:16:50.980Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 682fd6045e83dc325675312c
paper_authors.name: Yifei Chen
paper_authors._id: 682fd6045e83dc325675312d
paper_authors.name: Xiaoxi Li
paper_authors._id: 682fd6045e83dc325675312e
paper_authors.name: Jiajie Jin
paper_authors._id: 682fd6045e83dc325675312f
paper_authors.name: Hongjin Qian
paper_authors._id: 682fd6045e83dc3256753130
paper_authors.name: Yutao Zhu
paper_authors._id: 682fd6045e83dc3256753131
paper_authors.name: Hangyu Mao
paper_authors._id: 682fd6045e83dc3256753132
paper_authors.name: Guorui Zhou
paper_authors._id: 682fd6045e83dc3256753133
paper_authors.name: Zhicheng Dou
paper_authors._id: 682fd6045e83dc3256753134
paper_authors.name: Ji-Rong Wen
paper_title: Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement
    Learning
paper_summary: Recently, large language models (LLMs) have shown remarkable reasoning
  capabilities via large-scale reinforcement learning (RL). However, leveraging
  the RL algorithm to empower effective multi-tool collaborative reasoning in
  LLMs remains an open challenge. In this paper, we introduce Tool-Star, an
  RL-based framework designed to empower LLMs to autonomously invoke multiple
  external tools during stepwise reasoning. Tool-Star integrates six types of
  tools and incorporates systematic designs in both data synthesis and training.
  To address the scarcity of tool-use data, we propose a general tool-integrated
  reasoning data synthesis pipeline, which combines tool-integrated prompting
  with hint-based sampling to automatically and scalably generate tool-use
  trajectories. A subsequent quality normalization and difficulty-aware
  classification process filters out low-quality samples and organizes the
  dataset from easy to hard. Furthermore, we propose a two-stage training
  framework to enhance multi-tool collaborative reasoning by: (1) cold-start
  fine-tuning, which guides LLMs to explore reasoning patterns via
  tool-invocation feedback; and (2) a multi-tool self-critic RL algorithm with
  hierarchical reward design, which reinforces reward understanding and promotes
  effective tool collaboration. Experimental analyses on over 10 challenging
  reasoning benchmarks highlight the effectiveness and efficiency of Tool-Star.
  The code is available at https://github.com/dongguanting/Tool-Star.
paper_ai_summary: Tool-Star, an RL-based framework, enables LLMs to autonomously use multiple tools for stepwise reasoning, leveraging data synthesis and hierarchical reward design.
paper_ai_keywords: large language models LLMs large-scale reinforcement learning RL multi-tool collaborative reasoning tool-use data tool-integrated reasoning tool-invocation feedback multi-tool self-critic hierarchical reward design