title: ToolRL: Reward is All Tool Learning Needs
summary: Current Large Language Models (LLMs) often undergo supervised fine-tuning
  (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to
  unfamiliar or complex tool use scenarios. Recent advancements in reinforcement
  learning (RL), particularly with R1-like models, have demonstrated promising
  reasoning and generalization abilities. Yet, reward design for tool use
  presents unique challenges: multiple tools may be invoked with diverse
  parameters, and coarse-grained reward signals, such as answer matching, fail to
  offer the finegrained feedback required for effective learning. In this work,
  we present the first comprehensive study on reward design for tool selection
  and application tasks within the RL paradigm. We systematically explore a wide
  range of reward strategies, analyzing their types, scales, granularity, and
  temporal dynamics. Building on these insights, we propose a principled reward
  design tailored for tool use tasks and apply it to train LLMs using Group
  Relative Policy Optimization (GRPO). Empirical evaluations across diverse
  benchmarks demonstrate that our approach yields robust, scalable, and stable
  training, achieving a 17% improvement over base models and a 15% gain over SFT
  models. These results highlight the critical role of thoughtful reward design
  in enhancing the tool use capabilities and generalization performance of LLMs.
  All the codes are released to facilitate future research.
paper_authors._id: 6806fe1bec9fb3764b875ea0
paper_authors.name: Cheng Qian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:37:34.019Z
paper_authors._id: 665e121c6007027038fd4005
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sIVBJAGM-Kneq9KMf8aXb.png
paper_authors.fullname: Cheng Qian
paper_authors.type: user
paper_authors.user: chengq9
paper_authors._id: 6806fe1bec9fb3764b875ea1
paper_authors.name: Emre Can Acikgoz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:50:57.521Z
paper_authors._id: 63888d3fd68e37abd599f428
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63888d3fd68e37abd599f428/YaNyxG_oM6IgrHTkFZ6Eq.jpeg
paper_authors.fullname: emre can
paper_authors.type: user
paper_authors.user: emrecanacikgoz
paper_authors._id: 6806fe1bec9fb3764b875ea2
paper_authors.name: Qi He
paper_authors._id: 6806fe1bec9fb3764b875ea3
paper_authors.name: Hongru Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:52:19.954Z
paper_authors._id: 65f906e5c3dbdcae83ff7aac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
paper_authors.fullname: Hongru Wang
paper_authors.type: user
paper_authors.user: Merlin-Hongru
paper_authors._id: 6806fe1bec9fb3764b875ea4
paper_authors.name: Xiusi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:51:58.709Z
paper_authors._id: 6270ff726417aed8a7340c8b
paper_authors.avatarUrl: /avatars/3f14913c55cc4fc78678ac43fb603e80.svg
paper_authors.fullname: Xiusi Chen
paper_authors.type: user
paper_authors.user: XtremSup
paper_authors._id: 6806fe1bec9fb3764b875ea5
paper_authors.name: Dilek Hakkani-Tür
paper_authors._id: 6806fe1bec9fb3764b875ea6
paper_authors.name: Gokhan Tur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:51:42.581Z
paper_authors._id: 640f687c06c3b5ca88412985
paper_authors.avatarUrl: /avatars/6c1e538b78f52ce45ff124b0a5ec3369.svg
paper_authors.fullname: gokhan tur
paper_authors.type: user
paper_authors.user: Gokhantur
paper_authors._id: 6806fe1bec9fb3764b875ea7
paper_authors.name: Heng Ji
paper_title: ToolRL: Reward is All Tool Learning Needs
paper_summary: Current Large Language Models (LLMs) often undergo supervised fine-tuning
  (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to
  unfamiliar or complex tool use scenarios. Recent advancements in reinforcement
  learning (RL), particularly with R1-like models, have demonstrated promising
  reasoning and generalization abilities. Yet, reward design for tool use
  presents unique challenges: multiple tools may be invoked with diverse
  parameters, and coarse-grained reward signals, such as answer matching, fail to
  offer the finegrained feedback required for effective learning. In this work,
  we present the first comprehensive study on reward design for tool selection
  and application tasks within the RL paradigm. We systematically explore a wide
  range of reward strategies, analyzing their types, scales, granularity, and
  temporal dynamics. Building on these insights, we propose a principled reward
  design tailored for tool use tasks and apply it to train LLMs using Group
  Relative Policy Optimization (GRPO). Empirical evaluations across diverse
  benchmarks demonstrate that our approach yields robust, scalable, and stable
  training, achieving a 17% improvement over base models and a 15% gain over SFT
  models. These results highlight the critical role of thoughtful reward design
  in enhancing the tool use capabilities and generalization performance of LLMs.
  All the codes are released to facilitate future research.
paper_ai_summary: A comprehensive study on reward design for tool use within reinforcement learning improves LLMs' tool use capabilities and generalization performance over supervised fine-tuning.
paper_ai_keywords: supervised fine-tuning reinforcement learning R1-like models reward design tool selection application tasks reward strategies Group Relative Policy Optimization (GRPO)