title: OTC: Optimal Tool Calls via Reinforcement Learning
summary: Tool-integrated reasoning (TIR) augments large language models (LLMs) with
  the ability to invoke external tools, such as search engines and code
  interpreters, to solve tasks beyond the capabilities of language-only
  reasoning. While reinforcement learning (RL) has shown promise in improving TIR
  by optimizing final answer correctness, existing approaches often overlook the
  efficiency and cost associated with tool usage. This can lead to suboptimal
  behavior, including excessive tool calls that increase computational and
  financial overhead, or insufficient tool use that compromises answer quality.
  In this work, we propose Optimal Tool Call-controlled Policy Optimization
  (OTC-PO), a simple yet effective RL-based framework that encourages models to
  produce accurate answers with minimal tool calls. Our method introduces a
  tool-integrated reward that jointly considers correctness and tool efficiency,
  promoting high tool productivity. We instantiate this framework within both
  Proximal Policy Optimization (PPO) and Group Relative Preference Optimization
  (GRPO), resulting in OTC-PPO and OTC-GRPO. Experiments with Qwen-2.5 and
  Qwen-Math across multiple QA benchmarks show that our approach reduces tool
  calls by up to 73.1\% and improves tool productivity by up to 229.4\%, while
  maintaining comparable answer accuracy. To the best of our knowledge, this is
  the first RL-based framework that explicitly optimizes tool-use efficiency in
  TIR.
paper_authors._id: 68070933142b618d3357d2bc
paper_authors.name: Hongru Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T12:50:29.698Z
paper_authors._id: 65f906e5c3dbdcae83ff7aac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
paper_authors.fullname: Hongru Wang
paper_authors.type: user
paper_authors.user: Merlin-Hongru
paper_authors._id: 68070933142b618d3357d2bd
paper_authors.name: Cheng Qian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:37:30.075Z
paper_authors._id: 665e121c6007027038fd4005
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sIVBJAGM-Kneq9KMf8aXb.png
paper_authors.fullname: Cheng Qian
paper_authors.type: user
paper_authors.user: chengq9
paper_authors._id: 68070933142b618d3357d2be
paper_authors.name: Wanjun Zhong
paper_authors._id: 68070933142b618d3357d2bf
paper_authors.name: Xiusi Chen
paper_authors._id: 68070933142b618d3357d2c0
paper_authors.name: Jiahao Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:14:57.203Z
paper_authors._id: 65271df5abd7795aaa1fd86e
paper_authors.avatarUrl: /avatars/99e1aabaefcf5e7f438817375cd1ddef.svg
paper_authors.fullname: Jiahao Qiu
paper_authors.type: user
paper_authors.user: jiahaoq
paper_authors._id: 68070933142b618d3357d2c1
paper_authors.name: Shijue Huang
paper_authors._id: 68070933142b618d3357d2c2
paper_authors.name: Bowen Jin
paper_authors._id: 68070933142b618d3357d2c3
paper_authors.name: Mengdi Wang
paper_authors._id: 68070933142b618d3357d2c4
paper_authors.name: Kam-Fai Wong
paper_authors._id: 68070933142b618d3357d2c5
paper_authors.name: Heng Ji
paper_title: OTC: Optimal Tool Calls via Reinforcement Learning
paper_summary: Tool-integrated reasoning (TIR) augments large language models (LLMs) with
  the ability to invoke external tools, such as search engines and code
  interpreters, to solve tasks beyond the capabilities of language-only
  reasoning. While reinforcement learning (RL) has shown promise in improving TIR
  by optimizing final answer correctness, existing approaches often overlook the
  efficiency and cost associated with tool usage. This can lead to suboptimal
  behavior, including excessive tool calls that increase computational and
  financial overhead, or insufficient tool use that compromises answer quality.
  In this work, we propose Optimal Tool Call-controlled Policy Optimization
  (OTC-PO), a simple yet effective RL-based framework that encourages models to
  produce accurate answers with minimal tool calls. Our method introduces a
  tool-integrated reward that jointly considers correctness and tool efficiency,
  promoting high tool productivity. We instantiate this framework within both
  Proximal Policy Optimization (PPO) and Group Relative Preference Optimization
  (GRPO), resulting in OTC-PPO and OTC-GRPO. Experiments with Qwen-2.5 and
  Qwen-Math across multiple QA benchmarks show that our approach reduces tool
  calls by up to 73.1\% and improves tool productivity by up to 229.4\%, while
  maintaining comparable answer accuracy. To the best of our knowledge, this is
  the first RL-based framework that explicitly optimizes tool-use efficiency in
  TIR.
paper_ai_summary: A novel RL framework optimizes tool usage in language models for enhanced efficiency and cost-effectiveness without sacrificing accuracy.
paper_ai_keywords: tool-integrated reasoning LLMs reinforcement learning RL Proximal Policy Optimization PPO Group Relative Preference Optimization GRPO tool-integrated reward tool productivity QA benchmarks