title: ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
summary: While reasoning models (e.g., DeepSeek R1) trained with reinforcement
  learning (RL), excel in textual reasoning, they struggle in scenarios requiring
  structured problem-solving, such as geometric reasoning, concise computation,
  or complex equation solving-areas where computational tools like code
  interpreters (CI) demonstrate distinct advantages. To bridge this gap, we
  propose ReTool, which enhances long-form reasoning with tool-integrated
  learning, including two key features: (1) dynamic interleaving of real-time
  code execution within natural language reasoning processes, and (2) an
  automated RL paradigm that allows policy rollouts with multi-turn real-time
  code execution and teaches the model in learning when and how to invoke tools
  based on outcome feedback. ReTool employs a systematic training framework,
  beginning with synthetic cold-start data generation to produce code-augmented
  long-form reasoning traces for fine-tuning base models. Subsequent RL training
  leverages task outcomes as rewards to iteratively refine the model's tool use
  strategy, enabling autonomous discovery of optimal tool invocation patterns
  without human priors. Experiments on the challenging MATH Olympiad benchmark
  AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with
  400 training steps, outperforming text-based RL baseline (40% accuracy, 1080
  steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5%
  accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further
  analysis reveals emergent behaviors such as code self-correction, signaling an
  ''aha moment'' in which the model autonomously masters adaptive tool use. These
  findings highlight the promise of outcome-driven tool integration for advancing
  complex mathematical reasoning and offer new insights into hybrid
  neuro-symbolic systems.
paper_authors._id: 6800cc7159e20f50cc282e87
paper_authors.name: Jiazhan Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:40:16.327Z
paper_authors._id: 649a95f951e1ea30f1475530
paper_authors.avatarUrl: /avatars/d4d167c3985e702044e8290408dde661.svg
paper_authors.fullname: Jamie Jiazhan Feng
paper_authors.type: user
paper_authors.user: jzfeng
paper_authors._id: 6800cc7159e20f50cc282e88
paper_authors.name: Shijue Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:11:05.968Z
paper_authors._id: 64ce05c631c655ff8a2e183c
paper_authors.avatarUrl: /avatars/f2de7f8a1348b05f46946085e3e9718e.svg
paper_authors.fullname: Shijue Huang
paper_authors.type: user
paper_authors.user: JoeYing
paper_authors._id: 6800cc7159e20f50cc282e89
paper_authors.name: Xingwei Qu
paper_authors._id: 6800cc7159e20f50cc282e8a
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:11:27.962Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6800cc7159e20f50cc282e8b
paper_authors.name: Yujia Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:11:35.665Z
paper_authors._id: 643f37cce9d063936912048b
paper_authors.avatarUrl: /avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
paper_authors.fullname: Yujia Qin
paper_authors.type: user
paper_authors.user: YujiaHi
paper_authors._id: 6800cc7159e20f50cc282e8c
paper_authors.name: Baoquan Zhong
paper_authors._id: 6800cc7159e20f50cc282e8d
paper_authors.name: Chengquan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:11:51.371Z
paper_authors._id: 6698fe04a188ffb7e412deb7
paper_authors.avatarUrl: /avatars/e389e72e37916b74efc14724d56a0cf1.svg
paper_authors.fullname: Chengquan Jiang
paper_authors.type: user
paper_authors.user: imjcqt
paper_authors._id: 6800cc7159e20f50cc282e8e
paper_authors.name: Jinxin Chi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:11:57.367Z
paper_authors._id: 671ca8f1299315f77400863a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lB32V9C2R2D1rAXhmz73w.png
paper_authors.fullname: Jinxin Chi
paper_authors.type: user
paper_authors.user: chijx
paper_authors._id: 6800cc7159e20f50cc282e8f
paper_authors.name: Wanjun Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:12:03.535Z
paper_authors._id: 643f956635e2b54a42e7feba
paper_authors.avatarUrl: /avatars/c6185f81ae8499ae866ad451c1cbf43b.svg
paper_authors.fullname: Wanjun Zhong
paper_authors.type: user
paper_authors.user: WanjunZhong
paper_title: ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
paper_summary: While reasoning models (e.g., DeepSeek R1) trained with reinforcement
  learning (RL), excel in textual reasoning, they struggle in scenarios requiring
  structured problem-solving, such as geometric reasoning, concise computation,
  or complex equation solving-areas where computational tools like code
  interpreters (CI) demonstrate distinct advantages. To bridge this gap, we
  propose ReTool, which enhances long-form reasoning with tool-integrated
  learning, including two key features: (1) dynamic interleaving of real-time
  code execution within natural language reasoning processes, and (2) an
  automated RL paradigm that allows policy rollouts with multi-turn real-time
  code execution and teaches the model in learning when and how to invoke tools
  based on outcome feedback. ReTool employs a systematic training framework,
  beginning with synthetic cold-start data generation to produce code-augmented
  long-form reasoning traces for fine-tuning base models. Subsequent RL training
  leverages task outcomes as rewards to iteratively refine the model's tool use
  strategy, enabling autonomous discovery of optimal tool invocation patterns
  without human priors. Experiments on the challenging MATH Olympiad benchmark
  AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with
  400 training steps, outperforming text-based RL baseline (40% accuracy, 1080
  steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5%
  accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further
  analysis reveals emergent behaviors such as code self-correction, signaling an
  ''aha moment'' in which the model autonomously masters adaptive tool use. These
  findings highlight the promise of outcome-driven tool integration for advancing
  complex mathematical reasoning and offer new insights into hybrid
  neuro-symbolic systems.
paper_ai_summary: ReTool, a tool-integrated learning framework, enhances reasoning models with real-time code execution and reinforcement learning, significantly improving performance in structured problem-solving tasks like mathematical reasoning.
paper_ai_keywords: reasoning models reinforcement learning RL code interpreters CI tool-integrated learning long-form reasoning real-time code execution policy rollouts multi-turn execution synthetic cold-start data fine-tuning MATH Olympiad benchmark AIME accuracy OpenAI's o1-preview code self-correction hybrid neuro-symbolic systems