ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
While reasoning models (e.g., DeepSeek R1) trained with reinforcement
learning (RL), excel in textual reasoning, they struggle in scenarios requiring
structured problem-solving, such as geometric reasoning, concise computation,
or complex equation solving-areas where computational tools like code
interpreters (CI) demonstrate distinct advantages. To bridge this gap, we
propose ReTool, which enhances long-form reasoning with tool-integrated
learning, including two key features: (1) dynamic interleaving of real-time
code execution within natural language reasoning processes, and (2) an
automated RL paradigm that allows policy rollouts with multi-turn real-time
code execution and teaches the model in learning when and how to invoke tools
based on outcome feedback. ReTool employs a systematic training framework,
beginning with synthetic cold-start data generation to produce code-augmented
long-form reasoning traces for fine-tuning base models. Subsequent RL training
leverages task outcomes as rewards to iteratively refine the model's tool use
strategy, enabling autonomous discovery of optimal tool invocation patterns
without human priors. Experiments on the challenging MATH Olympiad benchmark
AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with
400 training steps, outperforming text-based RL baseline (40% accuracy, 1080
steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5%
accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further
analysis reveals emergent behaviors such as code self-correction, signaling an
''aha moment'' in which the model autonomously masters adaptive tool use. These
findings highlight the promise of outcome-driven tool integration for advancing
complex mathematical reasoning and offer new insights into hybrid
neuro-symbolic systems.
6800cc7159e20f50cc282e87
Jiazhan Feng
claimed_verified
2025-04-18T09:40:16.327Z
649a95f951e1ea30f1475530
/avatars/d4d167c3985e702044e8290408dde661.svg
Jamie Jiazhan Feng
user
jzfeng
6800cc7159e20f50cc282e88
Shijue Huang
admin_assigned
2025-04-17T11:11:05.968Z
64ce05c631c655ff8a2e183c
/avatars/f2de7f8a1348b05f46946085e3e9718e.svg
Shijue Huang
user
JoeYing
6800cc7159e20f50cc282e89
Xingwei Qu
6800cc7159e20f50cc282e8a
Ge Zhang
admin_assigned
2025-04-17T11:11:27.962Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6800cc7159e20f50cc282e8b
Yujia Qin
admin_assigned
2025-04-17T11:11:35.665Z
643f37cce9d063936912048b
/avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
Yujia Qin
user
YujiaHi
6800cc7159e20f50cc282e8c
Baoquan Zhong
6800cc7159e20f50cc282e8d
Chengquan Jiang
admin_assigned
2025-04-17T11:11:51.371Z
6698fe04a188ffb7e412deb7
/avatars/e389e72e37916b74efc14724d56a0cf1.svg
Chengquan Jiang
user
imjcqt
6800cc7159e20f50cc282e8e
Jinxin Chi
admin_assigned
2025-04-17T11:11:57.367Z
671ca8f1299315f77400863a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lB32V9C2R2D1rAXhmz73w.png
Jinxin Chi
user
chijx
6800cc7159e20f50cc282e8f
Wanjun Zhong
admin_assigned
2025-04-17T11:12:03.535Z
643f956635e2b54a42e7feba
/avatars/c6185f81ae8499ae866ad451c1cbf43b.svg
Wanjun Zhong
user
WanjunZhong
ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
While reasoning models (e.g., DeepSeek R1) trained with reinforcement
learning (RL), excel in textual reasoning, they struggle in scenarios requiring
structured problem-solving, such as geometric reasoning, concise computation,
or complex equation solving-areas where computational tools like code
interpreters (CI) demonstrate distinct advantages. To bridge this gap, we
propose ReTool, which enhances long-form reasoning with tool-integrated
learning, including two key features: (1) dynamic interleaving of real-time
code execution within natural language reasoning processes, and (2) an
automated RL paradigm that allows policy rollouts with multi-turn real-time
code execution and teaches the model in learning when and how to invoke tools
based on outcome feedback. ReTool employs a systematic training framework,
beginning with synthetic cold-start data generation to produce code-augmented
long-form reasoning traces for fine-tuning base models. Subsequent RL training
leverages task outcomes as rewards to iteratively refine the model's tool use
strategy, enabling autonomous discovery of optimal tool invocation patterns
without human priors. Experiments on the challenging MATH Olympiad benchmark
AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with
400 training steps, outperforming text-based RL baseline (40% accuracy, 1080
steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5%
accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further
analysis reveals emergent behaviors such as code self-correction, signaling an
''aha moment'' in which the model autonomously masters adaptive tool use. These
findings highlight the promise of outcome-driven tool integration for advancing
complex mathematical reasoning and offer new insights into hybrid
neuro-symbolic systems.
ReTool, a tool-integrated learning framework, enhances reasoning models with real-time code execution and reinforcement learning, significantly improving performance in structured problem-solving tasks like mathematical reasoning.
reasoning models reinforcement learning RL code interpreters CI tool-integrated learning long-form reasoning real-time code execution policy rollouts multi-turn execution synthetic cold-start data fine-tuning MATH Olympiad benchmark AIME accuracy OpenAI's o1-preview code self-correction hybrid neuro-symbolic systems