R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised
  and Reinforcement Learning
Despite advances in reasoning and planning of R1-like models, Large Language
Models (LLMs) still struggle with tasks requiring precise computation, symbolic
manipulation, optimization, and algorithmic reasoning, in which textual
reasoning lacks the rigor of code execution. A key challenge is enabling LLMs
to decide when to use textual reasoning versus code generation. While OpenAI
trains models to invoke a Code Interpreter as needed, public research lacks
guidance on aligning pre-trained LLMs to effectively leverage code and
generalize across diverse tasks. We present R1-Code-Interpreter, an extension
of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and
reinforcement learning (RL) to autonomously generate multiple code queries
during step-by-step reasoning. We curate 144 reasoning and planning tasks (107
for training, 37 for testing), each with over 200 diverse questions. We
fine-tune Qwen-2.5 models (3B/7B/14B) using various SFT and RL strategies,
investigating different answer formats, reasoning vs. non-reasoning models,
cold vs. warm starts, GRPO vs. PPO, and masked vs. unmasked code outputs.
Unlike prior RL work on narrow domains, we find that Code Interpreter training
is significantly harder due to high task diversity and expensive code
execution, highlighting the critical role of the SFT stage. Our final model,
R1-CI-14B, improves average accuracy on the 37 test tasks from 44.0\% to
64.1\%, outperforming GPT-4o (text-only: 58.6\%) and approaching GPT-4o with
Code Interpreter (70.9\%), with the emergent self-checking behavior via code
generation. Datasets, Codes, and Models are available at
https://github.com/yongchao98/R1-Code-Interpreter and
https://huggingface.co/yongchao98.
683f11c08ba4378eeab62953
Yongchao Chen
claimed_verified
2025-06-04T09:02:07.714Z
67266d21b7d88dbcf9e6c4aa
/avatars/0328058e74c424473ef890d1fbdd3e4d.svg
Yongchao Chen
user
yongchao98
683f11c08ba4378eeab62954
Yueying Liu
683f11c08ba4378eeab62955
Junwei Zhou
683f11c08ba4378eeab62956
Yilun Hao
extracted_confirmed
2025-06-03T18:25:00.398Z
65c24ce188812bbe20724208
/avatars/f40826e6035dca7e217cb95724a4e952.svg
Yilun Hao
user
yilunhao
683f11c08ba4378eeab62957
Jingquan Wang
683f11c08ba4378eeab62958
Yang Zhang
683f11c08ba4378eeab62959
Chuchu Fan
R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised
  and Reinforcement Learning
Despite advances in reasoning and planning of R1-like models, Large Language
Models (LLMs) still struggle with tasks requiring precise computation, symbolic
manipulation, optimization, and algorithmic reasoning, in which textual
reasoning lacks the rigor of code execution. A key challenge is enabling LLMs
to decide when to use textual reasoning versus code generation. While OpenAI
trains models to invoke a Code Interpreter as needed, public research lacks
guidance on aligning pre-trained LLMs to effectively leverage code and
generalize across diverse tasks. We present R1-Code-Interpreter, an extension
of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and
reinforcement learning (RL) to autonomously generate multiple code queries
during step-by-step reasoning. We curate 144 reasoning and planning tasks (107
for training, 37 for testing), each with over 200 diverse questions. We
fine-tune Qwen-2.5 models (3B/7B/14B) using various SFT and RL strategies,
investigating different answer formats, reasoning vs. non-reasoning models,
cold vs. warm starts, GRPO vs. PPO, and masked vs. unmasked code outputs.
Unlike prior RL work on narrow domains, we find that Code Interpreter training
is significantly harder due to high task diversity and expensive code
execution, highlighting the critical role of the SFT stage. Our final model,
R1-CI-14B, improves average accuracy on the 37 test tasks from 44.0\% to
64.1\%, outperforming GPT-4o (text-only: 58.6\%) and approaching GPT-4o with
Code Interpreter (70.9\%), with the emergent self-checking behavior via code
generation. Datasets, Codes, and Models are available at
https://github.com/yongchao98/R1-Code-Interpreter and
https://huggingface.co/yongchao98.
R1-Code-Interpreter extends text-only LLMs with improved code generation abilities through supervised fine-tuning and reinforcement learning, enhancing performance on diverse reasoning and planning tasks.
Large Language Models LLMs textual reasoning code execution multi-turn supervised fine-tuning reinforcement learning R1-Code-Interpreter Qwen-2.5 GRPO PPO masked vs. unmasked code outputs self-checking behavior