title: R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised
    and Reinforcement Learning
summary: Despite advances in reasoning and planning of R1-like models, Large Language
  Models (LLMs) still struggle with tasks requiring precise computation, symbolic
  manipulation, optimization, and algorithmic reasoning, in which textual
  reasoning lacks the rigor of code execution. A key challenge is enabling LLMs
  to decide when to use textual reasoning versus code generation. While OpenAI
  trains models to invoke a Code Interpreter as needed, public research lacks
  guidance on aligning pre-trained LLMs to effectively leverage code and
  generalize across diverse tasks. We present R1-Code-Interpreter, an extension
  of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and
  reinforcement learning (RL) to autonomously generate multiple code queries
  during step-by-step reasoning. We curate 144 reasoning and planning tasks (107
  for training, 37 for testing), each with over 200 diverse questions. We
  fine-tune Qwen-2.5 models (3B/7B/14B) using various SFT and RL strategies,
  investigating different answer formats, reasoning vs. non-reasoning models,
  cold vs. warm starts, GRPO vs. PPO, and masked vs. unmasked code outputs.
  Unlike prior RL work on narrow domains, we find that Code Interpreter training
  is significantly harder due to high task diversity and expensive code
  execution, highlighting the critical role of the SFT stage. Our final model,
  R1-CI-14B, improves average accuracy on the 37 test tasks from 44.0\% to
  64.1\%, outperforming GPT-4o (text-only: 58.6\%) and approaching GPT-4o with
  Code Interpreter (70.9\%), with the emergent self-checking behavior via code
  generation. Datasets, Codes, and Models are available at
  https://github.com/yongchao98/R1-Code-Interpreter and
  https://huggingface.co/yongchao98.
paper_authors._id: 683f11c08ba4378eeab62953
paper_authors.name: Yongchao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:02:07.714Z
paper_authors._id: 67266d21b7d88dbcf9e6c4aa
paper_authors.avatarUrl: /avatars/0328058e74c424473ef890d1fbdd3e4d.svg
paper_authors.fullname: Yongchao Chen
paper_authors.type: user
paper_authors.user: yongchao98
paper_authors._id: 683f11c08ba4378eeab62954
paper_authors.name: Yueying Liu
paper_authors._id: 683f11c08ba4378eeab62955
paper_authors.name: Junwei Zhou
paper_authors._id: 683f11c08ba4378eeab62956
paper_authors.name: Yilun Hao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-03T18:25:00.398Z
paper_authors._id: 65c24ce188812bbe20724208
paper_authors.avatarUrl: /avatars/f40826e6035dca7e217cb95724a4e952.svg
paper_authors.fullname: Yilun Hao
paper_authors.type: user
paper_authors.user: yilunhao
paper_authors._id: 683f11c08ba4378eeab62957
paper_authors.name: Jingquan Wang
paper_authors._id: 683f11c08ba4378eeab62958
paper_authors.name: Yang Zhang
paper_authors._id: 683f11c08ba4378eeab62959
paper_authors.name: Chuchu Fan
paper_title: R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised
    and Reinforcement Learning
paper_summary: Despite advances in reasoning and planning of R1-like models, Large Language
  Models (LLMs) still struggle with tasks requiring precise computation, symbolic
  manipulation, optimization, and algorithmic reasoning, in which textual
  reasoning lacks the rigor of code execution. A key challenge is enabling LLMs
  to decide when to use textual reasoning versus code generation. While OpenAI
  trains models to invoke a Code Interpreter as needed, public research lacks
  guidance on aligning pre-trained LLMs to effectively leverage code and
  generalize across diverse tasks. We present R1-Code-Interpreter, an extension
  of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and
  reinforcement learning (RL) to autonomously generate multiple code queries
  during step-by-step reasoning. We curate 144 reasoning and planning tasks (107
  for training, 37 for testing), each with over 200 diverse questions. We
  fine-tune Qwen-2.5 models (3B/7B/14B) using various SFT and RL strategies,
  investigating different answer formats, reasoning vs. non-reasoning models,
  cold vs. warm starts, GRPO vs. PPO, and masked vs. unmasked code outputs.
  Unlike prior RL work on narrow domains, we find that Code Interpreter training
  is significantly harder due to high task diversity and expensive code
  execution, highlighting the critical role of the SFT stage. Our final model,
  R1-CI-14B, improves average accuracy on the 37 test tasks from 44.0\% to
  64.1\%, outperforming GPT-4o (text-only: 58.6\%) and approaching GPT-4o with
  Code Interpreter (70.9\%), with the emergent self-checking behavior via code
  generation. Datasets, Codes, and Models are available at
  https://github.com/yongchao98/R1-Code-Interpreter and
  https://huggingface.co/yongchao98.
paper_ai_summary: R1-Code-Interpreter extends text-only LLMs with improved code generation abilities through supervised fine-tuning and reinforcement learning, enhancing performance on diverse reasoning and planning tasks.
paper_ai_keywords: Large Language Models LLMs textual reasoning code execution multi-turn supervised fine-tuning reinforcement learning R1-Code-Interpreter Qwen-2.5 GRPO PPO masked vs. unmasked code outputs self-checking behavior