title: MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical
    Reasoning
summary: The recently released GPT-4 Code Interpreter has demonstrated remarkable
  proficiency in solving challenging math problems, primarily attributed to its
  ability to seamlessly reason with natural language, generate code, execute
  code, and continue reasoning based on the execution output. In this paper, we
  present a method to fine-tune open-source language models, enabling them to use
  code for modeling and deriving math equations and, consequently, enhancing
  their mathematical reasoning abilities. We propose a method of generating novel
  and high-quality datasets with math problems and their code-based solutions,
  referred to as MathCodeInstruct. Each solution interleaves natural language,
  code, and execution results. We also introduce a customized supervised
  fine-tuning and inference approach. This approach yields the MathCoder models,
  a family of models capable of generating code-based solutions for solving
  challenging math problems. Impressively, the MathCoder models achieve
  state-of-the-art scores among open-source LLMs on the MATH (45.2%) and GSM8K
  (83.9%) datasets, substantially outperforming other open-source alternatives.
  Notably, the MathCoder model not only surpasses ChatGPT-3.5 and PaLM-2 on GSM8K
  and MATH but also outperforms GPT-4 on the competition-level MATH dataset. The
  dataset and models will be released at https://github.com/mathllm/MathCoder.
paper_authors._id: 651f6771be34a5f2cf5e0990
paper_authors.name: Ke Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:47:36.483Z
paper_authors._id: 64d592c28767727dffa1f002
paper_authors.avatarUrl: /avatars/fe38bcac944a2742dc12c624e62d24ef.svg
paper_authors.fullname: WangKe
paper_authors.type: user
paper_authors.user: scikkk
paper_authors._id: 651f6771be34a5f2cf5e0991
paper_authors.name: Houxing Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:26:55.129Z
paper_authors._id: 63e4b63dd6278c161be47f4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e4b63dd6278c161be47f4b/u0y1_o9WBqPJh8C8dimJ8.jpeg
paper_authors.fullname: Houxing Ren
paper_authors.type: user
paper_authors.user: Houxing
paper_authors._id: 651f6771be34a5f2cf5e0992
paper_authors.name: Aojun Zhou
paper_authors._id: 651f6771be34a5f2cf5e0993
paper_authors.name: Zimu Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:27:12.432Z
paper_authors._id: 64b0bfef2f2f9c345b87e673
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
paper_authors.fullname: Zimu Lu
paper_authors.type: user
paper_authors.user: luzimu
paper_authors._id: 651f6771be34a5f2cf5e0994
paper_authors.name: Sichun Luo
paper_authors._id: 651f6771be34a5f2cf5e0995
paper_authors.name: Weikang Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:27:29.478Z
paper_authors._id: 6337e1d1107c4835a04c8607
paper_authors.avatarUrl: /avatars/e85353b34c3da6efef84951934811fee.svg
paper_authors.fullname: Weikang Shi
paper_authors.type: user
paper_authors.user: shiwk20
paper_authors._id: 651f6771be34a5f2cf5e0996
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:40:18.584Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 651f6771be34a5f2cf5e0997
paper_authors.name: Linqi Song
paper_authors._id: 651f6771be34a5f2cf5e0998
paper_authors.name: Mingjie Zhan
paper_authors._id: 651f6771be34a5f2cf5e0999
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:28:15.657Z
paper_authors._id: 64d97a3a67c967b01512dd1c
paper_authors.avatarUrl: /avatars/8a885f9290d4f42de4c481bcfcae8d85.svg
paper_authors.fullname: Hongsheng Liu
paper_authors.type: user
paper_authors.user: hsliu
paper_title: MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical
    Reasoning
paper_summary: The recently released GPT-4 Code Interpreter has demonstrated remarkable
  proficiency in solving challenging math problems, primarily attributed to its
  ability to seamlessly reason with natural language, generate code, execute
  code, and continue reasoning based on the execution output. In this paper, we
  present a method to fine-tune open-source language models, enabling them to use
  code for modeling and deriving math equations and, consequently, enhancing
  their mathematical reasoning abilities. We propose a method of generating novel
  and high-quality datasets with math problems and their code-based solutions,
  referred to as MathCodeInstruct. Each solution interleaves natural language,
  code, and execution results. We also introduce a customized supervised
  fine-tuning and inference approach. This approach yields the MathCoder models,
  a family of models capable of generating code-based solutions for solving
  challenging math problems. Impressively, the MathCoder models achieve
  state-of-the-art scores among open-source LLMs on the MATH (45.2%) and GSM8K
  (83.9%) datasets, substantially outperforming other open-source alternatives.
  Notably, the MathCoder model not only surpasses ChatGPT-3.5 and PaLM-2 on GSM8K
  and MATH but also outperforms GPT-4 on the competition-level MATH dataset. The
  dataset and models will be released at https://github.com/mathllm/MathCoder.
paper_ai_summary: A method is presented for fine-tuning language models to solve math problems using code, achieving state-of-the-art results on MATH and GSM8K datasets.
paper_ai_keywords: fine-tuning language models MathCodeInstruct code-based solutions supervised fine-tuning inference MathCoder MATH dataset GSM8K dataset ChatGPT-3.5 PaLM-2 GPT-4