title: MathCoder2: Better Math Reasoning from Continued Pretraining on
    Model-translated Mathematical Code
summary: Code has been shown to be effective in enhancing the mathematical reasoning
  abilities of large language models due to its precision and accuracy. Previous
  works involving continued mathematical pretraining often include code that
  utilizes math-related packages, which are primarily designed for fields such as
  engineering, machine learning, signal processing, or module testing, rather
  than being directly focused on mathematical reasoning. In this paper, we
  introduce a novel method for generating mathematical code accompanied with
  corresponding reasoning steps for continued pretraining. Our approach begins
  with the construction of a high-quality mathematical continued pretraining
  dataset by incorporating math-related web data, code using mathematical
  packages, math textbooks, and synthetic data. Next, we construct reasoning
  steps by extracting LaTeX expressions, the conditions needed for the
  expressions, and the results of the expressions from the previously collected
  dataset. Based on this extracted information, we generate corresponding code to
  accurately capture the mathematical reasoning process. Appending the generated
  code to each reasoning step results in data consisting of paired natural
  language reasoning steps and their corresponding code. Combining this data with
  the original dataset results in a 19.2B-token high-performing mathematical
  pretraining corpus, which we name MathCode-Pile. Training several popular base
  models with this corpus significantly improves their mathematical abilities,
  leading to the creation of the MathCoder2 family of models. All of our data
  processing and training code is open-sourced, ensuring full transparency and
  easy reproducibility of the entire data collection and training pipeline. The
  code is released at https://github.com/mathllm/MathCoder2 .
paper_authors._id: 6708bbc0d1c640e5586d21fe
paper_authors.name: Zimu Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:28:21.421Z
paper_authors._id: 64b0bfef2f2f9c345b87e673
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
paper_authors.fullname: Zimu Lu
paper_authors.type: user
paper_authors.user: luzimu
paper_authors._id: 6708bbc0d1c640e5586d21ff
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:43:33.832Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 6708bbc0d1c640e5586d2200
paper_authors.name: Ke Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:43:31.392Z
paper_authors._id: 64d592c28767727dffa1f002
paper_authors.avatarUrl: /avatars/fe38bcac944a2742dc12c624e62d24ef.svg
paper_authors.fullname: WangKe
paper_authors.type: user
paper_authors.user: scikkk
paper_authors._id: 6708bbc0d1c640e5586d2201
paper_authors.name: Houxing Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:24:55.884Z
paper_authors._id: 63e4b63dd6278c161be47f4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e4b63dd6278c161be47f4b/u0y1_o9WBqPJh8C8dimJ8.jpeg
paper_authors.fullname: Houxing Ren
paper_authors.type: user
paper_authors.user: Houxing
paper_authors._id: 6708bbc0d1c640e5586d2202
paper_authors.name: Weikang Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:25:02.066Z
paper_authors._id: 6337e1d1107c4835a04c8607
paper_authors.avatarUrl: /avatars/e85353b34c3da6efef84951934811fee.svg
paper_authors.fullname: Weikang Shi
paper_authors.type: user
paper_authors.user: shiwk20
paper_authors._id: 6708bbc0d1c640e5586d2203
paper_authors.name: Junting Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:25:08.002Z
paper_authors._id: 639c2d36e62c431aa8d834a7
paper_authors.avatarUrl: /avatars/a23d57dcb190711494a2a7e75e4cb9a8.svg
paper_authors.fullname: Junting Pan
paper_authors.type: user
paper_authors.user: juntingpan
paper_authors._id: 6708bbc0d1c640e5586d2204
paper_authors.name: Mingjie Zhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:25:27.517Z
paper_authors._id: 6301e46858f2b86bdfd49933
paper_authors.avatarUrl: /avatars/d1c3fd94581f22a41506b24375a22bd3.svg
paper_authors.fullname: ZhanMingjie
paper_authors.type: user
paper_authors.user: Mingjie
paper_authors._id: 6708bbc0d1c640e5586d2205
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:25:33.604Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: MathCoder2: Better Math Reasoning from Continued Pretraining on
    Model-translated Mathematical Code
paper_summary: Code has been shown to be effective in enhancing the mathematical reasoning
  abilities of large language models due to its precision and accuracy. Previous
  works involving continued mathematical pretraining often include code that
  utilizes math-related packages, which are primarily designed for fields such as
  engineering, machine learning, signal processing, or module testing, rather
  than being directly focused on mathematical reasoning. In this paper, we
  introduce a novel method for generating mathematical code accompanied with
  corresponding reasoning steps for continued pretraining. Our approach begins
  with the construction of a high-quality mathematical continued pretraining
  dataset by incorporating math-related web data, code using mathematical
  packages, math textbooks, and synthetic data. Next, we construct reasoning
  steps by extracting LaTeX expressions, the conditions needed for the
  expressions, and the results of the expressions from the previously collected
  dataset. Based on this extracted information, we generate corresponding code to
  accurately capture the mathematical reasoning process. Appending the generated
  code to each reasoning step results in data consisting of paired natural
  language reasoning steps and their corresponding code. Combining this data with
  the original dataset results in a 19.2B-token high-performing mathematical
  pretraining corpus, which we name MathCode-Pile. Training several popular base
  models with this corpus significantly improves their mathematical abilities,
  leading to the creation of the MathCoder2 family of models. All of our data
  processing and training code is open-sourced, ensuring full transparency and
  easy reproducibility of the entire data collection and training pipeline. The
  code is released at https://github.com/mathllm/MathCoder2 .
paper_ai_keywords: mathematical reasoning pretraining LaTeX expressions synthetic data natural language reasoning steps high-performing pretraining corpus MathCode-Pile MathCoder2 family of models