title: Physics of Language Models: Part 2.2, How to Learn From Mistakes on
    Grade-School Math Problems
summary: Language models have demonstrated remarkable performance in solving reasoning
  tasks; however, even the strongest models still occasionally make reasoning
  mistakes. Recently, there has been active research aimed at improving reasoning
  accuracy, particularly by using pretrained language models to "self-correct"
  their mistakes via multi-round prompting. In this paper, we follow this line of
  work but focus on understanding the usefulness of incorporating
  "error-correction" data directly into the pretraining stage. This data consists
  of erroneous solution steps immediately followed by their corrections. Using a
  synthetic math dataset, we show promising results: this type of pretrain data
  can help language models achieve higher reasoning accuracy directly (i.e.,
  through simple auto-regression, without multi-round prompting) compared to
  pretraining on the same amount of error-free data. We also delve into many
  details, such as (1) how this approach differs from beam search, (2) how such
  data can be prepared, (3) whether masking is needed on the erroneous tokens,
  (4) the amount of error required, (5) whether such data can be deferred to the
  fine-tuning stage, and many others.
paper_authors._id: 66d23dd1960d799f1a69527b
paper_authors.name: Tian Ye
paper_authors._id: 66d23dd1960d799f1a69527c
paper_authors.name: Zicheng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:17:41.699Z
paper_authors._id: 66c7fc61d5a538bbb9e562b7
paper_authors.avatarUrl: /avatars/feac2e2273be88a7a44f09adb4f373e2.svg
paper_authors.fullname: Zicheng Xu
paper_authors.type: user
paper_authors.user: xu2333
paper_authors._id: 66d23dd1960d799f1a69527d
paper_authors.name: Yuanzhi Li
paper_authors._id: 66d23dd1960d799f1a69527e
paper_authors.name: Zeyuan Allen-Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:17:44.222Z
paper_authors._id: 6316d14c0db6c679cf77fe1f
paper_authors.avatarUrl: /avatars/b33dcadc0ff8941da6aa850284be3207.svg
paper_authors.fullname: Zeyuan Allen-Zhu
paper_authors.type: user
paper_authors.user: zhuzeyuan
paper_title: Physics of Language Models: Part 2.2, How to Learn From Mistakes on
    Grade-School Math Problems
paper_summary: Language models have demonstrated remarkable performance in solving reasoning
  tasks; however, even the strongest models still occasionally make reasoning
  mistakes. Recently, there has been active research aimed at improving reasoning
  accuracy, particularly by using pretrained language models to "self-correct"
  their mistakes via multi-round prompting. In this paper, we follow this line of
  work but focus on understanding the usefulness of incorporating
  "error-correction" data directly into the pretraining stage. This data consists
  of erroneous solution steps immediately followed by their corrections. Using a
  synthetic math dataset, we show promising results: this type of pretrain data
  can help language models achieve higher reasoning accuracy directly (i.e.,
  through simple auto-regression, without multi-round prompting) compared to
  pretraining on the same amount of error-free data. We also delve into many
  details, such as (1) how this approach differs from beam search, (2) how such
  data can be prepared, (3) whether masking is needed on the erroneous tokens,
  (4) the amount of error required, (5) whether such data can be deferred to the
  fine-tuning stage, and many others.
paper_ai_summary: Incorporating error-correction data into the pretraining of language models improves reasoning accuracy without multi-round prompting.
paper_ai_keywords: language models reasoning accuracy self-correction error-correction data synthetic math dataset auto-regression beam search masked tokens fine-tuning stage