title: LEMMA: Learning from Errors for MatheMatical Advancement in LLMs
summary: Large language models (LLMs) have demonstrated remarkable reasoning
  capability in solving mathematical problems. However, existing approaches
  primarily focus on improving the quality of correct training data, e.g.,
  distilling high-quality correct solutions from advanced models, neglecting the
  value contained in error data, potentially hindering the model's reflective
  ability. Though some studies attempt to leverage error data, they often involve
  complex mechanisms, such as Monte Carlo Tree Search (MCTS) to explore error
  nodes. In this work, we propose to enhance LLMs' reasoning ability by Learning
  from Errors for Mathematical Advancement (LEMMA). LEMMA constructs data
  consisting of an incorrect solution with an erroneous step and a reflection
  connection to a correct solution for fine-tuning. Specifically, we
  systematically analyze the model-generated error types and introduce an
  error-type grounded mistake augmentation method to collect diverse and
  representative errors. Correct solutions are either from fixing the errors or
  generating a fresh start. Through a model-aware smooth reflection connection,
  the erroneous solution is transferred to the correct one. By fine-tuning on the
  constructed dataset, the model is able to self-correct errors autonomously
  within the generation process without relying on external critique models.
  Experimental results demonstrate that LEMMA achieves significant performance
  improvements over other strong baselines.
paper_authors._id: 67e21f63fb4213c53714be08
paper_authors.name: Zhuoshi Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:17:23.682Z
paper_authors._id: 6565e24fe5aac326bfd15a9d
paper_authors.avatarUrl: /avatars/28ad90df0e0dbc10ef25ee6499a50dec.svg
paper_authors.fullname: Zhuoshi Pan
paper_authors.type: user
paper_authors.user: panzs
paper_authors._id: 67e21f63fb4213c53714be09
paper_authors.name: Yu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-29T08:00:04.396Z
paper_authors._id: 671b852aa4fa4f8f5fb5404c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/671b852aa4fa4f8f5fb5404c/TDLsgP8WgKW-qaA8Ys-iJ.jpeg
paper_authors.fullname: YU LI
paper_authors.type: user
paper_authors.user: yu0226
paper_authors._id: 67e21f63fb4213c53714be0a
paper_authors.name: Honglin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:17:30.635Z
paper_authors._id: 640d99628512ec51d7ef71c7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640d99628512ec51d7ef71c7/fcBkqnxfxuuuZTqfN_BGy.jpeg
paper_authors.fullname: Honglin Lin
paper_authors.type: user
paper_authors.user: LHL3341
paper_authors._id: 67e21f63fb4213c53714be0b
paper_authors.name: Qizhi Pei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:17:42.836Z
paper_authors._id: 6397f6081323f19c578f142e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6397f6081323f19c578f142e/it7FYYKjlLX8wSsMLm8EO.jpeg
paper_authors.fullname: QizhiPei
paper_authors.type: user
paper_authors.user: QizhiPei
paper_authors._id: 67e21f63fb4213c53714be0c
paper_authors.name: Zinan Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:17:50.791Z
paper_authors._id: 66580d3d80ee5b1e11a94e57
paper_authors.avatarUrl: /avatars/1a88e7337f9095c40c6d402fab797d83.svg
paper_authors.fullname: Zinan Tang
paper_authors.type: user
paper_authors.user: Word2Li
paper_authors._id: 67e21f63fb4213c53714be0d
paper_authors.name: Wei Wu
paper_authors._id: 67e21f63fb4213c53714be0e
paper_authors.name: Chenlin Ming
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:18:12.485Z
paper_authors._id: 677e133ee86d0754dc7ce296
paper_authors.avatarUrl: /avatars/c16511c1876b50c2d049925c5f320d15.svg
paper_authors.fullname: mingchenlin
paper_authors.type: user
paper_authors.user: mingchenlin2025
paper_authors._id: 67e21f63fb4213c53714be0f
paper_authors.name: H. Vicky Zhao
paper_authors._id: 67e21f63fb4213c53714be10
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:18:25.368Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 67e21f63fb4213c53714be11
paper_authors.name: Lijun Wu
paper_title: LEMMA: Learning from Errors for MatheMatical Advancement in LLMs
paper_summary: Large language models (LLMs) have demonstrated remarkable reasoning
  capability in solving mathematical problems. However, existing approaches
  primarily focus on improving the quality of correct training data, e.g.,
  distilling high-quality correct solutions from advanced models, neglecting the
  value contained in error data, potentially hindering the model's reflective
  ability. Though some studies attempt to leverage error data, they often involve
  complex mechanisms, such as Monte Carlo Tree Search (MCTS) to explore error
  nodes. In this work, we propose to enhance LLMs' reasoning ability by Learning
  from Errors for Mathematical Advancement (LEMMA). LEMMA constructs data
  consisting of an incorrect solution with an erroneous step and a reflection
  connection to a correct solution for fine-tuning. Specifically, we
  systematically analyze the model-generated error types and introduce an
  error-type grounded mistake augmentation method to collect diverse and
  representative errors. Correct solutions are either from fixing the errors or
  generating a fresh start. Through a model-aware smooth reflection connection,
  the erroneous solution is transferred to the correct one. By fine-tuning on the
  constructed dataset, the model is able to self-correct errors autonomously
  within the generation process without relying on external critique models.
  Experimental results demonstrate that LEMMA achieves significant performance
  improvements over other strong baselines.
paper_ai_summary: LEMMA enhances LLMs' reasoning by fine-tuning on error data with reflection connections to correct solutions, showcasing performance improvements over existing methods.
paper_ai_keywords: large language models LLMs mathematical problems training data error data reflection ability Monte Carlo Tree Search MCTS Learning from Errors for Mathematical Advancement LEMMA mistake augmentation correct solutions reflection connection self-correction generation process baseline methods