title: SuperCorrect: Supervising and Correcting Language Models with
    Error-Driven Insights
summary: Large language models (LLMs) like GPT-4, PaLM, and LLaMA have shown
  significant improvements in various reasoning tasks. However, smaller models
  such as Llama-3-8B and DeepSeekMath-Base still struggle with complex
  mathematical reasoning because they fail to effectively identify and correct
  reasoning errors. Recent reflection-based methods aim to address these issues
  by enabling self-reflection and self-correction, but they still face challenges
  in independently detecting errors in their reasoning steps. To overcome these
  limitations, we propose SuperCorrect, a novel two-stage framework that uses a
  large teacher model to supervise and correct both the reasoning and reflection
  processes of a smaller student model. In the first stage, we extract
  hierarchical high-level and detailed thought templates from the teacher model
  to guide the student model in eliciting more fine-grained reasoning thoughts.
  In the second stage, we introduce cross-model collaborative direct preference
  optimization (DPO) to enhance the self-correction abilities of the student
  model by following the teacher's correction traces during training. This
  cross-model DPO approach teaches the student model to effectively locate and
  resolve erroneous thoughts with error-driven insights from the teacher model,
  breaking the bottleneck of its thoughts and acquiring new skills and knowledge
  to tackle challenging problems. Extensive experiments consistently demonstrate
  our superiority over previous methods. Notably, our SuperCorrect-7B model
  significantly surpasses powerful DeepSeekMath-7B by 7.8%/5.3% and
  Qwen2.5-Math-7B by 15.1%/6.3% on MATH/GSM8K benchmarks, achieving new SOTA
  performance among all 7B models. Code:
  https://github.com/YangLing0818/SuperCorrect-llm
paper_authors._id: 670c74c39f3ee99ddfb15805
paper_authors.name: Ling Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:35:39.751Z
paper_authors._id: 64fde4e252e82dd432b74ce9
paper_authors.avatarUrl: /avatars/061a69d858b86d1600be916122cae7fc.svg
paper_authors.fullname: Ling Yang
paper_authors.type: user
paper_authors.user: Lingaaaaaaa
paper_authors._id: 670c74c39f3ee99ddfb15806
paper_authors.name: Zhaochen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:35:45.584Z
paper_authors._id: 64a131a7660cce8b86bf288d
paper_authors.avatarUrl: /avatars/6c1a2475645a1a6ae3f804fe6c35a226.svg
paper_authors.fullname: zhao chen yu
paper_authors.type: user
paper_authors.user: chenyu01
paper_authors._id: 670c74c39f3ee99ddfb15807
paper_authors.name: Tianjun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:35:51.068Z
paper_authors._id: 6374cd6b6ea8da14f8fef8dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
paper_authors.fullname: Tianjun Zhang
paper_authors.type: user
paper_authors.user: tianjunz
paper_authors._id: 670c74c39f3ee99ddfb15808
paper_authors.name: Minkai Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:35:57.450Z
paper_authors._id: 64c0e950aa57599de1c75dad
paper_authors.avatarUrl: /avatars/374d53317cbccc30fae70e5152ca13e0.svg
paper_authors.fullname: Minkai Xu
paper_authors.type: user
paper_authors.user: mkxu
paper_authors._id: 670c74c39f3ee99ddfb15809
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:36:03.351Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 670c74c39f3ee99ddfb1580a
paper_authors.name: Bin Cui
paper_authors._id: 670c74c39f3ee99ddfb1580b
paper_authors.name: Shuicheng Yan
paper_title: SuperCorrect: Supervising and Correcting Language Models with
    Error-Driven Insights
paper_summary: Large language models (LLMs) like GPT-4, PaLM, and LLaMA have shown
  significant improvements in various reasoning tasks. However, smaller models
  such as Llama-3-8B and DeepSeekMath-Base still struggle with complex
  mathematical reasoning because they fail to effectively identify and correct
  reasoning errors. Recent reflection-based methods aim to address these issues
  by enabling self-reflection and self-correction, but they still face challenges
  in independently detecting errors in their reasoning steps. To overcome these
  limitations, we propose SuperCorrect, a novel two-stage framework that uses a
  large teacher model to supervise and correct both the reasoning and reflection
  processes of a smaller student model. In the first stage, we extract
  hierarchical high-level and detailed thought templates from the teacher model
  to guide the student model in eliciting more fine-grained reasoning thoughts.
  In the second stage, we introduce cross-model collaborative direct preference
  optimization (DPO) to enhance the self-correction abilities of the student
  model by following the teacher's correction traces during training. This
  cross-model DPO approach teaches the student model to effectively locate and
  resolve erroneous thoughts with error-driven insights from the teacher model,
  breaking the bottleneck of its thoughts and acquiring new skills and knowledge
  to tackle challenging problems. Extensive experiments consistently demonstrate
  our superiority over previous methods. Notably, our SuperCorrect-7B model
  significantly surpasses powerful DeepSeekMath-7B by 7.8%/5.3% and
  Qwen2.5-Math-7B by 15.1%/6.3% on MATH/GSM8K benchmarks, achieving new SOTA
  performance among all 7B models. Code:
  https://github.com/YangLing0818/SuperCorrect-llm
paper_ai_keywords: Large language models (LLMs) GPT-4 PaLM LLaMA Llama-3-8B DeepSeekMath-Base complex mathematical reasoning reasoning errors reflection-based methods self-reflection self-correction SuperCorrect two-stage framework large teacher model student model hierarchical high-level and detailed thought templates fine-grained reasoning thoughts cross-model collaborative direct preference optimization (DPO) error-driven insights MATH/GSM8K benchmarks SuperCorrect-7B DeepSeekMath-7B Qwen2.2-Math-7B state-of-the-art (SOTA) performance