title: Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large
    Language Models -- The Story Goes On
summary: In this paper, we investigate the underlying factors that potentially enhance
  the mathematical reasoning capabilities of large language models (LLMs). We
  argue that the data scaling law for math reasoning capabilities in modern LLMs
  is far from being saturated, highlighting how the model's quality improves with
  increases in data quantity. To support this claim, we introduce the
  Skywork-Math model series, supervised fine-tuned (SFT) on common 7B LLMs using
  our proposed 2.5M-instance Skywork-MathQA dataset. Skywork-Math 7B has achieved
  impressive accuracies of 51.2% on the competition-level MATH benchmark and
  83.9% on the GSM8K benchmark using only SFT data, outperforming an early
  version of GPT-4 on MATH. The superior performance of Skywork-Math models
  contributes to our novel two-stage data synthesis and model SFT pipelines,
  which include three different augmentation methods and a diverse seed problem
  set, ensuring both the quantity and quality of Skywork-MathQA dataset across
  varying difficulty levels. Most importantly, we provide several practical
  takeaways to enhance math reasoning abilities in LLMs for both research and
  industry applications.
paper_authors._id: 669096575cb4c07caa232efd
paper_authors.name: Liang Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T07:44:28.572Z
paper_authors._id: 6621efe1a6eec3ad03e38759
paper_authors.avatarUrl: /avatars/c35acce69f244ec0833dffd53eedf6a3.svg
paper_authors.fullname: Liang Zeng
paper_authors.type: user
paper_authors.user: zengliangcs
paper_authors._id: 669096575cb4c07caa232efe
paper_authors.name: Liangjun Zhong
paper_authors._id: 669096575cb4c07caa232eff
paper_authors.name: Liang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:40:06.238Z
paper_authors._id: 64996ce2ed09df36eaf236ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64996ce2ed09df36eaf236ea/wIr1j2zkAezXfpsczTjq_.jpeg
paper_authors.fullname: zhaoliang
paper_authors.type: user
paper_authors.user: zhao1iang
paper_authors._id: 669096575cb4c07caa232f00
paper_authors.name: Tianwen Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T07:55:17.509Z
paper_authors._id: 63d8773a635499f21644601b
paper_authors.avatarUrl: /avatars/07efea204d172e2844dbacc418131da2.svg
paper_authors.fullname: Wei Tianwen
paper_authors.type: user
paper_authors.user: weitianwen
paper_authors._id: 669096575cb4c07caa232f01
paper_authors.name: Liu Yang
paper_authors._id: 669096575cb4c07caa232f02
paper_authors.name: Jujie He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T07:55:46.201Z
paper_authors._id: 64a676f223caaba1d2eb5d02
paper_authors.avatarUrl: /avatars/c51a938e3b0e64f1aa38e6c63828e921.svg
paper_authors.fullname: Jujie He
paper_authors.type: user
paper_authors.user: leafzs
paper_authors._id: 669096575cb4c07caa232f03
paper_authors.name: Cheng Cheng
paper_authors._id: 669096575cb4c07caa232f04
paper_authors.name: Rui Hu
paper_authors._id: 669096575cb4c07caa232f05
paper_authors.name: Yang Liu
paper_authors._id: 669096575cb4c07caa232f06
paper_authors.name: Shuicheng Yan
paper_authors._id: 669096575cb4c07caa232f07
paper_authors.name: Han Fang
paper_authors._id: 669096575cb4c07caa232f08
paper_authors.name: Yahui Zhou
paper_title: Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large
    Language Models -- The Story Goes On
paper_summary: In this paper, we investigate the underlying factors that potentially enhance
  the mathematical reasoning capabilities of large language models (LLMs). We
  argue that the data scaling law for math reasoning capabilities in modern LLMs
  is far from being saturated, highlighting how the model's quality improves with
  increases in data quantity. To support this claim, we introduce the
  Skywork-Math model series, supervised fine-tuned (SFT) on common 7B LLMs using
  our proposed 2.5M-instance Skywork-MathQA dataset. Skywork-Math 7B has achieved
  impressive accuracies of 51.2% on the competition-level MATH benchmark and
  83.9% on the GSM8K benchmark using only SFT data, outperforming an early
  version of GPT-4 on MATH. The superior performance of Skywork-Math models
  contributes to our novel two-stage data synthesis and model SFT pipelines,
  which include three different augmentation methods and a diverse seed problem
  set, ensuring both the quantity and quality of Skywork-MathQA dataset across
  varying difficulty levels. Most importantly, we provide several practical
  takeaways to enhance math reasoning abilities in LLMs for both research and
  industry applications.
paper_ai_keywords: large language models (LLMs) Skywork-Math model series supervised fine-tuned (SFT) Skywork-MathQA dataset MATH benchmark GSM8K benchmark two-stage data synthesis augmentation methods seed problem set