title: Common 7B Language Models Already Possess Strong Math Capabilities
summary: Mathematical capabilities were previously believed to emerge in common
  language models only at a very large scale or require extensive math-related
  pre-training. This paper shows that the LLaMA-2 7B model with common
  pre-training already exhibits strong mathematical abilities, as evidenced by
  its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks,
  respectively, when selecting the best response from 256 random generations. The
  primary issue with the current base model is the difficulty in consistently
  eliciting its inherent mathematical capabilities. Notably, the accuracy for the
  first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks,
  respectively. We find that simply scaling up the SFT data can significantly
  enhance the reliability of generating correct answers. However, the potential
  for extensive scaling is constrained by the scarcity of publicly available math
  questions. To overcome this limitation, we employ synthetic data, which proves
  to be nearly as effective as real data and shows no clear saturation when
  scaled up to approximately one million samples. This straightforward approach
  achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B
  models, surpassing previous models by 14.2% and 20.8%, respectively. We also
  provide insights into scaling behaviors across different reasoning complexities
  and error types.
paper_authors._id: 65ea7d94b985ae912c623cc6
paper_authors.name: Chen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-08T17:23:55.742Z
paper_authors._id: 64e0debc02fa032de402feb0
paper_authors.avatarUrl: /avatars/6c28a13216f7ae4c047cf6ea9846f6e3.svg
paper_authors.fullname: Chen Li
paper_authors.type: user
paper_authors.user: LC-Edward
paper_authors._id: 65ea7d94b985ae912c623cc7
paper_authors.name: Weiqi Wang
paper_authors._id: 65ea7d94b985ae912c623cc8
paper_authors.name: Jingcheng Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:44:21.351Z
paper_authors._id: 625026b7d2d191ac43320c5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
paper_authors.fullname: Jingcheng Hu
paper_authors.type: user
paper_authors.user: reign12
paper_authors._id: 65ea7d94b985ae912c623cc9
paper_authors.name: Yixuan Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:44:14.701Z
paper_authors._id: 640efe54a92fedb0e84ee598
paper_authors.avatarUrl: /avatars/cb1c6720525bfdbf59a9d3eac93a879e.svg
paper_authors.fullname: Yixuan Wei
paper_authors.type: user
paper_authors.user: EasonWei
paper_authors._id: 65ea7d94b985ae912c623cca
paper_authors.name: Nanning Zheng
paper_authors._id: 65ea7d94b985ae912c623ccb
paper_authors.name: Han Hu
paper_authors._id: 65ea7d94b985ae912c623ccc
paper_authors.name: Zheng Zhang
paper_authors._id: 65ea7d94b985ae912c623ccd
paper_authors.name: Houwen Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:42:29.072Z
paper_authors._id: 631952b6f18d0b5d999ca397
paper_authors.avatarUrl: /avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
paper_authors.fullname: Peng
paper_authors.type: user
paper_authors.user: Houwen
paper_title: Common 7B Language Models Already Possess Strong Math Capabilities
paper_summary: Mathematical capabilities were previously believed to emerge in common
  language models only at a very large scale or require extensive math-related
  pre-training. This paper shows that the LLaMA-2 7B model with common
  pre-training already exhibits strong mathematical abilities, as evidenced by
  its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks,
  respectively, when selecting the best response from 256 random generations. The
  primary issue with the current base model is the difficulty in consistently
  eliciting its inherent mathematical capabilities. Notably, the accuracy for the
  first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks,
  respectively. We find that simply scaling up the SFT data can significantly
  enhance the reliability of generating correct answers. However, the potential
  for extensive scaling is constrained by the scarcity of publicly available math
  questions. To overcome this limitation, we employ synthetic data, which proves
  to be nearly as effective as real data and shows no clear saturation when
  scaled up to approximately one million samples. This straightforward approach
  achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B
  models, surpassing previous models by 14.2% and 20.8%, respectively. We also
  provide insights into scaling behaviors across different reasoning complexities
  and error types.
paper_ai_summary: The LLaMA-2 7B model shows strong mathematical abilities through pre-training, and scaling synthetic data can enhance its performance on math benchmarks.
paper_ai_keywords: LLaMA-2 GSM8K MATH mathematical abilities pre-training synthetic data accuracy