Common 7B Language Models Already Possess Strong Math Capabilities
Mathematical capabilities were previously believed to emerge in common
language models only at a very large scale or require extensive math-related
pre-training. This paper shows that the LLaMA-2 7B model with common
pre-training already exhibits strong mathematical abilities, as evidenced by
its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks,
respectively, when selecting the best response from 256 random generations. The
primary issue with the current base model is the difficulty in consistently
eliciting its inherent mathematical capabilities. Notably, the accuracy for the
first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks,
respectively. We find that simply scaling up the SFT data can significantly
enhance the reliability of generating correct answers. However, the potential
for extensive scaling is constrained by the scarcity of publicly available math
questions. To overcome this limitation, we employ synthetic data, which proves
to be nearly as effective as real data and shows no clear saturation when
scaled up to approximately one million samples. This straightforward approach
achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B
models, surpassing previous models by 14.2% and 20.8%, respectively. We also
provide insights into scaling behaviors across different reasoning complexities
and error types.
65ea7d94b985ae912c623cc6
Chen Li
claimed_verified
2024-03-08T17:23:55.742Z
64e0debc02fa032de402feb0
/avatars/6c28a13216f7ae4c047cf6ea9846f6e3.svg
Chen Li
user
LC-Edward
65ea7d94b985ae912c623cc7
Weiqi Wang
65ea7d94b985ae912c623cc8
Jingcheng Hu
admin_assigned
2024-03-08T09:44:21.351Z
625026b7d2d191ac43320c5e
https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
Jingcheng Hu
user
reign12
65ea7d94b985ae912c623cc9
Yixuan Wei
admin_assigned
2024-03-08T09:44:14.701Z
640efe54a92fedb0e84ee598
/avatars/cb1c6720525bfdbf59a9d3eac93a879e.svg
Yixuan Wei
user
EasonWei
65ea7d94b985ae912c623cca
Nanning Zheng
65ea7d94b985ae912c623ccb
Han Hu
65ea7d94b985ae912c623ccc
Zheng Zhang
65ea7d94b985ae912c623ccd
Houwen Peng
admin_assigned
2024-03-08T09:42:29.072Z
631952b6f18d0b5d999ca397
/avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
Peng
user
Houwen
Common 7B Language Models Already Possess Strong Math Capabilities
Mathematical capabilities were previously believed to emerge in common
language models only at a very large scale or require extensive math-related
pre-training. This paper shows that the LLaMA-2 7B model with common
pre-training already exhibits strong mathematical abilities, as evidenced by
its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks,
respectively, when selecting the best response from 256 random generations. The
primary issue with the current base model is the difficulty in consistently
eliciting its inherent mathematical capabilities. Notably, the accuracy for the
first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks,
respectively. We find that simply scaling up the SFT data can significantly
enhance the reliability of generating correct answers. However, the potential
for extensive scaling is constrained by the scarcity of publicly available math
questions. To overcome this limitation, we employ synthetic data, which proves
to be nearly as effective as real data and shows no clear saturation when
scaled up to approximately one million samples. This straightforward approach
achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B
models, surpassing previous models by 14.2% and 20.8%, respectively. We also
provide insights into scaling behaviors across different reasoning complexities
and error types.
The LLaMA-2 7B model shows strong mathematical abilities through pre-training, and scaling synthetic data can enhance its performance on math benchmarks.
LLaMA-2 GSM8K MATH mathematical abilities pre-training synthetic data accuracy