(e.g., LLaMA-2), showing excellent elementary mathematical problem-solving capability.

• We identify an important factor when creating the MetaMathQA dataset – question diversity. The
diversity is particularly important in reasoning directions, and backward reasoning questions are
very helpful for LLMs to understand mathematical knowledge without memorization.

• We conduct experiments on two standard mathematical reasoning benchmarks: GSM8K [12] and
MATH [21]. MetaMath outperforms existing open-source LLMs by a large margin. MetaMath-7B
has achieved 66.5% on GSM8K (+11.5% compared to the previous best open-source LLM) on
GSM8K and 19.8% on MATH (+8.7% compared to the previous best open-source LLM).

• Our work studies data augmentation for improving the mathematical problem-solving ability of
LLMs. Despite being simple, our method significantly outperforms many intricate methods. Our
results highlight the importance of data augmentation and also shed light on other reasoning tasks.
METAMATH