title: MetaMath: Bootstrap Your Own Mathematical Questions for Large Language
    Models
summary: Large language models (LLMs) have pushed the limits of natural language
  understanding and exhibited excellent problem-solving ability. Despite the
  great success, most existing open-source LLMs (\eg, LLaMA-2) are still far away
  from satisfactory for solving mathematical problem due to the complex reasoning
  procedures. To bridge this gap, we propose MetaMath, a fine-tuned
  language model that specializes in mathematical reasoning. Specifically, we
  start by bootstrapping mathematical questions by rewriting the question from
  multiple perspectives without extra knowledge, which results in a new dataset
  called {MetaMathQA}. Then we fine-tune the LLaMA-2 models on MetaMathQA.
  Experimental results on two popular benchmarks (\ie, GSM8K and MATH) for
  mathematical reasoning demonstrate that MetaMath outperforms a suite of
  open-source LLMs by a significant margin. Our MetaMath-7B model achieves
  66.4% on GSM8K and 19.4% on MATH, exceeding the state-of-the-art models
  of the same size by 11.5% and 8.7%. Particularly, {MetaMath-70B} achieves
  an accuracy of 82.3% on {GSM8K}, slightly better than {GPT-3.5-Turbo}. We
  release the {MetaMathQA} dataset, the {MetaMath} models with different model
  sizes and the training code for public use.
paper_authors._id: 650d11852fe3e08aa64fc16a
paper_authors.name: Longhui Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:18:35.033Z
paper_authors._id: 64b753306c169983c982f609
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b753306c169983c982f609/oOC_N8cI1Go6avUMvIPVI.jpeg
paper_authors.fullname: Longhui Yu
paper_authors.type: user
paper_authors.user: Longhui98
paper_authors._id: 650d11852fe3e08aa64fc16b
paper_authors.name: Weisen Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-22T17:53:40.335Z
paper_authors._id: 646709230ed2f7a8cba98107
paper_authors.avatarUrl: /avatars/6a4124ed2957a93a294bd5563f500e1c.svg
paper_authors.fullname: Weisen Jiang
paper_authors.type: user
paper_authors.user: waysonkong
paper_authors._id: 650d11852fe3e08aa64fc16c
paper_authors.name: Han Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-22T17:53:37.317Z
paper_authors._id: 637751ad7169ed981703d26a
paper_authors.avatarUrl: /avatars/9823f115040a3d6b0240038010e46cb6.svg
paper_authors.fullname: Han Shi
paper_authors.type: user
paper_authors.user: shihan96
paper_authors._id: 650d11852fe3e08aa64fc16d
paper_authors.name: Jincheng Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:19:22.393Z
paper_authors._id: 650c74e5d439bbbbadfcfbbe
paper_authors.avatarUrl: /avatars/1002835739dbb90214b5f2824a7c8c1f.svg
paper_authors.fullname: YU Jincheng
paper_authors.type: user
paper_authors.user: yujincheng08
paper_authors._id: 650d11852fe3e08aa64fc16e
paper_authors.name: Zhengying Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:19:31.044Z
paper_authors._id: 6445f82f5691ca69b0df747a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6445f82f5691ca69b0df747a/lVILbKYiX6Aw9I97HlPo1.jpeg
paper_authors.fullname: Zhengying Liu
paper_authors.type: user
paper_authors.user: evariste-liu
paper_authors._id: 650d11852fe3e08aa64fc16f
paper_authors.name: Yu Zhang
paper_authors._id: 650d11852fe3e08aa64fc170
paper_authors.name: James T. Kwok
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:21:25.334Z
paper_authors._id: 647015f2dcddb2358a287f2f
paper_authors.avatarUrl: /avatars/01f7df1ad30d70c99ef1c3521bbeac94.svg
paper_authors.fullname: James Kwok
paper_authors.type: user
paper_authors.user: KazumaMyGuy
paper_authors._id: 650d11852fe3e08aa64fc171
paper_authors.name: Zhenguo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:21:38.107Z
paper_authors._id: 639944bef7c4648254d124f0
paper_authors.avatarUrl: /avatars/c5cbb87cd51cc1341844f67d42c55151.svg
paper_authors.fullname: lzg
paper_authors.type: user
paper_authors.user: lizhenguo
paper_authors._id: 650d11852fe3e08aa64fc172
paper_authors.name: Adrian Weller
paper_authors._id: 650d11852fe3e08aa64fc173
paper_authors.name: Weiyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:21:49.980Z
paper_authors._id: 648905d1a15c43c791d4381f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
paper_authors.fullname: Weiyang Liu
paper_authors.type: user
paper_authors.user: wy1iu
paper_title: MetaMath: Bootstrap Your Own Mathematical Questions for Large Language
    Models
paper_summary: Large language models (LLMs) have pushed the limits of natural language
  understanding and exhibited excellent problem-solving ability. Despite the
  great success, most existing open-source LLMs (\eg, LLaMA-2) are still far away
  from satisfactory for solving mathematical problem due to the complex reasoning
  procedures. To bridge this gap, we propose MetaMath, a fine-tuned
  language model that specializes in mathematical reasoning. Specifically, we
  start by bootstrapping mathematical questions by rewriting the question from
  multiple perspectives without extra knowledge, which results in a new dataset
  called {MetaMathQA}. Then we fine-tune the LLaMA-2 models on MetaMathQA.
  Experimental results on two popular benchmarks (\ie, GSM8K and MATH) for
  mathematical reasoning demonstrate that MetaMath outperforms a suite of
  open-source LLMs by a significant margin. Our MetaMath-7B model achieves
  66.4% on GSM8K and 19.4% on MATH, exceeding the state-of-the-art models
  of the same size by 11.5% and 8.7%. Particularly, {MetaMath-70B} achieves
  an accuracy of 82.3% on {GSM8K}, slightly better than {GPT-3.5-Turbo}. We
  release the {MetaMathQA} dataset, the {MetaMath} models with different model
  sizes and the training code for public use.
paper_ai_summary: MetaMath, a fine-tuned language model specializing in mathematical reasoning, outperforms existing models on benchmarks like GSM8K and MATH through the use of a new dataset, MetaMathQA.
paper_ai_keywords: MetaMath LLaMA-2 MetaMathQA mathematical reasoning GSM8K MATH MetaMath-7B MetaMath-70B GPT-3.5-Turbo