title: TinyGSM: achieving >80% on GSM8k with small language models
summary: Small-scale models offer various computational advantages, and yet to which
  extent size is critical for problem-solving abilities remains an open question.
  Specifically for solving grade school math, the smallest model size so far
  required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
  Our work studies how high-quality datasets may be the key for small language
  models to acquire mathematical reasoning. We introduce TinyGSM, a
  synthetic dataset of 12.3M grade school math problems paired with Python
  solutions, generated fully by GPT-3.5. After finetuning on TinyGSM, we
  find that a duo of a 1.3B generation model and a 1.3B verifier model can
  achieve 81.5\% accuracy, outperforming existing models that are orders of
  magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher''
  model (77.4\%), from which our model's training data is generated. Our approach
  is simple and has two key components: 1) the high-quality dataset
  TinyGSM, 2) the use of a verifier, which selects the final outputs
  from multiple candidate generations.
paper_authors._id: 657bcee9696ec3dda930e4c4
paper_authors.name: Bingbin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T08:28:02.121Z
paper_authors._id: 63d2e5ef2727d7888cb940b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674794281493-63d2e5ef2727d7888cb940b8.jpeg
paper_authors.fullname: Bingbin
paper_authors.type: user
paper_authors.user: ClaraBing
paper_authors._id: 657bcee9696ec3dda930e4c5
paper_authors.name: Sebastien Bubeck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T11:12:44.670Z
paper_authors._id: 64557408c9c0dcc8c24a7a92
paper_authors.avatarUrl: /avatars/911ba8fc3c224578d5946dddd94bf4c0.svg
paper_authors.fullname: Sebastien Bubeck
paper_authors.type: user
paper_authors.user: sebubeck
paper_authors._id: 657bcee9696ec3dda930e4c6
paper_authors.name: Ronen Eldan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T11:13:01.769Z
paper_authors._id: 633723a80267ebcf0264c06b
paper_authors.avatarUrl: /avatars/22bb971597e9f3abfa343280a9d0f65f.svg
paper_authors.fullname: Ronen Eldan
paper_authors.type: user
paper_authors.user: roneneldan
paper_authors._id: 657bcee9696ec3dda930e4c7
paper_authors.name: Janardhan Kulkarni
paper_authors._id: 657bcee9696ec3dda930e4c8
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T11:13:27.761Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_authors._id: 657bcee9696ec3dda930e4c9
paper_authors.name: Anh Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T07:49:08.871Z
paper_authors._id: 61ee4dc39539bc0a3bbc8a68
paper_authors.avatarUrl: /avatars/723f89e9c7afecdbd2854544bd7eb083.svg
paper_authors.fullname: Anh Tuan Nguyen
paper_authors.type: user
paper_authors.user: tuananh7198
paper_authors._id: 657bcee9696ec3dda930e4ca
paper_authors.name: Rachel Ward
paper_authors._id: 657bcee9696ec3dda930e4cb
paper_authors.name: Yi Zhang
paper_title: TinyGSM: achieving >80% on GSM8k with small language models
paper_summary: Small-scale models offer various computational advantages, and yet to which
  extent size is critical for problem-solving abilities remains an open question.
  Specifically for solving grade school math, the smallest model size so far
  required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
  Our work studies how high-quality datasets may be the key for small language
  models to acquire mathematical reasoning. We introduce TinyGSM, a
  synthetic dataset of 12.3M grade school math problems paired with Python
  solutions, generated fully by GPT-3.5. After finetuning on TinyGSM, we
  find that a duo of a 1.3B generation model and a 1.3B verifier model can
  achieve 81.5\% accuracy, outperforming existing models that are orders of
  magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher''
  model (77.4\%), from which our model's training data is generated. Our approach
  is simple and has two key components: 1) the high-quality dataset
  TinyGSM, 2) the use of a verifier, which selects the final outputs
  from multiple candidate generations.
paper_ai_summary: A synthetic dataset of math problems paired with solutions enables small language models to achieve high accuracy in mathematical reasoning, outperforming larger models.
paper_ai_keywords: TinyGSM GPT-3.5 mathematical reasoning verifier candidate generations