title: A Careful Examination of Large Language Model Performance on Grade
    School Arithmetic
summary: Large language models (LLMs) have achieved impressive success on many
  benchmarks for mathematical reasoning. However, there is growing concern that
  some of this performance actually reflects dataset contamination, where data
  closely resembling benchmark questions leaks into the training data, instead of
  true reasoning ability. To investigate this claim rigorously, we commission
  Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and
  complexity of the established GSM8k benchmark, the gold standard for measuring
  elementary mathematical reasoning. We ensure that the two benchmarks are
  comparable across important metrics such as human solve rates, number of steps
  in solution, answer magnitude, and more. When evaluating leading open- and
  closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with
  several families of models (e.g., Phi and Mistral) showing evidence of
  systematic overfitting across almost all model sizes. At the same time, many
  models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show
  minimal signs of overfitting. Further analysis suggests a positive relationship
  (Spearman's r^2=0.32) between a model's probability of generating an example
  from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that
  many models may have partially memorized GSM8k.
paper_authors._id: 663300c27dfc25dfbf8f0250
paper_authors.name: Hugh Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-02T07:48:47.067Z
paper_authors._id: 65d6a5f94c28026a003581b4
paper_authors.avatarUrl: /avatars/c24c314eafbf6a885c586258ae87a47f.svg
paper_authors.fullname: Hugh Zhang
paper_authors.type: user
paper_authors.user: hugh-scale
paper_authors._id: 663300c27dfc25dfbf8f0251
paper_authors.name: Jeff Da
paper_authors._id: 663300c27dfc25dfbf8f0252
paper_authors.name: Dean Lee
paper_authors._id: 663300c27dfc25dfbf8f0253
paper_authors.name: Vaughn Robinson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:23:24.254Z
paper_authors._id: 64e259ab93103b4537bfc815
paper_authors.avatarUrl: /avatars/664c31c467fbae50d730b0c9df891dd2.svg
paper_authors.fullname: Vaughn Robinson
paper_authors.type: user
paper_authors.user: dumela123
paper_authors._id: 663300c27dfc25dfbf8f0254
paper_authors.name: Catherine Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:23:30.378Z
paper_authors._id: 632f99c62a6ef6fb4ad53949
paper_authors.avatarUrl: /avatars/ad25b15d599fa21c2a75152ceb01f9ca.svg
paper_authors.fullname: Catherine Wu
paper_authors.type: user
paper_authors.user: catwu
paper_authors._id: 663300c27dfc25dfbf8f0255
paper_authors.name: Will Song
paper_authors._id: 663300c27dfc25dfbf8f0256
paper_authors.name: Tiffany Zhao
paper_authors._id: 663300c27dfc25dfbf8f0257
paper_authors.name: Pranav Raja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:23:59.291Z
paper_authors._id: 63f291970a16587ea96c4735
paper_authors.avatarUrl: /avatars/6d3f2029a7ef1f0f8b47c1d9829d9aa0.svg
paper_authors.fullname: Pranav Raja
paper_authors.type: user
paper_authors.user: pranavrajascale
paper_authors._id: 663300c27dfc25dfbf8f0258
paper_authors.name: Dylan Slack
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:24:20.238Z
paper_authors._id: 62ec892ff105ff23a9bd4548
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659668761477-noauth.png
paper_authors.fullname: Dylan Slack
paper_authors.type: user
paper_authors.user: dslack
paper_authors._id: 663300c27dfc25dfbf8f0259
paper_authors.name: Qin Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:24:27.051Z
paper_authors._id: 64b6dd32aaebda400ce31b05
paper_authors.avatarUrl: /avatars/4ff1783e873261c3874f8752e81e5a77.svg
paper_authors.fullname: Qin Lyu
paper_authors.type: user
paper_authors.user: lyuqin
paper_authors._id: 663300c27dfc25dfbf8f025a
paper_authors.name: Sean Hendryx
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:29:56.193Z
paper_authors._id: 67106d78607722fac0b844a7
paper_authors.avatarUrl: /avatars/459b1a39541f1dd871aad66e653aed48.svg
paper_authors.fullname: Sean Hendryx
paper_authors.type: user
paper_authors.user: hendryx-scale
paper_authors._id: 663300c27dfc25dfbf8f025b
paper_authors.name: Russell Kaplan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:24:36.607Z
paper_authors._id: 6313cc48c093ff968e0d9c5c
paper_authors.avatarUrl: /avatars/7f0aa8ee6274d0a7790ef0b3298ed19b.svg
paper_authors.fullname: Russell Kaplan
paper_authors.type: user
paper_authors.user: rkaplan
paper_authors._id: 663300c27dfc25dfbf8f025c
paper_authors.name: Michele
paper_authors._id: 663300c27dfc25dfbf8f025d
paper_authors.name: Lunati
paper_authors._id: 663300c27dfc25dfbf8f025e
paper_authors.name: Summer Yue
paper_title: A Careful Examination of Large Language Model Performance on Grade
    School Arithmetic
paper_summary: Large language models (LLMs) have achieved impressive success on many
  benchmarks for mathematical reasoning. However, there is growing concern that
  some of this performance actually reflects dataset contamination, where data
  closely resembling benchmark questions leaks into the training data, instead of
  true reasoning ability. To investigate this claim rigorously, we commission
  Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and
  complexity of the established GSM8k benchmark, the gold standard for measuring
  elementary mathematical reasoning. We ensure that the two benchmarks are
  comparable across important metrics such as human solve rates, number of steps
  in solution, answer magnitude, and more. When evaluating leading open- and
  closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with
  several families of models (e.g., Phi and Mistral) showing evidence of
  systematic overfitting across almost all model sizes. At the same time, many
  models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show
  minimal signs of overfitting. Further analysis suggests a positive relationship
  (Spearman's r^2=0.32) between a model's probability of generating an example
  from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that
  many models may have partially memorized GSM8k.
paper_ai_keywords: large language models (LLMs) alpha matte mathematical reasoning dataset contamination training data true reasoning ability Grade School Math 1000 (GSM1k) GSM8k benchmark demographic solve rates number of steps in solution answer magnitude open-source LLMs closed-source LLMs Phi Mistral systematic overfitting model's probability of generating an example performance gap Spearman's r^2