title: MathScale: Scaling Instruction Tuning for Mathematical Reasoning
summary: Large language models (LLMs) have demonstrated remarkable capabilities in
  problem-solving. However, their proficiency in solving mathematical problems
  remains inadequate. We propose MathScale, a simple and scalable method to
  create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt
  GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning,
  it first extracts topics and knowledge points from seed math questions and then
  build a concept graph, which is subsequently used to generate new math
  questions. MathScale exhibits effective scalability along the size axis of the
  math dataset that we generate. As a result, we create a mathematical reasoning
  dataset (MathScaleQA) containing two million math question-answer pairs. To
  evaluate mathematical reasoning abilities of LLMs comprehensively, we construct
  {\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten
  datasets (including GSM8K and MATH) covering K-12, college, and competition
  level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g.,
  LLaMA-2 and Mistral), resulting in significantly improved capabilities in
  mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves
  state-of-the-art performance across all datasets, surpassing its best peers of
  equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average
  accuracy, respectively.
paper_authors._id: 65e7db6a7c10574cc3e7a8ef
paper_authors.name: Zhengyang Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:59:41.936Z
paper_authors._id: 64912976b95c3f0a1e6233cb
paper_authors.avatarUrl: /avatars/c0615f8c6606073faffb419757d4e667.svg
paper_authors.fullname: Zhengyang Tang
paper_authors.type: user
paper_authors.user: tangzhy
paper_authors._id: 65e7db6a7c10574cc3e7a8f0
paper_authors.name: Xingxing Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:59:48.538Z
paper_authors._id: 64abbcff6cadc7aca584f71b
paper_authors.avatarUrl: /avatars/fc6e85ad4a8befd133a37b411712c648.svg
paper_authors.fullname: Xingxing Zhang
paper_authors.type: user
paper_authors.user: THU-CHUNXIA
paper_authors._id: 65e7db6a7c10574cc3e7a8f1
paper_authors.name: Benyou Wan
paper_authors._id: 65e7db6a7c10574cc3e7a8f2
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:00:04.844Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: MathScale: Scaling Instruction Tuning for Mathematical Reasoning
paper_summary: Large language models (LLMs) have demonstrated remarkable capabilities in
  problem-solving. However, their proficiency in solving mathematical problems
  remains inadequate. We propose MathScale, a simple and scalable method to
  create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt
  GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning,
  it first extracts topics and knowledge points from seed math questions and then
  build a concept graph, which is subsequently used to generate new math
  questions. MathScale exhibits effective scalability along the size axis of the
  math dataset that we generate. As a result, we create a mathematical reasoning
  dataset (MathScaleQA) containing two million math question-answer pairs. To
  evaluate mathematical reasoning abilities of LLMs comprehensively, we construct
  {\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten
  datasets (including GSM8K and MATH) covering K-12, college, and competition
  level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g.,
  LLaMA-2 and Mistral), resulting in significantly improved capabilities in
  mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves
  state-of-the-art performance across all datasets, surpassing its best peers of
  equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average
  accuracy, respectively.
paper_ai_summary: MathScale generates a large dataset of mathematical reasoning questions by building concept graphs from seed questions, enhancing open-source LLMs' performance on Math Word Problems.
paper_ai_keywords: large language models mathematical reasoning MathScale GPT-3.5 concept graph MathScaleQA MwpBench GSM8K MATH LLaMA-2 Mistral fine-tuning micro average accuracy macro average accuracy