title: InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic
    Mathematical Reasoning
summary: Recent advancements in Chain-of-Thoughts (CoT) and Program-of-Thoughts (PoT)
  methods have greatly enhanced language models' mathematical reasoning
  capabilities, facilitating their integration into instruction tuning datasets
  with LLMs. However, existing methods for large-scale dataset creation require
  substantial seed data and high computational costs for data synthesis, posing
  significant challenges for scalability. We introduce InfinityMATH, a scalable
  instruction tuning dataset for programmatic mathematical reasoning. The
  construction pipeline emphasizes decoupling numbers from mathematical problems
  to synthesize number-independent programs, enabling efficient and flexible
  scaling while minimizing dependency on specific numerical values. Fine-tuning
  experiments with open-source language and code models, such as Llama2 and
  CodeLlama, demonstrate the practical benefits of InfinityMATH. These fine-tuned
  models, showed significant relative improvements on both in-domain and
  out-of-domain benchmarks, ranging from 184.7% to 514.3% on average.
  Additionally, these models exhibited high robustness on the GSM8K+ and MATH+
  benchmarks, which are enhanced version of test sets with simply the number
  variations. InfinityMATH ensures that models are more versatile and effective
  across a broader range of mathematical problems. The data is available at
  https://huggingface.co/datasets/flagopen/InfinityMATH.
paper_authors._id: 66bd603b3f0c44697d3eba36
paper_authors.name: Bo-Wen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:50.084Z
paper_authors._id: 6335113375bed9932474315e
paper_authors.avatarUrl: /avatars/aa1837a3a07514b89c48de888934a7b2.svg
paper_authors.fullname: bowenzhang
paper_authors.type: user
paper_authors.user: bowen92
paper_authors._id: 66bd603b3f0c44697d3eba37
paper_authors.name: Yan Yan
paper_authors._id: 66bd603b3f0c44697d3eba38
paper_authors.name: Lin Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-16T03:40:38.652Z
paper_authors._id: 665ede6c8947302aa2b408d5
paper_authors.avatarUrl: /avatars/fc1bcf732a981a78b0afbc8fa0c072cd.svg
paper_authors.fullname: 李琳
paper_authors.type: user
paper_authors.user: lilin1129
paper_authors._id: 66bd603b3f0c44697d3eba39
paper_authors.name: Guang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:28:31.913Z
paper_authors._id: 632c234f42c386ebd2710434
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
paper_authors.fullname: Guang Liu
paper_authors.type: user
paper_authors.user: ZacLiu
paper_title: InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic
    Mathematical Reasoning
paper_summary: Recent advancements in Chain-of-Thoughts (CoT) and Program-of-Thoughts (PoT)
  methods have greatly enhanced language models' mathematical reasoning
  capabilities, facilitating their integration into instruction tuning datasets
  with LLMs. However, existing methods for large-scale dataset creation require
  substantial seed data and high computational costs for data synthesis, posing
  significant challenges for scalability. We introduce InfinityMATH, a scalable
  instruction tuning dataset for programmatic mathematical reasoning. The
  construction pipeline emphasizes decoupling numbers from mathematical problems
  to synthesize number-independent programs, enabling efficient and flexible
  scaling while minimizing dependency on specific numerical values. Fine-tuning
  experiments with open-source language and code models, such as Llama2 and
  CodeLlama, demonstrate the practical benefits of InfinityMATH. These fine-tuned
  models, showed significant relative improvements on both in-domain and
  out-of-domain benchmarks, ranging from 184.7% to 514.3% on average.
  Additionally, these models exhibited high robustness on the GSM8K+ and MATH+
  benchmarks, which are enhanced version of test sets with simply the number
  variations. InfinityMATH ensures that models are more versatile and effective
  across a broader range of mathematical problems. The data is available at
  https://huggingface.co/datasets/flagopen/InfinityMATH.
paper_ai_summary: InfinityMATH is a scalable dataset for programmatic mathematical reasoning that decouples numbers from problems, enhancing model performance and robustness across benchmarks.
paper_ai_keywords: Chain-of-Thoughts Program-of-Thoughts mathematical reasoning instruction tuning datasets large-scale dataset creation data synthesis decoupling number-independent programs fine-tuning Llama2 CodeLlama in-domain benchmarks out-of-domain benchmarks GSM8K+ MATH+