title: Generative AI for Math: Part I -- MathPile: A Billion-Token-Scale
    Pretraining Corpus for Math
summary: High-quality, large-scale corpora are the cornerstone of building foundation
  models. In this work, we introduce MathPile, a diverse and
  high-quality math-centric corpus comprising about 9.5 billion tokens.
  Throughout its creation, we adhered to the principle of ``less is
  more'', firmly believing in the supremacy of data quality over quantity, even
  in the pre-training phase. Our meticulous data collection and processing
  efforts included a complex suite of preprocessing, prefiltering, language
  identification, cleaning, filtering, and deduplication, ensuring the high
  quality of our corpus. Furthermore, we performed data contamination detection
  on downstream benchmark test sets to eliminate duplicates. We hope our
  MathPile can help to enhance the mathematical reasoning abilities of
  language models. We plan to open-source different versions of \mathpile with
  the scripts used for processing, to facilitate future developments in this
  field.
paper_authors._id: 658e38c84a24a38778964cba
paper_authors.name: Zengzhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-29T09:00:39.609Z
paper_authors._id: 62cbeb2d72dfd24b86bdf977
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
paper_authors.fullname: Zengzhi Wang
paper_authors.type: user
paper_authors.user: SinclairWang
paper_authors._id: 658e38c84a24a38778964cbb
paper_authors.name: Rui Xia
paper_authors._id: 658e38c84a24a38778964cbc
paper_authors.name: Pengfei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-02T09:27:44.659Z
paper_authors._id: 6144a0c4ff1146bbd84d9865
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
paper_authors.fullname: Pengfei Liu
paper_authors.type: user
paper_authors.user: Pengfei
paper_title: Generative AI for Math: Part I -- MathPile: A Billion-Token-Scale
    Pretraining Corpus for Math
paper_summary: High-quality, large-scale corpora are the cornerstone of building foundation
  models. In this work, we introduce MathPile, a diverse and
  high-quality math-centric corpus comprising about 9.5 billion tokens.
  Throughout its creation, we adhered to the principle of ``less is
  more'', firmly believing in the supremacy of data quality over quantity, even
  in the pre-training phase. Our meticulous data collection and processing
  efforts included a complex suite of preprocessing, prefiltering, language
  identification, cleaning, filtering, and deduplication, ensuring the high
  quality of our corpus. Furthermore, we performed data contamination detection
  on downstream benchmark test sets to eliminate duplicates. We hope our
  MathPile can help to enhance the mathematical reasoning abilities of
  language models. We plan to open-source different versions of \mathpile with
  the scripts used for processing, to facilitate future developments in this
  field.
paper_ai_summary: A new large-scale, high-quality math-centric corpus named MathPile has been introduced to enhance mathematical reasoning abilities of language models.