title: MegaMath: Pushing the Limits of Open Math Corpora
summary: Mathematical reasoning is a cornerstone of human intelligence and a key
  benchmark for advanced capabilities in large language models (LLMs). However,
  the research community still lacks an open, large-scale, high-quality corpus
  tailored to the demands of math-centric LLM pre-training. We present MegaMath,
  an open dataset curated from diverse, math-focused sources through following
  practices: (1) Revisiting web data: We re-extracted mathematical documents from
  Common Crawl with math-oriented HTML optimizations, fasttext-based filtering
  and deduplication, all for acquiring higher-quality data on the Internet. (2)
  Recalling Math-related code data: We identified high quality math-related code
  from large code training corpus, Stack-V2, further enhancing data diversity.
  (3) Exploring Synthetic data: We synthesized QA-style text, math-related code,
  and interleaved text-code blocks from web data or code data. By integrating
  these strategies and validating their effectiveness through extensive
  ablations, MegaMath delivers 371B tokens with the largest quantity and top
  quality among existing open math pre-training datasets.
paper_authors._id: 67efe6e839d83c66f96ef051
paper_authors.name: Fan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:54:00.099Z
paper_authors._id: 628f6e5ab90dde28ef57d293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
paper_authors.fullname: Fan Zhou
paper_authors.type: user
paper_authors.user: koalazf99
paper_authors._id: 67efe6e839d83c66f96ef052
paper_authors.name: Zengzhi Wang
paper_authors._id: 67efe6e839d83c66f96ef053
paper_authors.name: Nikhil Ranjan
paper_authors._id: 67efe6e839d83c66f96ef054
paper_authors.name: Zhoujun Cheng
paper_authors._id: 67efe6e839d83c66f96ef055
paper_authors.name: Liping Tang
paper_authors._id: 67efe6e839d83c66f96ef056
paper_authors.name: Guowei He
paper_authors._id: 67efe6e839d83c66f96ef057
paper_authors.name: Zhengzhong Liu
paper_authors._id: 67efe6e839d83c66f96ef058
paper_authors.name: Eric P. Xing
paper_title: MegaMath: Pushing the Limits of Open Math Corpora
paper_summary: Mathematical reasoning is a cornerstone of human intelligence and a key
  benchmark for advanced capabilities in large language models (LLMs). However,
  the research community still lacks an open, large-scale, high-quality corpus
  tailored to the demands of math-centric LLM pre-training. We present MegaMath,
  an open dataset curated from diverse, math-focused sources through following
  practices: (1) Revisiting web data: We re-extracted mathematical documents from
  Common Crawl with math-oriented HTML optimizations, fasttext-based filtering
  and deduplication, all for acquiring higher-quality data on the Internet. (2)
  Recalling Math-related code data: We identified high quality math-related code
  from large code training corpus, Stack-V2, further enhancing data diversity.
  (3) Exploring Synthetic data: We synthesized QA-style text, math-related code,
  and interleaved text-code blocks from web data or code data. By integrating
  these strategies and validating their effectiveness through extensive
  ablations, MegaMath delivers 371B tokens with the largest quantity and top
  quality among existing open math pre-training datasets.
paper_ai_summary: MegaMath is an open dataset designed for math-centric LLM pre-training, combining high-quality web data, math-related code, and synthetic content to enhance diversity and quality.
paper_ai_keywords: LLMs MegaMath Common Crawl HTML optimizations fastText deduplication Stack-V2 QA-style text interleaved text-code blocks math pre-training datasets