title: AutoMathText: Autonomous Data Selection with Language Models for
    Mathematical Texts
summary: To improve language models' proficiency in mathematical reasoning via
  continual pretraining, we introduce a novel strategy that leverages base
  language models for autonomous data selection. Departing from conventional
  supervised fine-tuning or trained classifiers with human-annotated data, our
  approach utilizes meta-prompted language models as zero-shot verifiers to
  autonomously evaluate and select high-quality mathematical content, and we
  release the curated open-source AutoMathText dataset encompassing over 200GB of
  data. To demonstrate the efficacy of our method, we continuously pretrained a
  7B-parameter Mistral language model on the AutoMathText dataset, achieving
  substantial improvements in downstream performance on the MATH dataset with a
  token amount reduced by orders of magnitude compared to previous continuous
  pretraining works. Our method showcases a 2 times increase in pretraining token
  efficiency compared to baselines, underscoring the potential of our approach in
  enhancing models' mathematical reasoning capabilities. The AutoMathText dataset
  is available at https://huggingface.co/datasets/math-ai/AutoMathText. The code
  is available at https://github.com/yifanzhang-pro/AutoMathText.
paper_authors._id: 65cadee7c4eddc15d036c8d5
paper_authors.name: Yifan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-14T08:42:53.417Z
paper_authors._id: 647bf082aba7062fe5c51ca9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
paper_authors.fullname: Yifan Zhang
paper_authors.type: user
paper_authors.user: yifAI
paper_authors._id: 65cadee7c4eddc15d036c8d6
paper_authors.name: Yifan Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:10:39.491Z
paper_authors._id: 624dae2cb58d1313f8ba9a2d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624dae2cb58d1313f8ba9a2d/cFar_YWV8qtWUjOZeNKT0.png
paper_authors.fullname: Yifan Luo
paper_authors.type: user
paper_authors.user: Luobots
paper_authors._id: 65cadee7c4eddc15d036c8d7
paper_authors.name: Yang Yuan
paper_authors._id: 65cadee7c4eddc15d036c8d8
paper_authors.name: Andrew Chi-Chih Yao
paper_title: AutoMathText: Autonomous Data Selection with Language Models for
    Mathematical Texts
paper_summary: To improve language models' proficiency in mathematical reasoning via
  continual pretraining, we introduce a novel strategy that leverages base
  language models for autonomous data selection. Departing from conventional
  supervised fine-tuning or trained classifiers with human-annotated data, our
  approach utilizes meta-prompted language models as zero-shot verifiers to
  autonomously evaluate and select high-quality mathematical content, and we
  release the curated open-source AutoMathText dataset encompassing over 200GB of
  data. To demonstrate the efficacy of our method, we continuously pretrained a
  7B-parameter Mistral language model on the AutoMathText dataset, achieving
  substantial improvements in downstream performance on the MATH dataset with a
  token amount reduced by orders of magnitude compared to previous continuous
  pretraining works. Our method showcases a 2 times increase in pretraining token
  efficiency compared to baselines, underscoring the potential of our approach in
  enhancing models' mathematical reasoning capabilities. The AutoMathText dataset
  is available at https://huggingface.co/datasets/math-ai/AutoMathText. The code
  is available at https://github.com/yifanzhang-pro/AutoMathText.
paper_ai_keywords: meta-prompted language models zero-shot verifiers AutoMathText dataset continuous pretraining MATH dataset token efficiency