title: Training and Evaluating Language Models with Template-based Data
    Generation
summary: The rapid advancement of large language models (LLMs) such as GPT-3, PaLM,
  and Llama has significantly transformed natural language processing, showcasing
  remarkable capabilities in understanding and generating language. However,
  these models often struggle with tasks requiring complex reasoning,
  particularly in mathematical problem-solving, due in part to the scarcity of
  large-scale, high-quality, domain-specific datasets necessary for training
  sophisticated reasoning abilities. To address this limitation, we introduce
  Template-based Data Generation (TDG), a novel approach that leverages LLMs
  (GPT-4) to automatically generate parameterized meta-templates, which are then
  used to synthesize a vast array of high-quality problems and solutions.
  Leveraging TDG, we create TemplateMath Part I: TemplateGSM, a dataset
  comprising over 7 million synthetically generated grade school math
  problems--each accompanied by code-based and natural language solutions--with
  the potential to generate an effectively unlimited number more. This dataset
  alleviates the scarcity of large-scale mathematical datasets and serves as a
  valuable resource for pre-training, fine-tuning, and evaluating LLMs in
  mathematical reasoning. Our method not only enables the generation of virtually
  infinite data but also elevates data augmentation to a new level by using GPT-4
  for meta-template generation, ensuring diverse and high-quality problem
  structures. The TemplateMath Part I: TemplateGSM dataset is publicly available
  at https://huggingface.co/datasets/math-ai/TemplateGSM. The code is available
  at https://github.com/iiis-ai/TemplateMath.
paper_authors._id: 6747f2f69c24d74b09c17158
paper_authors.name: Yifan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:31.227Z
paper_authors._id: 647bf082aba7062fe5c51ca9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
paper_authors.fullname: Yifan Zhang
paper_authors.type: user
paper_authors.user: yifAI
paper_title: Training and Evaluating Language Models with Template-based Data
    Generation
paper_summary: The rapid advancement of large language models (LLMs) such as GPT-3, PaLM,
  and Llama has significantly transformed natural language processing, showcasing
  remarkable capabilities in understanding and generating language. However,
  these models often struggle with tasks requiring complex reasoning,
  particularly in mathematical problem-solving, due in part to the scarcity of
  large-scale, high-quality, domain-specific datasets necessary for training
  sophisticated reasoning abilities. To address this limitation, we introduce
  Template-based Data Generation (TDG), a novel approach that leverages LLMs
  (GPT-4) to automatically generate parameterized meta-templates, which are then
  used to synthesize a vast array of high-quality problems and solutions.
  Leveraging TDG, we create TemplateMath Part I: TemplateGSM, a dataset
  comprising over 7 million synthetically generated grade school math
  problems--each accompanied by code-based and natural language solutions--with
  the potential to generate an effectively unlimited number more. This dataset
  alleviates the scarcity of large-scale mathematical datasets and serves as a
  valuable resource for pre-training, fine-tuning, and evaluating LLMs in
  mathematical reasoning. Our method not only enables the generation of virtually
  infinite data but also elevates data augmentation to a new level by using GPT-4
  for meta-template generation, ensuring diverse and high-quality problem
  structures. The TemplateMath Part I: TemplateGSM dataset is publicly available
  at https://huggingface.co/datasets/math-ai/TemplateGSM. The code is available
  at https://github.com/iiis-ai/TemplateMath.
paper_ai_keywords: large language models (LLMs) GPT-3 PaLM Llama natural language processing complex reasoning mathematical problem-solving domain-specific datasets Template-based Data Generation (TDG) GPT-4 parameterized meta-templates TemplateMath Part I: TemplateGSM code-based solutions