title: MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus
    Expansion
summary: Despite the remarkable capabilities of large language models across various
  tasks, their continued scaling faces a critical challenge: the scarcity of
  high-quality pretraining data. While model architectures continue to evolve,
  the natural language data struggles to scale up. To tackle this bottleneck, we
  propose MAssive Genre-Audience~(MAGA) reformulation
  method, which systematic synthesizes diverse, contextually-rich pretraining
  data from existing corpus. This work makes three main contributions: (1) We
  propose MAGA reformulation method, a lightweight and scalable approach for
  pretraining corpus expansion, and build a 770B tokens MAGACorpus. (2) We
  evaluate MAGACorpus with different data budget scaling strategies,
  demonstrating consistent improvements across various model sizes (134M-13B),
  establishing the necessity for next-generation large-scale synthetic
  pretraining language models. (3) Through comprehensive analysis, we investigate
  prompt engineering's impact on synthetic training collapse and reveal
  limitations in conventional collapse detection metrics using validation losses.
  Our work shows that MAGA can substantially expand training datasets while
  maintaining quality, offering a reliably pathway for scaling models beyond data
  limitations.
paper_authors._id: 67a56af6d7c26c7497a86308
paper_authors.name: Xintong Hao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-07T04:41:11.249Z
paper_authors._id: 64b764bffdb702b3d8640610
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b764bffdb702b3d8640610/lpHg0AX_NOmzw-ZxeOa1s.png
paper_authors.fullname: haoxintong
paper_authors.type: user
paper_authors.user: haoxintong
paper_authors._id: 67a56af6d7c26c7497a86309
paper_authors.name: Ke Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T12:56:30.330Z
paper_authors._id: 645604eebabbbbd3486dc615
paper_authors.avatarUrl: /avatars/17a5ca8274e2bfc8f183a4af9878a930.svg
paper_authors.fullname: shenke
paper_authors.type: user
paper_authors.user: shenke18
paper_authors._id: 67a56af6d7c26c7497a8630a
paper_authors.name: Chenggang Li
paper_title: MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus
    Expansion
paper_summary: Despite the remarkable capabilities of large language models across various
  tasks, their continued scaling faces a critical challenge: the scarcity of
  high-quality pretraining data. While model architectures continue to evolve,
  the natural language data struggles to scale up. To tackle this bottleneck, we
  propose MAssive Genre-Audience~(MAGA) reformulation
  method, which systematic synthesizes diverse, contextually-rich pretraining
  data from existing corpus. This work makes three main contributions: (1) We
  propose MAGA reformulation method, a lightweight and scalable approach for
  pretraining corpus expansion, and build a 770B tokens MAGACorpus. (2) We
  evaluate MAGACorpus with different data budget scaling strategies,
  demonstrating consistent improvements across various model sizes (134M-13B),
  establishing the necessity for next-generation large-scale synthetic
  pretraining language models. (3) Through comprehensive analysis, we investigate
  prompt engineering's impact on synthetic training collapse and reveal
  limitations in conventional collapse detection metrics using validation losses.
  Our work shows that MAGA can substantially expand training datasets while
  maintaining quality, offering a reliably pathway for scaling models beyond data
  limitations.
paper_ai_keywords: Contrast large language models pretraining data MAGA reformulation method MAGACorpus prompt engineering synthetic training collapse validation losses