title: CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for
    Language Model Pre-training
summary: Pre-training datasets are typically collected from web content and lack
  inherent domain divisions. For instance, widely used datasets like Common Crawl
  do not include explicit domain labels, while manually curating labeled datasets
  such as The Pile is labor-intensive. Consequently, identifying an optimal
  pre-training data mixture remains a challenging problem, despite its
  significant benefits for pre-training performance. To address these challenges,
  we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an
  automated framework that discovers, evaluates, and refines data mixtures in a
  pre-training setting. Specifically, CLIMB embeds and clusters large-scale
  datasets in a semantic space and then iteratively searches for optimal mixtures
  using a smaller proxy model and a predictor. When continuously trained on 400B
  tokens with this mixture, our 1B model exceeds the state-of-the-art
  Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific
  domain (e.g., Social Sciences) yields a 5% improvement over random sampling.
  Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20
  clusters as a research playground, and ClimbMix, a compact yet powerful
  400-billion-token dataset designed for efficient pre-training that delivers
  superior performance under an equal token budget. We analyze the final data
  mixture, elucidating the characteristics of an optimal data mixture. Our data
  is available at: https://research.nvidia.com/labs/lpr/climb/
paper_authors._id: 6801d661ed5fc062197db592
paper_authors.name: Shizhe Diao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:38:46.555Z
paper_authors._id: 633bd54b00732349209a18fe
paper_authors.avatarUrl: /avatars/150aa5b8d9bcca24366402932bf2d049.svg
paper_authors.fullname: Shizhe Diao
paper_authors.type: user
paper_authors.user: shizhediao
paper_authors._id: 6801d661ed5fc062197db593
paper_authors.name: Yu Yang
paper_authors._id: 6801d661ed5fc062197db594
paper_authors.name: Yonggan Fu
paper_authors._id: 6801d661ed5fc062197db595
paper_authors.name: Xin Dong
paper_authors._id: 6801d661ed5fc062197db596
paper_authors.name: Dan Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:29:21.716Z
paper_authors._id: 67e34615c15564fd97f697ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BNXjsEA5rGn7uFoFuVkB6.png
paper_authors.fullname: Dan Su
paper_authors.type: user
paper_authors.user: sudandandansu1
paper_authors._id: 6801d661ed5fc062197db597
paper_authors.name: Markus Kliegl
paper_authors._id: 6801d661ed5fc062197db598
paper_authors.name: Zijia Chen
paper_authors._id: 6801d661ed5fc062197db599
paper_authors.name: Peter Belcak
paper_authors._id: 6801d661ed5fc062197db59a
paper_authors.name: Yoshi Suhara
paper_authors._id: 6801d661ed5fc062197db59b
paper_authors.name: Hongxu Yin
paper_authors._id: 6801d661ed5fc062197db59c
paper_authors.name: Mostofa Patwary
paper_authors._id: 6801d661ed5fc062197db59d
paper_authors.name: Yingyan
paper_authors._id: 6801d661ed5fc062197db59e
paper_authors.name: Lin
paper_authors._id: 6801d661ed5fc062197db59f
paper_authors.name: Jan Kautz
paper_authors._id: 6801d661ed5fc062197db5a0
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:03:26.583Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_title: CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for
    Language Model Pre-training
paper_summary: Pre-training datasets are typically collected from web content and lack
  inherent domain divisions. For instance, widely used datasets like Common Crawl
  do not include explicit domain labels, while manually curating labeled datasets
  such as The Pile is labor-intensive. Consequently, identifying an optimal
  pre-training data mixture remains a challenging problem, despite its
  significant benefits for pre-training performance. To address these challenges,
  we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an
  automated framework that discovers, evaluates, and refines data mixtures in a
  pre-training setting. Specifically, CLIMB embeds and clusters large-scale
  datasets in a semantic space and then iteratively searches for optimal mixtures
  using a smaller proxy model and a predictor. When continuously trained on 400B
  tokens with this mixture, our 1B model exceeds the state-of-the-art
  Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific
  domain (e.g., Social Sciences) yields a 5% improvement over random sampling.
  Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20
  clusters as a research playground, and ClimbMix, a compact yet powerful
  400-billion-token dataset designed for efficient pre-training that delivers
  superior performance under an equal token budget. We analyze the final data
  mixture, elucidating the characteristics of an optimal data mixture. Our data
  is available at: https://research.nvidia.com/labs/lpr/climb/
paper_ai_keywords: CLIMB semantic space proxy model predictor ClimbLab ClimbMix