title: R&B: Domain Regrouping and Data Mixture Balancing for Efficient
    Foundation Model Training
summary: Data mixing strategies have successfully reduced the costs involved in
  training language models. While promising, such methods suffer from two flaws.
  First, they rely on predetermined data domains (e.g., data sources, task
  types), which may fail to capture critical semantic nuances, leaving
  performance on the table. Second, these methods scale with the number of
  domains in a computationally prohibitive way. We address these challenges via
  R&B, a framework that re-partitions training data based on semantic similarity
  (Regroup) to create finer-grained domains, and efficiently optimizes the data
  composition (Balance) by leveraging a Gram matrix induced by domain gradients
  obtained throughout training. Unlike prior works, it removes the need for
  additional compute to obtain evaluation information such as losses or
  gradients. We analyze this technique under standard regularity conditions and
  provide theoretical insights that justify R&B's effectiveness compared to
  non-adaptive mixing approaches. Empirically, we demonstrate the effectiveness
  of R&B on five diverse datasets ranging from natural language to reasoning and
  multimodal tasks. With as little as 0.01% additional compute overhead, R&B
  matches or exceeds the performance of state-of-the-art data mixing strategies.
paper_authors._id: 68154d77c8ab88a66b8d81a7
paper_authors.name: Albert Ge
paper_authors._id: 68154d77c8ab88a66b8d81a8
paper_authors.name: Tzu-Heng Huang
paper_authors._id: 68154d77c8ab88a66b8d81a9
paper_authors.name: John Cooper
paper_authors._id: 68154d77c8ab88a66b8d81aa
paper_authors.name: Avi Trost
paper_authors._id: 68154d77c8ab88a66b8d81ab
paper_authors.name: Ziyi Chu
paper_authors._id: 68154d77c8ab88a66b8d81ac
paper_authors.name: Satya Sai Srinath Namburi GNVV
paper_authors._id: 68154d77c8ab88a66b8d81ad
paper_authors.name: Ziyang Cai
paper_authors._id: 68154d77c8ab88a66b8d81ae
paper_authors.name: Kendall Park
paper_authors._id: 68154d77c8ab88a66b8d81af
paper_authors.name: Nicholas Roberts
paper_authors._id: 68154d77c8ab88a66b8d81b0
paper_authors.name: Frederic Sala
paper_title: R&B: Domain Regrouping and Data Mixture Balancing for Efficient
    Foundation Model Training
paper_summary: Data mixing strategies have successfully reduced the costs involved in
  training language models. While promising, such methods suffer from two flaws.
  First, they rely on predetermined data domains (e.g., data sources, task
  types), which may fail to capture critical semantic nuances, leaving
  performance on the table. Second, these methods scale with the number of
  domains in a computationally prohibitive way. We address these challenges via
  R&B, a framework that re-partitions training data based on semantic similarity
  (Regroup) to create finer-grained domains, and efficiently optimizes the data
  composition (Balance) by leveraging a Gram matrix induced by domain gradients
  obtained throughout training. Unlike prior works, it removes the need for
  additional compute to obtain evaluation information such as losses or
  gradients. We analyze this technique under standard regularity conditions and
  provide theoretical insights that justify R&B's effectiveness compared to
  non-adaptive mixing approaches. Empirically, we demonstrate the effectiveness
  of R&B on five diverse datasets ranging from natural language to reasoning and
  multimodal tasks. With as little as 0.01% additional compute overhead, R&B
  matches or exceeds the performance of state-of-the-art data mixing strategies.
paper_ai_summary: R&B, a framework that repartitions and balances training data based on semantic similarity and domain gradients, enhances language model performance with minimal additional computational cost.
paper_ai_keywords: semantic similarity Gram matrix domain gradients data mixing strategies evaluation information regularity conditions multimodal tasks