R&B: Domain Regrouping and Data Mixture Balancing for Efficient
  Foundation Model Training
Data mixing strategies have successfully reduced the costs involved in
training language models. While promising, such methods suffer from two flaws.
First, they rely on predetermined data domains (e.g., data sources, task
types), which may fail to capture critical semantic nuances, leaving
performance on the table. Second, these methods scale with the number of
domains in a computationally prohibitive way. We address these challenges via
R&B, a framework that re-partitions training data based on semantic similarity
(Regroup) to create finer-grained domains, and efficiently optimizes the data
composition (Balance) by leveraging a Gram matrix induced by domain gradients
obtained throughout training. Unlike prior works, it removes the need for
additional compute to obtain evaluation information such as losses or
gradients. We analyze this technique under standard regularity conditions and
provide theoretical insights that justify R&B's effectiveness compared to
non-adaptive mixing approaches. Empirically, we demonstrate the effectiveness
of R&B on five diverse datasets ranging from natural language to reasoning and
multimodal tasks. With as little as 0.01% additional compute overhead, R&B
matches or exceeds the performance of state-of-the-art data mixing strategies.
68154d77c8ab88a66b8d81a7
Albert Ge
68154d77c8ab88a66b8d81a8
Tzu-Heng Huang
claimed_verified
2025-07-08T08:07:36.130Z
6363edf3de4bc2f294accb16
/avatars/9baddfec7170ec662e974116f561ab2c.svg
Tzu-Heng Huang
user
zihengh1
68154d77c8ab88a66b8d81a9
John Cooper
68154d77c8ab88a66b8d81aa
Avi Trost
68154d77c8ab88a66b8d81ab
Ziyi Chu
68154d77c8ab88a66b8d81ac
Satya Sai Srinath Namburi GNVV
claimed_verified
2025-06-23T08:33:01.658Z
632be8c900e5ff9ee1277863
/avatars/04170ba241b61c4d50a7622f207c9ddc.svg
Namburi GNVV Satya Sai Srinath
user
NamburiSrinath
68154d77c8ab88a66b8d81ad
Ziyang Cai
68154d77c8ab88a66b8d81ae
Kendall Park
68154d77c8ab88a66b8d81af
Nicholas Roberts
68154d77c8ab88a66b8d81b0
Frederic Sala
R&B: Domain Regrouping and Data Mixture Balancing for Efficient
  Foundation Model Training
Data mixing strategies have successfully reduced the costs involved in
training language models. While promising, such methods suffer from two flaws.
First, they rely on predetermined data domains (e.g., data sources, task
types), which may fail to capture critical semantic nuances, leaving
performance on the table. Second, these methods scale with the number of
domains in a computationally prohibitive way. We address these challenges via
R&B, a framework that re-partitions training data based on semantic similarity
(Regroup) to create finer-grained domains, and efficiently optimizes the data
composition (Balance) by leveraging a Gram matrix induced by domain gradients
obtained throughout training. Unlike prior works, it removes the need for
additional compute to obtain evaluation information such as losses or
gradients. We analyze this technique under standard regularity conditions and
provide theoretical insights that justify R&B's effectiveness compared to
non-adaptive mixing approaches. Empirically, we demonstrate the effectiveness
of R&B on five diverse datasets ranging from natural language to reasoning and
multimodal tasks. With as little as 0.01% additional compute overhead, R&B
matches or exceeds the performance of state-of-the-art data mixing strategies.
R&B, a framework that repartitions and balances training data based on semantic similarity and domain gradients, enhances language model performance with minimal additional computational cost.
semantic similarity Gram matrix domain gradients data mixing strategies evaluation information regularity conditions multimodal tasks