Data Mixing Agent: Learning to Re-weight Domains for Continual
  Pre-training
Continual pre-training on small-scale task-specific data is an effective
method for improving large language models in new target fields, yet it risks
catastrophic forgetting of their original capabilities. A common solution is to
re-weight training data mixtures from source and target fields on a domain
space to achieve balanced performance. Previous domain reweighting strategies
rely on manual designation with certain heuristics based on human intuition or
empirical results. In this work, we prove that more general heuristics can be
parameterized by proposing Data Mixing Agent, the first model-based, end-to-end
framework that learns to re-weight domains. The agent learns generalizable
heuristics through reinforcement learning on large quantities of data mixing
trajectories with corresponding feedback from an evaluation environment.
Experiments in continual pre-training on math reasoning show that Data Mixing
Agent outperforms strong baselines in achieving balanced performance across
source and target field benchmarks. Furthermore, it generalizes well across
unseen source fields, target models, and domain spaces without retraining.
Direct application to the code generation field also indicates its adaptability
across target domains. Further analysis showcases the agents' well-aligned
heuristics with human intuitions and their efficiency in achieving superior
model performance with less source-field data.
687f42a833947f780d9b4b9a
Kailai Yang
claimed_verified
2025-07-22T08:10:16.373Z
646fc402e9c03ba436d5e93e
/avatars/870c86dc99fb1cb6a348a7a0385b1a04.svg
Kailai Yang
user
klyang
687f42a833947f780d9b4b9b
Xiao Liu
claimed_verified
2025-07-22T07:52:11.552Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
Xiao Liu
user
lx865712528
687f42a833947f780d9b4b9c
Lei Ji
687f42a833947f780d9b4b9d
Hao Li
687f42a833947f780d9b4b9e
Yeyun Gong
687f42a833947f780d9b4b9f
Peng Cheng
687f42a833947f780d9b4ba0
Mao Yang
Data Mixing Agent: Learning to Re-weight Domains for Continual
  Pre-training
Continual pre-training on small-scale task-specific data is an effective
method for improving large language models in new target fields, yet it risks
catastrophic forgetting of their original capabilities. A common solution is to
re-weight training data mixtures from source and target fields on a domain
space to achieve balanced performance. Previous domain reweighting strategies
rely on manual designation with certain heuristics based on human intuition or
empirical results. In this work, we prove that more general heuristics can be
parameterized by proposing Data Mixing Agent, the first model-based, end-to-end
framework that learns to re-weight domains. The agent learns generalizable
heuristics through reinforcement learning on large quantities of data mixing
trajectories with corresponding feedback from an evaluation environment.
Experiments in continual pre-training on math reasoning show that Data Mixing
Agent outperforms strong baselines in achieving balanced performance across
source and target field benchmarks. Furthermore, it generalizes well across
unseen source fields, target models, and domain spaces without retraining.
Direct application to the code generation field also indicates its adaptability
across target domains. Further analysis showcases the agents' well-aligned
heuristics with human intuitions and their efficiency in achieving superior
model performance with less source-field data.
Data Mixing Agent, a model-based framework using reinforcement learning, effectively re-weights training data to balance performance across source and target fields in continual pre-training of large language models.
continual pre-training large language models catastrophic forgetting domain reweighting reinforcement learning data mixing trajectories evaluation environment math reasoning code generation model performance