title: Model Merging in Pre-training of Large Language Models
summary: Model merging has emerged as a promising technique for enhancing large
  language models, though its application in large-scale pre-training remains
  relatively unexplored. In this paper, we present a comprehensive investigation
  of model merging techniques during the pre-training process. Through extensive
  experiments with both dense and Mixture-of-Experts (MoE) architectures ranging
  from millions to over 100 billion parameters, we demonstrate that merging
  checkpoints trained with constant learning rates not only achieves significant
  performance improvements but also enables accurate prediction of annealing
  behavior. These improvements lead to both more efficient model development and
  significantly lower training costs. Our detailed ablation studies on merging
  strategies and hyperparameters provide new insights into the underlying
  mechanisms while uncovering novel applications. Through comprehensive
  experimental analysis, we offer the open-source community practical
  pre-training guidelines for effective model merging.
paper_authors._id: 682c55ad39dfc753340da444
paper_authors.name: Yunshui Li
paper_authors._id: 682c55ad39dfc753340da445
paper_authors.name: Yiyuan Ma
paper_authors._id: 682c55ad39dfc753340da446
paper_authors.name: Shen Yan
paper_authors._id: 682c55ad39dfc753340da447
paper_authors.name: Chaoyi Zhang
paper_authors._id: 682c55ad39dfc753340da448
paper_authors.name: Jing Liu
paper_authors._id: 682c55ad39dfc753340da449
paper_authors.name: Jianqiao Lu
paper_authors._id: 682c55ad39dfc753340da44a
paper_authors.name: Ziwen Xu
paper_authors._id: 682c55ad39dfc753340da44b
paper_authors.name: Mengzhao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:18:09.440Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 682c55ad39dfc753340da44c
paper_authors.name: Minrui Wang
paper_authors._id: 682c55ad39dfc753340da44d
paper_authors.name: Shiyi Zhan
paper_authors._id: 682c55ad39dfc753340da44e
paper_authors.name: Jin Ma
paper_authors._id: 682c55ad39dfc753340da44f
paper_authors.name: Xunhao Lai
paper_authors._id: 682c55ad39dfc753340da450
paper_authors.name: Yao Luo
paper_authors._id: 682c55ad39dfc753340da451
paper_authors.name: Xingyan Bin
paper_authors._id: 682c55ad39dfc753340da452
paper_authors.name: Hongbin Ren
paper_authors._id: 682c55ad39dfc753340da453
paper_authors.name: Mingji Han
paper_authors._id: 682c55ad39dfc753340da454
paper_authors.name: Wenhao Hao
paper_authors._id: 682c55ad39dfc753340da455
paper_authors.name: Bairen Yi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T09:03:30.697Z
paper_authors._id: 6369d92f64aad59d4d44d362
paper_authors.avatarUrl: /avatars/73956400cfbfd53116aefc17b3c9f0fd.svg
paper_authors.fullname: Yi
paper_authors.type: user
paper_authors.user: Bairen
paper_authors._id: 682c55ad39dfc753340da456
paper_authors.name: LingJun Liu
paper_authors._id: 682c55ad39dfc753340da457
paper_authors.name: Bole Ma
paper_authors._id: 682c55ad39dfc753340da458
paper_authors.name: Xiaoying Jia
paper_authors._id: 682c55ad39dfc753340da459
paper_authors.name: Zhou Xun
paper_authors._id: 682c55ad39dfc753340da45a
paper_authors.name: Liang Xiang
paper_authors._id: 682c55ad39dfc753340da45b
paper_authors.name: Yonghui Wu
paper_title: Model Merging in Pre-training of Large Language Models
paper_summary: Model merging has emerged as a promising technique for enhancing large
  language models, though its application in large-scale pre-training remains
  relatively unexplored. In this paper, we present a comprehensive investigation
  of model merging techniques during the pre-training process. Through extensive
  experiments with both dense and Mixture-of-Experts (MoE) architectures ranging
  from millions to over 100 billion parameters, we demonstrate that merging
  checkpoints trained with constant learning rates not only achieves significant
  performance improvements but also enables accurate prediction of annealing
  behavior. These improvements lead to both more efficient model development and
  significantly lower training costs. Our detailed ablation studies on merging
  strategies and hyperparameters provide new insights into the underlying
  mechanisms while uncovering novel applications. Through comprehensive
  experimental analysis, we offer the open-source community practical
  pre-training guidelines for effective model merging.
paper_ai_summary: Model merging during pre-training enhances large language models, improving performance and reducing costs across various architectures and learning rates.
paper_ai_keywords: dense architectures Mixture-of-Experts checkpoints constant learning rates annealing behavior ablation studies hyperparameters pre-training guidelines