title: Drop-Upcycling: Training Sparse Mixture of Experts with Partial
    Re-initialization
summary: The Mixture of Experts (MoE) architecture reduces the training and inference
  cost significantly compared to a dense model of equivalent capacity. Upcycling
  is an approach that initializes and trains an MoE model using a pre-trained
  dense model. While upcycling leads to initial performance gains, the training
  progresses slower than when trained from scratch, leading to suboptimal
  performance in the long term. We propose Drop-Upcycling - a method that
  effectively addresses this problem. Drop-Upcycling combines two seemingly
  contradictory approaches: utilizing the knowledge of pre-trained dense models
  while statistically re-initializing some parts of the weights. This approach
  strategically promotes expert specialization, significantly enhancing the MoE
  model's efficiency in knowledge acquisition. Extensive large-scale experiments
  demonstrate that Drop-Upcycling significantly outperforms previous MoE
  construction methods in the long term, specifically when training on hundreds
  of billions of tokens or more. As a result, our MoE model with 5.9B active
  parameters achieves comparable performance to a 13B dense model in the same
  model family, while requiring approximately 1/4 of the training FLOPs. All
  experimental resources, including source code, training data, model checkpoints
  and logs, are publicly available to promote reproducibility and future research
  on MoE.
paper_authors._id: 67c07170af68756abc571ab8
paper_authors.name: Taishi Nakamura
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T22:09:06.783Z
paper_authors._id: 6308c49c454dc257521bc7f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6308c49c454dc257521bc7f9/UWUS6OPa6OpVu1T0gd-wJ.jpeg
paper_authors.fullname: Taishi
paper_authors.type: user
paper_authors.user: Taishi-N324
paper_authors._id: 67c07170af68756abc571ab9
paper_authors.name: Takuya Akiba
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:15:52.711Z
paper_authors._id: 6482810dba6c556892f6f257
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6482810dba6c556892f6f257/c7-wiVKenXiRtwnRpnjZN.jpeg
paper_authors.fullname: Takuya Akiba
paper_authors.type: user
paper_authors.user: iwiwi
paper_authors._id: 67c07170af68756abc571aba
paper_authors.name: Kazuki Fujii
paper_authors._id: 67c07170af68756abc571abb
paper_authors.name: Yusuke Oda
paper_authors._id: 67c07170af68756abc571abc
paper_authors.name: Rio Yokota
paper_authors._id: 67c07170af68756abc571abd
paper_authors.name: Jun Suzuki
paper_title: Drop-Upcycling: Training Sparse Mixture of Experts with Partial
    Re-initialization
paper_summary: The Mixture of Experts (MoE) architecture reduces the training and inference
  cost significantly compared to a dense model of equivalent capacity. Upcycling
  is an approach that initializes and trains an MoE model using a pre-trained
  dense model. While upcycling leads to initial performance gains, the training
  progresses slower than when trained from scratch, leading to suboptimal
  performance in the long term. We propose Drop-Upcycling - a method that
  effectively addresses this problem. Drop-Upcycling combines two seemingly
  contradictory approaches: utilizing the knowledge of pre-trained dense models
  while statistically re-initializing some parts of the weights. This approach
  strategically promotes expert specialization, significantly enhancing the MoE
  model's efficiency in knowledge acquisition. Extensive large-scale experiments
  demonstrate that Drop-Upcycling significantly outperforms previous MoE
  construction methods in the long term, specifically when training on hundreds
  of billions of tokens or more. As a result, our MoE model with 5.9B active
  parameters achieves comparable performance to a 13B dense model in the same
  model family, while requiring approximately 1/4 of the training FLOPs. All
  experimental resources, including source code, training data, model checkpoints
  and logs, are publicly available to promote reproducibility and future research
  on MoE.
paper_ai_summary: Drop-Upcycling enhances MoE models by combining pre-trained knowledge with selective re-initialization, improving long-term performance and efficiency.
paper_ai_keywords: Mixture of Experts MoE upcycling Drop-Upcycling expert specialization training FLOPs