title: BAM! Just Like That: Simple and Efficient Parameter Upcycling for
    Mixture of Experts
summary: The Mixture of Experts (MoE) framework has become a popular architecture for
  large language models due to its superior performance over dense models.
  However, training MoEs from scratch in a large-scale regime is prohibitively
  expensive. Existing methods mitigate this by pre-training multiple dense expert
  models independently and using them to initialize an MoE. This is done by using
  experts' feed-forward network (FFN) to initialize the MoE's experts while
  merging other parameters. However, this method limits the reuse of dense model
  parameters to only the FFN layers, thereby constraining the advantages when
  "upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a
  simple yet effective method that addresses this shortcoming. BAM makes full use
  of specialized dense models by not only using their FFN to initialize the MoE
  layers but also leveraging experts' attention parameters fully by initializing
  them into a soft-variant of Mixture of Attention (MoA) layers. We explore two
  methods for upcycling attention parameters: 1) initializing separate attention
  experts from dense models including all attention parameters for the best model
  performance; and 2) sharing key and value parameters across all experts to
  facilitate for better inference efficiency. To further improve efficiency, we
  adopt a parallel attention transformer architecture to MoEs, which allows the
  attention experts and FFN experts to be computed concurrently. Our experiments
  on seed models ranging from 590 million to 2 billion parameters demonstrate
  that BAM surpasses baselines in both perplexity and downstream task
  performance, within the same computational and data constraints.
paper_authors._id: 66bed259a9425c872d3eaedb
paper_authors.name: Qizhen Zhang
paper_authors._id: 66bed259a9425c872d3eaedc
paper_authors.name: Nikolas Gritsch
paper_authors._id: 66bed259a9425c872d3eaedd
paper_authors.name: Dwaraknath Gnaneshwar
paper_authors._id: 66bed259a9425c872d3eaede
paper_authors.name: Simon Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:42:25.438Z
paper_authors._id: 65ef61af8179815f59d189ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ef61af8179815f59d189ea/fknSvriVyP4KMI7Z53LNH.png
paper_authors.fullname: Simon Guo
paper_authors.type: user
paper_authors.user: simonguozirui
paper_authors._id: 66bed259a9425c872d3eaedf
paper_authors.name: David Cairuz
paper_authors._id: 66bed259a9425c872d3eaee0
paper_authors.name: Bharat Venkitesh
paper_authors._id: 66bed259a9425c872d3eaee1
paper_authors.name: Jakob Foerster
paper_authors._id: 66bed259a9425c872d3eaee2
paper_authors.name: Phil Blunsom
paper_authors._id: 66bed259a9425c872d3eaee3
paper_authors.name: Sebastian Ruder
paper_authors._id: 66bed259a9425c872d3eaee4
paper_authors.name: Ahmet Ustun
paper_authors._id: 66bed259a9425c872d3eaee5
paper_authors.name: Acyr Locatelli
paper_title: BAM! Just Like That: Simple and Efficient Parameter Upcycling for
    Mixture of Experts
paper_summary: The Mixture of Experts (MoE) framework has become a popular architecture for
  large language models due to its superior performance over dense models.
  However, training MoEs from scratch in a large-scale regime is prohibitively
  expensive. Existing methods mitigate this by pre-training multiple dense expert
  models independently and using them to initialize an MoE. This is done by using
  experts' feed-forward network (FFN) to initialize the MoE's experts while
  merging other parameters. However, this method limits the reuse of dense model
  parameters to only the FFN layers, thereby constraining the advantages when
  "upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a
  simple yet effective method that addresses this shortcoming. BAM makes full use
  of specialized dense models by not only using their FFN to initialize the MoE
  layers but also leveraging experts' attention parameters fully by initializing
  them into a soft-variant of Mixture of Attention (MoA) layers. We explore two
  methods for upcycling attention parameters: 1) initializing separate attention
  experts from dense models including all attention parameters for the best model
  performance; and 2) sharing key and value parameters across all experts to
  facilitate for better inference efficiency. To further improve efficiency, we
  adopt a parallel attention transformer architecture to MoEs, which allows the
  attention experts and FFN experts to be computed concurrently. Our experiments
  on seed models ranging from 590 million to 2 billion parameters demonstrate
  that BAM surpasses baselines in both perplexity and downstream task
  performance, within the same computational and data constraints.
paper_ai_summary: BAM enhances the Mixture of Experts framework by fully recycling dense model parameters, including attention layers, leading to better performance and efficiency than previous methods.
paper_ai_keywords: Mixture of Experts dense models feed-forward network initialization attention parameters Mixture of Attention parallel attention transformer perplexity downstream tasks