title: Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
summary: We investigate efficient methods for training Large Language Models (LLMs) to
  possess capabilities in multiple specialized domains, such as coding, math
  reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts
  from a seed model, which is branched to train experts in embarrassingly
  parallel fashion with high throughput and reduced communication cost. After
  individual experts are asynchronously trained, BTX brings together their
  feedforward parameters as experts in Mixture-of-Expert (MoE) layers and
  averages the remaining parameters, followed by an MoE-finetuning stage to learn
  token-level routing. BTX generalizes two special cases, the Branch-Train-Merge
  method, which does not have the MoE finetuning stage to learn routing, and
  sparse upcycling, which omits the stage of training experts asynchronously.
  Compared to alternative approaches, BTX achieves the best accuracy-efficiency
  tradeoff.
paper_authors._id: 65f114b632c29f9fb7031dfa
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T16:01:22.831Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 65f114b632c29f9fb7031dfb
paper_authors.name: Olga Golovneva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:32:59.410Z
paper_authors._id: 6318b1e06e6bd039440871d6
paper_authors.avatarUrl: /avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
paper_authors.fullname: Olga Golovneva
paper_authors.type: user
paper_authors.user: Golovneva
paper_authors._id: 65f114b632c29f9fb7031dfc
paper_authors.name: Vasu Sharma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:33:23.695Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 65f114b632c29f9fb7031dfd
paper_authors.name: Hu Xu
paper_authors._id: 65f114b632c29f9fb7031dfe
paper_authors.name: Xi Victoria Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:34:11.498Z
paper_authors._id: 61df349e34b589fbaf2e93c4
paper_authors.avatarUrl: /avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
paper_authors.fullname: Victoria Lin
paper_authors.type: user
paper_authors.user: VictoriaLinML
paper_authors._id: 65f114b632c29f9fb7031dff
paper_authors.name: Baptiste Rozière
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:34:21.092Z
paper_authors._id: 62d063dac375d0c84255b9a1
paper_authors.avatarUrl: /avatars/de0fc34bad8c761210c0895ebfa4feba.svg
paper_authors.fullname: Baptiste Roziere
paper_authors.type: user
paper_authors.user: broz
paper_authors._id: 65f114b632c29f9fb7031e00
paper_authors.name: Jacob Kahn
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T19:44:10.442Z
paper_authors._id: 65f1eac292b7b4b9d59cb886
paper_authors.avatarUrl: /avatars/e7adb8a06860663c828501460cc092ae.svg
paper_authors.fullname: Jacob Kahn
paper_authors.type: user
paper_authors.user: jacobkahn
paper_authors._id: 65f114b632c29f9fb7031e01
paper_authors.name: Daniel Li
paper_authors._id: 65f114b632c29f9fb7031e02
paper_authors.name: Wen-tau Yih
paper_authors._id: 65f114b632c29f9fb7031e03
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:34:53.258Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 65f114b632c29f9fb7031e04
paper_authors.name: Xian Li
paper_title: Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
paper_summary: We investigate efficient methods for training Large Language Models (LLMs) to
  possess capabilities in multiple specialized domains, such as coding, math
  reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts
  from a seed model, which is branched to train experts in embarrassingly
  parallel fashion with high throughput and reduced communication cost. After
  individual experts are asynchronously trained, BTX brings together their
  feedforward parameters as experts in Mixture-of-Expert (MoE) layers and
  averages the remaining parameters, followed by an MoE-finetuning stage to learn
  token-level routing. BTX generalizes two special cases, the Branch-Train-Merge
  method, which does not have the MoE finetuning stage to learn routing, and
  sparse upcycling, which omits the stage of training experts asynchronously.
  Compared to alternative approaches, BTX achieves the best accuracy-efficiency
  tradeoff.
paper_ai_keywords: Large Language Models (LLMs) Branch-Train-MiX (BTX) seed model branched model embarrassingly parallel fashion Mixture-of-Expert (MoE) layers feedforward parameters token-level routing Branch-Train-Merge sparse upcycling state-efficiency tradeoff