title: Scaling Laws for Fine-Grained Mixture of Experts
summary: Mixture of Experts (MoE) models have emerged as a primary solution for
  reducing the computational cost of Large Language Models. In this work, we
  analyze their scaling properties, incorporating an expanded range of variables.
  Specifically, we introduce a new hyperparameter, granularity, whose adjustment
  enables precise control over the size of the experts. Building on this, we
  establish scaling laws for fine-grained MoE, taking into account the number of
  training tokens, model size, and granularity. Leveraging these laws, we derive
  the optimal training configuration for a given computational budget. Our
  findings not only show that MoE models consistently outperform dense
  Transformers but also highlight that the efficiency gap between dense and MoE
  models widens as we scale up the model size and training budget. Furthermore,
  we demonstrate that the common practice of setting the size of experts in MoE
  to mirror the feed-forward layer is not optimal at almost any computational
  budget.
paper_authors._id: 65cae5a25f6633f8ce245f08
paper_authors.name: Jakub Krajewski
paper_authors._id: 65cae5a25f6633f8ce245f09
paper_authors.name: Jan Ludziejewski
paper_authors._id: 65cae5a25f6633f8ce245f0a
paper_authors.name: Kamil Adamczewski
paper_authors._id: 65cae5a25f6633f8ce245f0b
paper_authors.name: Maciej Pióro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:03:19.763Z
paper_authors._id: 643e7c8b3870da7f0ef20f36
paper_authors.avatarUrl: /avatars/71291f75e2ab20818537159179f904a6.svg
paper_authors.fullname: Maciej Pióro
paper_authors.type: user
paper_authors.user: maciek-pioro
paper_authors._id: 65cae5a25f6633f8ce245f0c
paper_authors.name: Michał Krutul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:03:27.911Z
paper_authors._id: 63bf063987619d1458ca9177
paper_authors.avatarUrl: /avatars/58f55bb6dd6f23b00fd2f01bd3be543a.svg
paper_authors.fullname: Michał Krutul
paper_authors.type: user
paper_authors.user: crewtool
paper_authors._id: 65cae5a25f6633f8ce245f0d
paper_authors.name: Szymon Antoniak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:03:34.919Z
paper_authors._id: 63f767c4bd28622c9b984637
paper_authors.avatarUrl: /avatars/126a23ca5b3adc89418b7a53b21e50e6.svg
paper_authors.fullname: Szymon Antoniak
paper_authors.type: user
paper_authors.user: Simontwice
paper_authors._id: 65cae5a25f6633f8ce245f0e
paper_authors.name: Kamil Ciebiera
paper_authors._id: 65cae5a25f6633f8ce245f0f
paper_authors.name: Krystian Król
paper_authors._id: 65cae5a25f6633f8ce245f10
paper_authors.name: Tomasz Odrzygóźdź
paper_authors._id: 65cae5a25f6633f8ce245f11
paper_authors.name: Piotr Sankowski
paper_authors._id: 65cae5a25f6633f8ce245f12
paper_authors.name: Marek Cygan
paper_authors._id: 65cae5a25f6633f8ce245f13
paper_authors.name: Sebastian Jaszczur
paper_title: Scaling Laws for Fine-Grained Mixture of Experts
paper_summary: Mixture of Experts (MoE) models have emerged as a primary solution for
  reducing the computational cost of Large Language Models. In this work, we
  analyze their scaling properties, incorporating an expanded range of variables.
  Specifically, we introduce a new hyperparameter, granularity, whose adjustment
  enables precise control over the size of the experts. Building on this, we
  establish scaling laws for fine-grained MoE, taking into account the number of
  training tokens, model size, and granularity. Leveraging these laws, we derive
  the optimal training configuration for a given computational budget. Our
  findings not only show that MoE models consistently outperform dense
  Transformers but also highlight that the efficiency gap between dense and MoE
  models widens as we scale up the model size and training budget. Furthermore,
  we demonstrate that the common practice of setting the size of experts in MoE
  to mirror the feed-forward layer is not optimal at almost any computational
  budget.
paper_ai_keywords: Mixture of Experts (MoE) computational cost Large Language Models granularity hyperparameter scaling laws fine-grained MoE training tokens model size optimal training configuration dense Transformers efficiency gap experts feed-forward layer