title: Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for
    Mixture-of-Experts Language Models
summary: Scaling the capacity of language models has consistently proven to be a
  reliable approach for improving performance and unlocking new capabilities.
  Capacity can be primarily defined by two dimensions: the number of model
  parameters and the compute per example. While scaling typically involves
  increasing both, the precise interplay between these factors and their combined
  contribution to overall capacity remains not fully understood. We explore this
  relationship in the context of sparse Mixture-of-Experts (MoEs), which allow
  scaling the number of parameters without proportionally increasing the FLOPs
  per example. We investigate how varying the sparsity level, i.e., the fraction
  of inactive parameters, impacts model's performance during pretraining and
  downstream few-shot evaluation. We find that under different constraints (e.g.,
  parameter size and total training compute), there is an optimal level of
  sparsity that improves both training efficiency and model performance. These
  results provide a better understanding of the impact of sparsity in scaling
  laws for MoEs and complement existing works in this area, offering insights for
  designing more efficient architectures.
paper_authors._id: 6798d09d208ffebef5bcfa47
paper_authors.name: Samira Abnar
paper_authors._id: 6798d09d208ffebef5bcfa48
paper_authors.name: Harshay Shah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T14:16:56.095Z
paper_authors._id: 64b1a4f64dd3e24895daa236
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b1a4f64dd3e24895daa236/lzlZ4DBOw-YVspNttAEY3.jpeg
paper_authors.fullname: Harshay Shah
paper_authors.type: user
paper_authors.user: harshay
paper_authors._id: 6798d09d208ffebef5bcfa49
paper_authors.name: Dan Busbridge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T14:17:03.153Z
paper_authors._id: 64c3726f2a5eaefd000cdedd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c3726f2a5eaefd000cdedd/iwFifH1sWQy7agW3eTmNQ.png
paper_authors.fullname: Dan Busbridge
paper_authors.type: user
paper_authors.user: dbusbridge
paper_authors._id: 6798d09d208ffebef5bcfa4a
paper_authors.name: Alaaeldin Mohamed Elnouby Ali
paper_authors._id: 6798d09d208ffebef5bcfa4b
paper_authors.name: Josh Susskind
paper_authors._id: 6798d09d208ffebef5bcfa4c
paper_authors.name: Vimal Thilak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T14:17:43.503Z
paper_authors._id: 6737e918edaf7e05e4b35791
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uk--P2mE2Jeg1QoxmXSTS.png
paper_authors.fullname: Vimal Thilak
paper_authors.type: user
paper_authors.user: vimalthilak
paper_title: Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for
    Mixture-of-Experts Language Models
paper_summary: Scaling the capacity of language models has consistently proven to be a
  reliable approach for improving performance and unlocking new capabilities.
  Capacity can be primarily defined by two dimensions: the number of model
  parameters and the compute per example. While scaling typically involves
  increasing both, the precise interplay between these factors and their combined
  contribution to overall capacity remains not fully understood. We explore this
  relationship in the context of sparse Mixture-of-Experts (MoEs), which allow
  scaling the number of parameters without proportionally increasing the FLOPs
  per example. We investigate how varying the sparsity level, i.e., the fraction
  of inactive parameters, impacts model's performance during pretraining and
  downstream few-shot evaluation. We find that under different constraints (e.g.,
  parameter size and total training compute), there is an optimal level of
  sparsity that improves both training efficiency and model performance. These
  results provide a better understanding of the impact of sparsity in scaling
  laws for MoEs and complement existing works in this area, offering insights for
  designing more efficient architectures.
paper_ai_keywords: sparse Mixture-of-Experts (MoEs) FLOPs sparsity level inactive parameters pretraining few-shot evaluation scaling laws