title: Scaling Laws for Sparsely-Connected Foundation Models
summary: We explore the impact of parameter sparsity on the scaling behavior of
  Transformers trained on massive datasets (i.e., "foundation models"), in both
  vision and language domains. In this setting, we identify the first scaling law
  describing the relationship between weight sparsity, number of non-zero
  parameters, and amount of training data, which we validate empirically across
  model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to
  characterize the "optimal sparsity", the sparsity level which yields the best
  performance for a given effective model size and training budget. For a fixed
  number of non-zero parameters, we identify that the optimal sparsity increases
  with the amount of data used for training. We also extend our study to
  different sparsity structures (such as the hardware-friendly n:m pattern) and
  strategies (such as starting from a pretrained dense model). Our findings shed
  light on the power and limitations of weight sparsity across various parameter
  and computational settings, offering both theoretical understanding and
  practical implications for leveraging sparsity towards computational efficiency
  improvements.
paper_authors._id: 6507b9dfaf387baff4a0a8d3
paper_authors.name: Elias Frantar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T09:57:07.223Z
paper_authors._id: 64b79f81037d6452a317b846
paper_authors.avatarUrl: /avatars/c9c9770136804f8e47f52ff1a4cd120a.svg
paper_authors.fullname: Elias Frantar
paper_authors.type: user
paper_authors.user: efrantar
paper_authors._id: 6507b9dfaf387baff4a0a8d4
paper_authors.name: Carlos Riquelme
paper_authors._id: 6507b9dfaf387baff4a0a8d5
paper_authors.name: Neil Houlsby
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T09:57:25.476Z
paper_authors._id: 64a2c7f263f69fb98d3bfdb4
paper_authors.avatarUrl: /avatars/a362a236c0654b7605dcb7673e309335.svg
paper_authors.fullname: Neil Houlsby
paper_authors.type: user
paper_authors.user: neilhoulsby
paper_authors._id: 6507b9dfaf387baff4a0a8d6
paper_authors.name: Dan Alistarh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T09:57:33.186Z
paper_authors._id: 64d100c5d8d0927372e3d4c0
paper_authors.avatarUrl: /avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
paper_authors.fullname: Dan Alistarh
paper_authors.type: user
paper_authors.user: dalistarh
paper_authors._id: 6507b9dfaf387baff4a0a8d7
paper_authors.name: Utku Evci
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-18T19:42:48.174Z
paper_authors._id: 65088f3b623330a3a5fcd55a
paper_authors.avatarUrl: /avatars/487230672d080d1084fede5e92a3a81f.svg
paper_authors.fullname: Evci
paper_authors.type: user
paper_authors.user: evcu
paper_title: Scaling Laws for Sparsely-Connected Foundation Models
paper_summary: We explore the impact of parameter sparsity on the scaling behavior of
  Transformers trained on massive datasets (i.e., "foundation models"), in both
  vision and language domains. In this setting, we identify the first scaling law
  describing the relationship between weight sparsity, number of non-zero
  parameters, and amount of training data, which we validate empirically across
  model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to
  characterize the "optimal sparsity", the sparsity level which yields the best
  performance for a given effective model size and training budget. For a fixed
  number of non-zero parameters, we identify that the optimal sparsity increases
  with the amount of data used for training. We also extend our study to
  different sparsity structures (such as the hardware-friendly n:m pattern) and
  strategies (such as starting from a pretrained dense model). Our findings shed
  light on the power and limitations of weight sparsity across various parameter
  and computational settings, offering both theoretical understanding and
  practical implications for leveraging sparsity towards computational efficiency
  improvements.
paper_ai_summary: Parameter sparsity in Transformers affects their scaling behavior on large datasets; optimal sparsity increases with data volume and different structures can impact performance.
paper_ai_keywords: parameter sparsity scaling law Transformers weight sparsity ViT JFT-4B T5 C4 optimal sparsity hardware-friendly n:m pattern pretrained dense model