title: Gemstones: A Model Suite for Multi-Faceted Scaling Laws
summary: Scaling laws are typically fit using a family of models with a narrow range
  of frozen hyper-parameter choices. In this work we study scaling laws using a
  wide range of architecture and hyper-parameter choices, and highlight their
  impact on resulting prescriptions. As a primary artifact of our research, we
  release the Gemstones: the most comprehensive open-source scaling law dataset
  to date, consisting of over 4000 checkpoints from transformers with up to 2
  billion parameters; these models have been trained with different learning
  rates, cooldown schedules, and architectural shapes. Our checkpoints enable
  more complex studies of scaling, such as a law that predicts language modeling
  performance as a function of model width and depth. By examining the various
  facets of our model suite, we find that the prescriptions of scaling laws can
  be highly sensitive to the experimental design process and the specific model
  checkpoints used during fitting. Code:
  https://github.com/mcleish7/gemstone-scaling-laws
paper_authors._id: 67ac9127184de583cc7daa75
paper_authors.name: Sean McLeish
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-12T12:21:11.370Z
paper_authors._id: 65255f1073a043e50d043641
paper_authors.avatarUrl: /avatars/257085f01c439d7c84787a4e6d085b3d.svg
paper_authors.fullname: Sean McLeish
paper_authors.type: user
paper_authors.user: smcleish
paper_authors._id: 67ac9127184de583cc7daa76
paper_authors.name: John Kirchenbauer
paper_authors._id: 67ac9127184de583cc7daa77
paper_authors.name: David Yu Miller
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:35:58.819Z
paper_authors._id: 67ae2de52f89d658be916ff0
paper_authors.avatarUrl: /avatars/55f6bea7b6727f5f3f152cd8659f75f6.svg
paper_authors.fullname: David Miller
paper_authors.type: user
paper_authors.user: dymil
paper_authors._id: 67ac9127184de583cc7daa78
paper_authors.name: Siddharth Singh
paper_authors._id: 67ac9127184de583cc7daa79
paper_authors.name: Abhinav Bhatele
paper_authors._id: 67ac9127184de583cc7daa7a
paper_authors.name: Micah Goldblum
paper_authors._id: 67ac9127184de583cc7daa7b
paper_authors.name: Ashwinee Panda
paper_authors._id: 67ac9127184de583cc7daa7c
paper_authors.name: Tom Goldstein
paper_title: Gemstones: A Model Suite for Multi-Faceted Scaling Laws
paper_summary: Scaling laws are typically fit using a family of models with a narrow range
  of frozen hyper-parameter choices. In this work we study scaling laws using a
  wide range of architecture and hyper-parameter choices, and highlight their
  impact on resulting prescriptions. As a primary artifact of our research, we
  release the Gemstones: the most comprehensive open-source scaling law dataset
  to date, consisting of over 4000 checkpoints from transformers with up to 2
  billion parameters; these models have been trained with different learning
  rates, cooldown schedules, and architectural shapes. Our checkpoints enable
  more complex studies of scaling, such as a law that predicts language modeling
  performance as a function of model width and depth. By examining the various
  facets of our model suite, we find that the prescriptions of scaling laws can
  be highly sensitive to the experimental design process and the specific model
  checkpoints used during fitting. Code:
  https://github.com/mcleish7/gemstone-scaling-laws
paper_ai_keywords: scaling laws hyper-parameter choices architecture prescriptions Gemstones scaling law dataset checkpoints parameters learning rates cooldown schedules architectural shapes model width model depth scaling laws sensitivity experimental design process