title: Scaling Smart: Accelerating Large Language Model Pre-training with Small
    Model Initialization
summary: The pre-training phase of language models often begins with randomly
  initialized parameters. With the current trends in scaling models, training
  their large number of parameters can be extremely slow and costly. In contrast,
  small language models are less expensive to train, but they often cannot
  achieve the accuracy of large models. In this paper, we explore an intriguing
  idea to connect these two different regimes: Can we develop a method to
  initialize large language models using smaller pre-trained models? Will such
  initialization bring any benefits in terms of training time and final accuracy?
  In this paper, we introduce HyperCloning, a method that can expand the
  parameters of a pre-trained language model to those of a larger model with
  increased hidden dimensions. Our method ensures that the larger model retains
  the functionality of the smaller model. As a result, the larger model already
  inherits the predictive power and accuracy of the smaller model before the
  training starts. We demonstrate that training such an initialized model results
  in significant savings in terms of GPU hours required for pre-training large
  language models.
paper_authors._id: 66ece1eca67d89bbe4284f8d
paper_authors.name: Mohammad Samragh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-22T12:47:58.302Z
paper_authors._id: 66eda2fefd187e5f9e4cd418
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HcRJQ44HV5PeLSn5FoZ5G.png
paper_authors.fullname: Mohammad Samragh
paper_authors.type: user
paper_authors.user: msamragh
paper_authors._id: 66ece1eca67d89bbe4284f8e
paper_authors.name: Iman Mirzadeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:24:44.826Z
paper_authors._id: 66280a4c7c988f33b795f90f
paper_authors.avatarUrl: /avatars/7a618b39c4ed20f5490ea64fbb67b5ac.svg
paper_authors.fullname: Iman Mirzadeh
paper_authors.type: user
paper_authors.user: imirzadeh-apple
paper_authors._id: 66ece1eca67d89bbe4284f8f
paper_authors.name: Keivan Alizadeh Vahid
paper_authors._id: 66ece1eca67d89bbe4284f90
paper_authors.name: Fartash Faghri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:24:56.327Z
paper_authors._id: 65f8945a3ba58880f9794a11
paper_authors.avatarUrl: /avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
paper_authors.fullname: Fartash Faghri
paper_authors.type: user
paper_authors.user: fartashf
paper_authors._id: 66ece1eca67d89bbe4284f91
paper_authors.name: Minsik Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:25:06.825Z
paper_authors._id: 630d422454c3dbd48058ba09
paper_authors.avatarUrl: /avatars/1985c5812523fddc0835cde18a490856.svg
paper_authors.fullname: minsik cho
paper_authors.type: user
paper_authors.user: thyeros
paper_authors._id: 66ece1eca67d89bbe4284f92
paper_authors.name: Moin Nabi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:25:12.769Z
paper_authors._id: 5fe36e9297b72d62ab9b90cd
paper_authors.avatarUrl: /avatars/edfea7e8c86dcf355241f5a48365b35a.svg
paper_authors.fullname: Moin Nabi
paper_authors.type: user
paper_authors.user: moinnabi
paper_authors._id: 66ece1eca67d89bbe4284f93
paper_authors.name: Devang Naik
paper_authors._id: 66ece1eca67d89bbe4284f94
paper_authors.name: Mehrdad Farajtabar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:25:23.623Z
paper_authors._id: 62d188a6d50433c35eb14020
paper_authors.avatarUrl: /avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
paper_authors.fullname: Mehrdad Farajtabar
paper_authors.type: user
paper_authors.user: mfarajtabar
paper_title: Scaling Smart: Accelerating Large Language Model Pre-training with Small
    Model Initialization
paper_summary: The pre-training phase of language models often begins with randomly
  initialized parameters. With the current trends in scaling models, training
  their large number of parameters can be extremely slow and costly. In contrast,
  small language models are less expensive to train, but they often cannot
  achieve the accuracy of large models. In this paper, we explore an intriguing
  idea to connect these two different regimes: Can we develop a method to
  initialize large language models using smaller pre-trained models? Will such
  initialization bring any benefits in terms of training time and final accuracy?
  In this paper, we introduce HyperCloning, a method that can expand the
  parameters of a pre-trained language model to those of a larger model with
  increased hidden dimensions. Our method ensures that the larger model retains
  the functionality of the smaller model. As a result, the larger model already
  inherits the predictive power and accuracy of the smaller model before the
  training starts. We demonstrate that training such an initialized model results
  in significant savings in terms of GPU hours required for pre-training large
  language models.
paper_ai_keywords: language models pre-training phase parameters scaling models training accuracy small language models large language models initialization HyperCloning phrase embeddings predictive power GPU hours pre-training large language models