title: Weight subcloning: direct initialization of transformers using larger
    pretrained ones
summary: Training large transformer models from scratch for a target task requires
  lots of data and is computationally demanding. The usual practice of transfer
  learning overcomes this challenge by initializing the model with weights of a
  pretrained model of the same size and specification to increase the convergence
  and training speed. However, what if no pretrained model of the required size
  is available? In this paper, we introduce a simple yet effective technique to
  transfer the knowledge of a pretrained model to smaller variants. Our approach
  called weight subcloning expedites the training of scaled-down transformers by
  initializing their weights from larger pretrained models.
    Weight subcloning involves an operation on the pretrained model to obtain the
  equivalent initialized scaled-down model. It consists of two key steps: first,
  we introduce neuron importance ranking to decrease the embedding dimension per
  layer in the pretrained model. Then, we remove blocks from the transformer
  model to match the number of layers in the scaled-down network. The result is a
  network ready to undergo training, which gains significant improvements in
  training speed compared to random initialization. For instance, we achieve 4x
  faster training for vision transformers in image classification and language
  models designed for next token prediction.
paper_authors._id: 657fa6ce17f67d5b8701c1f8
paper_authors.name: Mohammad Samragh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-22T12:49:51.585Z
paper_authors._id: 66eda2fefd187e5f9e4cd418
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HcRJQ44HV5PeLSn5FoZ5G.png
paper_authors.fullname: Mohammad Samragh
paper_authors.type: user
paper_authors.user: msamragh
paper_authors._id: 657fa6ce17f67d5b8701c1f9
paper_authors.name: Mehrdad Farajtabar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:01:12.473Z
paper_authors._id: 62d188a6d50433c35eb14020
paper_authors.avatarUrl: /avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
paper_authors.fullname: Mehrdad Farajtabar
paper_authors.type: user
paper_authors.user: mfarajtabar
paper_authors._id: 657fa6ce17f67d5b8701c1fa
paper_authors.name: Sachin Mehta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:01:34.876Z
paper_authors._id: 64f792652003abb61bf5c1b8
paper_authors.avatarUrl: /avatars/9721810de3519ece50b059c154baa9ca.svg
paper_authors.fullname: Sachin Mehta
paper_authors.type: user
paper_authors.user: sacmehta
paper_authors._id: 657fa6ce17f67d5b8701c1fb
paper_authors.name: Raviteja Vemulapalli
paper_authors._id: 657fa6ce17f67d5b8701c1fc
paper_authors.name: Fartash Faghri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T15:29:01.893Z
paper_authors._id: 65f8945a3ba58880f9794a11
paper_authors.avatarUrl: /avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
paper_authors.fullname: Fartash Faghri
paper_authors.type: user
paper_authors.user: fartashf
paper_authors._id: 657fa6ce17f67d5b8701c1fd
paper_authors.name: Devang Naik
paper_authors._id: 657fa6ce17f67d5b8701c1fe
paper_authors.name: Oncel Tuzel
paper_authors._id: 657fa6ce17f67d5b8701c1ff
paper_authors.name: Mohammad Rastegari
paper_title: Weight subcloning: direct initialization of transformers using larger
    pretrained ones
paper_summary: Training large transformer models from scratch for a target task requires
  lots of data and is computationally demanding. The usual practice of transfer
  learning overcomes this challenge by initializing the model with weights of a
  pretrained model of the same size and specification to increase the convergence
  and training speed. However, what if no pretrained model of the required size
  is available? In this paper, we introduce a simple yet effective technique to
  transfer the knowledge of a pretrained model to smaller variants. Our approach
  called weight subcloning expedites the training of scaled-down transformers by
  initializing their weights from larger pretrained models.
    Weight subcloning involves an operation on the pretrained model to obtain the
  equivalent initialized scaled-down model. It consists of two key steps: first,
  we introduce neuron importance ranking to decrease the embedding dimension per
  layer in the pretrained model. Then, we remove blocks from the transformer
  model to match the number of layers in the scaled-down network. The result is a
  network ready to undergo training, which gains significant improvements in
  training speed compared to random initialization. For instance, we achieve 4x
  faster training for vision transformers in image classification and language
  models designed for next token prediction.
paper_ai_keywords: transformer models transfer learning pretrained model weight subcloning neuron importance ranking embedding dimension blocks vision transformers image classification language models next token prediction