title: LoGAH: Predicting 774-Million-Parameter Transformers using Graph
    HyperNetworks with 1/100 Parameters
summary: A good initialization of deep learning models is essential since it can help
  them converge better and faster. However, pretraining large models is
  unaffordable for many researchers, which makes a desired prediction for initial
  parameters more necessary nowadays. Graph HyperNetworks (GHNs), one approach to
  predicting model parameters, have recently shown strong performance in
  initializing large vision models. Unfortunately, predicting parameters of very
  wide networks relies on copying small chunks of parameters multiple times and
  requires an extremely large number of parameters to support full prediction,
  which greatly hinders its adoption in practice. To address this limitation, we
  propose LoGAH (Low-rank GrAph Hypernetworks), a GHN with a low-rank parameter
  decoder that expands to significantly wider networks without requiring as
  excessive increase of parameters as in previous attempts. LoGAH allows us to
  predict the parameters of 774-million large neural networks in a
  memory-efficient manner. We show that vision and language models (i.e., ViT and
  GPT-2) initialized with LoGAH achieve better performance than those initialized
  randomly or using existing hypernetworks. Furthermore, we show promising
  transfer learning results w.r.t. training LoGAH on small datasets and using the
  predicted parameters to initialize for larger tasks. We provide the codes in
  https://github.com/Blackzxy/LoGAH .
paper_authors._id: 66556035d200a2ecdfef4fd0
paper_authors.name: Xinyu Zhou
paper_authors._id: 66556035d200a2ecdfef4fd1
paper_authors.name: Boris Knyazev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:28:47.528Z
paper_authors._id: 63e16165f0039731dfdd442a
paper_authors.avatarUrl: /avatars/37cf99dc016c291c800f60d260173482.svg
paper_authors.fullname: Boris Knyazev
paper_authors.type: user
paper_authors.user: bknyaz
paper_authors._id: 66556035d200a2ecdfef4fd2
paper_authors.name: Alexia Jolicoeur-Martineau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:28:53.448Z
paper_authors._id: 62f8ea1177b722f186611e8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660479589894-noauth.jpeg
paper_authors.fullname: Alexia Jolicoeur-Martineau
paper_authors.type: user
paper_authors.user: AlexiaJM
paper_authors._id: 66556035d200a2ecdfef4fd3
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T09:34:11.484Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: LoGAH: Predicting 774-Million-Parameter Transformers using Graph
    HyperNetworks with 1/100 Parameters
paper_summary: A good initialization of deep learning models is essential since it can help
  them converge better and faster. However, pretraining large models is
  unaffordable for many researchers, which makes a desired prediction for initial
  parameters more necessary nowadays. Graph HyperNetworks (GHNs), one approach to
  predicting model parameters, have recently shown strong performance in
  initializing large vision models. Unfortunately, predicting parameters of very
  wide networks relies on copying small chunks of parameters multiple times and
  requires an extremely large number of parameters to support full prediction,
  which greatly hinders its adoption in practice. To address this limitation, we
  propose LoGAH (Low-rank GrAph Hypernetworks), a GHN with a low-rank parameter
  decoder that expands to significantly wider networks without requiring as
  excessive increase of parameters as in previous attempts. LoGAH allows us to
  predict the parameters of 774-million large neural networks in a
  memory-efficient manner. We show that vision and language models (i.e., ViT and
  GPT-2) initialized with LoGAH achieve better performance than those initialized
  randomly or using existing hypernetworks. Furthermore, we show promising
  transfer learning results w.r.t. training LoGAH on small datasets and using the
  predicted parameters to initialize for larger tasks. We provide the codes in
  https://github.com/Blackzxy/LoGAH .
paper_ai_summary: LoGAH, a low-rank parameter decoder Graph HyperNetwork, efficiently initializes large neural networks with better performance compared to random or existing hypernetworks.
paper_ai_keywords: Graph HyperNetworks GHNs LoGAH low-rank parameter decoder memory-efficient large neural networks vision models language models ViT GPT-2 transfer learning