title: Stack More Layers Differently: High-Rank Training Through Low-Rank
    Updates
summary: Despite the dominance and effectiveness of scaling, resulting in large
  networks with hundreds of billions of parameters, the necessity to train
  overparametrized models remains poorly understood, and alternative approaches
  do not necessarily make it cheaper to train high-performance models. In this
  paper, we explore low-rank training techniques as an alternative approach to
  training large neural networks. We introduce a novel method called ReLoRA,
  which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to
  pre-training transformer language models with up to 350M parameters and
  demonstrate comparable performance to regular neural network training.
  Furthermore, we observe that the efficiency of ReLoRA increases with model
  size, making it a promising approach for training multi-billion-parameter
  networks efficiently. Our findings shed light on the potential of low-rank
  training techniques and their implications for scaling laws.
paper_authors._id: 64af5bc0398f0e7d07a97eb0
paper_authors.name: Vladislav Lialin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:48:51.706Z
paper_authors._id: 5f10910c636b661bdc42317e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650850398598-5f10910c636b661bdc42317e.jpeg
paper_authors.fullname: Vlad Lialin
paper_authors.type: user
paper_authors.user: dropout05
paper_authors._id: 64af5bc0398f0e7d07a97eb1
paper_authors.name: Namrata Shivagunde
paper_authors._id: 64af5bc0398f0e7d07a97eb2
paper_authors.name: Sherin Muckatira
paper_authors._id: 64af5bc0398f0e7d07a97eb3
paper_authors.name: Anna Rumshisky
paper_title: Stack More Layers Differently: High-Rank Training Through Low-Rank
    Updates
paper_summary: Despite the dominance and effectiveness of scaling, resulting in large
  networks with hundreds of billions of parameters, the necessity to train
  overparametrized models remains poorly understood, and alternative approaches
  do not necessarily make it cheaper to train high-performance models. In this
  paper, we explore low-rank training techniques as an alternative approach to
  training large neural networks. We introduce a novel method called ReLoRA,
  which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to
  pre-training transformer language models with up to 350M parameters and
  demonstrate comparable performance to regular neural network training.
  Furthermore, we observe that the efficiency of ReLoRA increases with model
  size, making it a promising approach for training multi-billion-parameter
  networks efficiently. Our findings shed light on the potential of low-rank
  training techniques and their implications for scaling laws.
paper_ai_keywords: low-rank training techniques ReLoRA low-rank updates pre-training transformer language models scaling laws