content.raw: 🔥 Level up your model training w/ GaLore + Transformers for SOTA results on consumer-grade hardware!
content.type: text
content.value: 🔥 Level up your model training w/ GaLore + Transformers for SOTA results on consumer-grade hardware!
content.type: new_line
content.type: new_line
content.raw: ⬇️ 82.5% less optimizer state memory footprint without performance degradation by expressing the gradient weight matrix as low rank.
content.type: text
content.value: ⬇️ 82.5% less optimizer state memory footprint without performance degradation by expressing the gradient weight matrix as low rank.
content.type: new_line
content.type: new_line
content.raw: 👩🏿‍💻 Install via 
content.type: text
content.value: 👩🏿‍💻 Install via 
content.code: pip install transformers>=4.39.0 galore-torch
content.raw: `pip install transformers>=4.39.0 galore-torch`
content.type: inline_code
content.raw: . #ProudlyGpuPoor
content.type: text
content.value: . #ProudlyGpuPoor
content.type: new_line
content.type: new_line
content.raw: The integration of GaLore into the training of large language models (LLMs) marks a significant advancement in the field of deep learning, particularly in terms of memory efficiency and the democratization of AI research. By allowing for the training of billion-parameter models on consumer-grade hardware, reducing memory footprint in optimizer states, and leveraging advanced projection matrix techniques, GaLore opens new horizons for researchers and practitioners with limited access to high-end computational resources.
content.type: text
content.value: The integration of GaLore into the training of large language models (LLMs) marks a significant advancement in the field of deep learning, particularly in terms of memory efficiency and the democratization of AI research. By allowing for the training of billion-parameter models on consumer-grade hardware, reducing memory footprint in optimizer states, and leveraging advanced projection matrix techniques, GaLore opens new horizons for researchers and practitioners with limited access to high-end computational resources.
content.type: new_line
content.type: new_line
content.raw: 🔬 Find out more about GaLore and investigate lots of juicy technical details: 
content.type: text
content.value: 🔬 Find out more about GaLore and investigate lots of juicy technical details: 
content.href: https://huggingface.co/blog/galore
content.raw: https://huggingface.co/blog/galore
content.type: link
content.type: new_line
content.type: new_line
content.raw: 🤗 Huge thanks to everyone involved ❤️:
content.type: text
content.value: 🤗 Huge thanks to everyone involved ❤️:
content.type: new_line
content.type: new_line
content.raw: • authors: 
content.type: text
content.value: • authors: 
content.raw: @jiaweizhao
content.type: mention
content.user: jiaweizhao
content.type: text
content.raw: @Kyriection
content.type: mention
content.user: Kyriection
content.type: text
content.raw: @beidic
content.type: mention
content.user: beidic
content.raw:  Zhangyang Wang  
content.type: text
content.value:  Zhangyang Wang  
content.raw: @animakumar
content.type: mention
content.user: animakumar
content.type: text
content.raw: @tydsh
content.type: mention
content.user: tydsh
content.type: text
content.type: new_line
content.raw: • community contributors: 
content.type: text
content.value: • community contributors: 
content.raw: @hiyouga
content.type: mention
content.user: hiyouga
content.type: text
content.raw: @mdouglas
content.type: mention
content.user: mdouglas
content.raw:  and others!
content.type: text
content.value:  and others!
content.type: new_line
content.raw: • 
content.type: text
content.value: • 
content.raw: @ybelkada
content.type: mention
content.user: ybelkada
content.raw:   for taking such swift action in composing and coordinating necessary PRs to get this live at ⚡ speed!
content.type: text
content.value:   for taking such swift action in composing and coordinating necessary PRs to get this live at ⚡ speed!
content.type: new_line
content.type: new_line
content.raw: 🏗️📈 Super rewarding to see how 
content.type: text
content.value: 🏗️📈 Super rewarding to see how 
content.raw: @timdettmers
content.type: mention
content.user: timdettmers
content.raw:  work with optimizers is being built upon to achieve even greater heights!
content.type: text
content.value:  work with optimizers is being built upon to achieve even greater heights!
content.type: new_line
content.type: new_line
content.raw: 🚧 Actually, there are ongoing works to integrate GaLore into bitsandbytes and optimize memory efficiency even further 💪. We'll keep you posted!
content.type: text
content.value: 🚧 Actually, there are ongoing works to integrate GaLore into bitsandbytes and optimize memory efficiency even further 💪. We'll keep you posted!
rawContent: 🔥 Level up your model training w/ GaLore + Transformers for SOTA results on consumer-grade hardware!
  
  ⬇️ 82.5% less optimizer state memory footprint without performance degradation by expressing the gradient weight matrix as low rank.
  
  👩🏿‍💻 Install via `pip install transformers>=4.39.0 galore-torch`. #ProudlyGpuPoor
  
  The integration of GaLore into the training of large language models (LLMs) marks a significant advancement in the field of deep learning, particularly in terms of memory efficiency and the democratization of AI research. By allowing for the training of billion-parameter models on consumer-grade hardware, reducing memory footprint in optimizer states, and leveraging advanced projection matrix techniques, GaLore opens new horizons for researchers and practitioners with limited access to high-end computational resources.
  
  🔬 Find out more about GaLore and investigate lots of juicy technical details: https://huggingface.co/blog/galore
  
  🤗 Huge thanks to everyone involved ❤️:
  
  • authors: @jiaweizhao @Kyriection @beidic Zhangyang Wang  @animakumar @tydsh 
  • community contributors: @hiyouga @mdouglas and others!
  • @ybelkada  for taking such swift action in composing and coordinating necessary PRs to get this live at ⚡ speed!
  
  🏗️📈 Super rewarding to see how @timdettmers work with optimizers is being built upon to achieve even greater heights!
  
  🚧 Actually, there are ongoing works to integrate GaLore into bitsandbytes and optimize memory efficiency even further 💪. We'll keep you posted!
reactions.reaction: 🔥
reactions.users: ybelkada lucabaggi t1u1 hiyouga osanseviero Priceva samusenps victor Zyn123 jiaweizhao Tom-Neverwinter chankhavu
reactions.reaction: 🤝
reactions.users: ybelkada mdouglas hiyouga Tom-Neverwinter
commentators._id: 6303cc3e1dd5d3c62483bd51
commentators.avatarUrl: /avatars/afbc48df2e8c47c35be48168113d83c0.svg
commentators.fullname: s
commentators.name: Tom-Neverwinter
commentators.type: user