title: Progressive Gradient Flow for Robust N:M Sparsity Training in
    Transformers
summary: N:M Structured sparsity has garnered significant interest as a result of
  relatively modest overhead and improved efficiency. Additionally, this form of
  sparsity holds considerable appeal for reducing the memory footprint owing to
  their modest representation overhead. There have been efforts to develop
  training recipes for N:M structured sparsity, they primarily focus on
  low-sparsity regions (sim50\%). Nonetheless, performance of models trained
  using these approaches tends to decline when confronted with high-sparsity
  regions (>80\%). In this work, we study the effectiveness of existing sparse
  training recipes at high-sparsity regions and argue that these methods
  fail to sustain the model quality on par with low-sparsity regions. We
  demonstrate that the significant factor contributing to this disparity is the
  presence of elevated levels of induced noise in the gradient magnitudes. To
  mitigate this undesirable effect, we employ decay mechanisms to progressively
  restrict the flow of gradients towards pruned elements. Our approach improves
  the model quality by up to 2% and 5% in vision and language models at
  high sparsity regime, respectively. We also evaluate the trade-off between
  model accuracy and training compute cost in terms of FLOPs. At iso-training
  FLOPs, our method yields better performance compared to conventional sparse
  training recipes, exhibiting an accuracy improvement of up to 2%. The source
  code is available at
  https://github.com/abhibambhaniya/progressive_gradient_flow_nm_sparsity.
paper_authors._id: 65c43f17b08720ad340e662b
paper_authors.name: Abhimanyu Rajeshkumar Bambhaniya
paper_authors._id: 65c43f17b08720ad340e662c
paper_authors.name: Amir Yazdanbakhsh
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-10T21:57:25.218Z
paper_authors._id: 6352dc90a023833e59d6349a
paper_authors.avatarUrl: /avatars/76e5e7b51515b58e207c7d2a4b360d30.svg
paper_authors.fullname: Amir Yazdanbakhsh
paper_authors.type: user
paper_authors.user: ayazdan
paper_authors._id: 65c43f17b08720ad340e662d
paper_authors.name: Suvinay Subramanian
paper_authors._id: 65c43f17b08720ad340e662e
paper_authors.name: Sheng-Chun Kao
paper_authors._id: 65c43f17b08720ad340e662f
paper_authors.name: Shivani Agrawal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T13:01:55.638Z
paper_authors._id: 653a06ff29f8a9115530c95c
paper_authors.avatarUrl: /avatars/709e00f0d4488933d4d9645879e2776c.svg
paper_authors.fullname: Shivani Agrawal
paper_authors.type: user
paper_authors.user: shiviagrawal05
paper_authors._id: 65c43f17b08720ad340e6630
paper_authors.name: Utku Evci
paper_authors._id: 65c43f17b08720ad340e6631
paper_authors.name: Tushar Krishna
paper_title: Progressive Gradient Flow for Robust N:M Sparsity Training in
    Transformers
paper_summary: N:M Structured sparsity has garnered significant interest as a result of
  relatively modest overhead and improved efficiency. Additionally, this form of
  sparsity holds considerable appeal for reducing the memory footprint owing to
  their modest representation overhead. There have been efforts to develop
  training recipes for N:M structured sparsity, they primarily focus on
  low-sparsity regions (sim50\%). Nonetheless, performance of models trained
  using these approaches tends to decline when confronted with high-sparsity
  regions (>80\%). In this work, we study the effectiveness of existing sparse
  training recipes at high-sparsity regions and argue that these methods
  fail to sustain the model quality on par with low-sparsity regions. We
  demonstrate that the significant factor contributing to this disparity is the
  presence of elevated levels of induced noise in the gradient magnitudes. To
  mitigate this undesirable effect, we employ decay mechanisms to progressively
  restrict the flow of gradients towards pruned elements. Our approach improves
  the model quality by up to 2% and 5% in vision and language models at
  high sparsity regime, respectively. We also evaluate the trade-off between
  model accuracy and training compute cost in terms of FLOPs. At iso-training
  FLOPs, our method yields better performance compared to conventional sparse
  training recipes, exhibiting an accuracy improvement of up to 2%. The source
  code is available at
  https://github.com/abhibambhaniya/progressive_gradient_flow_nm_sparsity.
paper_ai_summary: Progressive gradient flow decay mechanisms enhance the effectiveness of structured sparsity in high-sparsity regions, leading to improved performance with reduced computational cost.
paper_ai_keywords: structured sparsity memory footprint sparse training recipes high-sparsity regions gradient magnitudes decay mechanisms vision models language models FLOPs