Progressive Gradient Flow for Robust N:M Sparsity Training in
  Transformers
N:M Structured sparsity has garnered significant interest as a result of
relatively modest overhead and improved efficiency. Additionally, this form of
sparsity holds considerable appeal for reducing the memory footprint owing to
their modest representation overhead. There have been efforts to develop
training recipes for N:M structured sparsity, they primarily focus on
low-sparsity regions (sim50\%). Nonetheless, performance of models trained
using these approaches tends to decline when confronted with high-sparsity
regions (>80\%). In this work, we study the effectiveness of existing sparse
training recipes at high-sparsity regions and argue that these methods
fail to sustain the model quality on par with low-sparsity regions. We
demonstrate that the significant factor contributing to this disparity is the
presence of elevated levels of induced noise in the gradient magnitudes. To
mitigate this undesirable effect, we employ decay mechanisms to progressively
restrict the flow of gradients towards pruned elements. Our approach improves
the model quality by up to 2% and 5% in vision and language models at
high sparsity regime, respectively. We also evaluate the trade-off between
model accuracy and training compute cost in terms of FLOPs. At iso-training
FLOPs, our method yields better performance compared to conventional sparse
training recipes, exhibiting an accuracy improvement of up to 2%. The source
code is available at
https://github.com/abhibambhaniya/progressive_gradient_flow_nm_sparsity.
65c43f17b08720ad340e662b
Abhimanyu Rajeshkumar Bambhaniya
65c43f17b08720ad340e662c
Amir Yazdanbakhsh
extracted_confirmed
2024-06-10T21:57:25.218Z
6352dc90a023833e59d6349a
/avatars/76e5e7b51515b58e207c7d2a4b360d30.svg
Amir Yazdanbakhsh
user
ayazdan
65c43f17b08720ad340e662d
Suvinay Subramanian
65c43f17b08720ad340e662e
Sheng-Chun Kao
65c43f17b08720ad340e662f
Shivani Agrawal
admin_assigned
2024-02-08T13:01:55.638Z
653a06ff29f8a9115530c95c
/avatars/709e00f0d4488933d4d9645879e2776c.svg
Shivani Agrawal
user
shiviagrawal05
65c43f17b08720ad340e6630
Utku Evci
65c43f17b08720ad340e6631
Tushar Krishna
Progressive Gradient Flow for Robust N:M Sparsity Training in
  Transformers
N:M Structured sparsity has garnered significant interest as a result of
relatively modest overhead and improved efficiency. Additionally, this form of
sparsity holds considerable appeal for reducing the memory footprint owing to
their modest representation overhead. There have been efforts to develop
training recipes for N:M structured sparsity, they primarily focus on
low-sparsity regions (sim50\%). Nonetheless, performance of models trained
using these approaches tends to decline when confronted with high-sparsity
regions (>80\%). In this work, we study the effectiveness of existing sparse
training recipes at high-sparsity regions and argue that these methods
fail to sustain the model quality on par with low-sparsity regions. We
demonstrate that the significant factor contributing to this disparity is the
presence of elevated levels of induced noise in the gradient magnitudes. To
mitigate this undesirable effect, we employ decay mechanisms to progressively
restrict the flow of gradients towards pruned elements. Our approach improves
the model quality by up to 2% and 5% in vision and language models at
high sparsity regime, respectively. We also evaluate the trade-off between
model accuracy and training compute cost in terms of FLOPs. At iso-training
FLOPs, our method yields better performance compared to conventional sparse
training recipes, exhibiting an accuracy improvement of up to 2%. The source
code is available at
https://github.com/abhibambhaniya/progressive_gradient_flow_nm_sparsity.
Progressive gradient flow decay mechanisms enhance the effectiveness of structured sparsity in high-sparsity regions, leading to improved performance with reduced computational cost.
structured sparsity memory footprint sparse training recipes high-sparsity regions gradient magnitudes decay mechanisms vision models language models FLOPs