MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
Large Language Models (LLMs) are distinguished by their massive parameter
counts, which typically result in significant redundancy. This work introduces
MaskLLM, a learnable pruning method that establishes Semi-structured (or
``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during
inference. Instead of developing a new importance criterion, MaskLLM explicitly
models N:M patterns as a learnable distribution through Gumbel Softmax
sampling. This approach facilitates end-to-end training on large-scale datasets
and offers two notable advantages: 1) High-quality Masks - our method
effectively scales to large datasets and learns accurate masks; 2)
Transferability - the probabilistic modeling of mask distribution enables the
transfer learning of sparsity across domains or tasks. We assessed MaskLLM
using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3,
with sizes ranging from 843M to 15B parameters, and our empirical results show
substantial improvements over state-of-the-art methods. For instance, leading
approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to
the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL
solely by learning the masks with frozen weights. Furthermore, MaskLLM's
learnable nature allows customized masks for lossless application of 2:4
sparsity to downstream tasks or domains. Code is available at
https://github.com/NVlabs/MaskLLM.
66f61ce4b35008620e525842
Gongfan Fang
claimed_verified
2024-09-27T06:58:11.629Z
646a1939c37ca1e12308fe81
/avatars/752e9d86018e7d33ad8bcd741203fd86.svg
Gongfan Fang
user
Vinnnf
66f61ce4b35008620e525843
Hongxu Yin
admin_assigned
2024-09-27T07:25:47.927Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
66f61ce4b35008620e525844
Saurav Muralidharan
admin_assigned
2024-09-27T07:25:53.759Z
64caca9b4ca74090b97a8795
/avatars/310f7703ff3c07d283c02aca56fe86df.svg
Saurav Muralidharan
user
srvm
66f61ce4b35008620e525845
Greg Heinrich
admin_assigned
2024-09-27T07:25:59.492Z
64c23e06203c836866ed0bdd
https://cdn-avatars.huggingface.co/v1/production/uploads/64c23e06203c836866ed0bdd/rPUrP8PqL1dhWNO6L3XwI.png
Greg Heinrich
user
gheinrich
66f61ce4b35008620e525846
Jeff Pool
66f61ce4b35008620e525847
Jan Kautz
66f61ce4b35008620e525848
Pavlo Molchanov
claimed_verified
2024-09-30T08:07:13.750Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
66f61ce4b35008620e525849
Xinchao Wang
admin_assigned
2024-09-27T07:26:31.331Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
Large Language Models (LLMs) are distinguished by their massive parameter
counts, which typically result in significant redundancy. This work introduces
MaskLLM, a learnable pruning method that establishes Semi-structured (or
``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during
inference. Instead of developing a new importance criterion, MaskLLM explicitly
models N:M patterns as a learnable distribution through Gumbel Softmax
sampling. This approach facilitates end-to-end training on large-scale datasets
and offers two notable advantages: 1) High-quality Masks - our method
effectively scales to large datasets and learns accurate masks; 2)
Transferability - the probabilistic modeling of mask distribution enables the
transfer learning of sparsity across domains or tasks. We assessed MaskLLM
using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3,
with sizes ranging from 843M to 15B parameters, and our empirical results show
substantial improvements over state-of-the-art methods. For instance, leading
approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to
the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL
solely by learning the masks with frozen weights. Furthermore, MaskLLM's
learnable nature allows customized masks for lossless application of 2:4
sparsity to downstream tasks or domains. Code is available at
https://github.com/NVlabs/MaskLLM.
MaskLLM introduces a learnable pruning method using Gumbel Softmax sampling to achieve N:M sparsity in LLMs, improving inference efficiency and transferability across tasks.
MaskLLM Semi-structured Sparsity N:M Sparsity Gumbel Softmax sampling perplexity PPL transfer learning downstream tasks domain customization LLaMA-2 Nemotron-4 GPT-3