title: MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
summary: Large Language Models (LLMs) are distinguished by their massive parameter
  counts, which typically result in significant redundancy. This work introduces
  MaskLLM, a learnable pruning method that establishes Semi-structured (or
  ``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during
  inference. Instead of developing a new importance criterion, MaskLLM explicitly
  models N:M patterns as a learnable distribution through Gumbel Softmax
  sampling. This approach facilitates end-to-end training on large-scale datasets
  and offers two notable advantages: 1) High-quality Masks - our method
  effectively scales to large datasets and learns accurate masks; 2)
  Transferability - the probabilistic modeling of mask distribution enables the
  transfer learning of sparsity across domains or tasks. We assessed MaskLLM
  using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3,
  with sizes ranging from 843M to 15B parameters, and our empirical results show
  substantial improvements over state-of-the-art methods. For instance, leading
  approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to
  the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL
  solely by learning the masks with frozen weights. Furthermore, MaskLLM's
  learnable nature allows customized masks for lossless application of 2:4
  sparsity to downstream tasks or domains. Code is available at
  https://github.com/NVlabs/MaskLLM.
paper_authors._id: 66f61ce4b35008620e525842
paper_authors.name: Gongfan Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T06:58:11.629Z
paper_authors._id: 646a1939c37ca1e12308fe81
paper_authors.avatarUrl: /avatars/752e9d86018e7d33ad8bcd741203fd86.svg
paper_authors.fullname: Gongfan Fang
paper_authors.type: user
paper_authors.user: Vinnnf
paper_authors._id: 66f61ce4b35008620e525843
paper_authors.name: Hongxu Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T07:25:47.927Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_authors._id: 66f61ce4b35008620e525844
paper_authors.name: Saurav Muralidharan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T07:25:53.759Z
paper_authors._id: 64caca9b4ca74090b97a8795
paper_authors.avatarUrl: /avatars/310f7703ff3c07d283c02aca56fe86df.svg
paper_authors.fullname: Saurav Muralidharan
paper_authors.type: user
paper_authors.user: srvm
paper_authors._id: 66f61ce4b35008620e525845
paper_authors.name: Greg Heinrich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T07:25:59.492Z
paper_authors._id: 64c23e06203c836866ed0bdd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c23e06203c836866ed0bdd/rPUrP8PqL1dhWNO6L3XwI.png
paper_authors.fullname: Greg Heinrich
paper_authors.type: user
paper_authors.user: gheinrich
paper_authors._id: 66f61ce4b35008620e525846
paper_authors.name: Jeff Pool
paper_authors._id: 66f61ce4b35008620e525847
paper_authors.name: Jan Kautz
paper_authors._id: 66f61ce4b35008620e525848
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:13.750Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 66f61ce4b35008620e525849
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T07:26:31.331Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
paper_summary: Large Language Models (LLMs) are distinguished by their massive parameter
  counts, which typically result in significant redundancy. This work introduces
  MaskLLM, a learnable pruning method that establishes Semi-structured (or
  ``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during
  inference. Instead of developing a new importance criterion, MaskLLM explicitly
  models N:M patterns as a learnable distribution through Gumbel Softmax
  sampling. This approach facilitates end-to-end training on large-scale datasets
  and offers two notable advantages: 1) High-quality Masks - our method
  effectively scales to large datasets and learns accurate masks; 2)
  Transferability - the probabilistic modeling of mask distribution enables the
  transfer learning of sparsity across domains or tasks. We assessed MaskLLM
  using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3,
  with sizes ranging from 843M to 15B parameters, and our empirical results show
  substantial improvements over state-of-the-art methods. For instance, leading
  approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to
  the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL
  solely by learning the masks with frozen weights. Furthermore, MaskLLM's
  learnable nature allows customized masks for lossless application of 2:4
  sparsity to downstream tasks or domains. Code is available at
  https://github.com/NVlabs/MaskLLM.
paper_ai_summary: MaskLLM introduces a learnable pruning method using Gumbel Softmax sampling to achieve N:M sparsity in LLMs, improving inference efficiency and transferability across tasks.
paper_ai_keywords: MaskLLM Semi-structured Sparsity N:M Sparsity Gumbel Softmax sampling perplexity PPL transfer learning downstream tasks domain customization LLaMA-2 Nemotron-4 GPT-3