title: SliceGPT: Compress Large Language Models by Deleting Rows and Columns
summary: Large language models have become the cornerstone of natural language
  processing, but their use comes with substantial costs in terms of compute and
  memory resources. Sparsification provides a solution to alleviate these
  resource constraints, and recent works have shown that trained models can be
  sparsified post-hoc. Existing sparsification techniques face challenges as they
  need additional data structures and offer constrained speedup with current
  hardware. In this paper we present SliceGPT, a new post-training sparsification
  scheme which replaces each weight matrix with a smaller (dense) matrix,
  reducing the embedding dimension of the network. Through extensive
  experimentation, we show that SliceGPT can remove up to 25% of the model
  parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models
  while maintaining 99%, 99% and 90% zero-shot task performance of the dense
  model respectively. Our sliced models run on fewer GPUs and run faster without
  any additional code optimization: on 24GB consumer GPUs we reduce the total
  compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB
  A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance
  in transformer networks, which enables SliceGPT and we hope it will inspire and
  enable future avenues to reduce memory and computation demands for pre-trained
  models. Code is available at:
  https://github.com/microsoft/TransformerCompression
paper_authors._id: 65b70e8c4839ccc503a60510
paper_authors.name: Saleh Ashkboos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T09:32:52.773Z
paper_authors._id: 610d7ec84539844d377af78a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/610d7ec84539844d377af78a/-9H0bIv5pjI9hJamg9P0x.jpeg
paper_authors.fullname: Ashkboos
paper_authors.type: user
paper_authors.user: Saleh
paper_authors._id: 65b70e8c4839ccc503a60511
paper_authors.name: Maximilian L. Croci
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T11:13:59.051Z
paper_authors._id: 63eb657213a3eb9b0dc6d628
paper_authors.avatarUrl: /avatars/c4a9f974dd5d2ea67c318d8496e6248d.svg
paper_authors.fullname: Max Croci
paper_authors.type: user
paper_authors.user: nailimixaM
paper_authors._id: 65b70e8c4839ccc503a60512
paper_authors.name: Marcelo Gennari do Nascimento
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-29T09:56:30.666Z
paper_authors._id: 6329e25259950c1d279b4333
paper_authors.avatarUrl: /avatars/d24a0cf3e09fcfd77bc2a28dd9e2a200.svg
paper_authors.fullname: Marcelo Gennari do Nascimento
paper_authors.type: user
paper_authors.user: MarceloGennari
paper_authors._id: 65b70e8c4839ccc503a60513
paper_authors.name: Torsten Hoefler
paper_authors._id: 65b70e8c4839ccc503a60514
paper_authors.name: James Hensman
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-04T06:18:21.485Z
paper_authors._id: 65af77603db2280ece8a6644
paper_authors.avatarUrl: /avatars/37fac7c73e0190f933201398c74bcc50.svg
paper_authors.fullname: James Hensman
paper_authors.type: user
paper_authors.user: Jameshensman
paper_title: SliceGPT: Compress Large Language Models by Deleting Rows and Columns
paper_summary: Large language models have become the cornerstone of natural language
  processing, but their use comes with substantial costs in terms of compute and
  memory resources. Sparsification provides a solution to alleviate these
  resource constraints, and recent works have shown that trained models can be
  sparsified post-hoc. Existing sparsification techniques face challenges as they
  need additional data structures and offer constrained speedup with current
  hardware. In this paper we present SliceGPT, a new post-training sparsification
  scheme which replaces each weight matrix with a smaller (dense) matrix,
  reducing the embedding dimension of the network. Through extensive
  experimentation, we show that SliceGPT can remove up to 25% of the model
  parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models
  while maintaining 99%, 99% and 90% zero-shot task performance of the dense
  model respectively. Our sliced models run on fewer GPUs and run faster without
  any additional code optimization: on 24GB consumer GPUs we reduce the total
  compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB
  A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance
  in transformer networks, which enables SliceGPT and we hope it will inspire and
  enable future avenues to reduce memory and computation demands for pre-trained
  models. Code is available at:
  https://github.com/microsoft/TransformerCompression
paper_ai_summary: SliceGPT, a novel post-training sparsification method, reduces model parameters by up to 25% without significant performance decrease, optimizing inference speed and GPU usage on large language models.
paper_ai_keywords: sparsification post-training weight matrix embedding dimension transformer networks computational invariance