title: Shortened LLaMA: A Simple Depth Pruning for Large Language Models
summary: Structured pruning of modern large language models (LLMs) has emerged as a
  way of decreasing their high computational needs. Width pruning reduces the
  size of projection weight matrices (e.g., by removing attention heads) while
  maintaining the number of layers. Depth pruning, in contrast, removes entire
  layers or blocks, while keeping the size of the remaining weights unchanged.
  Most current research focuses on either width-only or a blend of width and
  depth pruning, with little comparative analysis between the two units (width
  vs. depth) concerning their impact on LLM inference efficiency. In this work,
  we show that a simple depth pruning approach can compete with recent width
  pruning methods in terms of zero-shot task performance. Our pruning method
  boosts inference speeds, especially under memory-constrained conditions that
  require limited batch sizes for running LLMs, where width pruning is
  ineffective. We hope this work can help deploy LLMs on local and edge devices.
paper_authors._id: 65c1a04f4ea98c77cee651f5
paper_authors.name: Bo-Kyeong Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-06T14:58:55.976Z
paper_authors._id: 64374f10cd93f4c9a34c1239
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64374f10cd93f4c9a34c1239/tdkdcgl18rKXLvyTEhfnG.jpeg
paper_authors.fullname: Bo-Kyeong Kim
paper_authors.type: user
paper_authors.user: bokyeong1015
paper_authors._id: 65c1a04f4ea98c77cee651f6
paper_authors.name: Geonmin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T11:19:33.978Z
paper_authors._id: 6437648b4aacf7bf7871c8d9
paper_authors.avatarUrl: /avatars/9c2102c4594259287fa78137ab258451.svg
paper_authors.fullname: GeonminKim
paper_authors.type: user
paper_authors.user: geonmin-kim
paper_authors._id: 65c1a04f4ea98c77cee651f7
paper_authors.name: Tae-Ho Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-06T14:58:53.958Z
paper_authors._id: 64cccbe8c7f30fbf7bbf5a6d
paper_authors.avatarUrl: /avatars/1c1f7c2da22c2afb3b352d446e80e5ef.svg
paper_authors.fullname: Tae-Ho Kim
paper_authors.type: user
paper_authors.user: ktho22
paper_authors._id: 65c1a04f4ea98c77cee651f8
paper_authors.name: Thibault Castells
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T09:09:17.127Z
paper_authors._id: 64709c9390482b0e0f670e80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8hZF_LZQ6ejoS_u_knXFV.png
paper_authors.fullname: Thibault Castells
paper_authors.type: user
paper_authors.user: thibcastells
paper_authors._id: 65c1a04f4ea98c77cee651f9
paper_authors.name: Shinkook Choi
paper_authors._id: 65c1a04f4ea98c77cee651fa
paper_authors.name: Junho Shin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-06T14:58:59.649Z
paper_authors._id: 65a727ef7bfcb01564d6b06b
paper_authors.avatarUrl: /avatars/0b1a5ab28c5068568b328d0b713fbace.svg
paper_authors.fullname: Junho Shin
paper_authors.type: user
paper_authors.user: illian01
paper_authors._id: 65c1a04f4ea98c77cee651fb
paper_authors.name: Hyoung-Kyu Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-06T14:59:01.856Z
paper_authors._id: 61f44bab7eba274ea80b74ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61f44bab7eba274ea80b74ce/BRbKX1jephdZ7D44FATl4.jpeg
paper_authors.fullname: Hyoung-Kyu Song
paper_authors.type: user
paper_authors.user: deepkyu
paper_title: Shortened LLaMA: A Simple Depth Pruning for Large Language Models
paper_summary: Structured pruning of modern large language models (LLMs) has emerged as a
  way of decreasing their high computational needs. Width pruning reduces the
  size of projection weight matrices (e.g., by removing attention heads) while
  maintaining the number of layers. Depth pruning, in contrast, removes entire
  layers or blocks, while keeping the size of the remaining weights unchanged.
  Most current research focuses on either width-only or a blend of width and
  depth pruning, with little comparative analysis between the two units (width
  vs. depth) concerning their impact on LLM inference efficiency. In this work,
  we show that a simple depth pruning approach can compete with recent width
  pruning methods in terms of zero-shot task performance. Our pruning method
  boosts inference speeds, especially under memory-constrained conditions that
  require limited batch sizes for running LLMs, where width pruning is
  ineffective. We hope this work can help deploy LLMs on local and edge devices.
paper_ai_keywords: structured pruning large language models (LLMs) width pruning projection weight matrices attention heads depth pruning layer removal representation engineering (RepE) zero-shot task performance inference efficiency memory-constrained conditions batch sizes local devices edge devices