Shortened LLaMA: A Simple Depth Pruning for Large Language Models
Structured pruning of modern large language models (LLMs) has emerged as a
way of decreasing their high computational needs. Width pruning reduces the
size of projection weight matrices (e.g., by removing attention heads) while
maintaining the number of layers. Depth pruning, in contrast, removes entire
layers or blocks, while keeping the size of the remaining weights unchanged.
Most current research focuses on either width-only or a blend of width and
depth pruning, with little comparative analysis between the two units (width
vs. depth) concerning their impact on LLM inference efficiency. In this work,
we show that a simple depth pruning approach can compete with recent width
pruning methods in terms of zero-shot task performance. Our pruning method
boosts inference speeds, especially under memory-constrained conditions that
require limited batch sizes for running LLMs, where width pruning is
ineffective. We hope this work can help deploy LLMs on local and edge devices.
65c1a04f4ea98c77cee651f5
Bo-Kyeong Kim
claimed_verified
2024-02-06T14:58:55.976Z
64374f10cd93f4c9a34c1239
https://cdn-avatars.huggingface.co/v1/production/uploads/64374f10cd93f4c9a34c1239/tdkdcgl18rKXLvyTEhfnG.jpeg
Bo-Kyeong Kim
user
bokyeong1015
65c1a04f4ea98c77cee651f6
Geonmin Kim
claimed_verified
2024-02-07T11:19:33.978Z
6437648b4aacf7bf7871c8d9
/avatars/9c2102c4594259287fa78137ab258451.svg
GeonminKim
user
geonmin-kim
65c1a04f4ea98c77cee651f7
Tae-Ho Kim
claimed_verified
2024-02-06T14:58:53.958Z
64cccbe8c7f30fbf7bbf5a6d
/avatars/1c1f7c2da22c2afb3b352d446e80e5ef.svg
Tae-Ho Kim
user
ktho22
65c1a04f4ea98c77cee651f8
Thibault Castells
claimed_verified
2024-04-19T09:09:17.127Z
64709c9390482b0e0f670e80
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8hZF_LZQ6ejoS_u_knXFV.png
Thibault Castells
user
thibcastells
65c1a04f4ea98c77cee651f9
Shinkook Choi
65c1a04f4ea98c77cee651fa
Junho Shin
claimed_verified
2024-02-06T14:58:59.649Z
65a727ef7bfcb01564d6b06b
/avatars/0b1a5ab28c5068568b328d0b713fbace.svg
Junho Shin
user
illian01
65c1a04f4ea98c77cee651fb
Hyoung-Kyu Song
claimed_verified
2024-02-06T14:59:01.856Z
61f44bab7eba274ea80b74ce
https://cdn-avatars.huggingface.co/v1/production/uploads/61f44bab7eba274ea80b74ce/BRbKX1jephdZ7D44FATl4.jpeg
Hyoung-Kyu Song
user
deepkyu
Shortened LLaMA: A Simple Depth Pruning for Large Language Models
Structured pruning of modern large language models (LLMs) has emerged as a
way of decreasing their high computational needs. Width pruning reduces the
size of projection weight matrices (e.g., by removing attention heads) while
maintaining the number of layers. Depth pruning, in contrast, removes entire
layers or blocks, while keeping the size of the remaining weights unchanged.
Most current research focuses on either width-only or a blend of width and
depth pruning, with little comparative analysis between the two units (width
vs. depth) concerning their impact on LLM inference efficiency. In this work,
we show that a simple depth pruning approach can compete with recent width
pruning methods in terms of zero-shot task performance. Our pruning method
boosts inference speeds, especially under memory-constrained conditions that
require limited batch sizes for running LLMs, where width pruning is
ineffective. We hope this work can help deploy LLMs on local and edge devices.
A depth pruning method for large language models matches width pruning in zero-shot task performance and improves inference speed under memory constraints.
structured pruning large language models width pruning depth pruning projection weight matrices attention heads layers blocks zero-shot task performance inference efficiency memory-constrained conditions batch sizes local devices edge devices