title: The Unreasonable Ineffectiveness of the Deeper Layers
summary: We empirically study a simple layer-pruning strategy for popular families of
  open-weight pretrained LLMs, finding minimal degradation of performance on
  different question-answering benchmarks until after a large fraction (up to
  half) of the layers are removed. To prune these models, we identify the optimal
  block of layers to prune by considering similarity across layers; then, to
  "heal" the damage, we perform a small amount of finetuning. In particular, we
  use parameter-efficient finetuning (PEFT) methods, specifically quantization
  and Low Rank Adapters (QLoRA), such that each of our experiments can be
  performed on a single A100 GPU. From a practical perspective, these results
  suggest that layer pruning methods can complement other PEFT strategies to
  further reduce computational resources of finetuning on the one hand, and can
  improve the memory and latency of inference on the other hand. From a
  scientific perspective, the robustness of these LLMs to the deletion of layers
  implies either that current pretraining methods are not properly leveraging the
  parameters in the deeper layers of the network or that the shallow layers play
  a critical role in storing knowledge.
paper_authors._id: 6603925f759b2e212d08b6bb
paper_authors.name: Andrey Gromov
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-27T03:28:32.198Z
paper_authors._id: 64d198292f92537fbc41fe86
paper_authors.avatarUrl: /avatars/1705d81d0b16d992833069150796b1d6.svg
paper_authors.fullname: Andrey Gromov
paper_authors.type: user
paper_authors.user: gromovand
paper_authors._id: 6603925f759b2e212d08b6bc
paper_authors.name: Kushal Tirumala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:01:03.657Z
paper_authors._id: 6429dac775bcc24c5e579f5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GI73-WJ5WjdB-KUUNGkdv.jpeg
paper_authors.fullname: Kushal Tirumala
paper_authors.type: user
paper_authors.user: kushaltirumala
paper_authors._id: 6603925f759b2e212d08b6bd
paper_authors.name: Hassan Shapourian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:01:09.596Z
paper_authors._id: 64d3dabaf63b01b7f66b7f98
paper_authors.avatarUrl: /avatars/ba34c4fb3b99de36522d0074725750c1.svg
paper_authors.fullname: Hassan Shapourian
paper_authors.type: user
paper_authors.user: hassansh
paper_authors._id: 6603925f759b2e212d08b6be
paper_authors.name: Paolo Glorioso
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:01:15.478Z
paper_authors._id: 64e40335c0edca443ef8af3e
paper_authors.avatarUrl: /avatars/c42a72a448dcb9e8bb3a0b41d1b6397a.svg
paper_authors.fullname: Paolo Glorioso
paper_authors.type: user
paper_authors.user: pglo
paper_authors._id: 6603925f759b2e212d08b6bf
paper_authors.name: Daniel A. Roberts
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-27T03:33:10.995Z
paper_authors._id: 64ee5701f5451e99166f50ad
paper_authors.avatarUrl: /avatars/aaa21574bb8aacdc58dc412f5292a260.svg
paper_authors.fullname: Dan Roberts
paper_authors.type: user
paper_authors.user: danintheory
paper_title: The Unreasonable Ineffectiveness of the Deeper Layers
paper_summary: We empirically study a simple layer-pruning strategy for popular families of
  open-weight pretrained LLMs, finding minimal degradation of performance on
  different question-answering benchmarks until after a large fraction (up to
  half) of the layers are removed. To prune these models, we identify the optimal
  block of layers to prune by considering similarity across layers; then, to
  "heal" the damage, we perform a small amount of finetuning. In particular, we
  use parameter-efficient finetuning (PEFT) methods, specifically quantization
  and Low Rank Adapters (QLoRA), such that each of our experiments can be
  performed on a single A100 GPU. From a practical perspective, these results
  suggest that layer pruning methods can complement other PEFT strategies to
  further reduce computational resources of finetuning on the one hand, and can
  improve the memory and latency of inference on the other hand. From a
  scientific perspective, the robustness of these LLMs to the deletion of layers
  implies either that current pretraining methods are not properly leveraging the
  parameters in the deeper layers of the network or that the shallow layers play
  a critical role in storing knowledge.
paper_ai_keywords: layer-pruning strategy open-weight pretrained LLMs question-answering benchmarks parameter-efficient finetuning (PEFT) quantization Low Rank Adapters (QLoRA) A100 GPU