title: LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
summary: We present LayerSkip, an end-to-end solution to speed-up inference of large
  language models (LLMs). First, during training we apply layer dropout, with low
  dropout rates for earlier layers and higher dropout rates for later layers, and
  an early exit loss where all transformer layers share the same exit. Second,
  during inference, we show that this training recipe increases the accuracy of
  early exit at earlier layers, without adding any auxiliary layers or modules to
  the model. Third, we present a novel self-speculative decoding solution where
  we exit at early layers and verify and correct with remaining layers of the
  model. Our proposed self-speculative decoding approach has less memory
  footprint than other speculative decoding approaches and benefits from shared
  compute and activations of the draft and verification stages. We run
  experiments on different Llama model sizes on different types of training:
  pretraining from scratch, continual pretraining, finetuning on specific data
  domain, and finetuning on specific task. We implement our inference solution
  and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x
  on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and
  checkpoints at https://github.com/facebookresearch/LayerSkip.
paper_authors._id: 662b1842ef7a4675bdfd9323
paper_authors.name: Mostafa Elhoushi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-26T19:58:42.088Z
paper_authors._id: 63c9725ebedad7e2bf160bdc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg
paper_authors.fullname: Mostafa Elhoushi
paper_authors.type: user
paper_authors.user: melhoushi
paper_authors._id: 662b1842ef7a4675bdfd9324
paper_authors.name: Akshat Shrivastava
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-26T02:58:11.458Z
paper_authors._id: 64b6d105f616b195a39f462a
paper_authors.avatarUrl: /avatars/f33805c3e42e476a135c3cf869cc423a.svg
paper_authors.fullname: Akshat
paper_authors.type: user
paper_authors.user: akshats07
paper_authors._id: 662b1842ef7a4675bdfd9325
paper_authors.name: Diana Liskovich
paper_authors._id: 662b1842ef7a4675bdfd9326
paper_authors.name: Basil Hosmer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T07:47:01.676Z
paper_authors._id: 631183a88894270062a2ea36
paper_authors.avatarUrl: /avatars/5da43a25768f5b677bbd02bd955210e7.svg
paper_authors.fullname: Basil Hosmer
paper_authors.type: user
paper_authors.user: bhosmer
paper_authors._id: 662b1842ef7a4675bdfd9327
paper_authors.name: Bram Wasti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T07:47:08.369Z
paper_authors._id: 631171f231257261d20f9f57
paper_authors.avatarUrl: /avatars/9d6273d9e7dc4e4b21752ba4b17ab1a7.svg
paper_authors.fullname: Bram Wasti
paper_authors.type: user
paper_authors.user: bwasti
paper_authors._id: 662b1842ef7a4675bdfd9328
paper_authors.name: Liangzhen Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T07:47:15.337Z
paper_authors._id: 64b705c73240387159397c71
paper_authors.avatarUrl: /avatars/da339f07eb58f58756b91eba9af557a1.svg
paper_authors.fullname: Liangzhen Lai
paper_authors.type: user
paper_authors.user: liangzhen-lai
paper_authors._id: 662b1842ef7a4675bdfd9329
paper_authors.name: Anas Mahmoud
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T07:47:22.751Z
paper_authors._id: 64a588d26c0223510c6047a9
paper_authors.avatarUrl: /avatars/6cc422be5cd2f0f78c0b0cc9c3743e02.svg
paper_authors.fullname: Anas Mahmoud
paper_authors.type: user
paper_authors.user: nasmahmoud
paper_authors._id: 662b1842ef7a4675bdfd932a
paper_authors.name: Bilge Acun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T07:47:30.164Z
paper_authors._id: 650e0307bfb7dd98bba68563
paper_authors.avatarUrl: /avatars/1b5287354dc32d85190f38b44ed081f3.svg
paper_authors.fullname: Bilge
paper_authors.type: user
paper_authors.user: bilgeacun
paper_authors._id: 662b1842ef7a4675bdfd932b
paper_authors.name: Saurabh Agarwal
paper_authors._id: 662b1842ef7a4675bdfd932c
paper_authors.name: Ahmed Roman
paper_authors._id: 662b1842ef7a4675bdfd932d
paper_authors.name: Ahmed A Aly
paper_authors._id: 662b1842ef7a4675bdfd932e
paper_authors.name: Beidi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T07:48:13.257Z
paper_authors._id: 64b732f832403871593e082c
paper_authors.avatarUrl: /avatars/dd21932b0c167131ee7545a622c46c3c.svg
paper_authors.fullname: Beidi Chen
paper_authors.type: user
paper_authors.user: beidic
paper_authors._id: 662b1842ef7a4675bdfd932f
paper_authors.name: Carole-Jean Wu
paper_title: LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
paper_summary: We present LayerSkip, an end-to-end solution to speed-up inference of large
  language models (LLMs). First, during training we apply layer dropout, with low
  dropout rates for earlier layers and higher dropout rates for later layers, and
  an early exit loss where all transformer layers share the same exit. Second,
  during inference, we show that this training recipe increases the accuracy of
  early exit at earlier layers, without adding any auxiliary layers or modules to
  the model. Third, we present a novel self-speculative decoding solution where
  we exit at early layers and verify and correct with remaining layers of the
  model. Our proposed self-speculative decoding approach has less memory
  footprint than other speculative decoding approaches and benefits from shared
  compute and activations of the draft and verification stages. We run
  experiments on different Llama model sizes on different types of training:
  pretraining from scratch, continual pretraining, finetuning on specific data
  domain, and finetuning on specific task. We implement our inference solution
  and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x
  on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and
  checkpoints at https://github.com/facebookresearch/LayerSkip.
paper_ai_summary: LayerSkip, an end-to-end method, accelerates large language model inference through layer dropout and self-speculative decoding without auxiliary layers.
paper_ai_keywords: layer dropout early exit loss transformer layers early exit self-speculative decoding memory footprint shared compute pretraining continual pretraining finetuning summarization coding semantic parsing