title: SkipDecode: Autoregressive Skip Decoding with Batching and Caching for
    Efficient LLM Inference
summary: Autoregressive large language models (LLMs) have made remarkable progress in
  various natural language generation tasks. However, they incur high computation
  cost and latency resulting from the autoregressive token-by-token generation.
  To address this issue, several approaches have been proposed to reduce
  computational cost using early-exit strategies. These strategies enable faster
  text generation using reduced computation without applying the full computation
  graph to each token. While existing token-level early exit methods show
  promising results for online inference, they cannot be readily applied for
  batch inferencing and Key-Value caching. This is because they have to wait
  until the last token in a batch exits before they can stop computing. This
  severely limits the practical application of such techniques. In this paper, we
  propose a simple and effective token-level early exit method, SkipDecode,
  designed to work seamlessly with batch inferencing and KV caching. It overcomes
  prior constraints by setting up a singular exit point for every token in a
  batch at each sequence position. It also guarantees a monotonic decrease in
  exit points, thereby eliminating the need to recompute KV Caches for preceding
  tokens. Rather than terminating computation prematurely as in prior works, our
  approach bypasses lower to middle layers, devoting most of the computational
  resources to upper layers, allowing later tokens to benefit from the compute
  expenditure by earlier tokens. Our experimental results show that SkipDecode
  can obtain 2x to 5x inference speedups with negligible regression across a
  variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7
  billion parameters, all the while being directly compatible with batching and
  KV caching optimization techniques.
paper_authors._id: 64a76ab00c36529d75fef32c
paper_authors.name: Luciano Del Corro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T08:03:57.586Z
paper_authors._id: 63b81aace60862785afd8ca2
paper_authors.avatarUrl: /avatars/f96ce78af6ad42514235bab811544789.svg
paper_authors.fullname: Luciano Del Corro
paper_authors.type: user
paper_authors.user: lucianodelcorro
paper_authors._id: 64a76ab00c36529d75fef32d
paper_authors.name: Allie Del Giorno
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T08:03:36.527Z
paper_authors._id: 63f5562471a5d395c721cd8e
paper_authors.avatarUrl: /avatars/ac296b6017e86ea04c73803fe2c44433.svg
paper_authors.fullname: Allie Del Giorno
paper_authors.type: user
paper_authors.user: microallie
paper_authors._id: 64a76ab00c36529d75fef32e
paper_authors.name: Sahaj Agarwal
paper_authors._id: 64a76ab00c36529d75fef32f
paper_authors.name: Bin Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T08:07:32.442Z
paper_authors._id: 64679498374fe5728d4044d5
paper_authors.avatarUrl: /avatars/78516e1c9532900cce19f81483855db8.svg
paper_authors.fullname: Bin Yu
paper_authors.type: user
paper_authors.user: binyu000
paper_authors._id: 64a76ab00c36529d75fef330
paper_authors.name: Ahmed Awadallah
paper_authors._id: 64a76ab00c36529d75fef331
paper_authors.name: Subhabrata Mukherjee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T08:55:28.501Z
paper_authors._id: 5fd2dbbd9e5a0a03458e1fc8
paper_authors.avatarUrl: /avatars/d0868aef6c180db8610184396443731a.svg
paper_authors.fullname: Subhabrata Mukherjee
paper_authors.type: user
paper_authors.user: subho
paper_title: SkipDecode: Autoregressive Skip Decoding with Batching and Caching for
    Efficient LLM Inference
paper_summary: Autoregressive large language models (LLMs) have made remarkable progress in
  various natural language generation tasks. However, they incur high computation
  cost and latency resulting from the autoregressive token-by-token generation.
  To address this issue, several approaches have been proposed to reduce
  computational cost using early-exit strategies. These strategies enable faster
  text generation using reduced computation without applying the full computation
  graph to each token. While existing token-level early exit methods show
  promising results for online inference, they cannot be readily applied for
  batch inferencing and Key-Value caching. This is because they have to wait
  until the last token in a batch exits before they can stop computing. This
  severely limits the practical application of such techniques. In this paper, we
  propose a simple and effective token-level early exit method, SkipDecode,
  designed to work seamlessly with batch inferencing and KV caching. It overcomes
  prior constraints by setting up a singular exit point for every token in a
  batch at each sequence position. It also guarantees a monotonic decrease in
  exit points, thereby eliminating the need to recompute KV Caches for preceding
  tokens. Rather than terminating computation prematurely as in prior works, our
  approach bypasses lower to middle layers, devoting most of the computational
  resources to upper layers, allowing later tokens to benefit from the compute
  expenditure by earlier tokens. Our experimental results show that SkipDecode
  can obtain 2x to 5x inference speedups with negligible regression across a
  variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7
  billion parameters, all the while being directly compatible with batching and
  KV caching optimization techniques.
paper_ai_keywords: autoregressive large language models (LLMs) natural language generation tasks token-by-token generation early-exit strategies batch inferencing Key-Value caching SkipDecode monotonic decrease in exit points computational resources upper layers inference speedups