title: EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language
    Models with 3D Parallelism
summary: We present EE-LLM, a framework for large-scale training and inference of
  early-exit large language models (LLMs). While recent works have shown
  preliminary evidence for the efficacy of early exiting in accelerating LLM
  inference, EE-LLM makes a foundational step towards scaling up early-exit LLMs
  by supporting their training and inference with massive 3D parallelism. Built
  upon Megatron-LM, EE-LLM implements a variety of algorithmic innovations and
  performance optimizations tailored to early exiting, including a lightweight
  method that facilitates backpropagation for the early-exit training objective
  with pipeline parallelism, techniques of leveraging idle resources in the
  original pipeline schedule for computation related to early-exit layers, and
  two approaches of early-exit inference that are compatible with KV caching for
  autoregressive generation. Our analytical and empirical study shows that EE-LLM
  achieves great training efficiency with negligible computational overhead
  compared to standard LLM training, as well as outstanding inference speedup
  without compromising output quality. To facilitate further research and
  adoption, we release EE-LLM at https://github.com/pan-x-c/EE-LLM.
paper_authors._id: 6576763db238c76bba229aab
paper_authors.name: Yanxi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-11T12:37:11.761Z
paper_authors._id: 6576f9f4654561a1b345610b
paper_authors.avatarUrl: /avatars/f801f551640caa70368fcc26a0f51d27.svg
paper_authors.fullname: Yanxi Chen
paper_authors.type: user
paper_authors.user: yanxi-chen
paper_authors._id: 6576763db238c76bba229aac
paper_authors.name: Xuchen Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:45:56.504Z
paper_authors._id: 64101b4ca92fedb0e8559b91
paper_authors.avatarUrl: /avatars/c7453a5641a94d04a75de9b7d4272bc2.svg
paper_authors.fullname: Xuchen Pan
paper_authors.type: user
paper_authors.user: panxuchen
paper_authors._id: 6576763db238c76bba229aad
paper_authors.name: Yaliang Li
paper_authors._id: 6576763db238c76bba229aae
paper_authors.name: Bolin Ding
paper_authors._id: 6576763db238c76bba229aaf
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:46:23.863Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language
    Models with 3D Parallelism
paper_summary: We present EE-LLM, a framework for large-scale training and inference of
  early-exit large language models (LLMs). While recent works have shown
  preliminary evidence for the efficacy of early exiting in accelerating LLM
  inference, EE-LLM makes a foundational step towards scaling up early-exit LLMs
  by supporting their training and inference with massive 3D parallelism. Built
  upon Megatron-LM, EE-LLM implements a variety of algorithmic innovations and
  performance optimizations tailored to early exiting, including a lightweight
  method that facilitates backpropagation for the early-exit training objective
  with pipeline parallelism, techniques of leveraging idle resources in the
  original pipeline schedule for computation related to early-exit layers, and
  two approaches of early-exit inference that are compatible with KV caching for
  autoregressive generation. Our analytical and empirical study shows that EE-LLM
  achieves great training efficiency with negligible computational overhead
  compared to standard LLM training, as well as outstanding inference speedup
  without compromising output quality. To facilitate further research and
  adoption, we release EE-LLM at https://github.com/pan-x-c/EE-LLM.
paper_ai_keywords: early-exit large language models (LLMs) massive 3D parallelism Megatron-LM pipeline parallelism idle resources early-exit layers KV caching autoregressive generation training efficiency computational overhead inference speedup output quality