title: EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit
    Large Language Models
summary: This work introduces EE-Tuning, a lightweight and economical solution to
  training/tuning early-exit large language models (LLMs). In contrast to the
  common approach of full-parameter pre-training, EE-Tuning augments any
  pre-trained (and possibly fine-tuned) standard LLM with additional early-exit
  layers that are tuned in a parameter-efficient manner, which requires
  significantly less computational resources and training data. Our
  implementation of EE-Tuning achieves outstanding training efficiency via
  extensive performance optimizations, as well as scalability due to its full
  compatibility with 3D parallelism. Results of systematic experiments validate
  the efficacy of EE-Tuning, confirming that effective early-exit LLM inference
  can be achieved with a limited training budget. In hope of making early-exit
  LLMs accessible to the community, we release the source code of our
  implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.
paper_authors._id: 65bc66854e82d4524364c761
paper_authors.name: Xuchen Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T13:28:36.128Z
paper_authors._id: 64101b4ca92fedb0e8559b91
paper_authors.avatarUrl: /avatars/c7453a5641a94d04a75de9b7d4272bc2.svg
paper_authors.fullname: Xuchen Pan
paper_authors.type: user
paper_authors.user: panxuchen
paper_authors._id: 65bc66854e82d4524364c762
paper_authors.name: Yanxi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T13:29:05.717Z
paper_authors._id: 6576f9f4654561a1b345610b
paper_authors.avatarUrl: /avatars/f801f551640caa70368fcc26a0f51d27.svg
paper_authors.fullname: Yanxi Chen
paper_authors.type: user
paper_authors.user: yanxi-chen
paper_authors._id: 65bc66854e82d4524364c763
paper_authors.name: Yaliang Li
paper_authors._id: 65bc66854e82d4524364c764
paper_authors.name: Bolin Ding
paper_authors._id: 65bc66854e82d4524364c765
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T13:37:29.139Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit
    Large Language Models
paper_summary: This work introduces EE-Tuning, a lightweight and economical solution to
  training/tuning early-exit large language models (LLMs). In contrast to the
  common approach of full-parameter pre-training, EE-Tuning augments any
  pre-trained (and possibly fine-tuned) standard LLM with additional early-exit
  layers that are tuned in a parameter-efficient manner, which requires
  significantly less computational resources and training data. Our
  implementation of EE-Tuning achieves outstanding training efficiency via
  extensive performance optimizations, as well as scalability due to its full
  compatibility with 3D parallelism. Results of systematic experiments validate
  the efficacy of EE-Tuning, confirming that effective early-exit LLM inference
  can be achieved with a limited training budget. In hope of making early-exit
  LLMs accessible to the community, we release the source code of our
  implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.
paper_ai_keywords: EE-Tuning early-exit large language models (LLMs) parameter-efficient manner early-exit layers computational resources training data performance optimizations 3D parallelism early-exit LLM inference training budget