EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit
  Large Language Models
This work introduces EE-Tuning, a lightweight and economical solution to
training/tuning early-exit large language models (LLMs). In contrast to the
common approach of full-parameter pre-training, EE-Tuning augments any
pre-trained (and possibly fine-tuned) standard LLM with additional early-exit
layers that are tuned in a parameter-efficient manner, which requires
significantly less computational resources and training data. Our
implementation of EE-Tuning achieves outstanding training efficiency via
extensive performance optimizations, as well as scalability due to its full
compatibility with 3D parallelism. Results of systematic experiments validate
the efficacy of EE-Tuning, confirming that effective early-exit LLM inference
can be achieved with a limited training budget. In hope of making early-exit
LLMs accessible to the community, we release the source code of our
implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.
65bc66854e82d4524364c761
Xuchen Pan
admin_assigned
2024-02-02T13:28:36.128Z
64101b4ca92fedb0e8559b91
/avatars/c7453a5641a94d04a75de9b7d4272bc2.svg
Xuchen Pan
user
panxuchen
65bc66854e82d4524364c762
Yanxi Chen
admin_assigned
2024-02-02T13:29:05.717Z
6576f9f4654561a1b345610b
/avatars/f801f551640caa70368fcc26a0f51d27.svg
Yanxi Chen
user
yanxi-chen
65bc66854e82d4524364c763
Yaliang Li
65bc66854e82d4524364c764
Bolin Ding
65bc66854e82d4524364c765
Jingren Zhou
admin_assigned
2024-02-02T13:37:29.139Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit
  Large Language Models
This work introduces EE-Tuning, a lightweight and economical solution to
training/tuning early-exit large language models (LLMs). In contrast to the
common approach of full-parameter pre-training, EE-Tuning augments any
pre-trained (and possibly fine-tuned) standard LLM with additional early-exit
layers that are tuned in a parameter-efficient manner, which requires
significantly less computational resources and training data. Our
implementation of EE-Tuning achieves outstanding training efficiency via
extensive performance optimizations, as well as scalability due to its full
compatibility with 3D parallelism. Results of systematic experiments validate
the efficacy of EE-Tuning, confirming that effective early-exit LLM inference
can be achieved with a limited training budget. In hope of making early-exit
LLMs accessible to the community, we release the source code of our
implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.
EE-Tuning efficiently augments pre-trained LLMs with early-exit layers, reducing computational resources and training data while maintaining performance.
EE-Tuning early-exit layers LLMs parameter-efficient tuning 3D parallelism