more effective in reducing memory usage when the number
of model parameters is larger. For example,
in LLaMA-
7B-Alpaca, compared with full fine-tuning, IA3, LoRA, and
QLoRA reduce memory usage by 24.08%, 26.30%, and
66.66%, respectively; while in LLaMA-13B-Alpaca, compared
with full fine-tuning, IA3, LoRA, and QLoRA reduce memory
usage by 33.55%, 39.46% and 76.86% of memory usage.
Notably, QLoRA dramatically reduces GPU memory con-
sumption, with QLoRA fine-tuning the LLaMA-7B requiring
Parameter-EfficientFine-TuningMethods