200040006000800010000Evaluation Steps3234363840425-shot MMLU Accuracy(%)LLaMA-7B-Alpaca (FT)200040006000800010000Evaluation Steps3234363840425-shot MMLU Accuracy(%)LLaMA-7B-Alpaca ((IA)3)200040006000800010000Evaluation Steps3234363840425-shot MMLU Accuracy(%)LLaMA-7B-Alpaca (LoRA)200040006000800010000Evaluation Steps3234363840425-shot MMLU Accuracy(%)LLaMA-7B-Alpaca (QLoRA)200040006000800010000Evaluation Steps44454647484950515-shot MMLU Accuracy(%)LLaMA-13B-Alpaca (FT)200040006000800010000Evaluation Steps44454647484950515-shot MMLU Accuracy(%)LLaMA-13B-Alpaca ((IA)3)200040006000800010000Evaluation Steps44454647484950515-shot MMLU Accuracy(%)LLaMA-13B-Alpaca (LoRA)200040006000800010000Evaluation Steps44454647484950515-shot MMLU Accuracy(%)LLaMA-13B-Alpaca (QLoRA)TABLE VII: The peak GPU memory usage when fine-tuning RoBERT-base, RoBERTa-large, T5-base, T5-large, LLaMA-7B,
and LLaMA-13B model using full fine-tuning and various PEFT methods.

Memory (GB) Model & Method

Memory (GB)

16
Parameter-EfficientFine-TuningMethods