Learning rate

5-shot MMLU Accuracy

Model

PEFT Method

FT
(IA)3
LoRA

FT
(IA)3
LoRA

BLEU
# TPs
222.9M 27.42
0.07M 27.58
0.88M 27.78
737.7M 28.13
0.19M 28.12
2.36M 28.12

T5-base

T5-large

amount of trainable parameters.

• Sequential adapter requires more trainable parameters
than prompt-tuning, prefix-tuning, (IA)3, BitFit, Child-
TuningD, LoRA, and AdaLoRA, but achieves better
performance than them on the GLUE benchmark.

• Prompt-tuning with the virtual marker length set to 20
achieves the smallest trainable parameter, but also the
worst performance, with its average performance on the
GLUE benchmark being about 10% lower than full fine-
tuning.

• Child-TuningD performs well when fine-tuning RoBERT-
base on the GLUE benchmark and obtains better perfor-
mance than full fine-tuning, but performs poorly when
fine-tuning RoBERT-large on the MNLI dataset, which
we guess it caused by the learning rate.
Parameter-EfficientFine-TuningMethods