• All PEFT methods reduce the number of

trainable
parameters, and most PEFT methods achieve perfor-
mance matching or even better than full fine-tuning on
the GLUE benchmark. For RoBERTa-base,
the aver-
age performance of prompt-tuning, prefix-tuning, IA3,
AdaLoRA, ProPELTprefix and ProPELTLoRA on GLUE all
underperforms full finetuning, while that of sequential
adapter, BitFit, Child-TuningD, LoRA, MAM adapter,
and ProPELTAdapter outperforms full fine-tuning. For
RoBERTa-large,
the average performance of prompt-
tuning, prefix-tuning, IA3, AdaLoRA, ProPELTprefix and
Child-TuningD on GLUE underperforms full fine-tuning,
while that of sequential adapter, BitFit, LoRA, MAM
adapter, ProPELTAdapter and ProPELTLoRA outperforms
full fine-tuning.
Parameter-EfficientFine-TuningMethods