6https://huggingface.co/datasets/wmt16

13

TABLE III: Fine-tuning RoBERTa-base (RoBB) and RoBERTa-large (RoBL) models on the GLUE benchmark. Specifically,
we report the Matthews correlation for COLA, accuracy/F1 score for MRPC and QQP, Pearson/Spearman correlation for
STS-B, averaged matched accuracy for MNLI, and accuracy for other NLU tasks. Higher values indicate better performance
across all metrics. We present the number of trainable parameters (# TPs) of each method, excluding Child-TuningD due to
its randomness during network pruning. We also bold the maximum values and underline the minimum values.

Model

PEFT Method

RoBB

RoBL

FT

AdapterS

Prompt-tuning
Prefix-tuning

(IA)3
BitFit

Child-TuningD

LoRA

AdaLoRA

MAM Adapter
ProPELTAdapter
ProPELTPrefix
ProPELTLoRA

FT

AdapterS

Prompt-tuning
Prefix-tuning

(IA)3
BitFit

Child-TuningD

LoRA

AdaLoRA

MAM Adapter
ProPELTAdapter
ProPELTPrefix
ProPELTLoRA

-
Parameter-EfficientFine-TuningMethods