7

the Hadamard adapter, layer normalization, and classifier are
updated.

B. Partial Fine-tuning

Partial fine-tuning methods aim to reduce the number of
fine-tuned parameters by selecting a subset of pre-trained pa-
rameters that are critical to downstream tasks while discarding
unimportant ones. We categorize partial fine-tuning methods
into three groups: Bias Update [34], [35], in which only
the bias term in the attention layer, feed-forward layer and
layer normalization of the transformer is updated, Pretrained
Weight Masking [36], [37], where the pretrained weights are
masked using various pruning criterion, and Delta Weight
Masking [38], [39], [40], [41], in which delta weights are
masked via pruning techniques and optimization approxima-
tion. A detailed analysis of pretrained weight and delta weight
masking is provided in Table I.

1) Bias Update: Bit-Fit

(Bias-term Fine-tuning)
Parameter-EfficientFine-TuningMethods