1) Manual Combination: Manual combination mainly in-
volves integrating the structure or features of one PEFT
method into another PEFT method to enhance performance
while achieving parameter efficiency. MAM Adapter (Mix-
And-Match Adapter) [16] is the combination of scaled parallel
adapter and prefix-tuning, which employs prefix-tuning with
smaller bottleneck dimensions at the attention layer and al-
locates more parameter budget to modify the representation
of FFN using the scaling parallel adapter. Scaled parallel
adapter denotes the parallel adapter with a scaling factor
to adjust the adapter output. Concretely, the output of the
MAM adapter h can be expressed with h = LN(X +
scale ∗ FFN(LN(Attn([Pk, X]) + [Pv, X]))) for the input X.
Further, U-MAM (Unstructured MAM) and S-MAM (struc-
tured MAM) [35] are proposed by combining NAS algorithm
[85] and pruning technique to automatically determine which
parameters of the network need to be fine-tuned based on
Parameter-EfficientFine-TuningMethods