Prefix-tuning [10] proposes to prepend soft prompts P =
[P1], [P2],··· , [Pl] (l denotes the length of the prefix) to
the hidden states of the multi-head attention layer, differing
from prompt-tuning that adds soft prompts to the input. To
ensure stable training, a FFN is introduced to parameterize
the soft prompts, as direct optimization of the soft prompts
can lead to instability. Two sets of prefix vectors ˆPk and
ˆPv are concatenated to the original key (K) and value
(V ) vectors of the attention layer. The self-attention mech-
anism with prefix-tuning can be represented by Equation 8.
During training, only ˆPk, ˆPv, and the parameters of FFN
are optimized, while all other parameters of PLMs remain
frozen. The structure of prefix-tuning is illustrated in Fig. 3.
After training, the FFN is discarded, and only Pk and Pv
are used for inference. P-tuning [25] also considers insert-
ing the soft prompts [P1],··· , [Pi], [Pi+1],··· , [Pl] into the
Parameter-EfficientFine-TuningMethods