2In our preliminary experiments, GPT-2 and BART fail in

this setting; the only exception is GPT-3.

4585The training objective is the same as equation (2),
but the set of trainable parameters changes: the lan-
guage model parameters φ are ﬁxed and the preﬁx
parameters θ are the only trainable parameters.
Here, each hi is a function of the trainable Pθ.
When i ∈ Pidx, this is clear because hi copies
directly from Pθ. When i (cid:54)∈ Pidx, hi still depends
on Pθ, because the preﬁx activations are always
in the left context and will therefore affect any
activations to the right.
Prefix-Tuning