Initialization

7.4
We ﬁnd that how the preﬁx is initialized has
a large impact in low-data settings. Random
initialization leads to low performance with high
variance. Initializing the preﬁx with activations of
real words signiﬁcantly improves generation, as
shown in Figure 5. In particular, initializing with
task relevant words such as “summarization” and
“table-to-text” obtains slightly better performance
than task irrelevant words such as “elephant”
and “divide”, but using real words is still better
than random. Moreover, in full data settings, the
initialization trick has no impact, and random
initialization leads to equally good performance.

Since we initialize the preﬁx with activations of
real words computed by the LM, this initialization
strategy is concordant with preﬁx-tuning’s philos-
ophy, which preserves the pretrained LM as much
as possible.
Prefix-Tuning