Recent work by Aghajanyan et al. (2020) uses
intrinsic dimension to show that there exists a low-
dimensional reparameterization that is as effective
for ﬁne-tuning as the full parametrization. This
explains why good accuracy on downstream tasks
can be obtained by updating only a small num-
ber of parameters. Our work echoes this ﬁnding
by showing that good generation performance can
also be attained by updating a very small preﬁx.
However, preﬁx-tuning is not just about the size of
trainable parameters, but more importantly, which
subset of parameters to modify. Therefore, it would
be interesting future work to explore other light-
weight ﬁne-tuning methods that achieve an even
better accuracy-size tradeoff.
Prefix-Tuning