Since the language models are pretrained on gen-
eral purpose corpora, preserving the LM parame-
ters might help generalization to domains unseen
during training. In concordance with this intuition,
we observe that both preﬁx-tuning and adapter-
tuning have signiﬁcant performance gain in extrap-
olation settings (§6.4); however, how these methods
improve extrapolation is an open question.
Prefix-Tuning