Howard & Ruder, 2018; Radford et al., 2018) In NLP, the
upstream model is usually a neural language model (Ben-
gio et al., 2003). Recent state-of-the-art results on ques-
tion answering (Rajpurkar et al., 2016) and text classi-
ﬁcation (Wang et al., 2018) have been attained by ﬁne-
tuning a Transformer network (Vaswani et al., 2017) with a
Masked Language Model loss (Devlin et al., 2018). Perfor-
mance aside, an advantage of ﬁne-tuning is that it does not
require task-speciﬁc model design, unlike representation-
based transfer. However, vanilla ﬁne-tuning does require a
new set of network weights for every new task.
Parameter-Efficient Transfer Learning for NLP