2. Adapter tuning for NLP
We present a strategy for tuning a large text model on several
downstream tasks. Our strategy has three key properties:
(i) it attains good performance, (ii) it permits training on
tasks sequentially, that is, it does not require simultaneous
access to all datasets, and (iii) it adds only a small number
of additional parameters per task. These properties are
especially useful in the context of cloud services, where
many models need to be trained on a series of downstream
tasks, so a high degree of sharing is desirable.
To achieve these properties, we propose a new bottleneck
adapter module. Tuning with adapter modules involves
adding a small number of new parameters to a model, which
are trained on the downstream task (Rebufﬁ et al., 2017).
When performing vanilla ﬁne-tuning of deep networks, a
modiﬁcation is made to the top layer of the network. This is
required because the label spaces and losses for the upstream
Parameter-Efficient Transfer Learning for NLP