Parameter-Efﬁcient Transfer Learning for NLP

GLUE (BERTLARGE)

Additional Tasks (BERTBASE)

Figure 3. Accuracy versus the number of trained parameters, aggregated across tasks. We compare adapters of different sizes (orange)
with ﬁne-tuning the top n layers, for varying n (blue). The lines and shaded areas indicate the 20th, 50th, and 80th percentiles across
tasks. For each task and algorithm, the best model is selected for each point along the curve. For GLUE, the validation set accuracy is
reported. For the additional tasks, we report the test-set accuracies. To remove the intra-task variance in scores, we normalize the scores
for each model and task by subtracting the performance of full ﬁne-tuning on the corresponding task.

MNLIm(BERTBASE)

CoLA (BERTBASE)
Parameter-Efficient Transfer Learning for NLP