number of parameters to BERTBASE to solve all tasks. Vari-
able ﬁne-tuning performs slightly better than ﬁne-tuning,
whilst training fewer layers. The optimal setting of variable
ﬁne-tuning results in training 52% of the network on average
per task, reducing the total to 9.9× parameters. Adapters,
however, offer a much more compact model. They intro-
duce 1.14% new parameters per task, resulting in 1.19×
parameters for all 17 tasks.
Parameter-Efficient Transfer Learning for NLP