among tasks, while the higher layers build features that are
unique to different tasks. This relates to our observation that
for some tasks, ﬁne-tuning only the top layers outperforms
full ﬁne-tuning, see Table 2.
Next, we investigate the robustness of the adapter modules
to the number of neurons and initialization scale. In our
main experiments the weights in the adapter module were
drawn from a zero-mean Gaussian with standard deviation
10−2, truncated to two standard deviations. To analyze the
impact of the initialization scale on the performance, we
test standard deviations in the interval [10−7, 1]. Figure 6
summarizes the results. We observe that on both datasets,
the performance of adapters is robust for standard deviations
below 10−2. However, when the initialization is too large,
performance degrades, more substantially on CoLA.
To investigate robustness of adapters to the number of neu-
rons, we re-examine the experimental data from Section 3.2.
Parameter-Efficient Transfer Learning for NLP