D HYPERPARAMETERS USED IN EXPERIMENTS

D.1 ROBERTA

We train using AdamW with a linear learning rate decay schedule. We sweep learning rate, number
of training epochs, and batch size for LoRA. Following Liu et al. (2019), we initialize the LoRA
modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the
usual initialization; the pre-trained model stays frozen for all tasks. We report the median over 5
random seeds; the result for each run is taken from the best epoch. For a fair comparison with the
setup in Houlsby et al. (2019) and Pfeiffer et al. (2021), we restrict the model sequence length to 128
and used a ﬁxed batch size for all tasks. Importantly, we start with the pre-trained RoBERTa large
model when adapting to MRPC, RTE, and STS-B, instead of a model already adapted to MNLI.
The runs with this restricted setup are marked with †. See the hyperparameters used in our runs
in Table 9.

D.2 DEBERTA
LORA