  Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting
large pre-trained models while only tuning a small number of parameters. They
have been shown to be competitive with full model fine-tuning for many
downstream tasks. However, prior work indicates that PEFTs may not work as well
for machine translation (MT), and there is no comprehensive study showing when
PEFTs work for MT. We conduct a comprehensive empirical study of PEFTs for MT,
considering (1) various parameter budgets, (2) a diverse set of language-pairs,
and (3) different pre-trained models. We find that 'adapters', in which small
feed-forward networks are added after every layer, are indeed on par with full
model fine-tuning when the parameter budget corresponds to 10% of total model
parameters. Nevertheless, as the number of tuned parameters decreases, the
performance of PEFTs decreases. The magnitude of this decrease depends on the
language pair, with PEFTs particularly struggling for distantly related
language-pairs. We find that using PEFTs with a larger pre-trained model
outperforms full fine-tuning with a smaller model, and for smaller training
data sizes, PEFTs outperform full fine-tuning for the same pre-trained model.

Computation and Language