  Adapters are light-weight modules that allow parameter-efficient fine-tuning
of pretrained models. Specialized language and task adapters have recently been
proposed to facilitate cross-lingual transfer of multilingual pretrained models
(Pfeiffer et al., 2020b). However, this approach requires training a separate
language adapter for every language one wishes to support, which can be
impractical for languages with limited data. An intuitive solution is to use a
related language adapter for the new language variety, but we observe that this
solution can lead to sub-optimal performance. In this paper, we aim to improve
the robustness of language adapters to uncovered languages without training new
adapters. We find that ensembling multiple existing language adapters makes the
fine-tuned model significantly more robust to other language varieties not
included in these adapters. Building upon this observation, we propose Entropy
Minimized Ensemble of Adapters (EMEA), a method that optimizes the ensemble
weights of the pretrained language adapters for each test sentence by
minimizing the entropy of its predictions. Experiments on three diverse groups
of language varieties show that our method leads to significant improvements on
both named entity recognition and part-of-speech tagging across all languages.

Computation and Language;Artificial Intelligence