summaries: This paper investigates a method for improving the cross-lingual transfer of pretrained multilingual models. The paper first empirically analyzed the influence of representation invariance and distributional class shift. Then, the paper proposed a method to improve the representation invariance and correcting the class shift. Experiments showed its superiority under large prior shifts.

summaries: In this paper, the authors provide substantial analyses on the cross-lingual transfer performance in the multilingual neural language models and reported that the performance is strongly correlated with representation invariance and negatively affected by distributional shift in class priors between data in the src/tgt languages. Based on these findings, the authors propose an unsupervised cross-lingual learning method, called importance-weighted domain adaptation (IWDA), where it performs feature alignment, prior shift estimation, and correction. The authors experimented on two different NLP tasks such as multilingual NER and multilingual sentiment analysis tasks, and experimentally showed the effectiveness. Besides that, they demonstrated that the proposed approach improves performance further, when combined with existing semi-supervised learning approaches.

summaries: This paper looks at the problem of unsupervised cross-lingual transfer (termed UCL) in the paper through the optics of domain adaptation. After empirically analysing and validating that distributional shifts in class priors might cause a huge problem for UCL (which wasn't tackled in previous research), the authors proceed with an introduction of a new method that aims to mitigate that problem. The idea is to get rid of that shift through a approach called importance-weighted domain adaptation (IWDA), which is largely the adaptation of the work from Tachet des Combes et al. (NeurIPS 2020) to the UCL problem.  The results on two tasks in the UCL setup (NER and MARC classification) show slight gains over the standard zero-shot transfer when IWDA is applied, with more prominent gains reported when a stronger domain shift is observed - however, such a setup has been created mostly artificially, to further demonstrate the benefits of modelling the shift in the model.

summaries: The paper first demonstrates the importance of feature invariance (language invariant representations) and class-prior invariance across languages on zero-shot cross-lingual performance. By analyzing the zero-shot performance on different languages on the MARC reviews and WikiANN NER tasks, and comparing against the class conditional distance between the source language (English) and target language feature representations, the authors illustrate the how high feature invariance results in better zero-shot performance. By also synthetically modifying the class prior for the target language, the authors demonstrate how increasing differences in class priors result in decreasing zero-shot cross-lingual transfer performance.  Building on their observations, the authors propose an approach that: (i) Introduces an adversarial loss term to penalize distortion in average class conditional feature representations between the source and target languages. (ii) Adds an importance weighting term to ensure the approach doesn't fail under class prior shifts.  Empirical studies demonstrate that the proposed approach improves significantly over the vanilla zero-shot model on both MARC sentiment analysis and NER tasks, also improving over self-training on sentiment analysis. Comparisons on synthetic datasets (sub-sampled from NER and MARC datasets) that enhance the class prior shift highlight the robustness of the approach under large class prior shifts where previous approaches fail.

abstractText: Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, evidenced by their decent zero-shot transfer performance. While studies have attributed this success to cross-lingually shared representations, quantitative analyses are sparse. Towards a better understanding of the role of multilingual representations, in this work, we first make the following observations through empirical analysis: (1) invariance of the feature representations strongly correlates with transfer performance, and (2) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work. Based on our findings, we propose an unsupervised cross-lingual learning method, called importance-weighted domain alignment (IWDA), that performs representation alignment, prior shift estimation, and correction. Experiment results demonstrate its superiority under large prior shifts. In addition, our method delivers further performance gains when combined with existing semi-supervised learning techniques.

authors:
Amr Alexandari
Anshul Kundaje
Avanti Shrikumar

title: Maximum Likelihood with BiasCorrected Calibration is Hard-To-Beat at Label Shift Adaptation

authors:
Naveen Arivazhagan
Ankur Bapna
Orhan Firat
Roee Aharoni
Melvin Johnson
Wolfgang Macherey

title: The Missing Ingredient in Zero-Shot Neural Machine Translation

authors:
Martin Arjovsky
Soumith Chintala
Léon Bottou

title: Wasserstein Generative Adversarial Networks

authors:
Mikel Artetxe
Holger Schwenk

title: Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

venue: Transactions of the Association for Computational Linguistics,

authors:
Mikel Artetxe
Gorka Labaka
Eneko Agirre
Kyunghyun Cho

title: Unsupervised Neural Machine Translation

venue: In International Conference on Learning Representations,

authors:
Mikel Artetxe
Gorka Labaka
Eneko Agirre

title: Translation Artifacts in Cross-lingual Transfer Learning

authors:
Mikel Artetxe
Sebastian Ruder
Dani Yogatama

title: On the Cross-lingual Transferability of Monolingual Representations

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,

authors:
Mikel Artetxe
Sebastian Ruder
Dani Yogatama
Gorka Labaka
Eneko Agirre

title: A Call for More Rigor in Unsupervised Cross-lingual Learning

authors:
Kamyar Azizzadenesheli
Anqi Liu
Fanny Yang
Animashree Anandkumar

title: Regularized Learning for Domain Adaptation under Label Shifts

authors:
M Saiful Bari
Tasnim Mohiuddin
Shafiq Joty

title: UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP

venue: In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1978–1992,

authors:
Shai Ben-David
John Blitzer
Koby Crammer
Fernando Pereira

title: Analysis of representations for domain adaptation

venue: Advances in Neural Information Processing Systems,

authors:
Steven Cao
Nikita Kitaev
Dan Klein

title: Multilingual Alignment of Contextual Word Representations

authors:
Xilun Chen
Yu Sun
Ben Athiwaratkun
Claire Cardie
Kilian Weinberger

title: Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification

authors:
Alexis Conneau
Guillaume Lample

title: Cross-lingual Language Model Pretraining

authors:
Alexis Conneau
Guillaume Lample
Ruty Rinott
Adina Williams
Samuel R. Bowman
Holger Schwenk
Veselin Stoyanov

title: XNLI: Evaluating Cross-lingual Sentence Representations

authors:
Alexis Conneau
Kartikay Khandelwal
Naman Goyal
Vishrav Chaudhary
Guillaume Wenzek
Francisco Guzmán
Edouard Grave
Myle Ott
Luke Zettlemoyer
Veselin Stoyanov

title: Unsupervised Cross-lingual Representation Learning at Scale

authors:
Alexis Conneau
Shijie Wu
Haoran Li
Luke Zettlemoyer
Veselin Stoyanov

title: Emerging Crosslingual Structure in Pretrained Language Models

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

authors:
Xin Dong
Gerard de Melo

title: A Robust Self-Learning Framework for Cross-Lingual Text Classification

venue: In Q,

authors:
Haim Dubossarsky
Ivan Vulić
Roi Reichart
Anna Korhonen

title: The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures

authors:
Philipp Dufter
Hinrich Schütze

title: Identifying Elements Essential for BERT’s Multilinguality

venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4423–4437,

authors:
Yaroslav Ganin
Evgeniya Ustinova
Hana Ajakan
Pascal Germain
Hugo Larochelle
François Laviolette
Mario March
Victor Lempitsky

title: Domain-Adversarial Training of Neural Networks

venue: Journal of Machine Learning Research,

authors:
Saurabh Garg
Yifan Wu
Sivaraman Balakrishnan
Zachary Lipton

title: A Unified View of Label Shift Estimation

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Arthur Gretton
Karsten M. Borgwardt
Malte J. Rasch
Bernhard Schölkopf
Alexander Smola

title: A Kernel Two-Sample Test

authors:
Chuan Guo
Geoff Pleiss
Yu Sun
Kilian Q. Weinberger

title: On Calibration of Modern Neural Networks

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: GANs trained by a two time-scale update rule converge to a local nash equilibrium

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the Knowledge in a Neural Network

authors:
Junjie Hu
Sebastian Ruder
Aditya Siddhant
Graham Neubig
Orhan Firat
Melvin Johnson

title: XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation

authors:
Junjie Hu
Melvin Johnson
Orhan Firat
Aditya Siddhant
Graham Neubig

title: Explicit Alignment Objectives for Multilingual Bidirectional Encoders

venue: In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3633–3643,

authors:
Lifu Huang
Heng Ji
Jonathan May

title: Cross-lingual Multi-Level Adversarial Transfer to Enhance Low-Resource Name Tagging

authors:
Shafiq Joty
Preslav Nakov
Lluı́s Màrquez
Israa Jaradat

title: Cross-language Learning with Adversarial Neural Networks

venue: In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL

authors:
Karthikeyan K
Zihan Wang
Stephen Mayhew
Dan Roth

title: Cross-Lingual Ability of Multilingual BERT: An Empirical Study

authors:
Phillip Keung
Yichao Lu
Vikas Bhardwaj

title: Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

authors:
Phillip Keung
Yichao Lu
György Szarvas
Noah A. Smith

title: The Multilingual Amazon Reviews Corpus

venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4563–4568,

authors:
Joo-Kyung Kim
Young-Bum Kim
Ruhi Sarikaya
Eric Fosler-Lussier

title: Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources

venue: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,

authors:
Simon Kornblith
Mohammad Norouzi
Honglak Lee
Geoffrey Hinton

title: Similarity of Neural Network Representations Revisited

authors:
Guillaume Lample
Alexis Conneau
Marc’Aurelio Ranzato
Ludovic Denoyer
Hervé Jégou

title: Word translation without parallel data

authors:
Anne Lauscher
Vinit Ravishankar
Ivan Vulić
Goran Glavaš

title: From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers

authors:
Juntao Li
Ruidan He
Hai Ye
Hwee Tou Ng
Lidong Bing
Rui Yan

title: Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model

venue: In Twenty-Ninth International Joint Conference on Artificial Intelligence,

authors:
Jindřich Libovický
Rudolf Rosa
Alexander Fraser

title: On the Language Neutrality of Pre-trained Multilingual Representations

venue: In Findings of the Association for Computational Linguistics: EMNLP

authors:
Zachary Lipton
Yu-Xiang Wang
Alexander Smola

title: Detecting and Correcting for Label Shift with Black Box Predictors

authors:
Mingsheng Long
Zhangjie Cao
Jianmin Wang
Michael I Jordan

title: Conditional adversarial domain adaptation

authors:
Adyasha Maharana
Mohit Bansal

title: Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension

authors:
Tomas Mikolov
Quoc V. Le
Ilya Sutskever

title: Exploiting Similarities among Languages for Machine Translation

authors:
Benjamin Muller
Yanai Elazar
Benoı̂t Sagot
Djamé Seddah

title: First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT

venue: In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,

authors:
Xiaoman Pan
Boliang Zhang
Jonathan May
Joel Nothman
Kevin Knight
Heng Ji

title: Crosslingual Name Tagging and Linking for 282 Languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

authors:
Barun Patra
Joel Ruben Antony Moniz
Sarthak Garg
Matthew R. Gormley
Graham Neubig

title: Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Jonas Pfeiffer
Ivan Vulić
Iryna Gurevych
Sebastian Ruder

title: MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

authors:
Ngoc-Quan Pham
Jan Niehues
Thanh-Le Ha
Alexander Waibel

title: Improving Zero-shot Translation with Language-Independent Constraints

venue: In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers),

authors:
Telmo Pires
Eva Schlinger
Dan Garrette

title: How Multilingual is Multilingual BERT

authors:
Afshin Rahimi
Yuan Li
Trevor Cohn

title: Massively Multilingual Transfer for NER

title: Neural Transfer Learning for Natural Language Processing

venue: PhD thesis, NUI Galway,

authors:
Sebastian Ruder
Barbara Plank

title: Strong Baselines for Neural Semi-Supervised Learning under Domain Shift

venue: In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),

authors:
Sebastian Ruder
Ivan Vulić
Anders Søgaard

title: A survey of cross-lingual word embedding models

venue: Journal of Artificial Intelligence Research,

authors:
Marco Saerens
Patrice Latinne
Christine Decaestecker

title: Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure

venue: Neural Computation,

authors:
Holger Schwenk
Xian Li

title: A Corpus for Multilingual Document Classification in Eight Languages

venue: In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018),

authors:
Samuel L. Smith
David H.P. Turban
Steven Hamblin
Nils Y. Hammerla

title: Offline bilingual word vectors, orthogonal transformations and the inverted softmax

venue: In 5th International Conference on Learning Representations,

authors:
Remi Tachet des Combes
Han Zhao
Yu-Xiang Wang
Geoffrey J. Gordon

title: Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift

authors:
Hoang Thanh-Tung
Truyen Tran
Svetha Venkatesh

title: Improving generalization and stability of generative adversarial networks

title: Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition

venue: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002),

authors:
Erik F. Tjong Kim Sang
Fien De Meulder

title: Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

venue: In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL

authors:
Giorgos Vernikos
Katerina Margatina
Alexandra Chronopoulou
Ion Androutsopoulos

title: Domain Adversarial Fine-Tuning as an Effective Regularizer

authors:
Xinyi Wang
Hieu Pham
Zihang Dai
Graham Neubig

title: SwitchOut: An Efficient Data Augmentation Algorithm for Neural Machine Translation

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

authors:
Qianhui Wu
Zijia Lin
Börje Karlsson
Jian-Guang Lou
Biqing Huang

title: Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language

authors:
Shijie Wu
Mark Dredze. Beto
Bentz

title: Becas: The Surprising Cross-Lingual Effectiveness of BERT

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP),

authors:
Shijie Wu
Mark Dredze

title: Are All Languages Created Equal in Multilingual BERT

venue: In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 120–130,

authors:
Mengzhou Xia
Guoqing Zheng
Subhabrata Mukherjee
Milad Shokouhi
Graham Neubig
Ahmed Hassan Awadallah

title: MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning

venue: In Proceedings of the 2021 Conference of the North American Chapter

authors:
7523–7532. PMLR
May 2019. Hao Zheng
Mirella Lapata

title: Sentence Centrality Revisited for Unsupervised Summarization

venue: Garg et al.,

heading: 1 INTRODUCTION

text:
Many recent state-of-the-art results in natural language processing (NLP) have been achieved on Transformer-based deep neural language models (LMs) using the “pre-train then fine-tune” paradigm (Devlin et al., 2019; Conneau & Lample, 2019). Multilingual versions of these LMs are pre-trained on unannotated and unparallel texts from more than one language, such as multilingual BERT (mBERT) and XLM-R (Conneau et al., 2020a), and it is found that the learned representations are shared across languages. In particular, the cross-lingual knowledge acquired by these models has enabled zero-shot cross-lingual transfer, where after fine-tuned on data in a source language, not only do they work well when evaluated on the source, but also quite decently for the same task on almost all languages they have seen during pre-training (Wu & Dredze, 2019).
Their success on zero-shot cross-lingual transfer has prompted numerous studies on their multilingual abilities. Among the many explanations, a recent hypothesis suggests that the deep representation combined with parameter sharing induced intermediate feature representations that are shared across languages (K et al., 2019; Conneau et al., 2020b; Muller et al., 2021). Following this, we begin by studying the following question for mutilingual neural LMs: What is the role of shared representations in unsupervised cross-lingual learning?
In an effort to answer this question in a quantitative manner, through empirical analysis on mBERT, we find that (1) its cross-lingual transfer performance strongly correlates with the alignment of feature representations for the NLP task at hand between the source and target languages (Section 2.1). In addition, (2) distributional shift in class priors between data in the source and target languages negatively affects transfer performance (sections 2.2 and 4.2). Despite being a common issue with real-world data under unsupervised settings, it has been largely overlooked in prior work.
Based on these findings on the cross-lingual transfer of mutilingual neural LMs, towards a principled approach for the unsupervised setting where access to unlabeled data in the target language for the downstream task at hand is available in addition to labeled ones in the source (hereafter referred
to as UCL1), we propose importance-weighted domain alignment (IWDA). This method performs representation alignment, class prior shift estimation, and correction (Section 3). Experiment results show that IWDA outperforms the more popular semi-supervised learning methods under large prior shifts, and can be additionally combined with them for further performance gains (Section 4).

heading: 1.1 RELATED WORK

text:
Our proposed method is heavily inspired by Tachet des Combes et al. (2020), which studied the issues with alignment-based domain adaptation (DA) under the presence of class prior shifts, and proposed the inclusion of prior shifts estimation and correction procedures in DA. Our method also shares similar components with the DA-based UCL method by Keung et al. (2019), while highlighting its limitations discovered in the present work (Section 2.2).
Analyzing Multilingual Neural LMs. Most studies analyze mBERT’s cross-lingual ability via probing experiments, and identified influencing factors including but not limited to linguistic similarity, size of pretraining corpus, domain similarity, parameter sharing and model depth (Pires et al., 2019; K et al., 2019; Lauscher et al., 2020; Conneau et al., 2020b; Dufter & Schütze, 2020). Another line of research analyze intermediate representations and observed that mBERT learns crosslingually shared representations (Wu & Dredze, 2019; Conneau et al., 2020b; Muller et al., 2021).
Cross-Lingual Representation Alignment. The idea of learning invariant representations has powered a long history of approaches for cross-lingual transfer on neural models. Under zero-shot settings, it is achieved by aligning word embeddings on bilingual dictionaries that could be induced via adversarial training (Mikolov et al., 2013; Smith et al., 2017; Zhang et al., 2017; Artetxe & Schwenk, 2019; Cao et al., 2019), a method that has powered many machine translation techniques (Artetxe et al., 2018; Lample et al., 2018; Arivazhagan et al., 2019; Pham et al., 2019; Hu et al., 2021) It is found on static embedding models that the goodness of alignment affects transfer performance (Patra et al., 2019; Dubossarsky et al., 2020). On deep models such as mBERT, it could also take the form of shared feature representations in the Transformer body (Artetxe et al., 2020b; Pfeiffer et al., 2020). Under UCL and supervised settings, techniques based on representation alignment were proposed for models ranging from distributed word embeddings (Joty et al., 2017; Chen et al., 2018), RNN (Kim et al., 2017; Huang et al., 2019), to mBERT (Keung et al., 2019; Zheng & Lapata, 2019; Xia et al., 2021). For a survey, see Ruder et al. (2019) and Ruder (2019).
Connection to Domain Adaptation. Domain adaptation is a popular framework for handling domain shifts between training and test data, e.g. learning a POS tagger on news articles but to be deployed on biomedical papers with a significantly different word distribution (Ben-David et al., 2007). The concept of domain is extended naturally to cross-lingual learning by treating the distributions of sentence and document pertaining to the same task in each language as separate domains2 (Ruder, 2019). Thereby, adaptation techniques proposed for domain shift can be applied to UCL. Besides methods aiming at extracting shared representations (Li et al., 2020; Vernikos et al., 2020), another family of successful approaches is semi-supervised learning, which include self-training (Dong & de Melo, 2019), tri-training (Ruder & Plank, 2018), knowledge distillation (Wu et al., 2020), and data augmentation (Wang et al., 2018; Maharana & Bansal, 2020; Bari et al., 2021).

heading: 2 FACTORS AFFECTING CROSS-LINGUAL TRANSFER

text:
To develop and provide evidence to the hypothesis that attributes the cross-lingual transfer success of multilingual neural LMs to shared representations, as well as to understand its implications on the design of unsupervised transfer algorithms, we attempt to quantify the relation between representation invariance and transfer performance in this section.
We briefly summarize the “pre-train then fine-tune” training paradigm and introduce notation along the way. Let g : X → Z denote the feature mapping provided by the multilingual LM, and z =
1With an abuse of notation. In this work, the acronym UCL refers to the learning setting considered, and is meant to distinguish from the umbrella term “unsupervised cross-lingual learning”, which also includes the study of the zero-shot setting where no target language task data is given.
2Hence the terms domain and language are used interchangeably in this work.
g(x) ∈ Z ⊆ Rd the feature vector computed on an input sentence or token x ∈ X . In practice as well as in our studies and experiments, the feature is the last-layer wordpiece embeddings for token classification tasks, and the embedding of a special start-of-sentence marker (symbolized as [CLS]) for sequence classification tasks. Hence, on BERT-Base models, d = 768 and Z ≈ Sd−1 sphere due to LayerNorm. Let h : Z → Y denote the output layer(s) added on top of g with an appropriate activation, usually a single linear layer with Softmax for classification tasks in practice. Hence, an end-to-end model is obtained, which computes predictions via ŷ = h(g(x)). By fine-tuning both g, h on labeled task data (x, y) sampled from the source language domain pS only, we get a zeroshot transfer model for the downstream task at hand, where the term zero-shot points to the fact that the model does not use any task data from the target language (x′, y′) ∼ pT during training. Based on the following decomposition of the joint feature-label distribution into the product of class-conditional feature distribution and marginal prior distribution,
p(z, y) = p(z|y)p(y) := p({x ∈ X : g(x) = z}|y)p(y),
we examine how cross-lingual transfer performance is influenced by (1) invariance of the feature representation of data between the source and target languages, with perfect invariance being pS(z|Y = j) = pT (z|Y = j) for all j ∈ Y , and (2) the distributional shift in the class priors, with no shift being pS(Y = j) = pT (Y = j) for all j ∈ Y . On classification tasks, Y = {1, 2, · · · , k} is discrete. For legibility in later sections, we use the shorthands to denote the class-conditional feature distribution and the marginal prior distribution: pZ|Y and pY .
Setup. Our empirical analysis is performed on the representations learned by mBERT zero-shot fine-tuned on English for two multilingual tasks: sentiment analysis on the Multilingual Amazon Reviews Corpus (MARC), and named-entity recognition (NER) on the WikiANN dataset (see descriptions in Section 4). To simulate the effects of varying amount of representation misalignment and class prior difference, evaluation is performed on 500 smaller datasets randomly subsampled from the original MARC according to different class priors, and roughly 700 datasets from WikiANN. MARC covers five high-resource languages (excluding English), and (our selected subset of) WikiANN covers 39 languages of distinct linguistic properties and varying resourcefulness.

heading: 2.1 INVARIANCE OF FEATURE REPRESENTATIONS

text:
For studying the effects of representation invariance, we compare model performance to the discrepancy between the class-conditional feature distributions on the source and target domains, pZ|YS and pZ|YT . We take the average to aggregate the discrepancies computed when conditioned on each class, 1 k ∑k j=1D(p Z|Y =j S , p Z|Y =j T ), and refer to this quantity by conditional feature shift.
The discrepancy measure we use computes the `1-distance between the mean feature vectors3,
D(p, q) := ∥∥∥Ex∼p[x]− Ex′∼q[x′]∥∥∥
1 = d∑ i=1 ∣∣∣Ex∼p[xi]− Ex′∼q[x′i]∣∣∣. Measuring the distance between the means of representations for the purpose of analyzing mBERT has appeared in prior work. A similar quantity is referred to as language centroids by Libovický et al. (2020). The important distinction here is that we explicitly remove the effects of class prior shift by conditioning on the class labels, which we argue is an artifact of the dataset since arbitrary amounts of unconditioned feature shift can be generated by altering the class priors.
First, we consider zero-shot transfer. In Fig. 1a, we plot theF1 score4 attained by the zero-shot model against conditional feature shift between English features and those of the target language, and observe that better representation alignment is indicative of better transfer performance. Findings in prior work on the cross-lingual ability of mBERT are reflected. For instance, linguistic similarity, a factor commonly argued to be important for cross-lingual transfer, could be related to the alignment
3Comparisons to more sophisticated metrics such as RBF-kernel MMD and linear CKA were made in our preliminary studies (Gretton et al., 2012; Kornblith et al., 2019), and we found that they correlate tightly with our first-moment discrepancy measure for analyzing conditional feature shift.
4To study the effects of class prior shift, we use macro-averaged F1 as the performance metric in this section, because classification accuracy could be inflated on datasets with skewed priors (Azizzadenesheli et al., 2018).
of features: Spanish, French and German are more linguistically similar to English than Japanese and Chinese, whose features are less aligned with those of English. These results also reflects the successes of zero-shot learning approaches on mBERT that aim to achieve representation invariance.
While the above finding shows that representation alignment is desirable for zero-shot transfer, is it also suitable for UCL settings where task data from the target language are available? We start by recognizing that the objective function of most unsupervised transfer algorithms consists of two parts: loss on labeled source data, and terms involving unlabeled data that in theory are linked to the loss on target domain. Based on this, we compared against conditional feature shift the performance of the model fine-tuned supervisedly on labeled data from source and target languages, as it would be the ideal objective for unsupervised algorithms. The results are plotted in Fig. 1b. Instead of observing increases in conditional feature shifts, which would have implied that language-specific features are preferred by over shared ones for good performance on each language, we observe much-improved representation alignments between English and each of the target languages. This means that achieving representation invariance is also appropriate under UCL settings.
To demonstrate that the correlation between last-layer class-conditional alignment and performance is not an artifact of the linear classification head, we show that model performance is also correlated with (1) conditional feature shift in intermediate layers, plotted on the bottom panels of Fig. 1, and (2) class-balanced feature shift, plotted in Fig. 5 in the appendix, where the removal of prior shift effects is achieved by reweighting the class priors to be uniform (eq. 1). Furthermore, we include results on the XNLI dataset, a higher-level semantic task (Fig. 7), and results on MARC with the more powerful XLM-R Large model (Fig. 8).

heading: 2.2 DISTRIBUTIONAL SHIFT IN CLASS PRIORS

text:
Most benchmarks for evaluating cross-lingual transfer are constructed so that the class prior distributions between data in the source and target languages are the same. This is desired for assessing zero-shot learning approaches, as the focus should not be on “‘tricks’ for how to best handle the class imbalance” (Schwenk & Li, 2018). However, for UCL settings with unlabeled target data, we argue that making the best use of the additional data involves the detection and correction of prior shift. Otherwise, as we show in this and later sections, negative transfer could occur.
To examine the effects of class prior shift on cross-lingual transfer, we compare zero-shot transfer performance to the difference in prior distributions, D(pYS , p Y T ), equivalent to total variation (TV). The results are presented in Fig. 2. We also plot the effects of representation alignment and prior shift jointly in Fig. 3, where stronger correlations are revealed: (1) transfer performance generally takes a hit when prior shifts are large, regardless of how well the features are aligned, and (2) on target domains of the same amount of prior shift, performance decreases when features are less aligned. The findings suggest that in addition to achieving representation invariance, good transfer performance also requires the detection and handling of potential prior shifts. While this is not possible under zero-shot settings without any data from the target domain, the unlabeled target task data provided under UCL settings could be leveraged toward this goal.
We examined existing approaches proposed for multilingual neural LMs for the UCL settings, but did not find procedures for prior shift correction, and moreover, there is a lack of attention to the potential issues and effects of class prior shift to the algorithms. Specifically, for alignment-based methods, Zhao et al. (2019) and Tachet des Combes et al. (2020) showed that adaptation could lead to negative transfer when prior shifts are not accounted for due to erroneous alignment. For methods based on semi-supervised learning, negative transfer is observed empirically under large shifts in our studies (Section 4.2). These cases of negative transfer call for more considerations to the effects of class prior shift in future work on cross-lingual transfer that leverage target data.
Interestingly, although the alignment-based method proposed by Keung et al. (2019) did not account for prior shift, it avoided negative transfer on the CoNLL NER task (which exhibits shifts) by performing feature alignment on the [CLS] sentence embeddings instead of the word embeddings (recall that predictions are made on word embeddings on token classification tasks). Since sentence embeddings on Transformer-based models could be loosely related to a weighted sum of the word embeddings, the procedure of Keung et al. (2019) is performing coarse-grained representation alignment, which avoids negative transfer while providing decent performance gain. However, as we compare in Table 1, this misses out opportunities for larger gains from word embedding alignment. If alignment were performed on word embeddings without prior shift correction, deteriorating performance would have been observed (Fig. 11).

heading: 3 IMPORTANCE-WEIGHTED DOMAIN ALIGNMENT

text:
For unsupervised cross-lingual learning on multilingual neural LMs with access to unlabeled target language data (which we refer to as the UCL setting), the findings in Section 2 suggest that good performance can be achieved via aligning representations and correcting class prior shifts. Based on these principles, we propose class-importance-weighted domain alignment (IWDA) for UCL, following closely the work by Tachet des Combes et al. (2020). In this section, we provide a highlevel description of the algorithm and its connection to the findings above, and refer interested readers to the aforementioned work and Appendix C for further implementation details.
We have seen that class-conditional feature alignment is desired for cross-lingual transfer, however, this is not possible to achieve under the UCL setting since the target labels are unknown. Assume for now that the target class prior distribution is known, then we can compute the true ratio of the class priors, w∗j := pT (Y = j)/pS(Y = j), also known as the importance weights (IWs), and aim for a weaker importance-weighted feature alignment, pw∗,ZS = p Z T , where
pw ∗ S (x, Y = j) = pS(x|Y = j)pS(Y = j)w∗j = pS(x|Y = j)pT (Y = j). (1) Besides as shown in Fig. 5 that importance-weighted feature alignment also correlates with transfer performance, it is also a necessary condition for the ideal goal of class-conditional alignment (Tachet des Combes et al., 2020).
Since the true w∗ are unknown under UCL, they are estimated by matching first-moment statistics using the confusion matrix (Saerens et al., 2002; Lipton et al., 2018). Assume now that the features are conditionally aligned, pZ|Y =jS = p Z|Y =j T for all j ∈ Y (known as the label shift assumption), and the label space is discrete, Y = {1, · · · , k}. Then for each class i ∈ [k], we have the following relation between predicted target prior distribution, the source confusion matrix, and w∗: pT (Ŷ = i) = k∑
j=1
pT (Ŷ = i|Y = j)pT (Y = j) = k∑
j=1
pS(Ŷ = i|Y = j)pT (Y = j) = k∑
j=1
pS(Ŷ = i, Y = j)w ∗ j ,
(2)
where Ŷ := h(g(X)) are label predictions. Since we have the same number of equations as unknowns, w∗ could be recovered under mild assumptions (Lipton et al., 2018).
Careful readers may have noticed that achieving either goal above requires having the other as an assumption. Indeed, attempting to estimate IW without conditional alignment could fail, as well as alignment without accurate IWs. The hope is that the optimization will be predominantly guided by the implicit bias of the pre-trained multilingual LM for selecting largely invariant features (hence conditionally aligned), so that small improvements to feature alignment and updates to prior shift estimates could bring incremental improvements to both components.
Optimization. The two goals above define an optimization problem over g, h and w,
min g,h
( E(x,y)∼pS [`(h(g(x)), y)] + λD(p w,Z S , p Z T ) ) , min
w ∑ ŷ∈[k] ( pT (ŷ)− ∑ y pS(ŷ, y)wy )2 ,
where alignment objective minimizes source task loss and the discrepancy of w-importanceweighted feature distributions (the metric should be more fine-grained than the first-moment metric used in our analysis), and the IW estimation objective solves eq. 2 using current statistics based on h, g with the constraints w ≥ 0 and w>pYS = 1. In our implementation, we use the minimax formulation of the alignment objective optimized with adversarial training, where D is replaced by a shallow ReLU network Wasserstein-1 critic and zero-centered gradient penalty (Arjovsky et al., 2017; Thanh-Tung et al., 2019). We found the Wasserstein-1 adversarial loss to be more responsive to importance weighting than cross-entropy loss (Ganin et al., 2016).
Improving Stability of IW Estimates. In Tachet des Combes et al. (2020), the IWs estimation objective is solved by quadratic programming (QP), and updated once every epoch of training in their implementation. Since accurate IWs estimates are integral to successful adaptation, we (1) update w using projected gradient descent at every training step for maintaining their optimality. This, however, led to the discovery of previously unexplored practical issues with the IW estimator. In our preliminary experiments, we observed that w tends overestimate the majority class on the target domain, and diverges during longer training episodes.
These issues are unique to unsupervised adaptation settings, and are due to inefficiencies of the critic and noises introduced by the alignment objective. They could also be reproduced with the previous QP procedure. We address these issues by (2) `2-regularizing w (Azizzadenesheli et al., 2018), and (3) using a decaying learning rate schedule for w. These modifications are necessary for achieving best performance, as long training episodes are required for fine-tuning on many NLP tasks, and also needed to maintain the critic near optimality for guiding properly the alignment.

heading: 4 EXPERIMENTS

text:
In our experiments, we compare importance-weighted domain alignment (IWDA) to two baseline methods for cross-lingual transfer on three NLP tasks, under the unsupervised setting where access to unlabeled task data from the target language is provided (UCL setting). Besides supporting our findings and our motivated algorithm, these experiments are aimed at investigating the limitations of existing approaches under distributional shifts in prior distributions between source and target. We also illustrate the potential benefits of IWDA in light of stronger cross-lingual learning methods.
Datasets. The tasks considered are named-entity recognition (NER), sentiment analysis, and textual entailment. NER: Two benchmark multilingual datasets are used, CoNLL (Tjong Kim Sang, 2002; Tjong Kim Sang & De Meulder, 2003) and WikiANN (Pan et al., 2017; Rahimi et al., 2019). The former includes English, German, Spanish and Dutch, and the latter covers over 200 languages, from which we select a subset of 40 according to the XTREME benchmark (Hu et al., 2020). Sentiment analysis: Multilingual Amazon Reviews Corpus (MARC) is used (Keung et al., 2020), a balanced dataset with product reviews in English, German, Spanish, French, Japanese and Chinese. Textual entailment: The XNLI dataset is used (Conneau et al., 2018), but due to space constraints, its setup and results are relegated to Appendix B. As in most work on cross-lingual transfer, English is used as the source language for transfer in our evaluations.
Baselines. We perform extensive comparative experiments with two semi-supervised learning (SSL) techniques. Both require having a model fine-tuned on source data only. Knowledge distillation (KD): It distills and transfers knowledge of the source teacher model to the target domain by training a newly initialized student network to mimic teacher outputs on target data (Hinton et al., 2015). A variant of KD was proposed for NER by Wu et al. (2020), which also freezes the bottom three layers of mBERT during training (Wu & Dredze, 2019). Self-training (ST): It iteratively assigns pseudo-labels to unlabeled target data and train on the most confident ones. Our implementation follows closely the work by Dong & de Melo (2019), which considered sequence classification.5
For IWDA experiments, the model is first fine-tuned on source only, then continue fine-tuned with the IWDA objective on both source and unlabeled target data. We also include results for when the true IWs are known and hard-coded as an upper bound on IWDA performance (IWDA (oracle)).
Experiments are performed on BERT-Base Multilingual Cased and XLM-R Large models. More details on IWDA and our baselines are included in Appendices C and D along with hyperparameter settings. Unless otherwise noted, results are from our implementation.
5These baselines are chosen because they require no more resources than the unlabeled target data provided, are not tailored to any particular model, and can be easily implemented on any existing models, which are qualities shared by IWDA. As many elaborate methods are also based on the same fundamental ideas as these baselines, we hope that they are representative of approaches based on SSL, if not exhaustive.

heading: 4.1 RESULTS ON STANDARD BENCHMARKS

text:
To begin, through evaluations on standard benchmarks, we demonstrate that IWDA brings performance gains via improving representations to be better-aligned across languages. Results on CoNLL and low-resource WikiANN NER are presented in Tables 1 and 2, and those on MARC in Table 3. Remaining results on WikiANN with mBERT are included in Table 4 in the appendix.
In all mBERT IWDA experiments, improvements upon zero-shot baseline are observed, and are accompanied by improved representation alignment. The average decrease in conditional feature shift is 21.60% and 22.33% on CoNLL with IWDA and IWDA (oracle) respectively (with 100% indicating perfect alignment), 51.53% and 64.77% on MARC, and 32.91% and 36.34% on WikiANN.
However, improvements sometimes fall short of SSL baselines, which we suspect is due to the noise introduced during the alignment process. Potential sources include fluctuations in IW estimates and insufficiently trained critics. By training IWDA with more passes on the data and improving the optimality of the critic, further improvements on CoNLL NER are observed (Table 6).
Another way to reduce noise is to perform a final-stage SSL training on the IWDA-aligned model. These experiments are labeled IWDA + KD and IWDA + ST, on which we observe some best results with mBERT on both tasks. This is an appealing strategy, because SSL algorithms theoretically depend on the assumption that the source and target domains are the same, IWDA training can be viewed as a preprocessing step on the feature representations to satisfy this assumption. We will revisit this point in the next section where we consider cross-lingual transfer under prior shifts.
On XLM-R models, IWDA also brings improvements on NER. However on MARC, gains are only observed for languages distant from English. Here, we remark that the performance gain from IWDA (if any) will come from enhancements to the alignment of last-layer features in their current configurations, but not from learning entirely different representations with fundamentally better class-conditional alignment. This cannot be achieved absent target labels, additional supervision or resources. If the representations learned by the powerful XLM-R model during zero-shot training are already efficient, then the potentials of IWDA will be limited. Opportunities for further improvement in this case could come from additional resources, e.g. Bari et al. (2021) perform unsupervised data augmentation by leveraging the multilingual mask-filling ability of XLM-R.

heading: 4.2 RESULTS UNDER CLASS PRIOR SHIFTS

text:
In this section, we evaluate on datasets with large prior shifts, a likely property of real-world data that is insufficiently investigated for UCL, and contrast the brittleness of SSL baselines with the robustness of IWDA. Because of the lack of multilingual benchmarks exhibiting prior shifts, similar to Section 2.2 , we evaluate on smaller datasets subsampled from the German portion of CoNLL and the Japanese portion of MARC according to varying amounts of shifts.
Results plotting percent improvements over zero-shot baselines against the amount of prior shift are presented in Fig. 4. Although SSL baselines are generally better than IWDA under mild to no shifts,
their performances quickly deteriorate as prior shift increases, until eventually resulting in negative transfer. IWDA on the other hand does not cause negative transfer from prior shifts owing to its estimation and correction capabilities. The relative success of IWDA demonstrates that it is possible to estimate prior shift from unlabeled target data. We include XLM-R MARC results in Fig. 10.
In mBERT experiments, the average percent correction of prior shift, computed from the decrease in TV between true target domain priors and end-of-training estimates, normalized by the amounts of shift, is 87.33% on CoNLL (with 100% indicating perfect IW estimation), and 28.08% on MARC under shifts larger than 0.8 in TV. On MARC with smaller shifts, we observe estimates that are worse than assuming a uniform prior distribution (-335.80% in correction). This is because these are harder problems with noisy labels (as it is naturally difficult to distinguish a 2-star review from a 3-star one), the suboptimality of the source model from which IWDA is trained means that the initial representations may be conditionally misaligned, thereby affecting the accuracy of the initial as well as subsequent IW estimates.
Since IWDA brings improvements to class-conditional alignment, as already pointed out in the previous section, it is appealing to complement SSL and potentially more elaborated methods with IWDA to improve robustness against prior shift. As demonstrated in our IWDA + KD and ST experiments, SSL no longer suffer from negative transfer when preceded by IWDA, and altogether provide further performance gains.

heading: 5 CONCLUSION AND FUTURE WORK

text:
From the observations on multilingual neural LMs that cross-lingual transfer performance is strongly correlated with representation invariance and negatively affected by class prior shift, we proposed importance-weighted domain alignment (IWDA) for unsupervised cross-lingual learning. Experiment results support our findings, and show that IWDA is effective at the estimation and correction of class prior shift from unlabeled data, and robust to large shifts, under which in contrast existing UCL approaches based on SSL could result in negative transfer. As prior shift is a prevalent issue under unsupervised settings, more care to it is necessary in the design of future UCL algorithms for them to achieve good performance not only on curated benchmarks, but also on real-world data.
While our present implementation of IW estimation with the moment-matching approach is largely effective, future work on the search and study of potentially more stable estimation methods would enable IWDA to achieve better and more consistent transfer performance. Since the strength of IWDA lies in its ability to detect prior shifts and improve on class-conditional representation alignment, with our experiments on combining IWDA and SSL baselines as a preliminary demonstration of its potential usefulness as a feature preprocessing step and in improving robustness to prior shifts, it would be of interest to investigate its benefits on stronger models and methods towards better cross-lingual transfer.

heading: A ADDITIONAL ANALYSES AND FINDINGS FROM SECTION 2

text:
In this section, we provide additional results to support our findings by considering more languages (WikiANN NER), a higher-level task (XNLI), and a more powerful model (XLM-R Large).
First and foremost, to demonstrate that the correlation between class-conditional representation alignment and performance is not confounded by the clustering structure induced by the linear classifier trained on last-layer features, we plot and observe similar correlation between transfer performance and class-balanced representation alignment in Fig. 5.
WikiANN. As an attempt to minimize the effects from potential confounders on the empirical findings in the main sections, we perform the same experiments for studying the relation between transfer performance, representation alignment and prior shift by zero-shot transferring mBERT from English to 39 target languages included in the WikiANN NER dataset. The results are provided in Fig. 6. These languages have distinct linguistic properties and varying degrees of resourcefulness. A summary of them can be found in Table 5 of Hu et al. (2020), and sizes of their pretraining corpora in Table 1 of Wu & Dredze (2020).
It is observed that the same correlation between representation alignment and performance still holds, but the effects of prior shift are mostly mild. We suspect that this is because the task we evaluated on, named-entity recognition, is a structured prediction problem, hence mBERT’s ability to leverage dependencies between words made it robust to prior shifts on these tasks. We later observed that if the source dataset on which the model is trained on has a skewed prior distribution, then the effects of prior shift will be pronounced. The mBERT WikiANN results under this setting are shown in Fig. 9b, and more discussions are coming up in Section A.1.
XNLI. So far, we have seen empirical evidence on sentiment analysis and NER tasks, and one may wonder if the same conclusions still hold when considering cross-lingual transfer to high-level semantic tasks. For thoroughness, we include a plot in Fig. 7 comparing zero-shot transfer performance to representation alignment when mBERT is trained on a textual entailment task, from English to 14 languages included in the XNLI dataset. Here, the same observations are made, demonstrating that the findings hold across NLP tasks.
XLM-R. Another variable in our empirical analyses is the model used for evaluation. While the lightness of mBERT afforded us to perform extensive experiments, it would be of practical interest to see whether our findings also hold on larger and more powerful models. In Fig. 8, we compared performance to alignment and prior shift on XLM-R Large model on MARC. Again, the observations support our findings, and show that they also hold across models.
A.1 EFFECTS FROM SKEWED SOURCE PRIOR DISTRIBUTIONS
As briefly mentioned above, we have only considered the cross-lingual transfer from relatively balanced source datasets in previous sections. Here, we show that when the model is fine-tuned on source datasets with skewed prior distributions, the effects of class prior shift are more pronounced.
We first experiment on MARC. We generate an skewed English source dataset via subsampling according to the prior distribution of 14.86%, 6.43%, 8.03%, 19.28% and 51.41%6, corresponding to the proportion of 1- through 5-star reviews. Then, we repeat the previous empirical analysis by evaluating an mBERT fine-tuned on this skewed English dataset, and the results are shown in Fig. 9a. Compared to when the source is class-balanced (Fig. 3), here the correlation between performance and class prior shift is much stronger.
On WikiANN, although the original English portion of the dataset is already considerably skewed (with class O makes up 50.75% of the labels), the correlation between prior shift and model performance is weak (Fig. 6b), likely due to NER being a structured prediction problem where the dependencies between words in the same sentence helped the model to be more robust. Repeating the same experiment above with the skewed English source dataset generated by keeping only sentences containing tokens with an O label (so that now 80.92% of the tokens belong to class O), we again observe pronounced effects of prior shift on performance (results in Fig. 9b).
6This is the actual distribution of product ratings on Amazon.com.

heading: B ADDITIONAL EXPERIMENT RESULTS FROM SECTION 4

text:
By comparing the transfer performance of IWDA to SSL on standard benchmarks as well as datasets exhibiting prior shifts in the main sections, we demonstrated that better performance can be achieved with better representation alignment under the UCL setting, as well as the limitations with existing approaches under prior shifts. While the prior shifts experiments in Section 4.2 are performed on mBERT, in Fig. 10, we provide XLM-R Large results on MARC Japanese prior shift experiments.
Furthermore, we provide full mBERT WikiANN results of applying IWDA (Table 4), and results on the XNLI dataset, a high-level semantic task of textual entailment.
XNLI. Because XNLI does not provide human-generated training data for languages other than English, to obtain unlabeled target language task data for our studies on the UCL setting, we use the machine-translation of the English training set provided by Conneau et al. (2020a). To remove potential effects due to parallelism, similar to the setup considered in Bari et al. (2021), we partition the dataset into equal portions of size 20,000 (5% of the original size) and randomly select disjoint partitions to use as source and target data. In addition, since the original XNLI test set is humantranslated but our target data is machine-translated, to avoid the effects from this discrepancy (or domain shift) between train and test, evaluations are performed on the machine-translated portion of the dataset in our experiments (we refer interested readers to Artetxe et al. (2020a) for a discussion on XNLI translation artifacts).
Due to our setup, our results are non-benchmark and cannot be compared to models evaluated on the original XNLI test set. However, the goal of this experiment is to study whether our principle of achieving domain alignment is beneficial on high-level tasks.
The results are included in Table 5. It is observed that IWDA generally improves transfer performance upon the zero-shot baseline, although the gains are not significant. As discussed in Section 4.1, the potentials of IWDA will be limited if the zero-shot representations are already largely aligned, which is reflected by the small performance gap between zero-shot and supervised results in our setup.
IWDA vs. DA. Besides demonstrating that SSL suffers from negative transfer under large prior shifts in our experiments in Section 4.2, we also discussed in Section 2.2 that domain alignment without prior shift correction leads to negative transfer.
In Fig. 11, we compare the performance on benchmark CoNLL German, which exhibits a prior shift from CoNLL English, from applying IWDA and domain alignment without prior shift correction (DA). CDAN refers to the conditioning technique by Long et al. (2018), which we use in our IWDA implementation by default. It is observed that without prior shift correction, as training progresses,
transfer performance of vanilla DA deteriorates as the model makes more and more erroneous feature alignments caused by the shift.
Better Training of Adversary. As briefly mentioned in Section 4, while the IWDA models for the experiments in the main sections are trained for four epochs with simultaneous adversary and model updates for fair comparisons with SSL techniques, better performance could be obtained from IWDA if the optimality of the critic (adversary) can be better maintained throughout training with more passes over the data and alternating updates.
In Table 6, we observe that IWDA delivers much-improved performance on CoNLL when we perform four adversary updates per model update and allow 40 passes over the dataset (we also set lambda iw to 1). Although this setting would lengthen the training time, this could be of practical interest for getting the best IWDA performance.
C IMPLEMENTATION DETAILS OF IWDA
A high-level description of IWDA was given in Section 3 with discussions that connect it to the motivating findings in Section 2. This section provides more details on the implementation of the algorithm along with the hyperparameter settings7 used in our experiments (Table 7).
First, recall the joint objective, reproduced below, which consists of an alignment objective and an importance weight (IW) estimation objective,
min g,h
( E(x,y)∼pS [`(h(g(x)), y)] + λD(p w,Z S , p Z T ) ) , min
w ∑ ŷ∈[k] ( pT (ŷ)− ∑ y pS(ŷ, y)wy )2 . (3)
Note that the alignment objective contains the discrepancy between (important-weighted) source and target feature distributions and source task loss, added to prevent g from degenerating to a trivial representation with perfect source and target alignment, e.g. constant maps x 7→ c.
Alignment Objective. We reformulate the alignment objective in eq. 3 as a minimax objective, and replace explicit evaluation of the discrepancy D with a critic function f (also known as discriminator or adversary; Ganin et al. 2016; Goodfellow et al. 2014). This turns the alignment objective into
min g,h
( E(x,y)∼pS [`(h(g(x)), y)] + λmax
f∈F ( Ex∼pw,XS [`ad(f(g(x)), 0)]− Ex∼pXT [`ad(f(g(x)), 1)]︸ ︷︷ ︸
(4)
)) ,
where `ad is a specific choice of adversarial loss accompanied by an adversary function class F , that takes as input the scores computed by the adversary and the domain labels (here, source domain is labeled 0). The role of the adversary is to distinguish features computed from inputs sampled on the source domain from those on the target domain, and output a score on each feature that reflects its domain identity. When the features are aligned, the adversary would not be able to distinguish source and target features.
In our implementation, we use Wasserstein-1 loss for `ad, whose corresponding adversary function classF is 1-Lipschitz functions (Arjovsky et al., 2017). It is shown that when f attains its maximum, the term (4) exactly computes the Wasserstein-1 distance between the importance-weighted feature
7We use the same settings on all IWDA experiments. For methodological discussions on hyperparameter tuning for unsupervised cross-lingual learning, see Artetxe et al. (2020c).
distributions, pw,ZS and p Z T . We parameterize F as a shallow ReLU network of width 2048, and enforce the Lipschitz condition using soft constraints in the form of zero-centered gradient penalty, which penalizes the gradient of f between pairs of source and target features (Thanh-Tung et al., 2019).
Last but not least, for added stability, we condition the alignment of features on their (pseudo-)classes as proposed by Long et al. (2018) (CDAN). This is implemented as follows. Instead of feeding the adversary directly with features output by g, it receives the outer product between features and classifier output distributions, g(x)⊗h(g(x)). This means that the input dimension of f is increased to the dimension of features multiplied by the number of output classes k. When k is large, Long et al. (2018) used random projections to reduce dimensionality, which we did not explore.
Putting everything together—minimax reformulation, Wasserstein-1 loss, zero-centered gradient penalty, and conditional alignment—upon receiving a mini-batch of source labeled examples, (xi, yi) n i=1, and target unlabeled examples, (x ′ j) m j=1, the empirical alignment loss is
1
n n∑ i=1 `(h(g(xi)), yi) + λ
( 1
n n∑ i=1 f(g(xi)⊗ h(g(xi)))− 1 m m∑ j=1 f(g(x′j)⊗ h(g(x′i))) ) − µ ñ ñ∑ i=1 ‖(∇f)(z̃i)‖2,
where ñ = min(n,m) and z̃i is a random point on the line connecting g(xi) and g(x′i). Here, we have two hyperparameters: λ (lambda da) controls the strength of alignment, and µ (lambda gp) controls the strength on enforcing the Lipschitz condition. The learning rate of the adversary (lr ad) is set to be larger than that of the model (lr; Heusel et al. 2017). AdamW optimizer is used throughout our experiments.
On this loss, we back-propagate gradients to f on maximizing the second and third terms, g on minimizing the first two terms, and h on minimizing the first term only. In our experiments in the main sections, all three models are optimized with simultaneous updates, but we note that alternating updates (standard in GAN literature) could allow IWDA to deliver further performance gains with longer training time. In Table 6, we observe improved mBERT CoNLL results when the training alternates between four updates to f and one update to g for 40 epochs.
IW Estimation Objective. We proposed several modifications to the IW estimation objective in Section 3 to improve its stability, and one of them is adding `2-regularization to w (Azizzadenesheli et al., 2018).
Given first moment statistics in the form of the source confusion matrix, Cij := pS(Ŷ = i, Y = j), and target output distribution, νi := pT (Ŷ = i), the regularized IWs objective minimizes
k∑ i=1 ( νi − k∑ j=1 Cijwj )2 + ξ k∑ i=1 (wi − w(0))2, (4)
where w(0) is an initial IW estimate (set to all-ones vector by default, i.e. no prior shift). Here, one hyperparameter that controls the strength of the regularization is introduced, ξ (lambda iw). For simplicity, the `2-regularization is implemented with and documented as weight decay. Hence, ξ equals to lambda iw multiplied by 2 times the learning rate of w.
Instead of initializing w(0) = 1, because IWDA fine-tuning is preceded by zero-shot fine-tuning in our experiments and its decent performance reflects partial class-conditional feature alignment, we initialize it to a crude IW estimate w′ obtained from the zero-shot model using the QP by Tachet des Combes et al. (2020). Since the initial alignment will not be perfect and w′ not entirely reliable, we set w(0) = (1− ρ)w′ + ρ1, where ρ (lambda iw init) is a hyperparameter. The statistics C, ν evolve during training as the feature representation computed by g changes, so it needs to be updated throughout the learning process. Re-evaluating them at each optimization step
on a held-out set would make training prohibitively slow, hence we update them with the statistics Ĉ, ν̂ computed on training mini-batches: C(t) = (1 − η)C(t−1) + ηĈ, and similarly for ν(t). This update rate η (lr iw stats) is not the same as IW learning rate (lr iw).
Lastly, to minimize potential effects from the miscalibration of neural models (Guo et al., 2017; Alexandari et al., 2020), hard confusion matrix and target output distribution are used in the estimation of IWs (Lipton et al., 2018; Garg et al., 2020).

heading: D DESCRIPTION OF SSL BASELINES

text:
Knowledge Distillation (KD). Our implementation follows the original paper by Hinton et al. (2015) closely (from which the Wu et al. (2020) variant is moderately different). Given a teacher LM fine-tuned either under zero-shot or using IWDA, we distill its knowledge on a newly initialized pre-trained student LM by minimizing the mean-squared-error difference of the teacher and student Softmax outputs on source and target data without labels. If labeled data from the source is available, we also train the student to minimize source task loss.
We set the temperature to 3, and the weight of the source task loss to 0.1. Learning rate is 1e-4, batch size is 32, and we train for 4 epochs.
In addition, Wu et al. (2020) observed that when distilling to pretrained LMs, better performance can be achieved if the bottom three layers of the student is frozen (motivated by findings from Wu & Dredze (2019)). We also apply this technique in our KD experiments.
Self-Training (ST). Our implementation follows Dong & de Melo (2019) closely. Given an LM fine-tuned on source domain only, ST continues fine-tuning the model by first assigning pseudolabels to target data, collecting the ones with the highest confidence, and then train on them with the pseudo-labels. Source task loss is also minimized during self-training.
The hyperparameters of ST involve selecting how any pseudo-labeled examples to train on. On MARC with 200,000 training examples, at each epoch, we select top 4,000 most confident pseudolabeled examples from the target domain conditioned on each (pseudo-)class, so that 20,000 examples are selected each round (MARC has five classes). These numbers are scales proportionally on datasets of other sizes. We perform self-training for 4 epochs, and set learning rate to 1e-5 and batch size to 128.
On IWDA + ST prior shift experiments, we select much less examples for self-training to avoid training on low-confidence examples from minority classes. This adjustment of hyperparameters relies on knowledge about the prior distribution on the target domain, which is approximated by the IW estimates from IWDA.