summaries: This paper investigates a method for improving the cross-lingual transfer of pretrained multilingual models. The paper first empirically analyzed the influence of representation invariance and distributional class shift. Then, the paper proposed a method to improve the representation invariance and correcting the class shift. Experiments showed its superiority under large prior shifts. In this paper, the authors provide substantial analyses on the cross-lingual transfer performance in the multilingual neural language models and reported that the performance is strongly correlated with representation invariance and negatively affected by distributional shift in class priors between data in the src/tgt languages. Based on these findings, the authors propose an unsupervised cross-lingual learning method, called importance-weighted domain adaptation (IWDA), where it performs feature alignment, prior shift estimation, and correction. The authors experimented on two different NLP tasks such as multilingual NER and multilingual sentiment analysis tasks, and experimentally showed the effectiveness. Besides that, they demonstrated that the proposed approach improves performance further, when combined with existing semi-supervised learning approaches. This paper looks at the problem of unsupervised cross-lingual transfer (termed UCL) in the paper through the optics of domain adaptation. After empirically analysing and validating that distributional shifts in class priors might cause a huge problem for UCL (which wasn't tackled in previous research), the authors proceed with an introduction of a new method that aims to mitigate that problem. The idea is to get rid of that shift through a approach called importance-weighted domain adaptation (IWDA), which is largely the adaptation of the work from Tachet des Combes et al. (NeurIPS 2020) to the UCL problem.  The results on two tasks in the UCL setup (NER and MARC classification) show slight gains over the standard zero-shot transfer when IWDA is applied, with more prominent gains reported when a stronger domain shift is observed - however, such a setup has been created mostly artificially, to further demonstrate the benefits of modelling the shift in the model. The paper first demonstrates the importance of feature invariance (language invariant representations) and class-prior invariance across languages on zero-shot cross-lingual performance. By analyzing the zero-shot performance on different languages on the MARC reviews and WikiANN NER tasks, and comparing against the class conditional distance between the source language (English) and target language feature representations, the authors illustrate the how high feature invariance results in better zero-shot performance. By also synthetically modifying the class prior for the target language, the authors demonstrate how increasing differences in class priors result in decreasing zero-shot cross-lingual transfer performance.  Building on their observations, the authors propose an approach that: (i) Introduces an adversarial loss term to penalize distortion in average class conditional feature representations between the source and target languages. (ii) Adds an importance weighting term to ensure the approach doesn't fail under class prior shifts.  Empirical studies demonstrate that the proposed approach improves significantly over the vanilla zero-shot model on both MARC sentiment analysis and NER tasks, also improving over self-training on sentiment analysis. Comparisons on synthetic datasets (sub-sampled from NER and MARC datasets) that enhance the class prior shift highlight the robustness of the approach under large class prior shifts where previous approaches fail.
abstractText: Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, evidenced by their decent zero-shot transfer performance. While studies have attributed this success to cross-lingually shared representations, quantitative analyses are sparse. Towards a better understanding of the role of multilingual representations, in this work, we first make the following observations through empirical analysis: (1) invariance of the feature representations strongly correlates with transfer performance, and (2) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work. Based on our findings, we propose an unsupervised cross-lingual learning method, called importance-weighted domain alignment (IWDA), that performs representation alignment, prior shift estimation, and correction. Experiment results demonstrate its superiority under large prior shifts. In addition, our method delivers further performance gains when combined with existing semi-supervised learning techniques.
references.authors: Amr Alexandari Anshul Kundaje Avanti Shrikumar
references.title: Maximum Likelihood with BiasCorrected Calibration is Hard-To-Beat at Label Shift Adaptation
references.venue: In International Conference on Machine Learning,
references.authors: Naveen Arivazhagan Ankur Bapna Orhan Firat Roee Aharoni Melvin Johnson Wolfgang Macherey
references.title: The Missing Ingredient in Zero-Shot Neural Machine Translation
references.authors: Martin Arjovsky Soumith Chintala Léon Bottou
references.title: Wasserstein Generative Adversarial Networks
references.venue: In International Conference on Machine Learning,
references.authors: Mikel Artetxe Holger Schwenk
references.title: Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Mikel Artetxe Gorka Labaka Eneko Agirre Kyunghyun Cho
references.title: Unsupervised Neural Machine Translation
references.venue: In International Conference on Learning Representations,
references.authors: Mikel Artetxe Gorka Labaka Eneko Agirre
references.title: Translation Artifacts in Cross-lingual Transfer Learning
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Mikel Artetxe Sebastian Ruder Dani Yogatama
references.title: On the Cross-lingual Transferability of Monolingual Representations
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Mikel Artetxe Sebastian Ruder Dani Yogatama Gorka Labaka Eneko Agirre
references.title: A Call for More Rigor in Unsupervised Cross-lingual Learning
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Kamyar Azizzadenesheli Anqi Liu Fanny Yang Animashree Anandkumar
references.title: Regularized Learning for Domain Adaptation under Label Shifts
references.venue: In International Conference on Learning Representations,
references.authors: M Saiful Bari Tasnim Mohiuddin Shafiq Joty
references.title: UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP
references.venue: In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1978–1992,
references.authors: Shai Ben-David John Blitzer Koby Crammer Fernando Pereira
references.title: Analysis of representations for domain adaptation
references.venue: Advances in Neural Information Processing Systems,
references.authors: Steven Cao Nikita Kitaev Dan Klein
references.title: Multilingual Alignment of Contextual Word Representations
references.venue: In International Conference on Learning Representations,
references.authors: Xilun Chen Yu Sun Ben Athiwaratkun Claire Cardie Kilian Weinberger
references.title: Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Alexis Conneau Guillaume Lample
references.title: Cross-lingual Language Model Pretraining
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Alexis Conneau Guillaume Lample Ruty Rinott Adina Williams Samuel R. Bowman Holger Schwenk Veselin Stoyanov
references.title: XNLI: Evaluating Cross-lingual Sentence Representations
references.venue: [cs],
references.authors: Alexis Conneau Kartikay Khandelwal Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzmán Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov
references.title: Unsupervised Cross-lingual Representation Learning at Scale
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Alexis Conneau Shijie Wu Haoran Li Luke Zettlemoyer Veselin Stoyanov
references.title: Emerging Crosslingual Structure in Pretrained Language Models
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
references.authors: Xin Dong Gerard de Melo
references.title: A Robust Self-Learning Framework for Cross-Lingual Text Classification
references.venue: In Q,
references.authors: Haim Dubossarsky Ivan Vulić Roi Reichart Anna Korhonen
references.title: The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Philipp Dufter Hinrich Schütze
references.title: Identifying Elements Essential for BERT’s Multilinguality
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4423–4437,
references.authors: Yaroslav Ganin Evgeniya Ustinova Hana Ajakan Pascal Germain Hugo Larochelle François Laviolette Mario March Victor Lempitsky
references.title: Domain-Adversarial Training of Neural Networks
references.venue: Journal of Machine Learning Research,
references.authors: Saurabh Garg Yifan Wu Sivaraman Balakrishnan Zachary Lipton
references.title: A Unified View of Label Shift Estimation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: Advances in Neural Information Processing Systems,
references.authors: Arthur Gretton Karsten M. Borgwardt Malte J. Rasch Bernhard Schölkopf Alexander Smola
references.title: A Kernel Two-Sample Test
references.venue: Journal of Machine Learning Research,
references.authors: Chuan Guo Geoff Pleiss Yu Sun Kilian Q. Weinberger
references.title: On Calibration of Modern Neural Networks
references.venue: In International Conference on Machine Learning,
references.authors: Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler Sepp Hochreiter
references.title: GANs trained by a two time-scale update rule converge to a local nash equilibrium
references.venue: Advances in Neural Information Processing Systems,
references.authors: Geoffrey Hinton Oriol Vinyals Jeff Dean
references.title: Distilling the Knowledge in a Neural Network
references.authors: Junjie Hu Sebastian Ruder Aditya Siddhant Graham Neubig Orhan Firat Melvin Johnson
references.title: XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
references.venue: In International Conference on Machine Learning,
references.authors: Junjie Hu Melvin Johnson Orhan Firat Aditya Siddhant Graham Neubig
references.title: Explicit Alignment Objectives for Multilingual Bidirectional Encoders
references.venue: In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3633–3643,
references.authors: Lifu Huang Heng Ji Jonathan May
references.title: Cross-lingual Multi-Level Adversarial Transfer to Enhance Low-Resource Name Tagging
references.authors: Shafiq Joty Preslav Nakov Lluı́s Màrquez Israa Jaradat
references.title: Cross-language Learning with Adversarial Neural Networks
references.venue: In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL
references.authors: Karthikeyan K Zihan Wang Stephen Mayhew Dan Roth
references.title: Cross-Lingual Ability of Multilingual BERT: An Empirical Study
references.venue: In International Conference on Learning Representations,
references.authors: Phillip Keung Yichao Lu Vikas Bhardwaj
references.title: Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Phillip Keung Yichao Lu György Szarvas Noah A. Smith
references.title: The Multilingual Amazon Reviews Corpus
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4563–4568,
references.authors: Joo-Kyung Kim Young-Bum Kim Ruhi Sarikaya Eric Fosler-Lussier
references.title: Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources
references.venue: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,
references.authors: Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey Hinton
references.title: Similarity of Neural Network Representations Revisited
references.venue: In International Conference on Machine Learning,
references.authors: Guillaume Lample Alexis Conneau Marc’Aurelio Ranzato Ludovic Denoyer Hervé Jégou
references.title: Word translation without parallel data
references.venue: In International Conference on Learning Representations,
references.authors: Anne Lauscher Vinit Ravishankar Ivan Vulić Goran Glavaš
references.title: From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Juntao Li Ruidan He Hai Ye Hwee Tou Ng Lidong Bing Rui Yan
references.title: Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model
references.venue: In Twenty-Ninth International Joint Conference on Artificial Intelligence,
references.authors: Jindřich Libovický Rudolf Rosa Alexander Fraser
references.title: On the Language Neutrality of Pre-trained Multilingual Representations
references.venue: In Findings of the Association for Computational Linguistics: EMNLP
references.authors: Zachary Lipton Yu-Xiang Wang Alexander Smola
references.title: Detecting and Correcting for Label Shift with Black Box Predictors
references.venue: In International Conference on Machine Learning,
references.authors: Mingsheng Long Zhangjie Cao Jianmin Wang Michael I Jordan
references.title: Conditional adversarial domain adaptation
references.venue: Advances in Neural Information Processing Systems,
references.authors: Adyasha Maharana Mohit Bansal
references.title: Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension
references.venue: In Findings of the Association for Computational Linguistics: EMNLP
references.authors: Tomas Mikolov Quoc V. Le Ilya Sutskever
references.title: Exploiting Similarities among Languages for Machine Translation
references.venue: [cs],
references.authors: Benjamin Muller Yanai Elazar Benoı̂t Sagot Djamé Seddah
references.title: First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
references.venue: In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,
references.authors: Xiaoman Pan Boliang Zhang Jonathan May Joel Nothman Kevin Knight Heng Ji
references.title: Crosslingual Name Tagging and Linking for 282 Languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
references.authors: Barun Patra Joel Ruben Antony Moniz Sarthak Garg Matthew R. Gormley Graham Neubig
references.title: Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Jonas Pfeiffer Ivan Vulić Iryna Gurevych Sebastian Ruder
references.title: MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Ngoc-Quan Pham Jan Niehues Thanh-Le Ha Alexander Waibel
references.title: Improving Zero-shot Translation with Language-Independent Constraints
references.venue: In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers),
references.authors: Telmo Pires Eva Schlinger Dan Garrette
references.title: How Multilingual is Multilingual BERT
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Afshin Rahimi Yuan Li Trevor Cohn
references.title: Massively Multilingual Transfer for NER
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Sebastian Ruder
references.title: Neural Transfer Learning for Natural Language Processing
references.venue: PhD thesis, NUI Galway,
references.authors: Sebastian Ruder Barbara Plank
references.title: Strong Baselines for Neural Semi-Supervised Learning under Domain Shift
references.venue: In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
references.authors: Sebastian Ruder Ivan Vulić Anders Søgaard
references.title: A survey of cross-lingual word embedding models
references.venue: Journal of Artificial Intelligence Research,
references.authors: Marco Saerens Patrice Latinne Christine Decaestecker
references.title: Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure
references.venue: Neural Computation,
references.authors: Holger Schwenk Xian Li
references.title: A Corpus for Multilingual Document Classification in Eight Languages
references.venue: In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018),
references.authors: Samuel L. Smith David H.P. Turban Steven Hamblin Nils Y. Hammerla
references.title: Offline bilingual word vectors, orthogonal transformations and the inverted softmax
references.venue: In 5th International Conference on Learning Representations,
references.authors: Remi Tachet des Combes Han Zhao Yu-Xiang Wang Geoffrey J. Gordon
references.title: Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Hoang Thanh-Tung Truyen Tran Svetha Venkatesh
references.title: Improving generalization and stability of generative adversarial networks
references.venue: In International Conference on Learning Representations,
references.authors: Erik F. Tjong Kim Sang
references.title: Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
references.venue: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002),
references.authors: Erik F. Tjong Kim Sang Fien De Meulder
references.title: Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
references.venue: In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL
references.authors: Giorgos Vernikos Katerina Margatina Alexandra Chronopoulou Ion Androutsopoulos
references.title: Domain Adversarial Fine-Tuning as an Effective Regularizer
references.venue: In Findings of the Association for Computational Linguistics: EMNLP
references.authors: Xinyi Wang Hieu Pham Zihang Dai Graham Neubig
references.title: SwitchOut: An Efficient Data Augmentation Algorithm for Neural Machine Translation
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Qianhui Wu Zijia Lin Börje Karlsson Jian-Guang Lou Biqing Huang
references.title: Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Shijie Wu Mark Dredze. Beto Bentz
references.title: Becas: The Surprising Cross-Lingual Effectiveness of BERT
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP),
references.authors: Shijie Wu Mark Dredze
references.title: Are All Languages Created Equal in Multilingual BERT
references.venue: In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 120–130,
references.authors: Mengzhou Xia Guoqing Zheng Subhabrata Mukherjee Milad Shokouhi Graham Neubig Ahmed Hassan Awadallah
references.title: MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
references.venue: In Proceedings of the 2021 Conference of the North American Chapter
references.authors: 7523–7532. PMLR May 2019. Hao Zheng Mirella Lapata
references.title: Sentence Centrality Revisited for Unsupervised Summarization
references.title: We parameterize F as a shallow ReLU network of width 2048, and enforce the Lipschitz condition using soft constraints in the form of zero-centered gradient penalty, which penalizes the gradient of f between pairs of source and target features (Thanh-Tung
references.authors: Alexandari
references.title: 2020), hard confusion matrix and target output distribution are used in the estimation of IWs (Lipton et al., 2018
references.venue: Garg et al.,
references.authors: Wu
references.title: 2020) observed that when distilling to pretrained LMs, better performance can be achieved if the bottom three layers of the student is frozen (motivated by findings
sections.text: Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, evidenced by their decent zero-shot transfer performance. While studies have attributed this success to cross-lingually shared representations, quantitative analyses are sparse. Towards a better understanding of the role of multilingual representations, in this work, we first make the following observations through empirical analysis: (1) invariance of the feature representations strongly correlates with transfer performance, and (2) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work. Based on our findings, we propose an unsupervised cross-lingual learning method, called importance-weighted domain alignment (IWDA), that performs representation alignment, prior shift estimation, and correction. Experiment results demonstrate its superiority under large prior shifts. In addition, our method delivers further performance gains when combined with existing semi-supervised learning techniques.
sections.heading: 1 INTRODUCTION
sections.text: Many recent state-of-the-art results in natural language processing (NLP) have been achieved on Transformer-based deep neural language models (LMs) using the “pre-train then fine-tune” paradigm (Devlin et al., 2019; Conneau & Lample, 2019). Multilingual versions of these LMs are pre-trained on unannotated and unparallel texts from more than one language, such as multilingual BERT (mBERT) and XLM-R (Conneau et al., 2020a), and it is found that the learned representations are shared across languages. In particular, the cross-lingual knowledge acquired by these models has enabled zero-shot cross-lingual transfer, where after fine-tuned on data in a source language, not only do they work well when evaluated on the source, but also quite decently for the same task on almost all languages they have seen during pre-training (Wu & Dredze, 2019).
  Their success on zero-shot cross-lingual transfer has prompted numerous studies on their multilingual abilities. Among the many explanations, a recent hypothesis suggests that the deep representation combined with parameter sharing induced intermediate feature representations that are shared across languages (K et al., 2019; Conneau et al., 2020b; Muller et al., 2021). Following this, we begin by studying the following question for mutilingual neural LMs: What is the role of shared representations in unsupervised cross-lingual learning?
  In an effort to answer this question in a quantitative manner, through empirical analysis on mBERT, we find that (1) its cross-lingual transfer performance strongly correlates with the alignment of feature representations for the NLP task at hand between the source and target languages (Section 2.1). In addition, (2) distributional shift in class priors between data in the source and target languages negatively affects transfer performance (sections 2.2 and 4.2). Despite being a common issue with real-world data under unsupervised settings, it has been largely overlooked in prior work.
  Based on these findings on the cross-lingual transfer of mutilingual neural LMs, towards a principled approach for the unsupervised setting where access to unlabeled data in the target language for the downstream task at hand is available in addition to labeled ones in the source (hereafter referred
  to as UCL1), we propose importance-weighted domain alignment (IWDA). This method performs representation alignment, class prior shift estimation, and correction (Section 3). Experiment results show that IWDA outperforms the more popular semi-supervised learning methods under large prior shifts, and can be additionally combined with them for further performance gains (Section 4).
sections.heading: 1.1 RELATED WORK
sections.text: Our proposed method is heavily inspired by Tachet des Combes et al. (2020), which studied the issues with alignment-based domain adaptation (DA) under the presence of class prior shifts, and proposed the inclusion of prior shifts estimation and correction procedures in DA. Our method also shares similar components with the DA-based UCL method by Keung et al. (2019), while highlighting its limitations discovered in the present work (Section 2.2).
  Analyzing Multilingual Neural LMs. Most studies analyze mBERT’s cross-lingual ability via probing experiments, and identified influencing factors including but not limited to linguistic similarity, size of pretraining corpus, domain similarity, parameter sharing and model depth (Pires et al., 2019; K et al., 2019; Lauscher et al., 2020; Conneau et al., 2020b; Dufter & Schütze, 2020). Another line of research analyze intermediate representations and observed that mBERT learns crosslingually shared representations (Wu & Dredze, 2019; Conneau et al., 2020b; Muller et al., 2021).
  Cross-Lingual Representation Alignment. The idea of learning invariant representations has powered a long history of approaches for cross-lingual transfer on neural models. Under zero-shot settings, it is achieved by aligning word embeddings on bilingual dictionaries that could be induced via adversarial training (Mikolov et al., 2013; Smith et al., 2017; Zhang et al., 2017; Artetxe & Schwenk, 2019; Cao et al., 2019), a method that has powered many machine translation techniques (Artetxe et al., 2018; Lample et al., 2018; Arivazhagan et al., 2019; Pham et al., 2019; Hu et al., 2021) It is found on static embedding models that the goodness of alignment affects transfer performance (Patra et al., 2019; Dubossarsky et al., 2020). On deep models such as mBERT, it could also take the form of shared feature representations in the Transformer body (Artetxe et al., 2020b; Pfeiffer et al., 2020). Under UCL and supervised settings, techniques based on representation alignment were proposed for models ranging from distributed word embeddings (Joty et al., 2017; Chen et al., 2018), RNN (Kim et al., 2017; Huang et al., 2019), to mBERT (Keung et al., 2019; Zheng & Lapata, 2019; Xia et al., 2021). For a survey, see Ruder et al. (2019) and Ruder (2019).
  Connection to Domain Adaptation. Domain adaptation is a popular framework for handling domain shifts between training and test data, e.g. learning a POS tagger on news articles but to be deployed on biomedical papers with a significantly different word distribution (Ben-David et al., 2007). The concept of domain is extended naturally to cross-lingual learning by treating the distributions of sentence and document pertaining to the same task in each language as separate domains2 (Ruder, 2019). Thereby, adaptation techniques proposed for domain shift can be applied to UCL. Besides methods aiming at extracting shared representations (Li et al., 2020; Vernikos et al., 2020), another family of successful approaches is semi-supervised learning, which include self-training (Dong & de Melo, 2019), tri-training (Ruder & Plank, 2018), knowledge distillation (Wu et al., 2020), and data augmentation (Wang et al., 2018; Maharana & Bansal, 2020; Bari et al., 2021).
sections.heading: 2 FACTORS AFFECTING CROSS-LINGUAL TRANSFER
sections.text: To develop and provide evidence to the hypothesis that attributes the cross-lingual transfer success of multilingual neural LMs to shared representations, as well as to understand its implications on the design of unsupervised transfer algorithms, we attempt to quantify the relation between representation invariance and transfer performance in this section.
  We briefly summarize the “pre-train then fine-tune” training paradigm and introduce notation along the way. Let g : X → Z denote the feature mapping provided by the multilingual LM, and z =
  1With an abuse of notation. In this work, the acronym UCL refers to the learning setting considered, and is meant to distinguish from the umbrella term “unsupervised cross-lingual learning”, which also includes the study of the zero-shot setting where no target language task data is given.
  2Hence the terms domain and language are used interchangeably in this work.
  g(x) ∈ Z ⊆ Rd the feature vector computed on an input sentence or token x ∈ X . In practice as well as in our studies and experiments, the feature is the last-layer wordpiece embeddings for token classification tasks, and the embedding of a special start-of-sentence marker (symbolized as [CLS]) for sequence classification tasks. Hence, on BERT-Base models, d = 768 and Z ≈ Sd−1 sphere due to LayerNorm. Let h : Z → Y denote the output layer(s) added on top of g with an appropriate activation, usually a single linear layer with Softmax for classification tasks in practice. Hence, an end-to-end model is obtained, which computes predictions via ŷ = h(g(x)). By fine-tuning both g, h on labeled task data (x, y) sampled from the source language domain pS only, we get a zeroshot transfer model for the downstream task at hand, where the term zero-shot points to the fact that the model does not use any task data from the target language (x′, y′) ∼ pT during training. Based on the following decomposition of the joint feature-label distribution into the product of class-conditional feature distribution and marginal prior distribution,
  p(z, y) = p(z|y)p(y) := p({x ∈ X : g(x) = z}|y)p(y),
  we examine how cross-lingual transfer performance is influenced by (1) invariance of the feature representation of data between the source and target languages, with perfect invariance being pS(z|Y = j) = pT (z|Y = j) for all j ∈ Y , and (2) the distributional shift in the class priors, with no shift being pS(Y = j) = pT (Y = j) for all j ∈ Y . On classification tasks, Y = {1, 2, · · · , k} is discrete. For legibility in later sections, we use the shorthands to denote the class-conditional feature distribution and the marginal prior distribution: pZ|Y and pY .
  Setup. Our empirical analysis is performed on the representations learned by mBERT zero-shot fine-tuned on English for two multilingual tasks: sentiment analysis on the Multilingual Amazon Reviews Corpus (MARC), and named-entity recognition (NER) on the WikiANN dataset (see descriptions in Section 4). To simulate the effects of varying amount of representation misalignment and class prior difference, evaluation is performed on 500 smaller datasets randomly subsampled from the original MARC according to different class priors, and roughly 700 datasets from WikiANN. MARC covers five high-resource languages (excluding English), and (our selected subset of) WikiANN covers 39 languages of distinct linguistic properties and varying resourcefulness.
sections.heading: 2.1 INVARIANCE OF FEATURE REPRESENTATIONS
sections.text: For studying the effects of representation invariance, we compare model performance to the discrepancy between the class-conditional feature distributions on the source and target domains, pZ|YS and pZ|YT . We take the average to aggregate the discrepancies computed when conditioned on each class, 1 k ∑k j=1D(p Z|Y =j S , p Z|Y =j T ), and refer to this quantity by conditional feature shift.
  The discrepancy measure we use computes the `1-distance between the mean feature vectors3,
  D(p, q) := ∥∥∥Ex∼p[x]− Ex′∼q[x′]∥∥∥
  1 = d∑ i=1 ∣∣∣Ex∼p[xi]− Ex′∼q[x′i]∣∣∣. Measuring the distance between the means of representations for the purpose of analyzing mBERT has appeared in prior work. A similar quantity is referred to as language centroids by Libovický et al. (2020). The important distinction here is that we explicitly remove the effects of class prior shift by conditioning on the class labels, which we argue is an artifact of the dataset since arbitrary amounts of unconditioned feature shift can be generated by altering the class priors.
  First, we consider zero-shot transfer. In Fig. 1a, we plot theF1 score4 attained by the zero-shot model against conditional feature shift between English features and those of the target language, and observe that better representation alignment is indicative of better transfer performance. Findings in prior work on the cross-lingual ability of mBERT are reflected. For instance, linguistic similarity, a factor commonly argued to be important for cross-lingual transfer, could be related to the alignment
  3Comparisons to more sophisticated metrics such as RBF-kernel MMD and linear CKA were made in our preliminary studies (Gretton et al., 2012; Kornblith et al., 2019), and we found that they correlate tightly with our first-moment discrepancy measure for analyzing conditional feature shift.
  4To study the effects of class prior shift, we use macro-averaged F1 as the performance metric in this section, because classification accuracy could be inflated on datasets with skewed priors (Azizzadenesheli et al., 2018).
  of features: Spanish, French and German are more linguistically similar to English than Japanese and Chinese, whose features are less aligned with those of English. These results also reflects the successes of zero-shot learning approaches on mBERT that aim to achieve representation invariance.
  While the above finding shows that representation alignment is desirable for zero-shot transfer, is it also suitable for UCL settings where task data from the target language are available? We start by recognizing that the objective function of most unsupervised transfer algorithms consists of two parts: loss on labeled source data, and terms involving unlabeled data that in theory are linked to the loss on target domain. Based on this, we compared against conditional feature shift the performance of the model fine-tuned supervisedly on labeled data from source and target languages, as it would be the ideal objective for unsupervised algorithms. The results are plotted in Fig. 1b. Instead of observing increases in conditional feature shifts, which would have implied that language-specific features are preferred by over shared ones for good performance on each language, we observe much-improved representation alignments between English and each of the target languages. This means that achieving representation invariance is also appropriate under UCL settings.
  To demonstrate that the correlation between last-layer class-conditional alignment and performance is not an artifact of the linear classification head, we show that model performance is also correlated with (1) conditional feature shift in intermediate layers, plotted on the bottom panels of Fig. 1, and (2) class-balanced feature shift, plotted in Fig. 5 in the appendix, where the removal of prior shift effects is achieved by reweighting the class priors to be uniform (eq. 1). Furthermore, we include results on the XNLI dataset, a higher-level semantic task (Fig. 7), and results on MARC with the more powerful XLM-R Large model (Fig. 8).
sections.heading: 2.2 DISTRIBUTIONAL SHIFT IN CLASS PRIORS
sections.text: Most benchmarks for evaluating cross-lingual transfer are constructed so that the class prior distributions between data in the source and target languages are the same. This is desired for assessing zero-shot learning approaches, as the focus should not be on “‘tricks’ for how to best handle the class imbalance” (Schwenk & Li, 2018). However, for UCL settings with unlabeled target data, we argue that making the best use of the additional data involves the detection and correction of prior shift. Otherwise, as we show in this and later sections, negative transfer could occur.
  To examine the effects of class prior shift on cross-lingual transfer, we compare zero-shot transfer performance to the difference in prior distributions, D(pYS , p Y T ), equivalent to total variation (TV). The results are presented in Fig. 2. We also plot the effects of representation alignment and prior shift jointly in Fig. 3, where stronger correlations are revealed: (1) transfer performance generally takes a hit when prior shifts are large, regardless of how well the features are aligned, and (2) on target domains of the same amount of prior shift, performance decreases when features are less aligned. The findings suggest that in addition to achieving representation invariance, good transfer performance also requires the detection and handling of potential prior shifts. While this is not possible under zero-shot settings without any data from the target domain, the unlabeled target task data provided under UCL settings could be leveraged toward this goal.
  We examined existing approaches proposed for multilingual neural LMs for the UCL settings, but did not find procedures for prior shift correction, and moreover, there is a lack of attention to the potential issues and effects of class prior shift to the algorithms. Specifically, for alignment-based methods, Zhao et al. (2019) and Tachet des Combes et al. (2020) showed that adaptation could lead to negative transfer when prior shifts are not accounted for due to erroneous alignment. For methods based on semi-supervised learning, negative transfer is observed empirically under large shifts in our studies (Section 4.2). These cases of negative transfer call for more considerations to the effects of class prior shift in future work on cross-lingual transfer that leverage target data.
  Interestingly, although the alignment-based method proposed by Keung et al. (2019) did not account for prior shift, it avoided negative transfer on the CoNLL NER task (which exhibits shifts) by performing feature alignment on the [CLS] sentence embeddings instead of the word embeddings (recall that predictions are made on word embeddings on token classification tasks). Since sentence embeddings on Transformer-based models could be loosely related to a weighted sum of the word embeddings, the procedure of Keung et al. (2019) is performing coarse-grained representation alignment, which avoids negative transfer while providing decent performance gain. However, as we compare in Table 1, this misses out opportunities for larger gains from word embedding alignment. If alignment were performed on word embeddings without prior shift correction, deteriorating performance would have been observed (Fig. 11).
sections.heading: 3 IMPORTANCE-WEIGHTED DOMAIN ALIGNMENT
sections.text: For unsupervised cross-lingual learning on multilingual neural LMs with access to unlabeled target language data (which we refer to as the UCL setting), the findings in Section 2 suggest that good performance can be achieved via aligning representations and correcting class prior shifts. Based on these principles, we propose class-importance-weighted domain alignment (IWDA) for UCL, following closely the work by Tachet des Combes et al. (2020). In this section, we provide a highlevel description of the algorithm and its connection to the findings above, and refer interested readers to the aforementioned work and Appendix C for further implementation details.
  We have seen that class-conditional feature alignment is desired for cross-lingual transfer, however, this is not possible to achieve under the UCL setting since the target labels are unknown. Assume for now that the target class prior distribution is known, then we can compute the true ratio of the class priors, w∗j := pT (Y = j)/pS(Y = j), also known as the importance weights (IWs), and aim for a weaker importance-weighted feature alignment, pw∗,ZS = p Z T , where
  pw ∗ S (x, Y = j) = pS(x|Y = j)pS(Y = j)w∗j = pS(x|Y = j)pT (Y = j). (1) Besides as shown in Fig. 5 that importance-weighted feature alignment also correlates with transfer performance, it is also a necessary condition for the ideal goal of class-conditional alignment (Tachet des Combes et al., 2020).
  Since the true w∗ are unknown under UCL, they are estimated by matching first-moment statistics using the confusion matrix (Saerens et al., 2002; Lipton et al., 2018). Assume now that the features are conditionally aligned, pZ|Y =jS = p Z|Y =j T for all j ∈ Y (known as the label shift assumption), and the label space is discrete, Y = {1, · · · , k}. Then for each class i ∈ [k], we have the following relation between predicted target prior distribution, the source confusion matrix, and w∗: pT (Ŷ = i) = k∑
  j=1
  pT (Ŷ = i|Y = j)pT (Y = j) = k∑
  j=1
  pS(Ŷ = i|Y = j)pT (Y = j) = k∑
  j=1
  pS(Ŷ = i, Y = j)w ∗ j ,
  (2)
  where Ŷ := h(g(X)) are label predictions. Since we have the same number of equations as unknowns, w∗ could be recovered under mild assumptions (Lipton et al., 2018).
  Careful readers may have noticed that achieving either goal above requires having the other as an assumption. Indeed, attempting to estimate IW without conditional alignment could fail, as well as alignment without accurate IWs. The hope is that the optimization will be predominantly guided by the implicit bias of the pre-trained multilingual LM for selecting largely invariant features (hence conditionally aligned), so that small improvements to feature alignment and updates to prior shift estimates could bring incremental improvements to both components.
  Optimization. The two goals above define an optimization problem over g, h and w,
  min g,h
  ( E(x,y)∼pS [`(h(g(x)), y)] + λD(p w,Z S , p Z T ) ) , min
  w ∑ ŷ∈[k] ( pT (ŷ)− ∑ y pS(ŷ, y)wy )2 ,
  where alignment objective minimizes source task loss and the discrepancy of w-importanceweighted feature distributions (the metric should be more fine-grained than the first-moment metric used in our analysis), and the IW estimation objective solves eq. 2 using current statistics based on h, g with the constraints w ≥ 0 and w>pYS = 1. In our implementation, we use the minimax formulation of the alignment objective optimized with adversarial training, where D is replaced by a shallow ReLU network Wasserstein-1 critic and zero-centered gradient penalty (Arjovsky et al., 2017; Thanh-Tung et al., 2019). We found the Wasserstein-1 adversarial loss to be more responsive to importance weighting than cross-entropy loss (Ganin et al., 2016).
  Improving Stability of IW Estimates. In Tachet des Combes et al. (2020), the IWs estimation objective is solved by quadratic programming (QP), and updated once every epoch of training in their implementation. Since accurate IWs estimates are integral to successful adaptation, we (1) update w using projected gradient descent at every training step for maintaining their optimality. This, however, led to the discovery of previously unexplored practical issues with the IW estimator. In our preliminary experiments, we observed that w tends overestimate the majority class on the target domain, and diverges during longer training episodes.
  These issues are unique to unsupervised adaptation settings, and are due to inefficiencies of the critic and noises introduced by the alignment objective. They could also be reproduced with the previous QP procedure. We address these issues by (2) `2-regularizing w (Azizzadenesheli et al., 2018), and (3) using a decaying learning rate schedule for w. These modifications are necessary for achieving best performance, as long training episodes are required for fine-tuning on many NLP tasks, and also needed to maintain the critic near optimality for guiding properly the alignment.
sections.heading: 4 EXPERIMENTS
sections.text: In our experiments, we compare importance-weighted domain alignment (IWDA) to two baseline methods for cross-lingual transfer on three NLP tasks, under the unsupervised setting where access to unlabeled task data from the target language is provided (UCL setting). Besides supporting our findings and our motivated algorithm, these experiments are aimed at investigating the limitations of existing approaches under distributional shifts in prior distributions between source and target. We also illustrate the potential benefits of IWDA in light of stronger cross-lingual learning methods.
  Datasets. The tasks considered are named-entity recognition (NER), sentiment analysis, and textual entailment. NER: Two benchmark multilingual datasets are used, CoNLL (Tjong Kim Sang, 2002; Tjong Kim Sang & De Meulder, 2003) and WikiANN (Pan et al., 2017; Rahimi et al., 2019). The former includes English, German, Spanish and Dutch, and the latter covers over 200 languages, from which we select a subset of 40 according to the XTREME benchmark (Hu et al., 2020). Sentiment analysis: Multilingual Amazon Reviews Corpus (MARC) is used (Keung et al., 2020), a balanced dataset with product reviews in English, German, Spanish, French, Japanese and Chinese. Textual entailment: The XNLI dataset is used (Conneau et al., 2018), but due to space constraints, its setup and results are relegated to Appendix B. As in most work on cross-lingual transfer, English is used as the source language for transfer in our evaluations.
  Baselines. We perform extensive comparative experiments with two semi-supervised learning (SSL) techniques. Both require having a model fine-tuned on source data only. Knowledge distillation (KD): It distills and transfers knowledge of the source teacher model to the target domain by training a newly initialized student network to mimic teacher outputs on target data (Hinton et al., 2015). A variant of KD was proposed for NER by Wu et al. (2020), which also freezes the bottom three layers of mBERT during training (Wu & Dredze, 2019). Self-training (ST): It iteratively assigns pseudo-labels to unlabeled target data and train on the most confident ones. Our implementation follows closely the work by Dong & de Melo (2019), which considered sequence classification.5
  For IWDA experiments, the model is first fine-tuned on source only, then continue fine-tuned with the IWDA objective on both source and unlabeled target data. We also include results for when the true IWs are known and hard-coded as an upper bound on IWDA performance (IWDA (oracle)).
  Experiments are performed on BERT-Base Multilingual Cased and XLM-R Large models. More details on IWDA and our baselines are included in Appendices C and D along with hyperparameter settings. Unless otherwise noted, results are from our implementation.
  5These baselines are chosen because they require no more resources than the unlabeled target data provided, are not tailored to any particular model, and can be easily implemented on any existing models, which are qualities shared by IWDA. As many elaborate methods are also based on the same fundamental ideas as these baselines, we hope that they are representative of approaches based on SSL, if not exhaustive.
sections.heading: 4.1 RESULTS ON STANDARD BENCHMARKS
sections.text: To begin, through evaluations on standard benchmarks, we demonstrate that IWDA brings performance gains via improving representations to be better-aligned across languages. Results on CoNLL and low-resource WikiANN NER are presented in Tables 1 and 2, and those on MARC in Table 3. Remaining results on WikiANN with mBERT are included in Table 4 in the appendix.
  In all mBERT IWDA experiments, improvements upon zero-shot baseline are observed, and are accompanied by improved representation alignment. The average decrease in conditional feature shift is 21.60% and 22.33% on CoNLL with IWDA and IWDA (oracle) respectively (with 100% indicating perfect alignment), 51.53% and 64.77% on MARC, and 32.91% and 36.34% on WikiANN.
  However, improvements sometimes fall short of SSL baselines, which we suspect is due to the noise introduced during the alignment process. Potential sources include fluctuations in IW estimates and insufficiently trained critics. By training IWDA with more passes on the data and improving the optimality of the critic, further improvements on CoNLL NER are observed (Table 6).
  Another way to reduce noise is to perform a final-stage SSL training on the IWDA-aligned model. These experiments are labeled IWDA + KD and IWDA + ST, on which we observe some best results with mBERT on both tasks. This is an appealing strategy, because SSL algorithms theoretically depend on the assumption that the source and target domains are the same, IWDA training can be viewed as a preprocessing step on the feature representations to satisfy this assumption. We will revisit this point in the next section where we consider cross-lingual transfer under prior shifts.
  On XLM-R models, IWDA also brings improvements on NER. However on MARC, gains are only observed for languages distant from English. Here, we remark that the performance gain from IWDA (if any) will come from enhancements to the alignment of last-layer features in their current configurations, but not from learning entirely different representations with fundamentally better class-conditional alignment. This cannot be achieved absent target labels, additional supervision or resources. If the representations learned by the powerful XLM-R model during zero-shot training are already efficient, then the potentials of IWDA will be limited. Opportunities for further improvement in this case could come from additional resources, e.g. Bari et al. (2021) perform unsupervised data augmentation by leveraging the multilingual mask-filling ability of XLM-R.
sections.heading: 4.2 RESULTS UNDER CLASS PRIOR SHIFTS
sections.text: In this section, we evaluate on datasets with large prior shifts, a likely property of real-world data that is insufficiently investigated for UCL, and contrast the brittleness of SSL baselines with the robustness of IWDA. Because of the lack of multilingual benchmarks exhibiting prior shifts, similar to Section 2.2 , we evaluate on smaller datasets subsampled from the German portion of CoNLL and the Japanese portion of MARC according to varying amounts of shifts.
  Results plotting percent improvements over zero-shot baselines against the amount of prior shift are presented in Fig. 4. Although SSL baselines are generally better than IWDA under mild to no shifts,
  their performances quickly deteriorate as prior shift increases, until eventually resulting in negative transfer. IWDA on the other hand does not cause negative transfer from prior shifts owing to its estimation and correction capabilities. The relative success of IWDA demonstrates that it is possible to estimate prior shift from unlabeled target data. We include XLM-R MARC results in Fig. 10.
  In mBERT experiments, the average percent correction of prior shift, computed from the decrease in TV between true target domain priors and end-of-training estimates, normalized by the amounts of shift, is 87.33% on CoNLL (with 100% indicating perfect IW estimation), and 28.08% on MARC under shifts larger than 0.8 in TV. On MARC with smaller shifts, we observe estimates that are worse than assuming a uniform prior distribution (-335.80% in correction). This is because these are harder problems with noisy labels (as it is naturally difficult to distinguish a 2-star review from a 3-star one), the suboptimality of the source model from which IWDA is trained means that the initial representations may be conditionally misaligned, thereby affecting the accuracy of the initial as well as subsequent IW estimates.
  Since IWDA brings improvements to class-conditional alignment, as already pointed out in the previous section, it is appealing to complement SSL and potentially more elaborated methods with IWDA to improve robustness against prior shift. As demonstrated in our IWDA + KD and ST experiments, SSL no longer suffer from negative transfer when preceded by IWDA, and altogether provide further performance gains.
sections.heading: 5 CONCLUSION AND FUTURE WORK
sections.text: From the observations on multilingual neural LMs that cross-lingual transfer performance is strongly correlated with representation invariance and negatively affected by class prior shift, we proposed importance-weighted domain alignment (IWDA) for unsupervised cross-lingual learning. Experiment results support our findings, and show that IWDA is effective at the estimation and correction of class prior shift from unlabeled data, and robust to large shifts, under which in contrast existing UCL approaches based on SSL could result in negative transfer. As prior shift is a prevalent issue under unsupervised settings, more care to it is necessary in the design of future UCL algorithms for them to achieve good performance not only on curated benchmarks, but also on real-world data.
  While our present implementation of IW estimation with the moment-matching approach is largely effective, future work on the search and study of potentially more stable estimation methods would enable IWDA to achieve better and more consistent transfer performance. Since the strength of IWDA lies in its ability to detect prior shifts and improve on class-conditional representation alignment, with our experiments on combining IWDA and SSL baselines as a preliminary demonstration of its potential usefulness as a feature preprocessing step and in improving robustness to prior shifts, it would be of interest to investigate its benefits on stronger models and methods towards better cross-lingual transfer.
sections.heading: A ADDITIONAL ANALYSES AND FINDINGS FROM SECTION 2
sections.text: In this section, we provide additional results to support our findings by considering more languages (WikiANN NER), a higher-level task (XNLI), and a more powerful model (XLM-R Large).
  First and foremost, to demonstrate that the correlation between class-conditional representation alignment and performance is not confounded by the clustering structure induced by the linear classifier trained on last-layer features, we plot and observe similar correlation between transfer performance and class-balanced representation alignment in Fig. 5.
  WikiANN. As an attempt to minimize the effects from potential confounders on the empirical findings in the main sections, we perform the same experiments for studying the relation between transfer performance, representation alignment and prior shift by zero-shot transferring mBERT from English to 39 target languages included in the WikiANN NER dataset. The results are provided in Fig. 6. These languages have distinct linguistic properties and varying degrees of resourcefulness. A summary of them can be found in Table 5 of Hu et al. (2020), and sizes of their pretraining corpora in Table 1 of Wu & Dredze (2020).
  It is observed that the same correlation between representation alignment and performance still holds, but the effects of prior shift are mostly mild. We suspect that this is because the task we evaluated on, named-entity recognition, is a structured prediction problem, hence mBERT’s ability to leverage dependencies between words made it robust to prior shifts on these tasks. We later observed that if the source dataset on which the model is trained on has a skewed prior distribution, then the effects of prior shift will be pronounced. The mBERT WikiANN results under this setting are shown in Fig. 9b, and more discussions are coming up in Section A.1.
  XNLI. So far, we have seen empirical evidence on sentiment analysis and NER tasks, and one may wonder if the same conclusions still hold when considering cross-lingual transfer to high-level semantic tasks. For thoroughness, we include a plot in Fig. 7 comparing zero-shot transfer performance to representation alignment when mBERT is trained on a textual entailment task, from English to 14 languages included in the XNLI dataset. Here, the same observations are made, demonstrating that the findings hold across NLP tasks.
  XLM-R. Another variable in our empirical analyses is the model used for evaluation. While the lightness of mBERT afforded us to perform extensive experiments, it would be of practical interest to see whether our findings also hold on larger and more powerful models. In Fig. 8, we compared performance to alignment and prior shift on XLM-R Large model on MARC. Again, the observations support our findings, and show that they also hold across models.
  A.1 EFFECTS FROM SKEWED SOURCE PRIOR DISTRIBUTIONS
  As briefly mentioned above, we have only considered the cross-lingual transfer from relatively balanced source datasets in previous sections. Here, we show that when the model is fine-tuned on source datasets with skewed prior distributions, the effects of class prior shift are more pronounced.
  We first experiment on MARC. We generate an skewed English source dataset via subsampling according to the prior distribution of 14.86%, 6.43%, 8.03%, 19.28% and 51.41%6, corresponding to the proportion of 1- through 5-star reviews. Then, we repeat the previous empirical analysis by evaluating an mBERT fine-tuned on this skewed English dataset, and the results are shown in Fig. 9a. Compared to when the source is class-balanced (Fig. 3), here the correlation between performance and class prior shift is much stronger.
  On WikiANN, although the original English portion of the dataset is already considerably skewed (with class O makes up 50.75% of the labels), the correlation between prior shift and model performance is weak (Fig. 6b), likely due to NER being a structured prediction problem where the dependencies between words in the same sentence helped the model to be more robust. Repeating the same experiment above with the skewed English source dataset generated by keeping only sentences containing tokens with an O label (so that now 80.92% of the tokens belong to class O), we again observe pronounced effects of prior shift on performance (results in Fig. 9b).
  6This is the actual distribution of product ratings on Amazon.com.
sections.heading: B ADDITIONAL EXPERIMENT RESULTS FROM SECTION 4
sections.text: By comparing the transfer performance of IWDA to SSL on standard benchmarks as well as datasets exhibiting prior shifts in the main sections, we demonstrated that better performance can be achieved with better representation alignment under the UCL setting, as well as the limitations with existing approaches under prior shifts. While the prior shifts experiments in Section 4.2 are performed on mBERT, in Fig. 10, we provide XLM-R Large results on MARC Japanese prior shift experiments.
  Furthermore, we provide full mBERT WikiANN results of applying IWDA (Table 4), and results on the XNLI dataset, a high-level semantic task of textual entailment.
  XNLI. Because XNLI does not provide human-generated training data for languages other than English, to obtain unlabeled target language task data for our studies on the UCL setting, we use the machine-translation of the English training set provided by Conneau et al. (2020a). To remove potential effects due to parallelism, similar to the setup considered in Bari et al. (2021), we partition the dataset into equal portions of size 20,000 (5% of the original size) and randomly select disjoint partitions to use as source and target data. In addition, since the original XNLI test set is humantranslated but our target data is machine-translated, to avoid the effects from this discrepancy (or domain shift) between train and test, evaluations are performed on the machine-translated portion of the dataset in our experiments (we refer interested readers to Artetxe et al. (2020a) for a discussion on XNLI translation artifacts).
  Due to our setup, our results are non-benchmark and cannot be compared to models evaluated on the original XNLI test set. However, the goal of this experiment is to study whether our principle of achieving domain alignment is beneficial on high-level tasks.
  The results are included in Table 5. It is observed that IWDA generally improves transfer performance upon the zero-shot baseline, although the gains are not significant. As discussed in Section 4.1, the potentials of IWDA will be limited if the zero-shot representations are already largely aligned, which is reflected by the small performance gap between zero-shot and supervised results in our setup.
  IWDA vs. DA. Besides demonstrating that SSL suffers from negative transfer under large prior shifts in our experiments in Section 4.2, we also discussed in Section 2.2 that domain alignment without prior shift correction leads to negative transfer.
  In Fig. 11, we compare the performance on benchmark CoNLL German, which exhibits a prior shift from CoNLL English, from applying IWDA and domain alignment without prior shift correction (DA). CDAN refers to the conditioning technique by Long et al. (2018), which we use in our IWDA implementation by default. It is observed that without prior shift correction, as training progresses,
  transfer performance of vanilla DA deteriorates as the model makes more and more erroneous feature alignments caused by the shift.
  Better Training of Adversary. As briefly mentioned in Section 4, while the IWDA models for the experiments in the main sections are trained for four epochs with simultaneous adversary and model updates for fair comparisons with SSL techniques, better performance could be obtained from IWDA if the optimality of the critic (adversary) can be better maintained throughout training with more passes over the data and alternating updates.
  In Table 6, we observe that IWDA delivers much-improved performance on CoNLL when we perform four adversary updates per model update and allow 40 passes over the dataset (we also set lambda iw to 1). Although this setting would lengthen the training time, this could be of practical interest for getting the best IWDA performance.
  C IMPLEMENTATION DETAILS OF IWDA
  A high-level description of IWDA was given in Section 3 with discussions that connect it to the motivating findings in Section 2. This section provides more details on the implementation of the algorithm along with the hyperparameter settings7 used in our experiments (Table 7).
  First, recall the joint objective, reproduced below, which consists of an alignment objective and an importance weight (IW) estimation objective,
  min g,h
  ( E(x,y)∼pS [`(h(g(x)), y)] + λD(p w,Z S , p Z T ) ) , min
  w ∑ ŷ∈[k] ( pT (ŷ)− ∑ y pS(ŷ, y)wy )2 . (3)
  Note that the alignment objective contains the discrepancy between (important-weighted) source and target feature distributions and source task loss, added to prevent g from degenerating to a trivial representation with perfect source and target alignment, e.g. constant maps x 7→ c.
  Alignment Objective. We reformulate the alignment objective in eq. 3 as a minimax objective, and replace explicit evaluation of the discrepancy D with a critic function f (also known as discriminator or adversary; Ganin et al. 2016; Goodfellow et al. 2014). This turns the alignment objective into
  min g,h
  ( E(x,y)∼pS [`(h(g(x)), y)] + λmax
  f∈F ( Ex∼pw,XS [`ad(f(g(x)), 0)]− Ex∼pXT [`ad(f(g(x)), 1)]︸ ︷︷ ︸
  (4)
  )) ,
  where `ad is a specific choice of adversarial loss accompanied by an adversary function class F , that takes as input the scores computed by the adversary and the domain labels (here, source domain is labeled 0). The role of the adversary is to distinguish features computed from inputs sampled on the source domain from those on the target domain, and output a score on each feature that reflects its domain identity. When the features are aligned, the adversary would not be able to distinguish source and target features.
  In our implementation, we use Wasserstein-1 loss for `ad, whose corresponding adversary function classF is 1-Lipschitz functions (Arjovsky et al., 2017). It is shown that when f attains its maximum, the term (4) exactly computes the Wasserstein-1 distance between the importance-weighted feature
  7We use the same settings on all IWDA experiments. For methodological discussions on hyperparameter tuning for unsupervised cross-lingual learning, see Artetxe et al. (2020c).
  distributions, pw,ZS and p Z T . We parameterize F as a shallow ReLU network of width 2048, and enforce the Lipschitz condition using soft constraints in the form of zero-centered gradient penalty, which penalizes the gradient of f between pairs of source and target features (Thanh-Tung et al., 2019).
  Last but not least, for added stability, we condition the alignment of features on their (pseudo-)classes as proposed by Long et al. (2018) (CDAN). This is implemented as follows. Instead of feeding the adversary directly with features output by g, it receives the outer product between features and classifier output distributions, g(x)⊗h(g(x)). This means that the input dimension of f is increased to the dimension of features multiplied by the number of output classes k. When k is large, Long et al. (2018) used random projections to reduce dimensionality, which we did not explore.
  Putting everything together—minimax reformulation, Wasserstein-1 loss, zero-centered gradient penalty, and conditional alignment—upon receiving a mini-batch of source labeled examples, (xi, yi) n i=1, and target unlabeled examples, (x ′ j) m j=1, the empirical alignment loss is
  1
  n n∑ i=1 `(h(g(xi)), yi) + λ
  ( 1
  n n∑ i=1 f(g(xi)⊗ h(g(xi)))− 1 m m∑ j=1 f(g(x′j)⊗ h(g(x′i))) ) − µ ñ ñ∑ i=1 ‖(∇f)(z̃i)‖2,
  where ñ = min(n,m) and z̃i is a random point on the line connecting g(xi) and g(x′i). Here, we have two hyperparameters: λ (lambda da) controls the strength of alignment, and µ (lambda gp) controls the strength on enforcing the Lipschitz condition. The learning rate of the adversary (lr ad) is set to be larger than that of the model (lr; Heusel et al. 2017). AdamW optimizer is used throughout our experiments.
  On this loss, we back-propagate gradients to f on maximizing the second and third terms, g on minimizing the first two terms, and h on minimizing the first term only. In our experiments in the main sections, all three models are optimized with simultaneous updates, but we note that alternating updates (standard in GAN literature) could allow IWDA to deliver further performance gains with longer training time. In Table 6, we observe improved mBERT CoNLL results when the training alternates between four updates to f and one update to g for 40 epochs.
  IW Estimation Objective. We proposed several modifications to the IW estimation objective in Section 3 to improve its stability, and one of them is adding `2-regularization to w (Azizzadenesheli et al., 2018).
  Given first moment statistics in the form of the source confusion matrix, Cij := pS(Ŷ = i, Y = j), and target output distribution, νi := pT (Ŷ = i), the regularized IWs objective minimizes
  k∑ i=1 ( νi − k∑ j=1 Cijwj )2 + ξ k∑ i=1 (wi − w(0))2, (4)
  where w(0) is an initial IW estimate (set to all-ones vector by default, i.e. no prior shift). Here, one hyperparameter that controls the strength of the regularization is introduced, ξ (lambda iw). For simplicity, the `2-regularization is implemented with and documented as weight decay. Hence, ξ equals to lambda iw multiplied by 2 times the learning rate of w.
  Instead of initializing w(0) = 1, because IWDA fine-tuning is preceded by zero-shot fine-tuning in our experiments and its decent performance reflects partial class-conditional feature alignment, we initialize it to a crude IW estimate w′ obtained from the zero-shot model using the QP by Tachet des Combes et al. (2020). Since the initial alignment will not be perfect and w′ not entirely reliable, we set w(0) = (1− ρ)w′ + ρ1, where ρ (lambda iw init) is a hyperparameter. The statistics C, ν evolve during training as the feature representation computed by g changes, so it needs to be updated throughout the learning process. Re-evaluating them at each optimization step
  on a held-out set would make training prohibitively slow, hence we update them with the statistics Ĉ, ν̂ computed on training mini-batches: C(t) = (1 − η)C(t−1) + ηĈ, and similarly for ν(t). This update rate η (lr iw stats) is not the same as IW learning rate (lr iw).
  Lastly, to minimize potential effects from the miscalibration of neural models (Guo et al., 2017; Alexandari et al., 2020), hard confusion matrix and target output distribution are used in the estimation of IWs (Lipton et al., 2018; Garg et al., 2020).
sections.heading: D DESCRIPTION OF SSL BASELINES
sections.text: Knowledge Distillation (KD). Our implementation follows the original paper by Hinton et al. (2015) closely (from which the Wu et al. (2020) variant is moderately different). Given a teacher LM fine-tuned either under zero-shot or using IWDA, we distill its knowledge on a newly initialized pre-trained student LM by minimizing the mean-squared-error difference of the teacher and student Softmax outputs on source and target data without labels. If labeled data from the source is available, we also train the student to minimize source task loss.
  We set the temperature to 3, and the weight of the source task loss to 0.1. Learning rate is 1e-4, batch size is 32, and we train for 4 epochs.
  In addition, Wu et al. (2020) observed that when distilling to pretrained LMs, better performance can be achieved if the bottom three layers of the student is frozen (motivated by findings from Wu & Dredze (2019)). We also apply this technique in our KD experiments.
  Self-Training (ST). Our implementation follows Dong & de Melo (2019) closely. Given an LM fine-tuned on source domain only, ST continues fine-tuning the model by first assigning pseudolabels to target data, collecting the ones with the highest confidence, and then train on them with the pseudo-labels. Source task loss is also minimized during self-training.
  The hyperparameters of ST involve selecting how any pseudo-labeled examples to train on. On MARC with 200,000 training examples, at each epoch, we select top 4,000 most confident pseudolabeled examples from the target domain conditioned on each (pseudo-)class, so that 20,000 examples are selected each round (MARC has five classes). These numbers are scales proportionally on datasets of other sizes. We perform self-training for 4 epochs, and set learning rate to 1e-5 and batch size to 128.
  On IWDA + ST prior shift experiments, we select much less examples for self-training to avoid training on low-confidence examples from minority classes. This adjustment of hyperparameters relies on knowledge about the prior distribution on the target domain, which is approximated by the IW estimates from IWDA.