Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly. In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse crosslingual tasks. Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent. Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches. Extensive experiments on various tasks demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We further show that our proposed framework can generalize to contextualized representations and achieves state-of-the-art results on the CoNLL cross-lingual NER benchmark.  (Mikolov et al., 2013a;Pennington et al., 2014;Bojanowski et al., 2017) have become ubiquitous across a wide range of NLP tasks. In particular, methods for crosslingual word embeddings (CLWE) have proven a powerful tool for cross-lingual transfer for downstream tasks, such as text classification (Klementiev et al., 2012a), dependency parsing (Ahmad et al., 2019), named entity recognition (NER) (Xie et al., 2018;Chen et al., 2019), natural language inference , language modeling (Adams et al., 2017), and machine translation (MT) (Zou et al., 2013;Artetxe et al., 2018b;. The goal of these CLWE methods is to learn embeddings in a shared vector space for two or more languages. There are two main paradigms for learning CLWE: cross-lingual alignment and joint training.
The most successful approach has been the cross-lingual embedding alignment method (Mikolov et al., 2013b), which relies on the assumption that monolingually-trained continuous word embedding spaces share similar structure across different languages. The underlying idea is to first independently train embeddings in different languages using monolingual corpora alone, and then learn a mapping to align them to a shared vector space. Such a mapping can be trained in a supervised fashion using parallel resources such as bilingual lexicons (Xing et al., 2015;Smith et al., 2017;Joulin et al., 2018b;Jawanpuria et al., 2019), or even in an unsupervised 2 manner based on distribution matching (Zhang et al., 2017a;Artetxe et al., 2018a;Zhou et al., 2019). Recently, it has been shown that alignment methods can also be effectively applied to contextualized word representations (Schuster et al., 2019;Aldarmaki & Diab, 2019).
Another successful line of research for CLWE considers joint training methods, which optimize a monolingual objective predicting the context of a word in a monolingual corpus along with either a 1 Code will be released on publication. 2 In this paper, "supervision" refers to that provided by a parallel corpus or bilingual dictionaries.
This paper proposes a new and simple framework for learning cross-lingual embeddings that, based on well-supported insights, unifies two standard frameworks: joint learning and alignment-based approaches. While I like and acknowledge the fact that the paper examines these frameworks critically and has also some didactic value, I still have some concerns regarding the current experiments, and would like to see some additional analyses in the paper. Honestly, this type of work would work better as a short ACL/EMNLP paper, as the core methodological contribution is a very simple combination of the existing techniques.
With joint methods, it is true that shared words can be used as implicit bilingual learning signal which gets propagated further in the model. Even in the case of alignment-based methods, it was shown that this signal can assist in learning shared cross-lingual spaces, but: 1) such spaces are of lower-quality than the spaces learned by relying on seed dictionaries (see e.g., the work of Vulic and Korhonen (ACL 2016)), 2) this is useful only for similar language pairs which use the same script. The paper fails to provide an insightful discussion on how the scores differ when we move towards more distant languages. For instance, English-Chinese is included as a more distant language pair, and the combined method seems to work fine, but the reader is left wondering why that happens. The same is true for English-Russian. The paper should definitely provide more information and insights here. 
One experiment that would be quite interesting imho is to take seed dictionaries to initialise the joint training method. It will not be unsupervised any more, but it would be very useful to report the differences in results between fully unsupervised joint initialisation (which, as mentioned should work only with more similar languages) and such weakly supervised init: e.g., the method could take all one-to-one translation pairs from the seed dictionary as 'shared words'. It would also be interesting to combine such 'shared words' and words that are really shared (identically spelled words) as initialisation and measure how it affects the results, also in light of differences in language distance. Adding one such experiment would make the paper more interesting.
Some recent strong baselines are missing: e.g., I wonder how the model that does self-learning on top of seed dictionaries (similar to the work of Vulic et al., EMNLP 2019) compares to the proposed method. Also, can we use the same self-learning technique with the combined method here? Would that lead to improved results? Another work I would like to see comparisons to is the work of Zhang et al. (ACL 2019) and despite the fact that the authors explicitly stated that they decided not to compare to the work of Artetxe et al. (ACL 2019) as they see it as concurrent work, I would still like to see that comparison as the model of Artetxe et al. seems very relevant to the presented work.
I do not see the extension to contextualized representations (described in Section 3.2) as a real contribution: this is a very straightforward method to apply an alignment-based method on multilingual BERT representations, and very similar techniques have been probed in previous work (e.g., by Aldarmaki & Diab), only the bilingual signal/dictionary came from parallel data instead.
Finally, as mentioned before, one of the must-have experiments is including more (distant and diverse) language pairs and analysing the results across such language pairs, with an aim to further understand how the levels of sharing, over-sharing, and non-isomorphism affect performance. Also, while the authors state that reduced isomorphism is the key problem of alignment-based methods, I still fail to see how exactly the alignment refinement step does not suffer from that problem? More discussion is needed here.
Other comments:
- It would be useful to also point to the survey paper of Ruder et al. (JAIR 2019) where the difference between alignment-based and joint models is described in more detail and could inform an interested reader beyond the confines of the related work section in this paper. Similarly, differences between joint models and alignment-based models have also been analysed by Upadhyay et al. (ACL 2016); Vulic and Korhonen (ACL 2016).
- I like how Figure 1 clearly visualizes the main intuitions behind the proposed framework, and how mitigating the oversharing problem leads to better alignments. This was very nice.
- In light of the problems with silver standard MUSE datasets (see also the recent work of Kementchedjhieva, EMNLP 2019), I would suggest to run BLI experiments on more language pairs: e.g., there are BLI datasets of Glavas et al available for 28 language pairs.