Abstract Recent research has discovered that a shared bilingual word embedding space can be induced by projecting monolingual word embedding spaces from two languages using a self-learning paradigm without any bilingual supervision. However, it has also been shown that for distant language pairs such fully unsupervised self-learning methods are unstable and often get stuck in poor local optima due to reduced isomorphism between starting monolingual spaces. In this work, we propose a new robust framework for learning unsupervised multilingual word embeddings that mitigates the instability issues. We learn a shared multilingual embedding space for a variable number of languages by incrementally adding new languages one by one to the current multilingual space. Through the gradual language addition our method can leverage the interdependencies between the new language and all other languages in the current multilingual hub/space. We find that it is beneficial to project more distant languages later in the iterative process. Our fully unsupervised multilingual embedding spaces yield results that are on par with the state-of-the-art methods in the bilingual lexicon induction (BLI) task, and simultaneously obtain state-of-the-art scores on two downstream tasks: multilingual document classification and multilingual dependency parsing, outperforming even supervised baselines. This finding also accentuates the need to establish evaluation protocols for cross-lingual word embeddings beyond the omnipresent intrinsic BLI task in future work. The ubiquitous use and success of word embeddings in monolingual tasks inspired further research on inducing cross-lingual word embeddings for two or more languages in the same vector space. Embeddings of translations and words with similar meaning are geometrically close in the shared cross-lingual vector space . This property makes them effective features for cross-lingual NLP tasks such as cross-lingual document classification (Kle-mentiev et al., 2012), cross-lingual information retrieval (Vulic and Moens, 2015), bilingual lexicon induction (Mikolov et al., 2013b; Gouws et al., 2015; Heyman et al., 2017), and (unsupervised) machine translation (Artetxe et al., 2017b; Lample et al., 2018; Artetxe et al., 2018c). Most prior work has focused on methods for constructing bilingual word embeddings (BWEs), yielding word representations for exactly two languages. For problems such as multilingual document classification, however, it is highly-desirable to represent words in a multilingual space. A favourable property is that it enables fitting a single classifier on the union of training datasets in many languages, which results in 1) knowledge transfer across languages that may lead to better classification performance, and 2) a setup that is easier to maintain as it is no longer required to train many different monolingual or bilingual classifiers. Multilingual word embedding (MWE) methods typically generalize existing BWE methods by mapping multiple source language spaces to the space of one target language (Ammar et al., 2016), which is used as a pivot/hub language. This approach may lead to suboptimal solutions as it does not account for interdependencies between the source languages. Most BWE and MWE methods rely on cross-lingual supervision to some extent: e.g., bilingual lexicons (Mikolov et al., 2013a), parallel corpora (Gouws et al., 2015), or subject-aligned document pairs (Vulic and Moens, 2016). In such paradigms, modeling dependencies between all languages is impractical as it requires supervision for all language pair combinations. Recent research has shown that BWEs can also be learned without cross-lingual supervision and can even outperform supervised BWE variants on bilingual lexicon induction benchmarks (Con-neau et al., 2018; Artetxe et al., 2018a). Chen and Cardie (2018) took a first step towards learning multilingual spaces without supervision while incorporating dependencies between all languages but their approach extends the work of Conneau et al. (2018), which has known limitations concerning optimization stability with distant language pairs (Sgaard et al., 2018). In this work, we investigate robust methods to induce MWEs without any cross-lingual supervision. The robustness of our approach is illustrated in good performance for distant languages such as Finnish and Bulgarian. This paper makes the following contributions. First, based on a reformulation of the BWE method of Artetxe et al. (2018a), we propose two novel methods for inducing MWEs: 1) the single hub space model (SHS) uses the classical idea of mapping source languages to a single hub language; 2) the incremental hub space model (IHS) incorporates dependencies between all languages by incrementally expanding the multilingual space by one language in each step. IHS results in mappings that are more robust and coherent across languages. Both SHS and IHS only require monolingual data. Second, we evaluate our method on benchmarks for bilingual lexicon induction (BLI), multilingual document classification, and dependency parsing. We find that the IHS method is competitive with state-of-the-art BWE methods on the bilingual lexicon induction benchmarks, while yielding the highest scores on the multilingual document classification and dependency parsing benchmarks. Third, unlike the majority of prior work (Con-neau et al., 2018; Artetxe et al., 2018a; Chen and Cardie, 2018, inter alia ), we do not limit our evaluation to the intrinsic BLI task only. Consequently, we investigate if embedding reweighting, a recently proposed best practice for BWEs, is useful for extrinsic tasks such as document classification and dependency parsing in multilingual settings. Cross-lingual word embeddings have received a lot of attention in recent years. Most methods construct a space shared between two languages using cross-lingual supervision in the form of bilingual lexicons (Mikolov et al., 2013a; Artetxe et al., 2016; Smith et al., 2017), parallel corpora (Klementiev et al., 2012; Faruqui and Dyer, 2014; Gouws et al., 2015; Luong et al., 2015) or subject-aligned document pairs (Vulic and Moens, 2016). See Ruder et al. (2018) for a full overview of BWE model typology in relation to the required supervision. To enable knowledge transfer across an arbitrary number of languages, multilingual methods have been introduced. Huang et al. (2015), propose decomposing a matrix with multilingual co-occurrence counts weighted by probabilistic dictionaries. Ammar et al. (2016) compare this method to three other MWE models: MultiCluster, MultiCCA, and MultiSkip. MultiCluster uses bilingual dictionaries to cluster translations and then train the monolingual Skip-gram model (SG) (Mikolov et al., 2013a) on a union of monolingual corpora where they replace words with their cluster id such that words in the same cluster get the same representation. MultiCCA is the multilingual extension of the method of Faruqui and Dyer (2014): Using canonical correlation analysis (CCA) and dictionaries with English as the target language, monolingual embeddings are projected to the English vector space. MultiSkip is a straightforward extension of the BiSkip method (Luong et al., 2015) which generalizes the monolingual SG objective to account for word alignments in parallel corpora. Similarly, Duong et al. (2017), extend CBOW to multiple languages. All these methods learn multilingual embeddings using bilingual dictionaries of parallel corpora: This limits their applicability for many languages. More recently, Conneau et al. (2018); Artetxe et al. (2018a) showed that BWEs can be effectively induced without any cross-lingual supervision. The approaches are based on the assumption that monolingual embedding spaces are approximately isomorphic. 1 Improving on earlier attempts (Cao et al., 2016; Zhang et al., 2017), Conneau et al. (2018) propose a two-step framework to map two monolingual spaces to the shared space. First, they use an adversarial objective to get an initial bilingual space in which the discriminator can no longer distinguish to which language a given word embedding belongs. They then fine-tune the initial solution. An important limitation is that the adversarial objective is prone to converge to degenerate solutions. Furthermore, Sgaard et al. (2018) empirically prove that the method typically fails for distant language pairs such as English-Finnish. an-1 One of the necessary conditions for this assumption to hold is that the monolingual corpora on which the embeddings are trained are comparable (Sgaard et al., 2018). other framework with the same goal. Expanding on their earlier work (Artetxe et al., 2017a, 2018b), they use an unsupervised heuristic to obtain a noisy initial seed lexicon which is used to obtain an initial bilingual space. This solution is iteratively improved similar to Artetxe et al. (2017a) and Conneau et al. (2018) while using value dropping regularization to escape early convergence to local minima. Their method is the starting point for this work, and it is discussed in detail in  3. The two unsupervised approaches are limited to finding mappings between a pair of languages. To the best of our knowledge, Chen and Cardie (2018) is the only unsupervised method that constructs a multilingual embedding space. Their method extends the adversarial pre-training and iterative refinement steps of Conneau et al. (2018) to the multilingual setting. Their work does not investigate the limitations of Conneau et al. (2018)'s method: Less stable optimization and difficulties with mapping spaces with reduced isomorphism. Furthermore, their generalization turns the iterative refinement into a non-convex optimization problem. In contrast, our multilingual methods proposed in this work are applicable to distant language pairs and decompose every iteration in the refinement step in multiple convex optimization problems, making them very robust and widely applicable. We now summarize mapping-based approaches to learning BWEs, which serve as the backbone of our multilingual approach. These methods rely on a mapping procedure , that is, a way to transform two monolingual spaces such that translations and similar words obtain similar representations. Supervised approaches take translations from readily available seed training dictionaries. Unsupervised approaches construct a seed lexicon from scratch, and use an iterative procedure to refine the seed lexicon and the mapped bilingual space. Mapping Procedure. Various mapping procedures have been proposed in the literature (Mikolov et al., 2013b; Dinu et al., 2015; Lazaridou et al., 2015; Vulic and Korhonen, 2016). These methods can be seen as variants of a a single framework (Artetxe et al., 2018b), summarized here. At its core, each mapping procedure learns the orthogonal transformations W x and W z for the monolingual embedding spaces X and Z that minimize the distance between embeddings of translations in the mapped spaces XW x and ZW z . The orthogonality constraint ensures that the transformations preserve the monolingual constellation of embeddings. Formally, let D be a matrix representing a bilingual dictionary s.t. D ij = 1 if the i th source word is translated by the j th target word and D ij = 0 otherwise, then W x and W z are found by solving the following optimization problem: arg max W x , W z (cid:88) i (cid:88) j D ij ( X i, : W x )  ( Z j, : W z ) = arg max W x , W z tr ( XW x ( DZW z ) (cid:124) ) (1) subject to W x W (cid:124) x = I , W z W (cid:124) z = I where tr (  ) denotes the trace operator. Eq. (1) has a closed-form solution based on the singular vectors of X (cid:124) DZ : W x = U , W z = V with USV (cid:124) = SVD( X (cid:124) DZ ) . In addition to the transformation, there are several optional pre-processing (S1-S2) and post-processing (S3-S5) steps: S1. Normalization : apply length normalization (normalizing X and Z such that all embeddings have a unit Euclidean norm), or mean centering, or a combination of both; S2. Whitening : apply ZCA whitening (Bell and Se-jnowski, 1997) on X and Z which transforms the monolingual embedding matrices such that each dimension/component has unit variance and such that the dimensions are uncorrelated (see Eq. (2) later). The intuition is that it is easier to align the vector spaces along directions of high variance; S3. Re-weighting the components according to the singular value matrix S of X (cid:124) DZ : This is an attempt to further align the embeddings in the multilingual space as each singular value measures how well a dimension in the multilingual space correlates across languages for the given dictionary; S4. De-whitening , the inverse transformation of S2: After the mapping, it was shown as important to restore the variance information in case whitening was applied (Artetxe et al., 2018c); S5. Dimensionality reduction truncates the embedding vectors such that only components with the highest singular values are kept. Refinement Procedure. The refinement aims at iteratively improving the seed dictionary and the bilingual space with Expectation Maximization (Dempster et al., 1977). In each iteration, the mapping procedure is executed using the dictionary from the previous iteration to obtain a new bilingual space, and then a new dictionary is induced using nearest neighbor retrieval in the cross-lingual similarity matrix M . This is repeated until the (unsuper-vised) training objective (cid:80) i (cid:80) j D ij (cid:0) ( X i, : W x )  ( Z j, : W z ) (cid:1) stops increasing. The matrix M is calculated using cross-domain similarity local scaling (CSLS; Conneau et al. (2018)), an extended variant of cosine similarity that avoids the hubness problem (Radovanovic et al., 2010; Dinu et al., 2015). 2 In particular, the element m ij at row i and column j of M corresponds to the CSLS value between the crosslingual vectors x CLi and z CLj of the i th source word and the j th target word respectively: m ij = CSLS ( x CLi , z CLj ); CSLS ( x , z ) = 2 cos ( x , z )  r Zk ( x )  r Xk ( z ) . r Xk ( x ) and r Zk ( z ) calculate the average cosine similarity of a vector with its k nearest neighbors (measured by cosine) in the mapped spaces of X , Z , respectively. It is also beneficial to jointly infer dictionaries source-to-target and target-to-source (Artetxe et al., 2018a). 3 The mapping is then learned from the concatenation of these two dictionaries. 4 To avoid suboptimal local minima, Artetxe et al. (2018a) propose to randomly drop values from the matrix M with probability 1  p (further value dropping ). The value of p is exponentially increased as training progresses. p is initialized to a small value (e.g., 0 . 1 ). Whenever the objective stops improving for N patience refinement steps, p is multiplied with a given factor (e.g., 2 ) until p  1 after which all values in M are kept. Value dropping was shown to be crucial when constructing bilingual spaces between distant language pairs. We later analyze its impact on the proposed multilingual methods. Inducing a Seed Lexicon. Artetxe et al. (2018a) obtain a seed lexicon based on the assumption that for a translation pair w Xi , w Zj , the monolingual similarity vectors,  X i X (cid:124) and (cid:112) Z j Z (cid:124) of translations i, j are (approximately) equal up to a permu-2 Hubness is the phenomenon observed in a high-dimensional vector space where there are vectors, called hubs, which are the nearest neighbors to many vectors in the space. 3 Note that z CL j being the nearest neighbor of x CL i does not imply that the inverse is also true. 4 Due to the large search space, limiting the search space by truncating both vocabularies and their corresponding embedding matrices to the C refinement most frequent words results in better solutions and speeds up computation. tation. 5 Therefore, seed translations for a source word i are generated by finding the nearest neighbor based on similarity of monolingual similarities. This heuristic yields a very noisy seed lexicon, but it was proven to contain a sufficiently strong bilingual signal to bootstrap the refinement procedure. The seed lexicon is inferred symmetrically (i.e., by concatenating respective source-to-target and target-to-source seed lexicons) and the vocabularies are truncated to the C seed most frequent words. We now present two models for learning unsupervised multilingual word embedding spaces: the single hub space model (SHS) and the incremental hub space model (IHS). The methods generalize the bilingual framework described in  3, and rely on (a subset of) preprocessing and postprocessing steps S1-S5 in the multilingual setting. Single Hub Space (SHS). The SHS model de-fines one language as the hub language L 0 and projects the embedding spaces Z 1 , ..., ZN of all other languages L 1 , ..., LN (further secondary languages ) to the hub space X . Hence, we reduce the construction of a multilingual space of N languages to the alignment of N  1 vector spaces. Learning these projections is similar to the bilingual case: We use the unsupervised iterative refinement procedure and seed lexicon heuristic from  3. However, we require the orthogonal mapping to be asymmetric: The hub language space should either remain unchanged or it should be transformed with the same operation for each of the N  1 language pairs. We therefore derive an asymmetric version of the mapping framework from  3 that yields the exact same solution as the original. Let X be the embedding matrix of the, Z 1 , . . . , ZN the embedding matrices of the secondary languages, and D k,l the dictionary between languages L k and L l . We induce a multilingual space X m , Z m 1 , ... , Z mN in three main steps. First, the embeddings of each language are preprocessed by normalizing and whitening the embeddings, as described by Eqs. (2)-(6). Normalization consists of subsequently performing length normalization, mean 5 The square root in the formulas is empirically motivated. ZCAwhiten ( W ) = W ( W (cid:124) W )  0 . 5 (2) X (cid:48) = normalize ( X ) (3) X (cid:48)(cid:48) = ZCAwhiten ( X (cid:48) ) (4) Z (cid:48) l = normalize ( Z l ) (5) Z (cid:48)(cid:48) l = ZCAwhiten ( Z (cid:48) l ) (6) After preprocessing, we rotate each secondary language L l to a bilingual space between the hub language space and its own embedding space, as described by Eqs. (7)-(10). The calculations are analogous to the bilingual mapping procedure: the left and right singular vectors U l and V l of X (cid:48)(cid:48) D k,l Z (cid:48)(cid:48) (cid:124) l are the rotation matrices that project the preprocessed matrices X (cid:48)(cid:48) and Z (cid:48)(cid:48) l to their bilingual space; this is formulated in Eqs. (7)-(8). The bilingual projection of Z (cid:48)(cid:48) l can be reweighted by multiplying it with a given power q of the singular values matrix S l of X (cid:48)(cid:48) (cid:124) D 0 ,l Z (cid:48)(cid:48) l , Eq. (9). Intuitively, the reweighting operation makes the dimensions that correlate better across languages more important. Next, we restore the variance information of Z (cid:48) l by performing a dewhitening operation: we project back to the monolingual space, multiply with the inverse of the whitening matrix, and then project back to the bilingual space (Eq. (10)). 6 U l S l V (cid:124) l = SVD ( X (cid:48)(cid:48) (cid:124) D 0 ,l Z (cid:48)(cid:48) l ) (7) Z l,bi ( l ) = Z (cid:48)(cid:48) l V l (8) Z (cid:48) l,bi ( l ) = Z l,bi ( l ) S ql (9) Z (cid:48)(cid:48) l,bi ( l ) = Z (cid:48) l,bi ( l ) V (cid:124) l ( Z (cid:48) (cid:124) l Z (cid:48) l ) 0 . Finally, we project Z (cid:48)(cid:48) l,bi ( l ) to the space of the hub language in Eq. (11)). The multilingual space for the hub language is simply the monolingual embedding space after preprocessing, see Eq. (12). For the bilingual case this formulation is equivalent to the symmetric mapping introduced in  3: one can easily verify that the dot products between the mapped spaces simplify to the same formula. 6 Note that the projection matrices that map from the bilingual to the monolingual spaces are given by the inverses of U l and V l . Since the matrices are orthogonal their inverses are equal to their transposes. Incremental Hub Space (IHS). SHS enables language interactions only indirectly through the hub language. Ideally, a multilingual method should incorporate interdependencies between all languages. We hypothesize that, especially when mapping a language distant to the hub language, it is beneficial to incorporate the structural similarities with all other languages as a regularization mechanism to find a more robust mapping. We therefore propose the incremental hub space (IHS) model. It incrementally expands the multilingual space X m and takes into account all languages in the current multilingual space when adding a new language. First, we define a language order and initialize the space to the preprocessed embedding space of language L 0 . Next, following the order, we gradually add new languages to the space: in each iteration we rotate the preprocessed embedding space Z (cid:48)(cid:48) l of language l to the multilingual space by minimizing the dot product between embeddings of the translations between language l and all the languages in the multilingual space. The recipe to calculate the cross-lingual embedding Z ml is similar to the SHS model: the preprocessing and postprocessing steps are the same, but the projection matrices are calculated with Eq. (14) instead of Eq. (7) and conform with the new objective from Eq. (13). After convergence, Z ml is added to the multilingual space X m : X ml = Z ml . arg max W xl , W zl l  1 (cid:88) k =0 tr ( X mk W xl ( D k,l Z (cid:48)(cid:48) l W zl ) (cid:124) ) (13) subject to W xl W (cid:124) xl = I , W zl W (cid:124) zl = I U l S l V (cid:124) l = SVD ( C ) (14) C = ( X m 0 ) (cid:124) D 0 ,l Z (cid:48)(cid:48) l || ... || ( X ml  1 ) (cid:124) D l  1 ,l Z (cid:48)(cid:48) l where || denotes concatenation along the row axis In supervised settings this approach would be impractical as it requires bilingual dictionaries D k,l for all language pairs k, l , and not only with the hub language. However, within an unsupervised framework this constraint is lifted. Tasks and Datasets. The induced embeddings are evaluated in three tasks: bilingual lexicon induction (BLI), multilingual dependency parsing, and multilingual document classification. BLI is currently the most widely used method to evaluate bilingual embedding spaces. Although BLI performance is not the primary goal of our multilingual embedding spaces, it provides a fast means to address the following questions: 1) Is the incremental construction of multilingual embedding spaces indeed an effective regularization method? Is it still necessary to perform value dropping in this case? 7 ; 2) Is the reweighting of embedding spaces also beneficial for BLI in multilingual settings?; 3) Does multilingual training improve bilingual lexicon induction performance? How do our multilingual models compare to each other and to the state-of-the-art unsupervised BLI methods? We report Precision@1 (P@1) BLI performance on two standard BLI datasets. 1) DINUARTETXE is the extended version of Dinu et al. (2015)'s dataset, used by Artetxe et al. (2018a). 8 It consists of bilingual dictionaries for English-German, English-Italian, English-Spanish and English-Finnish. Monolingual embeddings are provided, based on the CBOW model trained on the WaCKy corpora for English, Italian and German (Baroni et al., 2009), the monolingual WMT Common Crawl corpus for Finnish, and the WMT News Crawl for Spanish (Bojar et al., 2015). The test dictionary sizes are between 1.869 and 1,993 word pairs for each language pair. As our methods are unsupervised, we do not use the provided training dictionaries. 2) EURMUSEWIKI is the dataset compiled from dictionaries for all combinations of the following European languages: English, German, Spanish, French, Italian, and Portuguese. The test set sizes range between 1,513 and 3,660 word pairs. We rely on publicly available monolingual fastText embeddings (Bojanowski et al., 2016). 9 All monolingual word embeddings are 300-dimensional and represent the 200k most frequent words as in prior work (Dinu et al., 2015; Conneau et al., 2018). Multilingual dependency parsing and multilingual document classification tasks assess the embeddings w.r.t. their actual goal: enabling transfer learning across multiple languages. The word embeddings are used as feature vectors for classifiers in the respective downstream tasks. We address the following research questions: 4) Is reweighting of embedding spaces also beneficial in downstream tasks?; 5) How do our methods compare against 7 Value dropping significantly slows down training time and leads to non-deterministic outcomes. However, it has been shown to be crucial in the bilingual setting to obtain good results when mapping distant language pairs in previous work (Artetxe et al., 2018a). 8 https://github.com/artetxem/vecmap/ 9 https://fasttext.cc/docs/en/ pretrained-vectors.html supervised multilingual embedding models? We rely on the evaluation platform of Ammar et al. (2016) for the downstream tasks 10 : the users submit their multilingual embeddings and obtain the final scores, which ensures that the classifiers we use are identical to the ones used in prior work (Ammar et al., 2016; Duong et al., 2017). REUTERSMLDC is a multilingual document classification dataset covering seven languages: English, German, French, Italian, Spanish, Danish, and Swedish. The final performance is reported as the average accuracy across all languages. The respective training and test set consist of 7,000 and 13,058 documents. The dataset is well balanced in the number of documents per language. 11 The architecture of the document classifier is the average perceptron used by Klementiev et al. (2012). MLPARSING is a multilingual dependency parsing dataset sampled from the Universal Dependencies 1.1 corpus (Agi  c et al., 2015) 12 . It contains 12 languages: English, German, French, Spanish, Italian, Bulgarian, Czech, Danish, Swedish, Greek, Finnish, and Hungarian. The respective training and test set contain 6,748 and 1,200 sentences. The test set contains 100 sentences for each language, while for the training set the number of sentences for a language ranges between 98 and 6,694. The parser used is the stack-LSTM parser by Dyer et al. (2015). The parser is not allowed to use any part-of-speech and morphology features, and keeps the input word embeddings fixed to isolate the effect of the evaluated embeddings on the parsing performance (Ammar et al., 2016). The reported scores are UAS scores averaged across languages. For comparison with related work, we train 512-dimensional monolingual embeddings on the text collections used by Ammar et al. (2016) and Duong et al. (2017). The monolingual embeddings are again trained using fastText. Training Setup. In all experiments, we set the following hyper-parameters to values that were used in prior research (Conneau et al., 2018; Artetxe et al., 2018a). When constructing the seed lexicon the 4,000 most frequent words of each language are considered ( C seed = 4 , 000 ), and during the refinement step the 20,000 most frequent words 10 https://github.com/wammar/ multilingual-embeddings-eval-portal 11 As the dataset is not publicly available this information was provided by the first author of Ammar et al. (2016). of each language are used ( C refinement = 20 , 000 ). When using value dropping, the keep probability p is initialized is 0.1, N patience is set to 50, and the stochastic multiplier is set to 2. Dictionaries are constructed symmetrically: from hub language(s) to the secondary language and from the secondary language to the hub language(s): during refinement each dictionary consists of 2  20 , 000 translation pairs. We use CSLS with k = 10 nearest neighbors following the setup of Conneau et al. (2018). Experiment 1: Value Dropping. In the first experiment, we investigate if the expensive value dropping procedure is a necessary condition for mapping between distant language pairs in our multilingual framework. Table 1 provides results on the DINUARTETXE dataset for SHS and IHS models. For IHS we process the languages in the following order: English, German, Italian, Spanish, Finnish. When using value dropping we report the average and best results across five runs. We observe that value dropping is crucial for SHS to succeed for English-Finnish. However, it is not necessary for IHS. This supports our hypothesis that mapping a language to a multilingual hub space serves a type of regularization that can substitute value dropping. As validated later, it is still important to avoid adding distant languages early with the incremental IHS procedure. 13 , 14 Experiment 2: Comparative BLI Performance and Reweighting. In this experiment, we test if reweighting embedding spaces is beneficial for BLI in our multilingual setup, and also compare our 13 For instance, when using IHS with a language order that starts with English and Finnish, value dropping still prevents bad performance for Finnish. However, this is not a problem in practice as the language order can be easily predetermined according to various language similarity heuristics. 14 We further validated the robustness of our approach on other distant language pairs but moved this experiment to the appendix due to space constraints. methods against state-of-the-art BLI methods. Table 2 and Table 3 show the results for SHS and IHS with reweighting coefficients q of 0, 0.5 and 1 on DINUARTETXE and EURMUSEWIKI , respectively. We also include the state-of-the-art results of Artetxe et al. (2018a) and Chen and Cardie (2018) for reference. The EURMUSEWIKI benchmark evaluates BLI performance on all language pair combinations of its six languages and does this in both directions (EN-DE, DE-EN, EN-ES, ... IT-PT, PT-IT) yielding 28 P @1 scores per model. For clarity, we report the average P @1 scores per language as well as the global P @1 average. Following Experiment 1, all results for SHS are obtained using value dropping (again averaged across 5 different runs), while we do not use it with IHS. The SHS hub language is English, and the language orders for IHS are EN, DE, IT, ES, FI for DINUARTETXE , and EN, DE, ES, FR, IT, PT for EURMUSEWIKI . The scores reveal that reweighting the embedding spaces is indeed still beneficial for BLI when mapping to a multilingual space. Both SHS and IHS obtain best results with the reweighting coef-ficient q = 0 . 5 . When comparing SHS and IHS, we see that for language pairs involving English (the SHS hub language) SHS obtains slightly better results, but for the other language pairs IHS outperforms SHS slightly. This is no surprise as IHS by design incorporates dependencies between all languages when learning the projection matrices, though it is striking that mapping to a single hub language is still a strong BLI baseline. For both datasets IHS obtains BLI performance on par with the state-of-the-art: on DINUARTETXE , SHS and IHS ( q = 0 . 5 ) obtain scores similar to (Artetxe et al., 2018a); on EURMUSEWIKIIHS ( q = 0 . 5 ) slightly outperforms Chen and Cardie (2018) for all languages except Spanish. Although optimizing BLI performance is not the main goal of this work, these results verify the soundness of our methods. Experiment 3: Language Order. Next, we investigate 1) the influence of the hub language choice for SHS, and 2) the impact of the language order for IHS. We run both SHS and IHS with reweighting 0.5 on DINUARTETXE . SHS is with value dropping (results are again averaged over 5 runs), and for IHS we do not use value dropping. We find that the SHS model is sensitive to the hub language: the best average scores are obtained when using English (41.6%) or German (41.0%). With Italian, the average score drops to 40.4%, mainly due to worse performance on English and German. With Spanish, the average score further drops to 31.6%, as Spanish and Finnish completely fail to align even when using value dropping. With Finnish, EN-ES alignment becomes unstable, while P @1 for EN-DE and EN-IT drops 5.5% and 3.7% compared to the case with English as the hub. These results indicate that the hub language has to be cho-sen carefully to avoid instability issues. For IHS, we evaluate all 120 order permutations on DINUARTETXE . The best performing order (EN-DE-ES-FI-IT) achieves an average accuracy of 41.96%, see the last row in Table 2. The full results, not reported due to space constraints 15 confirm our hypothesis that distant languages (Finnish) should be mapped at the end: when using Finnish as one of the first two languages performance drops significantly. EN-FI scores drop below 1% and the results for all other language pairs are also suboptimal. 15 The full results can be found in Appendix A.2. input word embeddings on downstream model performance, and compare SHS and IHS to several supervised methods that use cross-lingual supervision. Table 4 reports the results for SHS and IHS with q set to 0 and 0.5 on the REUTERSMLDC and MLPARSING benchmarks, 16 along with the results from related work. For SHS the hub language is English and for IHS the language order is English, German, Spanish, Italian, French, Bulgarian, Czech, Danish, Finnish, Greek, Hungarian, and Swedish. We again use SHS with value dropping and IHS without it. The results in Table 4 are comparable: all methods were trained on the same text corpora (i.e., the collections of Ammar et al. (2016)), but our methods do not use parallel corpora nor bilingual dictionaries. A first interesting result is that, contrary to the BLI task, reweighting the embeddings is not beneficial for multilingual dependency parsing and document classification. This can be explained by the fact that the reweighted embedding spaces are no longer isomorphic to the original monolingual embedding spaces, hence important patterns in the embedding space could be distorted. Further, we notice that both SHS and IHS improve over the best reported results on the REUTERSMLDC and MLPARSING benchmarks. This result is surprising given that all the reported baselines require supervision to induce the multilingual embedding spaces. Further, we again find that the best results are obtained with IHS, most notably for dependency parsing for which the difference in UAS scores between 16 Since the languages covered in MLPARSING is a superset of the languages in REUTERSMLDC, we use the same multilingual embedding space for both tasks. Experiment 5: Time Complexity. In this experiment, we study how training time of SHS and IHS behaves as a function of the number of languages that are mapped. The singular value decompositions are more expensive for IHS as the matrices grow linearly with the number of languages (see Eq. (14)) whereas for SHS they are constant. On the other hand, IHS does not require the use of value dropping. Figure 1 plots training time for IHS and SHS conditioned on the number of languages. The estimates are based on training on a single Nvidia Titan Xp GPU. We find that SHS with value dropping and IHS without value dropping have similar training times, IHS being a bit more efficient when mapping 7 languages or less. Mapping 12 languages takes approximately four hours for both methods. We proposed two novel methods for learning multilingual word embeddings (MWEs) without any cross-lingual supervision. The better-performing incremental hub space model (IHS) is the first unsupervised MWE method that combines three desirable properties: 1) It incorporates interdependencies between all targeted languages; 2) It works for distant language pairs; and 3) It is both deterministic and robust, that is, it does not produce degenerate solutions. Our evaluation on standard benchmarks has proven that the IHS method induces multilingual word embeddings that are competitive with the state of the art in bilingual lexicon induction. Moreover, we have shown that IHS outperforms even supervised models on downstream tasks of multilingual dependency parsing and document classification, and this anomaly requires further investigation in future work. Furthermore, we looked at the influence of reweighting the dimensions of the embedding spaces according to their cross-correlations with the hub language space(s) and found that, while it improves performance for the BLI task, it is harmful to downstream crosslingual transfer tasks. These empirical observations stress the requirement to include comprehensive evaluation protocols for cross-lingual word embedding models in future research. We thank the reviewers for their insightful comments. IV would like to thank Goran Glavas and Anders Sgaard for interesting discussions, and the ERC Consolidator Grant LEXICAL (no 648909) for the support. GH, BV and MFM would like to thank Stijn Jaques, Evelyn Reynders and Evelien Verbaenen for the fruitful collaboration and Flanders Innovation & Entrepreneurship (VLAIO) for funding TELLMI within the ITEA 3 project PAPUD.
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective abstain abstain abstain abstain abstain method result method objective other other other other other other other other other other other other other other other other other other other other other other abstain other method other other other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain method abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method other abstain abstain abstain method abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result other method other other other other other abstain other objective abstain abstain abstain abstain abstain result result abstain other other other