The overall loss is given as

3.1 MUSE Embedding

Multilingual Unsupervised embeddings (MUSE) Conneau et al. [2017] is a technique to acquire
multilingual word embeddings. Assuming that the learning process starts from two sets of word
embeddings trained on monolingual corpora, each corresponding to a different language. We can
represent the embeddings as two matrices X ∈ Rc×M ; Y ∈ Rc×M , where M is the number of
representative words and c is the dimension of each embedding. We assume that in the two languages
we use the M most frequent words to compute the embeddings and compute them in such a way that
they have the same dimension c. The goal of the MUSE training procedure is to learn a mapping W ∗
such that W ∗ = arg minW∈Rc×c ∥W X − Y ∥F , where ∥ · ∥F denotes the Frobenius norm. Xing
et al. [2015] showed that this technique can be improved by imposing an orthogonality constraint on
W , thus boiling this objective down to the Procrustes problem as
∥W X − Y ∥F = U V
⊺
= SVD (Y X
Translatotron3