Answers.Body: <p>Embeddings generated by transformers like Bert or XLM-R are fundamentally different from embeddings learned through language models like GloVe or Word2Vec.
  The latter are static, i.e. they are just dictionaries containing a vocabulary with n-dimensional vectors associated to each word. Because of this they can be plotted through PCA and the distance between them can be easily calculate with whatever metrics you prefer.</p>
  
  <p>When training Bert or XLM-R instead you are not learning vectors, but the parameters of a transformer. The embedding for each token are then generated once a token is fed into the transformer. This implies several things, the most important being that the hidden representation (the embedding) for the token change depending on the context (recall that XML-R use as input also the hidden states generated by the previous token). This means that there are no static vectors to compare by plotting them or by calculating the cosine similarity. 
  Nevertheless, there are way to analyse and visualise the syntax and semantics encoded in the parameters, this paper show some strategies: <a href="https://arxiv.org/pdf/1906.02715.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1906.02715.pdf</a></p>
  
  <p>On a more linguistic side, I would also ask why vectors of same words should show the same semantic properties across languages. Surely there are similarities for lot of words translated literally, but the use of some expressions is inherently different across languages. To make a quick example: in English the clock 'works', in Dutch the clock 'lopen' (it walks) and in Italian the clock 'funziona' (it functions). Same expression, three different words in different languages that do not necessarily share the same neighbours in their monolingual latent spaces. 
  The point of transformers is exactly to move from static representations to dynamic ones that are able to learn that all those three verbs (in their specific language) can appear early in a sentences and close to the word clock. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T07:01:45.770Z
Answers.Text: Thanks for your reply. Note that I do not ask to compare the same word in different languages, but the same concept. In other words, if one would get a pooled output of the sentence "Hij zit op een stoel" (DU) "He sits in a chair" (EN) one can assume that these are semantically equivalent. The question is, though, whether that semantic relatedness can be found in multilingual LMs. The answer, I believe, is _no_.
Answers.UserDisplayName: Bram Vanroy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T21:52:35.890Z
Answers.LastActivityDate: 2020-03-09T21:52:35.890Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>There is a general idea in the field of NLP that there is a mapping between embeddings in different langauges. Figure 1 explains this. <a href="https://i.sstatic.net/Dg7xH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Dg7xH.png" alt="Figure 1"></a></p>
  
  <p>In Figure 1. we have the embedding of English words and Spanish words, and we see that their exists a mapping between the manifolds associated to this two languages, i.e. Spanish manifold is a distorted image of the English maniflod. This idea was used to create an unsupervised translator in <a href="https://github.com/facebookresearch/MUSE" rel="nofollow noreferrer">MUSE Project</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T09:25:16.563Z
Answers.Text: I am aware of MUSE. However, that is different from the multilingual language models that I refer to in my OP. MUSE is more a generalisation or a mapping of fastText over languages, but not a language model.
Answers.UserDisplayName: Bram Vanroy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T08:55:47.770Z
Answers.LastActivityDate: 2020-03-10T08:55:47.770Z
Answers.UserDisplayName: hola
Body: <p>Facebook has <a href="https://arxiv.org/pdf/1911.02116.pdf" rel="nofollow noreferrer">just pushed out</a> a bigger version of their multi-lingual language model XLM, called XLM-R. My question is: do these kind of multi-lingual models imply, or even ensure, that their embeddings are comparable between languages? That is, are semantically related words close together in the vector space across languages?</p>
  
  <p>Perhaps the most interesting citation from the paper that is relevant to my question (p. 3):</p>
  
  <blockquote>
    <p>Unlike Lample and Conneau (2019), we do not use language embeddings,
    which allows our model to better deal with code-switching.</p>
  </blockquote>
  
  <p>Because they do not seem to make a distinction between languages, and there's just one vocabulary for all trained data, I fail to see how this can be truly representative of semantics anymore. The move away from semantics is increased further by the use of BPE, since morphological features (or just plain, statistical <em>word chunks</em>) of one language might often not be semantically related to the same chunk in another language - this can be true for tokens themselves, but especially so for subword information.</p>
  
  <p>So, in short: how well can the embeddings in multi-lingual language models be used for semantically comparing input (e.g. a word or sentence) of two different languages?</p>
  
Tags: <deep-learning><natural-language-processing><transformer><language-model>
Title: Are embeddings in multi-lingual language models comparable across languages?
UserDisplayName: Bram Vanroy
ThreadText: # Are embeddings in multi-lingual language models comparable across languages?
  
  (Asked by: username_0 on 2019-11-08)
  
  Facebook has [just pushed out](https://arxiv.org/pdf/1911.02116.pdf) a bigger version of their multi-lingual language model XLM, called XLM-R. My question is: do these kind of multi-lingual models imply, or even ensure, that their embeddings are comparable between languages? That is, are semantically related words close together in the vector space across languages?
  
  Perhaps the most interesting citation from the paper that is relevant to my question (p. 3):
  
  > Unlike Lample and Conneau (2019), we do not use language embeddings,
  > which allows our model to better deal with code-switching.
  
  Because they do not seem to make a distinction between languages, and there's just one vocabulary for all trained data, I fail to see how this can be truly representative of semantics anymore. The move away from semantics is increased further by the use of BPE, since morphological features (or just plain, statistical *word chunks*) of one language might often not be semantically related to the same chunk in another language - this can be true for tokens themselves, but especially so for subword information.
  
  So, in short: how well can the embeddings in multi-lingual language models be used for semantically comparing input (e.g. a word or sentence) of two different languages?
  
  **Tags:** deep-learning, natural-language-processing, transformer, language-model
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-03-09. Score: 2
  
  Embeddings generated by transformers like Bert or XLM-R are fundamentally different from embeddings learned through language models like GloVe or Word2Vec.
  The latter are static, i.e. they are just dictionaries containing a vocabulary with n-dimensional vectors associated to each word. Because of this they can be plotted through PCA and the distance between them can be easily calculate with whatever metrics you prefer.
  
  When training Bert or XLM-R instead you are not learning vectors, but the parameters of a transformer. The embedding for each token are then generated once a token is fed into the transformer. This implies several things, the most important being that the hidden representation (the embedding) for the token change depending on the context (recall that XML-R use as input also the hidden states generated by the previous token). This means that there are no static vectors to compare by plotting them or by calculating the cosine similarity.
  Nevertheless, there are way to analyse and visualise the syntax and semantics encoded in the parameters, this paper show some strategies: <https://arxiv.org/pdf/1906.02715.pdf>
  
  On a more linguistic side, I would also ask why vectors of same words should show the same semantic properties across languages. Surely there are similarities for lot of words translated literally, but the use of some expressions is inherently different across languages. To make a quick example: in English the clock 'works', in Dutch the clock 'lopen' (it walks) and in Italian the clock 'funziona' (it functions). Same expression, three different words in different languages that do not necessarily share the same neighbours in their monolingual latent spaces.
  The point of transformers is exactly to move from static representations to dynamic ones that are able to learn that all those three verbs (in their specific language) can appear early in a sentences and close to the word clock.
  
  > **Comments:**
  > **username_0**: Thanks for your reply. Note that I do not ask to compare the same word in different languages, but the same concept. In other words, if one would get a pooled output of the sentence "Hij zit op een stoel" (DU) "He sits in a chair" (EN) one can assume that these are semantically equivalent. The question is, though, whether that semantic relatedness can be found in multilingual LMs. The answer, I believe, is _no_.  
  
  ---
  
  ### Answer by username_2 on 2020-03-10. Score: -1
  
  There is a general idea in the field of NLP that there is a mapping between embeddings in different langauges. Figure 1 explains this. [![Figure 1](https://i.sstatic.net/Dg7xH.png)](https://i.sstatic.net/Dg7xH.png)
  
  In Figure 1. we have the embedding of English words and Spanish words, and we see that their exists a mapping between the manifolds associated to this two languages, i.e. Spanish manifold is a distorted image of the English maniflod. This idea was used to create an unsupervised translator in [MUSE Project](https://github.com/facebookresearch/MUSE).
  
  > **Comments:**
  > **username_0**: I am aware of MUSE. However, that is different from the multilingual language models that I refer to in my OP. MUSE is more a generalisation or a mapping of fastText over languages, but not a language model.  
  
  ---
  
  